Multi-modal data fusion processing method and system

Abstract

The invention relates to the technical field of multi-modal data processing, and discloses a multi-modal data fusion processing method and system, and the method comprises the steps: collecting a multi-modal data set containing the activities of the old through an intelligent old-age care robot; constructing a timestamp calibration matrix based on the multi-modal data set, and carrying out dynamic weight correction on the timestamp calibration matrix by adopting a three-modal data envelope delay compensation algorithm to obtain a three-modal data stream with a synchronous time sequence; performing hierarchical feature extraction through a nested feature extractor to obtain a three-mode tumble feature vector group; performing vector fusion on the three-mode tumble feature vector group to obtain tumble fusion feature vectors; and inputting the tumble fusion feature vector into a multi-modal forward prediction model to perform tumble risk trajectory prediction, and outputting a tumble detection result, the method realizes prediction analysis of body posture changes in a key time window before the old man tumble, and improves the ability to distinguish a slow getting-up action and a real tumble behavior of the old man.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and in particular to a method and system for fusion processing of multimodal data. Background Art

[0002] Existing fall detection technologies primarily rely on single sensors or simple multi-sensor combinations. For example, detection methods based solely on vision are easily affected by environmental factors such as lighting changes and occlusion. Methods based solely on inertial sensors have difficulty distinguishing falls from daily activities. Methods based solely on audio are prone to false positives. These single-modal detection methods are significantly lacking in robustness and accuracy in complex elderly care environments. While current multimodal fall detection technologies can overcome the limitations of single-modal methods to a certain extent, they face several key technical challenges, particularly the timing synchronization of different sensor data, the alignment of heterogeneous sampling frequencies, and the weight allocation problem of multimodal feature fusion. Existing technologies often use fixed weights or simple weighted average fusion strategies, which are unable to dynamically adjust the contribution of each modality based on environmental changes and sensor status, resulting in false positives or missed positives in complex scenarios. Summary of the Invention

[0003] The present invention provides a fusion processing method and system for multimodal data. The present invention realizes the predictive analysis of body posture changes in the critical time window before the elderly fall, and improves the ability to distinguish the elderly's slow rising movements from real falling behaviors.

[0004] In a first aspect, the present invention provides a method for fusion processing of multimodal data, the method comprising: Collect multimodal datasets containing elderly activities through intelligent elderly care robots; A timestamp calibration matrix is ​​constructed based on the multimodal data set, and a trimodal data envelope delay compensation algorithm is used to perform dynamic weight correction on the timestamp calibration matrix to obtain a time-synchronized trimodal data stream; Inputting the time-series synchronized trimodal data stream into a nested feature extractor for hierarchical feature extraction to obtain a trimodal fall feature vector group; performing vector fusion on the three-modal fall feature vector group to obtain a fall fusion feature vector; The fall fusion feature vector is input into a multimodal forward prediction model to predict the fall risk trajectory and output a fall detection result.

[0005] In conjunction with the first aspect, in a first implementation of the first aspect of the present invention, collecting a multimodal dataset containing elderly activities by the intelligent elderly care robot includes: The RGB-D camera is used to collect depth visual information of the elderly's activity area to obtain raw depth image data. At the same time, the six-axis IMU sensor is used to collect inertial motion of the elderly's body posture to obtain raw IMU inertial data. The microphone array is used to collect omnidirectional audio of the ambient sound to obtain raw audio data. Performing depth restoration and color correction on the original depth image data to obtain corrected depth image data, performing zero bias correction and filtering on the original IMU inertial data to obtain corrected IMU inertial data, and performing background noise suppression and frequency domain analysis on the original audio data to obtain corrected audio data; The corrected depth image data, the corrected IMU inertial data, and the corrected audio data are time-series aligned to construct a multimodal dataset including a depth vision data stream, a navigation inertial data stream, and an environmental voice data stream.

[0006] In combination with the first aspect, in a second implementation of the first aspect of the present invention, constructing a timestamp calibration matrix based on the multimodal data set, and dynamically weight-correcting the timestamp calibration matrix using a trimodal data envelope delay compensation algorithm to obtain a time-synchronized trimodal data stream includes: Organizing the depth vision reference timestamp, the navigation sensor reference timestamp, the voice sensor reference timestamp, and the time deviation values ​​between each modality in the multimodal dataset into a timestamp calibration matrix; The deviation of the heterogeneous sampling frequency differences of the RGB-D camera, the six-axis IMU sensor and the microphone array is calculated according to the timestamp calibration matrix to obtain the timing deviation analysis result; Using a trimodal data envelope delay compensation algorithm, dynamically weighting the timing deviation analysis results is performed to obtain a synchronous timestamp combination; The multimodal data set is time-rearranged and interpolated and aligned based on the synchronization timestamp combination to obtain a time-synchronized trimodal data stream.

[0007] In combination with the first aspect, in a third implementation of the first aspect of the present invention, the use of a tri-modal data envelope delay compensation algorithm to dynamically assign weights to the timing deviation analysis results to obtain a synchronization timestamp combination includes: Based on the timing deviation analysis results, statistical analysis is performed on the real-time response delay characteristics of the depth vision modality, the navigation sensor modality, and the speech recognition modality to obtain response delay characteristic data; Dynamically allocating a depth vision weight coefficient, a navigation sensor weight coefficient, and a speech recognition weight coefficient according to the response delay characteristic data to obtain a dynamic weight coefficient combination; Inputting the dynamic weight coefficient combination into the trimodal data envelope delay compensation algorithm for weighted summation to obtain a timestamp compensation value; Deviation correction and synchronization alignment are performed based on the timestamp compensation value to obtain a synchronization timestamp combination including a depth vision synchronization timestamp, a navigation sensor synchronization timestamp, and a voice sensor synchronization timestamp.

[0008] In combination with the first aspect, in a fourth implementation of the first aspect of the present invention, inputting the time-synchronized trimodal data stream into a nested feature extractor for hierarchical feature extraction to obtain a trimodal fall feature vector group includes: Extracting a combination of separate data streams including an RGB-D image sequence, a six-axis IMU motion sequence, and a multi-channel audio sequence based on the time-series synchronized three-modal data stream; Using the deep image skeleton key point extraction layer in the nested feature extractor to perform human skeleton detection on the RGB-D image sequence to obtain a deep visual fall feature vector; The six-axis attitude change detection layer in the nested feature extractor is used to calculate the acceleration and angular velocity gradients of the six-axis IMU motion sequence to obtain the navigation inertial fall feature vector; Using the speech event mutation recognition layer in the nested feature extractor to perform sound pressure level detection on the multi-channel audio sequence to obtain an environmental speech fall feature vector; Feature alignment is performed on the depth vision fall feature vector, the navigation inertia fall feature vector, and the environmental voice fall feature vector to obtain a three-modal fall feature vector group including depth vision features, navigation inertia features, and environmental voice features.

[0009] In combination with the first aspect, in a fifth implementation of the first aspect of the present invention, performing vector fusion on the three-modal fall feature vector group to obtain a fall fusion feature vector includes: Based on the three-modal fall feature vector group, a strategy space including high-confidence detection, medium-confidence detection, low-confidence detection, and exit from competition is set for each participant in the deep vision game; a strategy space including dominant motion detection, auxiliary posture detection, baseline monitoring, and standby state is set for each participant in the navigation sensor game; and a strategy space including emergency call detection, abnormal sound monitoring, ambient sound analysis, and silent mode is set for each participant in the speech recognition game, thereby obtaining a three-party game model. According to the three-party game model, the accuracy benefit of the depth vision game participants minus the computational cost loss and the environmental interference loss, the motion detection accuracy benefit of the navigation sensor game participants minus the drift error loss and the vibration interference loss, and the acoustic event recognition rate benefit of the speech recognition game participants minus the noise impact loss and the false trigger penalty loss are calculated respectively to obtain a game benefit matrix; Performing a competition analysis on the game payoff matrix to obtain a competition analysis result, and performing an optimal weight solution based on the competition analysis result to obtain a dynamic fusion weight coefficient; The three-modal fall feature vector group is vector-fused based on the dynamic fusion weight coefficient to obtain a fall fusion feature vector.

[0010] In combination with the first aspect, in a sixth implementation of the first aspect of the present invention, performing a competition analysis on the game payoff matrix to obtain a competition analysis result, and performing an optimal weight solution based on the competition analysis result to obtain a dynamic fusion weight coefficient includes: Based on the game payoff matrix, a first comprehensive score of skeleton integrity, motion amplitude, and environment quality is calculated for participants in the depth vision game; a second comprehensive score of acceleration change rate, posture stability, and sensor accuracy is calculated for participants in the navigation sensor game; and a third comprehensive score of acoustic feature matching and signal-to-noise ratio is calculated for participants in the speech recognition game. executing a dominance competition rule process based on the first comprehensive score, the second comprehensive score, and the third comprehensive score to obtain a dominance allocation result; Inputting the dominant power allocation result into a Nash equilibrium algorithm for iterative optimization to obtain equilibrium solution data that meets the Nash equilibrium conditions; Optimal weight distribution is performed based on the equalization solution data to obtain a dynamic fusion weight coefficient including a depth vision weight coefficient, a navigation sensor weight coefficient, and a speech recognition weight coefficient.

[0011] In combination with the first aspect, in a seventh implementation of the first aspect of the present invention, performing vector fusion on the three-modal fall feature vector group based on the dynamic fusion weight coefficient to obtain a fall fusion feature vector includes: Performing time window processing based on the three-modal fall feature vector group to obtain time series matching window data; According to the time series matching window data, the vertical position change speed of the body center point and the body tilt angle data are extracted from the depth visual features, the center of gravity acceleration synthesis vector and the posture change angular velocity data are calculated from the navigation inertial features, and the distress keywords and abnormal sound data are detected from the environmental voice features through Mel frequency cepstral coefficient analysis to obtain the target fall feature data; Adopting an asymmetric time-series weight compensation field mechanism to adjust the weight of the target fall feature data, and obtaining a target weight coefficient combination including the weight parameters after compensation adjustment; The three-modal fall feature vector group is weightedly fused in combination with the target weight coefficient combination to obtain a fall fusion feature vector that fuses the body tilt angle change of the depth image, the IMU center of gravity displacement acceleration, and the voice spectrum call for help feature.

[0012] In combination with the first aspect, in an eighth implementation of the first aspect of the present invention, inputting the fall fusion feature vector into a multimodal forward prediction model to perform fall risk trajectory prediction and outputting a fall detection result includes: Inputting the fall fusion feature vector into a multimodal forward prediction model, starting a deep visual skeleton sequence prediction branch, a navigation sensor motion trajectory extrapolation branch, and a speech recognition emotional state assessment branch to perform three-way parallel prediction to obtain a parallel prediction data combination; Based on the parallel prediction data combination, the deep visual skeleton sequence prediction branch performs motion trajectory prediction, the navigation sensor motion trajectory extrapolation branch performs acceleration and angular velocity state estimation and trajectory extrapolation, and the speech recognition emotional state assessment branch performs speech emotion change trend analysis and psychological stress state prediction, thereby obtaining three-way prediction trajectory results; Comprehensive decision-making and fall judgment are performed based on the three-way predicted trajectory results and the fall fusion feature vector to obtain a fall detection result.

[0013] In a second aspect, the present invention provides a multimodal data fusion processing system, the multimodal data fusion processing system comprising: The acquisition module is used to collect multimodal datasets containing elderly people's activities through intelligent elderly care robots; a correction module, configured to construct a timestamp calibration matrix based on the multimodal data set, and perform dynamic weight correction on the timestamp calibration matrix using a trimodal data envelope delay compensation algorithm to obtain a time-synchronized trimodal data stream; a feature extraction module, configured to input the time-series synchronized trimodal data stream into a nested feature extractor for hierarchical feature extraction to obtain a trimodal fall feature vector group; a vector fusion module, configured to perform vector fusion on the three-modal fall feature vector group to obtain a fall fusion feature vector; The output module is used to input the fall fusion feature vector into the multimodal forward prediction model to predict the fall risk trajectory and output the fall detection result.

[0014] In the technical solution provided by the present invention, the precise synchronization problem of the heterogeneous sampling frequencies of RGB-D camera 30fps, six-axis IMU sensor 100Hz, and microphone array 16kHz is solved through the three-modal data envelope delay compensation algorithm and timestamp calibration matrix, ensuring the millisecond-level alignment of sensor data of different physical characteristics on a unified time base, eliminating the feature mismatch problem caused by timing deviation in traditional methods. A three-party game competition analysis framework is adopted to construct the three modalities of depth vision, navigation sensor, and speech recognition as intelligent game participants. The adaptive dynamic adjustment of the weights of each modality is achieved through the Nash equilibrium algorithm, overcoming the limitation of the traditional fixed weight fusion method that cannot adapt to environmental changes, and significantly improving the adaptability and robustness of the system in different scenarios. Through the hierarchical processing mechanism of the nested feature extractor, the precise extraction and millisecond-level timing alignment of the key points of the deep image skeleton, the six-axis posture gradient of the IMU, and the sudden change features of the speech sound pressure level are achieved, ensuring that the different modal features describe the state of the elderly at the same time, avoiding the timing inconsistency problem caused by the separate processing of each modal feature in the traditional method. Through a three-way parallel prediction mechanism, the system can predict and analyze changes in the elderly's body posture within the critical time window before they fall, breaking through the limitations of traditional methods that only judge at the current moment and having the ability to identify fall risks in advance. By adopting an asymmetric temporal weight compensation field mechanism, the system dynamically adjusts the weight distribution according to the different characteristic intensities of rapid falling movements, impact signals, and distress signals, effectively solving the problem of different contributions of the three sensors at different stages of a fall, and significantly improving the ability to distinguish between the elderly's slow rise and real fall behaviors. The changes in the body tilt angle of the depth image, the acceleration of the center of gravity displacement of the IMU, and the distress features of the voice spectrum are weighted and fused on a unified time axis to form a complete detection result including the probability of fall, confidence score, occurrence timestamp, and warning level, providing a reliable basis for the emergency response of the elderly care robot.

[0015] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0016] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of an embodiment of a method for fusion processing of multimodal data in an embodiment of the present invention; Figure 2 Schematic diagram of an embodiment of a multimodal data fusion processing system in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0019] The terms "including," "having," and any variations thereof, as used in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to the process, method, product, or device.

[0020] To facilitate understanding of this embodiment, a multimodal data fusion processing method disclosed in an embodiment of the present invention is first introduced in detail. Figure 1 As shown, this method includes the following steps: 101. Collect multimodal datasets containing elderly activities through intelligent elderly care robots; It is understandable that the execution subject of the present invention may be a multimodal data fusion processing system, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0021] Specifically, the visual acquisition module equipped with an RGB-D camera performs spatial imaging processing on the elderly's activity area. The collected raw depth image data includes each frame of RGB color image and the corresponding depth information. The color image provides texture and appearance information of the target, and the depth map reflects the geometric position distribution of the human body relative to the camera in space. At the same time, in order to track the elderly's posture changes in three-dimensional space in real time, the system synchronously activates the six-axis inertial measurement unit (IMU) sensor installed on the robot platform or wearable device. By capturing the raw data of three-axis acceleration and three-axis angular velocity to form an inertial measurement data stream, the system combines the change trajectory of each axis signal to infer the dynamic center of gravity state and posture change trend of the human body. The system also uses the microphone array to perform 360-degree omnidirectional audio collection of the environment, recording various voice signals and background sounds generated during the elderly's activities, thereby constructing the original audio data channel. To ensure the availability and consistency of multi-source data, the three types of raw data undergo multi-level correction and preprocessing. A depth map inpainting algorithm is used to remove holes in the raw depth image data caused by occlusion or reflection interference. Color correction is also performed on the color image to ensure that the contrast and saturation of the RGB channels meet visual analysis standards, resulting in structurally complete and color-stable corrected depth image data. Furthermore, due to zero offset and high-frequency noise in the raw IMU inertial data, static drift modeling and high-pass filtering are used to zero-bias correct the acceleration and angular velocity signals, respectively, to generate corrected IMU inertial data. For the raw audio data, background noise suppression is performed using speech enhancement techniques. The signal undergoes frequency domain analysis and conversion, and characteristic components such as the Mel-spectrum spectrum or energy envelope are extracted using a short-time Fourier transform. This produces corrected audio data with clear spectral structure and prominent semantic components. Timestamp correction and packet synchronization mechanisms are used to align the corrected depth image data, corrected IMU inertial data, and corrected audio data. Based on the high-precision timestamps provided by each sensor type, a unified time reference line is constructed. Through mechanisms such as delay compensation or dynamic interpolation, the position of each modal data on the timeline is adjusted to ensure that the depth vision, inertial navigation, and ambient speech data are correlated and paired at any given moment, thereby achieving synchronous integration of multi-source data across modalities. A depth vision data stream consisting of a corrected depth image sequence, a navigation inertial data stream containing vectorized pose information, and an ambient speech data stream containing spectral semantic features are constructed. Together, these three constitute a structured, temporally consistent, and semantically complementary multimodal dataset.

[0022] 102. Construct a timestamp calibration matrix based on the multimodal data set, and use a trimodal data envelope delay compensation algorithm to dynamically correct the timestamp calibration matrix to obtain a time-synchronized trimodal data stream; Specifically, the high-precision timestamp information associated with each modality is extracted and aggregated from the multimodal dataset. Using the deep vision modality as the time reference, the timestamp corresponding to each frame of the RGB-D camera is extracted. The timestamps of the acceleration and angular velocity data recorded by the six-axis IMU sensor in the navigation modality, as well as the audio sampling timestamps associated with the microphone array in the audio modality, are also extracted simultaneously. Furthermore, to address the non-simultaneous arrival of data packets between different modalities, the average time difference between the modalities is calculated, including the time deviation between vision and navigation, the time deviation between vision and speech, and the time deviation between navigation and speech. A structured timestamp calibration matrix is ​​constructed to express the time reference relationship and relative deviation state of the three modal signals during acquisition. To analyze the data alignment challenges caused by heterogeneous hardware sampling frequencies among the three sensor types, the authors used the aforementioned timestamp calibration matrix to model and quantify the temporal resolution differences between the visual modality (30 frames per second), the navigation modality (100 Hz), and the voice modality (16 kHz). This yielded a detailed set of timing deviation analysis results, including the maximum offset duration between a single visual image and the corresponding navigation data, the envelope synchronization error between consecutive navigation data segments and voice samples, and the matching of each modality's sampling period within a fixed time window. Furthermore, a trimodal data envelope delay compensation algorithm was introduced, dynamically weighting the timing deviation analysis results based on sampling delay statistics and modal response stability. This compensation algorithm dynamically adjusts the influence of each timestamp type in the synchronization calculation by setting three dynamic weight coefficients corresponding to the temporal dominance of the visual, navigation, and voice modalities, generating a set of synchronized timestamp combinations based on a unified reference. This method adaptively adjusts the weights based on the signal delay status of each modality within the current time window, ensuring high temporal responsiveness in rapidly changing scenarios, such as falls. Based on the generated synchronized timestamp combination, the three data streams in the original multimodal dataset are uniformly time-aligned and interpolated. For high-frequency modalities such as navigation and audio data, sampling points are selected and linearly interpolated to correspond to low-frequency visual frames. For gaps in the visual data, temporal continuity is reconstructed through interpolation between images, achieving temporal consistency among the three data streams on the same time basis. The final output is a time-aligned data set with synchronized timestamps.

[0023] 103. Input the time-synchronized trimodal data stream into a nested feature extractor for hierarchical feature extraction to obtain a trimodal fall feature vector group; Specifically, the multimodal data stream is deconstructed according to modal properties to extract three separate data stream combinations: RGB-D image sequences, six-axis IMU motion sequences, and multi-channel audio sequences. This ensures that each modal input maintains the independence and complementarity of its original physical properties and maintains synchronization under a unified time base. These three data sets are input into a feature extractor built on a nested structure. Following a multi-level modular processing architecture, intra-modal feature extraction and inter-modal temporal consistency calibration are performed on the depth image, inertial data, and audio signals, respectively. During the processing of the deep visual modality, the deep image skeleton key point extraction layer in the nested feature extractor is called, and the key point positioning operation is performed on each frame in the RGB-D image sequence through the skeleton recognition algorithm based on the deep neural network. The 17 human joint nodes including the head, neck, shoulders, elbows, wrists, chest, waist, hips, knees and ankles are identified, and the displacement changes of each node in the three-dimensional coordinate space are accelerated to obtain the deep visual fall feature vector describing the dynamic evolution of human posture. This vector has a 51-dimensional structure and can capture visual signals such as the overall falling of the body, rapid displacement of a single point and structural posture collapse. In the navigation inertial modal path, the nested feature extractor calls its internal six-axis attitude change detection layer, inputs the six-axis IMU motion sequence into the module, performs time gradient calculation and statistical change analysis on the three-axis acceleration and three-axis angular velocity signals respectively, extracts the acceleration vector gradient and angular velocity vector gradient, and constructs a set of six-dimensional inertial feature vectors through synthesis, which reflects the movement amplitude and attitude rotation trend of the elderly person's body in a short time, and is suitable for capturing rapid attitude changes, center of gravity displacement and impact signals caused by instability. Simultaneously, in the audio modal path, a nested feature extractor activates the speech event mutation recognition layer, performing multidimensional sound pressure level detection and spectral energy analysis on multichannel audio sequences. This module constructs a joint analysis model in the time and frequency domains to detect signal points indicating sudden increases in sound pressure amplitude or abnormal energy distribution patterns within a very short period of time. This module identifies typical sudden events such as "bumps," "rapid shouts," and "furniture collisions." These events are then mapped into 32-dimensional ambient speech fall feature vectors, which contain parameters such as Mel-frequency cepstral coefficients, energy envelope information, and spectral peak amplitudes. These vectors reflect non-verbal anomalies associated with falls in the audio modality. To achieve collaborative analysis of the three modal features, the three sets of modal feature vectors are aligned on the time axis. Specifically, based on the combination of the original sampling time of each modality and the synchronization timestamp, the corresponding eigenvalues ​​of the three sets of feature vectors within the same time segment are selected, forming a structurally unified and semantically complementary trimodal fall feature vector set.

[0024] 104. Perform vector fusion on the three-modal fall feature vector group to obtain a fall fusion feature vector; Specifically, the fall feature vectors extracted from three modalities—depth vision, navigation inertia, and ambient speech—are modeled as independent but interconnected game participants, constructing a three-party modal competition model. In this model, the depth vision modality serves as the first player, with a strategy space consisting of four options: high-confidence detection, medium-confidence detection, low-confidence detection, and exit. These options characterize the strength of the image recognition module's judgment of a fall event. The navigation sensor modality serves as the second player, with a strategy space consisting of dominant motion detection, auxiliary posture detection, baseline monitoring, and standby mode to reflect the dominant or supporting role of the IMU signal in the current behavior recognition task. The speech recognition modality serves as the third player, with a strategy space consisting of emergency call detection, abnormal sound detection, ambient sound analysis, and silence mode to reflect the confidence level and expected functional role of the speech modality in the current audio environment. Based on this, a specific payoff function is established for each player in the three-party game model, and the behavioral benefits and potential losses obtained by each player under their current strategy are calculated. The reward function for the depth vision participant uses recognition accuracy as the positive revenue source, deducting the computational cost associated with image processing and the interference losses caused by environmental occlusion and illumination variations. The reward function for the navigation sensor participant uses motion detection accuracy as the primary revenue metric, while also incorporating interference from IMU drift error and structural vibration on pose estimation as a penalty. The reward function for the speech recognition participant focuses on the recognition rate of sudden acoustic events, while also considering penalty terms caused by signal contamination and false triggering caused by ambient noise. This results in a separate evaluation of the revenue of each modality under a specific state combination. The reward functions of each participant are numerically quantified and constructed as a game reward matrix, which represents the overall game landscape for each strategy combination in the three-modal strategy combination space. A competitive analysis is performed on the game reward matrix. By analyzing the reward equilibrium state under the interaction of the three strategies, the optimal strategy for each participant is determined under the current environmental conditions and sensor response state. The corresponding optimal weight combination is then calculated using a weight optimization model. This weight combination should be dynamically adaptive, automatically adjusting the fusion contribution ratio between modalities based on real-time changes in sensor data quality, thereby generating a set of dynamic fusion weight coefficients. These coefficients are based on a game balance, ensuring that if one modality's performance degrades, the other modalities automatically increase their weight, thereby achieving system perceptual stability in complex scenarios. Dynamic fusion weight coefficients are used to fuse the three-modal fall feature vector group. This involves multiplying the depth vision feature vector, the navigation inertial feature vector, and the ambient speech feature vector by their corresponding weight coefficients, and performing a weighted combination on a unified time basis to generate a fused fall feature vector.

[0025] 105. Input the fall fusion feature vector into the multimodal forward prediction model to predict the fall risk trajectory and output the fall detection result.

[0026] Specifically, a multimodal forward prediction model with three-way parallel processing capabilities was constructed. This model uses the fused fall feature vector as a unified input and internally deploys three parallel branches: a deep visual skeleton sequence prediction branch, a navigation sensor motion trajectory extrapolation branch, and a speech recognition emotional state assessment branch. These branches are used to infer the potential future evolution trends of the elderly person's spatial posture, inertial motion, and speech emotion, respectively. Upon startup, the fused feature vector is mapped to the sub-model structures in each of the three branches, forming three parallel prediction data streams. These streams generate prediction results based on the historical behavioral characteristics and current state change trends of the different modalities, outputting a complete three-way prediction data set. During the prediction phase, the deep visual skeleton sequence prediction branch receives visual feature vectors containing coordinate changes of key skeleton nodes. Using a recurrent neural network (RNN) with a long short-term memory (LSTM) architecture, it performs sequence learning and state memory on the historical skeleton position sequence. It then predicts the spatial trajectory of key nodes within one second, thereby constructing a continuous trajectory of human posture change. This prediction can reveal whether the elderly person is experiencing a persistent state of imbalance and detect trends of increasing falls or tilts. Simultaneously, the navigation sensor trajectory extrapolation branch inputs the acceleration and angular velocity features from the inertial modality into the state estimation model. Combined with filtering extrapolation algorithms such as the Kalman filter, it continuously estimates the center of gravity acceleration changes and attitude angular velocity evolution trends within future time windows, thereby delineating a predicted inertial trajectory. This trajectory is used to determine whether the elderly person is entering an uncontrolled impact state or experiencing rapid flips. The speech recognition emotion state assessment branch processes the speech content, spectral energy, and pitch variation information extracted from the audio modality. It then uses a recurrent neural network to model potential emotional trends in the speech. Specifically, it analyzes emotional transitions based on high-frequency distress words, high-pressure pitch changes, and speech rhythm disturbances to predict whether the elderly person is experiencing psychological stress or subjective panic. The three predicted trajectory results are then integrated and compared with the initial input fall fusion feature vector. A comprehensive decision-making mechanism is established to evaluate the predicted data in conjunction with the current state to determine whether a fall is highly likely in the future. This comprehensive decision-making process not only infers behavioral trends within the next second but also leverages current multimodal fusion features to model the completeness of the behavioral context, thereby forming a fall detection judgment based on the dynamic evolution process. The final fall detection result output includes the predicted probability of a fall, a risk confidence score, a predicted timestamp, and a fall level assessment label.

[0027] In a specific embodiment, the process of executing step 101 may specifically include the following steps: The RGB-D camera is used to collect depth visual information of the elderly's activity area to obtain raw depth image data. At the same time, the six-axis IMU sensor is used to collect inertial motion of the elderly's body posture to obtain raw IMU inertial data. The microphone array is used to collect omnidirectional audio of the ambient sound to obtain raw audio data. Perform depth restoration and color correction on the original depth image data to obtain corrected depth image data, perform zero bias correction and filtering on the original IMU inertial data to obtain corrected IMU inertial data, and perform background noise suppression and frequency domain analysis on the original audio data to obtain corrected audio data; The corrected depth image data, corrected IMU inertial data and corrected audio data are time-series aligned to construct a multimodal dataset including deep visual data stream, navigation inertial data stream and environmental voice data stream.

[0028] Specifically, a highly integrated perception system with a depth vision acquisition module, an inertial measurement module, and an ambient audio acquisition module as its core is constructed on the intelligent elderly care robot platform. An RGB-D camera is used to continuously acquire spatial images of the elderly's activity area, and an original image dataset containing red, green, and blue three-channel color image information and corresponding depth map information is generated in real time. The raw depth image data is continuously sampled at a frequency of 30 frames per second, ensuring complete capture of the elderly's movements, such as moving, turning, sitting, or falling, in actual application scenarios. Simultaneously, a six-axis inertial measurement unit (IMU) installed on the robot body or wearable device dynamically tracks and inertially collects the elderly's body posture. This sensor continuously outputs a raw IMU data stream consisting of three-axis acceleration and three-axis angular velocity, and updates the sampling results at a frequency of up to 100 Hz. With this high-frequency data, the system can capture subtle body movements, turns, or sudden changes in the center of gravity, and construct a trajectory of posture changes. Omnidirectional audio capture of the ambient sound within the elderly's space is performed through a microphone array. The array is configured with 16 channels of input, supporting sound source acquisition within a 360-degree range, and records continuous audio signals with high fidelity at a sampling rate of 16,000 times per second, ensuring that rapid shouting, impact sounds, or abnormal voice events are promptly recorded. The raw data is preprocessed, and operations such as structural correction and noise removal are performed on the three types of raw data. For the original depth image data, a depth restoration algorithm is used to perform deep reconstruction of image holes caused by insufficient lighting, reflective occlusion, or edge interference, patch invalid or missing pixel areas in the image, and enhance the continuity of the image structure through bilateral filtering, edge enhancement, etc. After the depth image restoration is completed, color restoration processing is performed based on the illumination balance deviation in the original color image. The visual consistency of the image is improved through operations such as white balance correction, saturation adjustment, and gamma correction, and the corrected depth image data is output. For the original IMU inertial data, due to the zero drift and external vibration interference problems of the sensor itself, the system uses a static calibration method to accurately model the static offset values ​​of acceleration and angular velocity, and performs zero bias correction in real time. On this basis, the high-frequency noise and low-frequency trend drift superimposed on the signal are removed by combining bandpass filters and low-pass filters, thereby generating corrected IMU inertial data with a stable baseline and clear dynamic response. At the same time, the original audio data, as a continuous signal, is mixed with invalid information such as background wind noise, motor sounds, and appliance hum. The system uses algorithms such as spectral subtraction and speech enhancement to identify and suppress background noise. The system then converts the audio signal into a frequency domain representation using a short-time Fourier transform, extracting structurally meaningful frequency domain features such as spectrograms, energy envelopes, or Mel-frequency cepstral coefficients, thereby generating corrected audio data with semantic expression capabilities. The various data streams are time-aligned to ensure that they describe the elderly person's behavior within the same time period during the fusion analysis phase.The system maps the time axis to a unified timeline based on the timestamp information attached to each sensor type and constructs a high-precision temporal reference model. Time alignment, interpolation, and frame reordering are performed to address the heterogeneous temporal granularity differences between the 30 frames per second RGB-D image sequence, the 100 Hz update rate of IMU data, and the 16 kHz sampling rate of audio data. Using deep vision as the temporal reference, the system finds the nearest neighbor IMU data packet and audio signal segment at the time node of each image frame. The navigation and audio data are then remapped to the image frame's time point using linear interpolation or sliding window averaging, forming a trimodal time series dataset with a unified time reference. This alignment process incorporates a delay compensation mechanism to model and correct for packet arrival delays, ensuring temporal consistency across the three modal data even in complex network or sensor-driven scenarios with response delays. Through these steps, a multimodal dataset consisting of deep vision data streams, navigation inertial data streams, and ambient audio data streams is constructed.

[0029] In a specific embodiment, the process of executing step 102 may specifically include the following steps: The depth vision reference timestamp, navigation sensor reference timestamp, voice sensor reference timestamp and time deviation values ​​between each modality in the multimodal dataset are organized into a timestamp calibration matrix; The deviation of the heterogeneous sampling frequency differences of the RGB-D camera, six-axis IMU sensor and microphone array is calculated based on the timestamp calibration matrix to obtain the timing deviation analysis results. Using the trimodal data envelope delay compensation algorithm, dynamic weight allocation is performed on the timing deviation analysis results to obtain a synchronous timestamp combination; Based on the combination of synchronized timestamps, the multimodal datasets are time-rearranged and interpolated to obtain a time-synchronized trimodal data stream.

[0030] Specifically, the system extracts high-precision timestamp information from each modality's data stream from a multimodal dataset and uses this information as a basis for constructing a full-modality timing calibration structure. The system extracts the acquisition time corresponding to each RGB-D camera frame from the depth vision data stream and sets this as the baseline timestamp sequence for the vision modality. Simultaneously, the timestamps of the three-axis acceleration and three-axis angular velocity packets output by the IMU sensor are extracted from the navigation inertial data stream and organized as the time base for the navigation modality. Furthermore, the sampling start time or analysis window time of the audio clips recorded by the microphone array is extracted from the voice data stream as the time base for the voice modality. To achieve a unified temporal representation across modalities, the three time bases are paired and the relative time offset between the modalities is calculated. Specifically, using the depth vision modality as a reference, the average delay between the navigation modality and the depth vision modality is calculated, as well as the average offset of the voice modality relative to the depth vision modality. Furthermore, the time difference between the navigation modality and the voice modality is calculated. By structurally integrating the three sets of time bases and the time offset data between the three modalities, a timestamp calibration matrix is ​​constructed. Based on this timestamp calibration matrix, the heterogeneous sampling frequency differences among the three sensor types are analyzed. Combined with their respective data packet sampling periods, a cross-modal temporal resolution difference model is constructed. Since RGB-D image frames provide relatively low-frequency visual snapshots at 30 frames per second, the IMU inertial module continuously outputs a large amount of posture data at a 100 Hz update rate, and the microphone array provides a high-density acoustic waveform sample rate at 16,000 audio sampling times per second, the three sensors exhibit significant differences in temporal granularity. Based on the density and offset of the sampling time axes of each modality in the calibration matrix, the system analyzes the valid intersection interval of multimodal data within a common time window. A timing deviation analysis report is generated by calculating the average time difference, maximum offset distance, and jitter distribution between sampling points of different modalities within each time slice. This report indicates whether the current sensors exhibit significant latency in actual operation and reveals potential trends in asynchrony. A trimodal data envelope delay compensation algorithm is introduced, employing a weight-driven temporal fusion mechanism. Under this mechanism, the system assigns dynamically adjusted time-based weights to each modality. The visual modality's time-reference weight dominates when identifying structural behavioral changes (such as leaning, standing, and falling). The navigation modality takes priority when processing short-term, high-frequency events like sudden posture changes and imbalances. The voice modality assumes a time-driven role for acoustic events like sudden calls for help and unusual sounds. The system adjusts the fusion weights of the three modalities in real time based on the data timing jitter, its own stability indicators, and historical synchronization accuracy within each time period. The combined weights form a weighted average synchronization timestamp structure.This combined structure has a targeted bias towards modalities with higher data quality and faster time response, improving the real-time and robustness of the overall time alignment while reducing error accumulation. In particular, it can effectively avoid feature description distortion caused by time misalignment when dealing with rapidly changing behaviors such as falls. Based on the synchronous timestamp combination, time axis reconstruction and alignment interpolation operations are performed on the original three types of data streams. A global time series with the visual modality time frame as the main index is established, and the navigation data points and audio data segments closest to their time are searched in this series. If the sampling time of the two does not accurately correspond to the time point of the visual frame, the system performs linear interpolation on the acceleration and angular velocity data of the navigation modality and performs window function smoothing on the spectral data of the speech modality, so that at the time anchor point of the visual frame, all three types of modal data can provide consistent temporal representation content. During the processing process, the system performs secondary correction on the rearranged data based on the real-time evaluated time deviation residual, thereby constructing a three-modal data stream with a fully time-aligned structure.

[0031] In a specific embodiment, the execution step adopts a trimodal data envelope delay compensation algorithm to dynamically assign weights to the timing deviation analysis results to obtain a synchronization timestamp combination, which may specifically include the following steps: Based on the results of timing deviation analysis, the real-time response delay characteristics of the depth vision modality, navigation sensor modality, and speech recognition modality are statistically analyzed to obtain response delay characteristic data; Dynamically assigning depth vision weight coefficients, navigation sensor weight coefficients, and speech recognition weight coefficients according to response delay characteristic data to obtain a dynamic weight coefficient combination; The dynamic weight coefficient combination is input into the three-modal data envelope delay compensation algorithm for weighted summation to obtain the timestamp compensation value; Deviation correction and synchronization alignment are performed based on the timestamp compensation value to obtain a synchronization timestamp combination including the depth vision synchronization timestamp, the navigation sensor synchronization timestamp, and the voice sensor synchronization timestamp.

[0032] Specifically, high-dimensional statistical modeling is performed on the results of timing deviation analysis to analyze the response lag, signal return stability, and degree of coordination and consistency between each modality during continuous data acquisition. The image frame time series of the depth vision modality is differentially compared with its actual trigger time within a sliding time window. The mean, extreme value, and variance of the inter-frame delay changes within multiple time windows are statistically analyzed to form the response delay distribution characteristic data of the visual modality. Simultaneously, the acceleration and angular velocity data packets of the navigation sensor modality are more susceptible to data jitter, packet loss, and cache delay due to their higher sampling frequency. Therefore, the system analyzes the time interval fluctuation, mutation frequency, and data frame loss probability during continuous output of this modality to extract dynamic delay characteristic indicators of the navigation modality. For the speech recognition modality, its response delay is mainly reflected in the audio stream segment transmission, spectrum analysis window offset, and acoustic event hysteresis recognition. The system evaluates the minimum processing cycle, waveform mutation response duration, and recognition lag period between voice packet triggering and actual processing results to construct a statistical profile of the speech modality's delay. By pooling and analyzing the delay characteristics of the three modalities, a response delay characteristic dataset containing three-way modal response performance data is generated. Based on this response delay characteristic data, the three-modal fusion weight allocation strategy is dynamically adjusted to adapt to the varying response performance of the different modalities during actual operation. Weight coefficients are calculated and allocated in real time based on objective delay performance evaluation results. For example, if the average image response delay of the depth vision modality increases significantly within a specific time window, indicating that the current vision sensor is limited by processing bottlenecks or environmental interference, the system will automatically reduce its weight coefficient to prevent this modality from delaying overall time synchronization due to untimely response. If the navigation sensor modality exhibits a stable high-frequency response and strong data continuity within the same time window, its weight coefficient will be increased accordingly, becoming the dominant driving factor in the current timing compensation process. Similarly, if the speech recognition modality experiences accumulated recognition errors or background noise interference under specific circumstances, the system will appropriately reduce its timing reference weight to maintain the temporal control stability of the modal fusion. This adjustment process forms a dynamic weight coefficient combination consisting of the depth vision weight coefficient, the navigation sensor weight coefficient, and the speech recognition weight coefficient. The dynamic weight coefficients are combined and input into the trimodal data envelope delay compensation algorithm, and a weighted summation operation is performed to generate the timestamp compensation value used for multimodal time alignment. During this process, the compensation algorithm combines the distance deviation between the current timestamp of each modality and the reference time axis with the dynamic weight coefficients for fusion weighting, thereby optimizing the original timestamp of each modality. The design of this weighted summation mechanism ensures that the system does not rely entirely on a single modality as the absolute time anchor point. Instead, it achieves relatively balanced time corrections through the delay negotiation relationship between the modalities, so that the compensation result can respond to the rapid changes of the dominant modality while maintaining time coordination between the modalities.The compensation result is a continuous dynamic value that can provide real-time feedback on the time fine-tuning required for each frame or each data packet, and further refine the compensation step size through an embedded incremental correction mechanism to ensure that the output time structure has stable and progressive characteristics. Based on the above timestamp compensation value, the multimodal original time series is subjected to deviation correction and synchronization alignment. The system appends the corresponding visual time compensation value to each frame of depth image to generate a depth-vision synchronization timestamp; for each IMU data segment, the time correction of the navigation modality is applied according to its delay status and weight coefficient, and the navigation sensor synchronization timestamp is output; similarly, for each set of audio processing windows or spectrum analysis frames, the compensation value driven by the speech recognition weight is superimposed on the original time point to obtain the speech sensor synchronization timestamp. The three types of timestamps that have undergone deviation correction and alignment are uniformly organized into a set of synchronized timestamp combinations with a consistent time base.

[0033] In a specific embodiment, the process of executing step 103 may specifically include the following steps: Based on the time-series synchronization of the three-modal data stream, separate data stream combinations including RGB-D image sequence, six-axis IMU motion sequence, and multi-channel audio sequence are extracted respectively; The deep image skeleton key point extraction layer in the nested feature extractor is used to detect the human skeleton in the RGB-D image sequence and obtain the deep visual fall feature vector. The six-axis attitude change detection layer in the nested feature extractor is used to calculate the acceleration and angular velocity gradients of the six-axis IMU motion sequence to obtain the navigation inertial fall feature vector. The speech event mutation recognition layer in the nested feature extractor is used to detect the sound pressure level of the multi-channel audio sequence and obtain the environmental speech fall feature vector. Feature alignment is performed on the deep vision fall feature vector, navigation inertia fall feature vector, and environmental voice fall feature vector to obtain a three-modal fall feature vector group including deep vision features, navigation inertia features, and environmental voice features.

[0034] Specifically, modality separation and structured organization are performed from the three-modal data stream. The fusion data set generated based on the unified time axis is extracted respectively, namely the RGB-D image frame sequence corresponding to the visual modality, the six-axis inertial sensor data stream corresponding to the navigation modality, and the multi-channel audio sampling sequence corresponding to the speech modality. These three types of data are modally isolated to form image sequence subsets, inertial motion sequence subsets, and acoustic audio sequence subsets. These three types of modality subsets are input into the nested feature extractor respectively, and the feature construction task of the specific modality is performed in the different branch layers of the extractor. The deep image skeleton key point extraction layer receives a continuous RGB-D image frame sequence. This layer is based on a deep convolutional neural network architecture and integrates a key point positioning model. It can identify the main skeleton nodes of the elderly body in each frame of the image, including the head, neck, shoulders, elbows, wrists, chest, waist, hips, knees and ankles, etc., a total of multiple structural key points. The system represents the position of each key node in three-dimensional spatial coordinates and quantifies the velocity and acceleration of coordinate differences between frames for the same nodes. This generates deep visual fall feature vectors that describe the dynamic trends, posture stability, and local motion changes of the elderly person's limbs. The six-axis posture change detection layer receives a continuous motion sequence consisting of three-axis acceleration and three-axis angular velocity output from the navigation modality. This processing layer, centered on posture state estimation, performs first-order gradient calculations on the acceleration vector change and angular velocity change within each time slice, extracting key features such as acceleration and angular velocity gradients that can indicate sudden inertial changes, motion imbalance, or structural impact. The system then applies denoising filtering based on an inertial sensor noise model and calculates the local mean and range of feature change rates within a short sliding window. This constructs a set of navigation inertial fall feature vectors that represent center of gravity stability, posture perturbation amplitude, and rotational velocity trends. These vectors characterize the sudden jump in acceleration, violent angular velocity fluctuations, and sudden inertial behavior that occurs during the transition from standing to falling. At the same time, the voice event mutation recognition layer processes the audio data stream collected by the multi-channel microphone array. This layer uses a dual mechanism of spectral feature extraction and sound pressure analysis. By mapping the original waveform signal to the time-frequency space and calculating the energy concentration, frequency jump points, and sound pressure change trends within each time period, it can capture sudden acoustic signals related to falls. The system sets the sound pressure threshold and mutation slope indicators. If the audio data shows a sharp increase in sound pressure level or a sudden change in spectral energy at a certain moment, or if an audio pattern containing voice clips such as calls for help or screams is detected, it is immediately marked as a suspicious event window and an environmental voice fall feature vector associated with the event window is generated. This feature vector can express the elderly person's verbal calls for help due to a fall, the impact sound when hitting the ground, or the sound of furniture colliding. Feature alignment is performed on the depth vision fall feature vector, the navigation inertial fall feature vector, and the environmental voice fall feature vector.Using a sequential frame matching mechanism, the system searches the navigation and voice data for the nearest inertial data segment and audio analysis window for each depth vision frame's timestamp. Modalities that don't completely overlap in time are then interpolated and reconstructed or windowed to achieve millisecond-level temporal consistency between the modal features. The system then organizes the aligned depth vision, navigation inertial, and ambient voice fall feature vectors into a unified structure indexed by timestamps, forming a structured tri-modal fall feature vector set.

[0035] In a specific embodiment, the process of executing step 104 may specifically include the following steps: Based on the trimodal fall feature vector group, a strategy space consisting of high-confidence detection, medium-confidence detection, low-confidence detection, and exit from competition is set for the participants in the deep vision game. A strategy space consisting of dominant motion detection, auxiliary posture detection, baseline monitoring, and standby state is set for the participants in the navigation sensor game. A strategy space consisting of emergency call detection, abnormal sound monitoring, ambient sound analysis, and silent mode is set for the participants in the speech recognition game. This results in a three-party game model. According to the three-party game model, the accuracy benefits of the depth vision game participants minus the computational cost loss and environmental interference loss, the motion detection accuracy benefits of the navigation sensor game participants minus the drift error loss and vibration interference loss, and the acoustic event recognition rate benefits of the speech recognition game participants minus the noise impact loss and false trigger penalty loss are calculated to obtain the game benefit matrix. Conduct a competitive analysis on the game payoff matrix to obtain the competitive analysis results, and then solve the optimal weight based on the competitive analysis results to obtain the dynamic fusion weight coefficient; The three-modal fall feature vector group is fused based on the dynamic fusion weight coefficient to obtain the fall fusion feature vector.

[0036] Specifically, based on the trimodal fall feature vector set, a multi-agent game decision-making mechanism is introduced. Each modality is abstracted as an independent game participant. With the goal of optimizing strategic benefits, a three-party game model is established to drive the dynamic allocation of modal weights. In this model, the depth vision modality is constructed as the first game agent, and its policy space is set to four options: high-confidence detection, medium-confidence detection, low-confidence detection, and exit from the competition. This reflects the modality's judgment strength and contribution to fall events under different confidence scenarios. The navigation sensor modality is then constructed as the second game agent. Its policy space consists of dominant motion detection, auxiliary posture detection, baseline monitoring, and standby state, aiming to characterize its ability to play a primary or auxiliary information support role in the current scenario. The speech recognition modality is the third game agent. Its policy space consists of emergency call detection, abnormal sound detection, ambient sound analysis, and silence mode, which reflects whether the modality detects useful audio events or is simply silent in actual operation. The three participants use the fall feature vectors extracted during the current period as input and dynamically adjust their joint decision-making responses based on the actual state of their respective strategy spaces. Based on the aforementioned strategy definitions, the system constructs payoff functions for each of the three participants and uses them to calculate the game payoff matrix for the three-party game model. For the depth vision participant, its payoff is dominated by the accuracy of fall detection output by the image feature recognition model, while deducting two types of losses: the computational overhead required for image processing and the impact of environmental interference caused by image quality degradation. The navigation sensor participant's payoff is primarily based on the accuracy of gesture recognition, while the losses include drift errors caused by factors such as sensor zero drift and temperature drift, as well as signal instability caused by mechanical vibration. The speech recognition participant's payoff is derived from the recognition rate of acoustic events. Accurately recognizing sounds such as calls for help and collisions significantly increases its payoff. However, misidentification in the presence of noise and false triggering of background silence are the main sources of losses. By constructing a three-way payoff-loss balance model, a game payoff matrix is ​​obtained. The payoffs under each strategy combination are jointly modeled using true eigenvalues ​​and environmental feedback conditions to ensure that the matrix results are operational and contextually adaptable. This payoff matrix is ​​input into the game solver for competitive analysis. By modeling the strategy selection of the three players in the strategy space, the optimal strategy combination for each modality in the current scenario is determined by combining the concept of Nash equilibrium. This analysis process determines which modality should be the dominant modality and identifies which modalities should be relegated to a supporting role or temporarily withdraw from integrated participation in the current environment.Based on the results of the competitive analysis, the marginal contribution of each modality at game equilibrium is extracted, and the marginal benefits of the three modalities are normalized to form a set of dynamic fusion weight coefficients for weighted fusion. This coefficient set reflects the actual information effectiveness and system resource cost-effectiveness of the three modalities for the fall event recognition task in the current time period, and is the core driving factor for the dynamic fusion of multimodal information. This set of dynamic fusion weight coefficients is applied to the fusion process of the three-modal fall feature vector group. The depth vision feature vector is multiplied by its corresponding visual weight coefficient, and the navigation inertial feature vector and speech recognition feature vector are multiplied by the navigation weight coefficient and speech weight coefficient, respectively. The three are then weighted and superimposed on the feature dimension to generate the fused fall feature vector.

[0037] In a specific embodiment, the execution step performs a competition analysis on the game payoff matrix to obtain a competition analysis result, and solves the optimal weight based on the competition analysis result to obtain the dynamic fusion weight coefficient. The process can specifically include the following steps: Based on the game payoff matrix, a first comprehensive score of skeleton integrity, motion amplitude, and environment quality is calculated for participants in the depth vision game; a second comprehensive score of acceleration change rate, posture stability, and sensor accuracy is calculated for participants in the navigation sensor game; and a third comprehensive score of acoustic feature matching and signal-to-noise ratio is calculated for participants in the speech recognition game. executing the dominant power competition rules according to the first comprehensive score, the second comprehensive score, and the third comprehensive score to obtain the dominant power allocation result; The dominant power distribution result is input into the Nash equilibrium algorithm for iterative optimization to obtain the equilibrium solution data that meets the Nash equilibrium conditions; Based on the balanced solution data, the optimal weight allocation is performed to obtain the dynamic fusion weight coefficient including the depth vision weight coefficient, the navigation sensor weight coefficient, and the speech recognition weight coefficient.

[0038] Specifically, based on the game payoff matrix, a modality quality evaluation index is introduced. A comprehensive scoring system for dominance competition and equilibrium optimization is calculated for each type of game participant. The quality score of deep vision game participants is analyzed by analyzing the structural integrity of the skeleton features extracted during the current time period. This parameter is calculated by counting the ratio of the number of successfully identified skeleton key points to the total number of points in each frame, combined with the node stability tracking rate between consecutive frames to assess the integrity of the image sequence. The motion amplitude index is used to measure the range of dynamic body changes exhibited by elderly people in the image sequence. The system calculates the Euclidean displacement amplitude between the skeleton center of gravity, limb extremities, and joints to extract spatial activity energy information, which reflects their sensitivity to sudden falls. A third indicator, environmental quality, is then combined to quantify the degree of external environmental interference with visual recognition by evaluating brightness uniformity, occlusion ratio, and background clutter in the image. After normalization of these three indicators, a weighted aggregation strategy is used to generate a first comprehensive score for the visual modality, representing the level of perceptual performance within the current frame segment. For participants in the navigation sensor game, a second comprehensive scoring system is constructed based on three dimensions: acceleration rate, attitude stability, and sensor accuracy. The acceleration rate reflects the intensity of the elderly person's body movement per unit time. The system determines whether there is a clear impact or falling trend by calculating statistical features such as the inter-frame gradient, variance, and range of the acceleration vector. The attitude stability index evaluates the smoothness of the change in the IMU output angular velocity within a specific time window and the proportion of the attitude angle stable area. If the attitude data shows violent jumps, it indicates that the elderly person is in an unstable state, otherwise it indicates that the elderly person is in a state of stillness or slow movement. Sensor accuracy is comprehensively judged based on the IMU's transmission frame loss rate, data packet integrity, and system calibration error. If the sensor has significant zero-bias drift or signal noise, the score should be reduced. These three features are ultimately combined into the second comprehensive score for the navigation mode. For participants in the speech recognition game, the scoring model is centered around acoustic feature matching and signal-to-noise ratio (SNR). Acoustic feature matching primarily refers to the level of similarity between the Mel-frequency cepstral coefficients and energy distribution spectra extracted during the current time period and known fall-related speech templates. The system uses a dynamic time warping algorithm or a cosine distance function to compare the two and generate a corresponding score. The SNR evaluates the analytical validity and recognition robustness of the speech data segment by analyzing the energy ratio of the target signal to background noise. A low SNR indicates significant uncertainty in the audio features, which directly impacts the usability of the speech modality. These two metrics are combined to generate a third comprehensive score for the speech modality. Dominance competition rules are applied based on the first, second, and third comprehensive scores. Based on a relative competition model, the three modalities are considered candidate dominant information sources and prioritized according to their scores.If a modality scores significantly higher than the other two modalities, the system directly assigns dominance to that modality, making it the dominant participant in the current data fusion. If multiple modalities have similar scores, a modal game negotiation mechanism is activated, using a weighted balance strategy to allocate dominance. This allows for multi-modal collaborative dominance and ensures the system's flexible adaptability to complex scenarios. The resulting dominance allocation is output as a structured policy vector. This dominance allocation is then fed into the iterative optimization framework of a Nash equilibrium algorithm, which performs multiple rounds of policy updates. In each round, the algorithm performs response optimization operations on the policy space of each participating modality, searching for the strategy combination that maximizes its own payoff while holding the strategies of the other modalities constant. The algorithm then adjusts the strategy probabilities based on feedback from the payoff function, gradually approaching the optimal strategy set for all modalities. After several iterations, when no modality can achieve a single significant payoff under the current strategy, the system is considered to have reached a Nash equilibrium and outputs a set of equilibrium solution data that satisfies the equilibrium conditions. This solution represents the stable participation of each modality in data fusion under the current environment and characteristic performance, and fully considers the mutual influence and information redundancy of the game participants. The system constructs an optimal weight allocation module based on the equilibrium solution data. By normalizing the equilibrium probabilities corresponding to each modal strategy in the equilibrium solution and adjusting the weight ratios based on the results of the dominance competition, it ultimately generates a set of dynamic fusion weight coefficients, including those for depth vision, navigation sensors, and speech recognition.

[0039] In a specific embodiment, the execution step performs vector fusion on the three-modal fall feature vector group based on the dynamic fusion weight coefficient to obtain the fall fusion feature vector, which may specifically include the following steps: Perform time window processing based on the three-modal fall feature vector group to obtain time series matching window data; Based on the time-series matching window data, the vertical position change speed of the body center and the body tilt angle data are extracted from the deep visual features. The center of gravity acceleration vector and posture change angular velocity data are calculated from the navigation inertial features. The Mel-frequency cepstral coefficient analysis is used to detect the distress keywords and abnormal sound data from the ambient voice features to obtain the target fall feature data. An asymmetric time-series weight compensation field mechanism is used to adjust the weight of the target fall feature data, and a target weight coefficient combination including the weight parameters after compensation adjustment is obtained; The three-modal fall feature vector group is weightedly fused with the target weight coefficient combination to obtain the fall fusion feature vector that integrates the body tilt angle change of the depth image, the acceleration of the center of gravity displacement of the IMU, and the voice spectrum call for help features.

[0040] Specifically, the system performs frame-by-frame sliding processing on the three-modal feature vector group on a continuous time axis by setting a time window parameter of a fixed length. The time window is set to about three seconds, covering the entire period from the start of abnormal movement before the fall to the stillness after the fall, and controlling the scale of feature data. In each sliding window, the system extracts the time series feature data of all depth vision, navigation inertia and environmental voice modes within the time period, forming a window data structure with local time series feature alignment properties, and obtaining time series matching window data. Based on the time series matching window data, the data in each modality is subjected to secondary deconstruction and feature drill-down processing of key behavioral indicators. In the depth vision modality, the system estimates the center of gravity of the skeleton structure extracted from each frame of the image, calculates the inter-frame displacement value of the center point of the body in the vertical direction (y-axis), and calculates the vertical position change speed based on this, which is used as the core indicator to measure whether the elderly have a trend of falling rapidly. The system also calculates the angle between the torso and the ground based on the spatial configuration between the neck and hips, and the shoulders and knees, to determine the body tilt angle. This angle effectively determines whether the elderly person has lost control and whether the tilt has reached the critical point of falling. In the navigation inertial modality, the system calculates the three-axis composite center of gravity acceleration vector based on continuous acceleration vector data. Combined with the angular velocity data, it estimates the angular velocity of the posture change per unit time. These two metrics are used together to identify the presence of sudden motion disturbances or impact states. In the ambient speech modality, the system performs Mel-frequency cepstral coefficient analysis on the spectral data within the audio window to extract the energy peak frequency band characteristics in the audio. Using a keyword matching mechanism, it identifies distress words such as "help" and "help" or abnormal spectral mutations. If present, it is identified as a speech emergency. The extracted metrics from these three modalities are aggregated within the time window to form unified target fall feature data, which is used as a multimodal joint feature basis to determine the presence of fall behavior. To improve the accuracy and adaptability of feature fusion, an asymmetric temporal weight compensation field mechanism is introduced based on the aforementioned target fall feature data. The design principle of this mechanism is that the temporal locations of sensitivity to fall events vary across modalities. For example, the depth vision modality is more sensitive to the body's falling process, exhibiting significant changes in tilt angle and center of gravity velocity starting one second before a fall. The navigation modality is most sensitive to acceleration jumps and posture disturbances at the moment of impact, with its strongest response occurring later in the fall process. The speech modality emits sounds after or during a fall, so its response occurs at the end of the time window. To adapt to this asynchronous response structure of modalities, each target feature is assigned a different time-dominant segment within the time window, and dynamic compensation weight curves are set based on their response temporal structures, forming an asymmetric weight compensation field. The mechanism automatically adjusts the weight contribution ratio of each modality based on its feature strength and response location within the window, thereby generating a target weight coefficient combination that includes the compensated weight parameters.This target weight coefficient combination is applied to the trimodal fall feature vector group within the window, performing a weighted fusion operation at the modality level. The system weights and enhances the tilt angle change and center of gravity sinking speed in the depth vision features according to the compensated visual weight coefficient. At the same time, the navigation modality compensation weight is applied to the synthetic acceleration and attitude angular velocity in the navigation modality. The distress keyword matching results and spectral mutation features extracted in the voice modality are enhanced according to the voice modality weight. All weighted feature vectors are combined and normalized at the dimensional level to finally generate a fused fall feature vector.

[0041] In a specific embodiment, the process of executing step 105 may specifically include the following steps: The fall fusion feature vector is input into the multimodal forward prediction model, and the deep vision skeleton sequence prediction branch, the navigation sensor motion trajectory extrapolation branch, and the speech recognition emotional state assessment branch are started to perform three-way parallel prediction to obtain a parallel prediction data combination; Based on the combination of parallel prediction data, the deep visual skeleton sequence prediction branch performs motion trajectory prediction, the navigation sensor motion trajectory extrapolation branch performs acceleration and angular velocity state estimation and trajectory extrapolation, and the speech recognition emotional state assessment branch performs speech emotion change trend analysis and psychological stress state prediction, obtaining three-way prediction trajectory results; Comprehensive decision-making and fall judgment are performed based on the three-way predicted trajectory results and the fall fusion feature vector to obtain the fall detection result.

[0042] Specifically, a forward prediction model structure with multi-branch parallel processing capabilities is constructed, and the fall fusion feature vector is input into the multimodal forward prediction model. Within the forward prediction model, the system constructs three independent but collaborative prediction branches based on modal attributes. The first branch is the deep visual skeleton sequence prediction module, which uses a recursive neural network structure with time series as input. It preferably uses long short-term memory (LSTM) to perform state learning on the three-dimensional coordinate sequence of skeleton points in the past few frames. It then uses the learned spatial structure pattern to predict the dynamic trajectory of each key skeleton node in the future several frames. The second branch is the navigation inertial trajectory extrapolation module. This module extracts the acceleration and angular velocity subvectors from the fall fusion feature vector and combines them with the state transition model. It uses a Kalman filter or an extended Kalman filter to filter and predict the continuous acceleration state within the next second. It further calculates the angular velocity change trend to obtain the spatial trajectory evolution trend of the IMU in the short term. The third branch is the speech recognition emotion state assessment module. This module uses an emotion classification neural network to perform time series trend modeling on the Mel-frequency cepstral coefficient sequence or emotional pitch and intensity sequence extracted from the speech modality. By analyzing the acoustic emotional tonality change trajectory of the audio over the past few seconds, it predicts whether the future speech will show a higher degree of psychological stress response or panic calling trend. The three prediction branches use the same time window as the unit and are activated simultaneously during model execution, enabling simultaneous prediction of dynamic fall risk at the visual, inertial, and speech levels. The output data constitutes a parallel prediction data combination. After completing the three-branch parallel prediction, behavioral trend analysis and structural risk identification are performed on each predicted trajectory. Based on the position sequence of each key skeletal node over several future frames, the system calculates the vertical velocity of the torso's center of gravity, the trend of the skeletal tilt angle, and the degree of spatial contraction between the head, hips, and ankles to determine whether there is a trend of sustained displacement acceleration, structural collapse, or postural instability. If a rapid descent and sustained tilt are combined, it is marked as a visual pre-fall signal. Based on the navigation sensor prediction results, the system analyzes the overall slope of the future acceleration sequence and the fluctuation range of the angular velocity to determine whether there is a sudden change in the center of gravity or a trend of body rotation. If the predicted composite acceleration exceeds the gravitational constant or the angular velocity exceeds the normal motion threshold within a certain future time slice, the system determines it as an inertial impact warning signal. In the speech emotion prediction results, if the emotional transition index output by the prediction model continues to rise and is accompanied by spectral anomalies or sudden energy changes, it indicates that the elderly person is experiencing intense psychological stress during the fall process, and the system accordingly marks this as an emotionally arousing audio warning signal. The three prediction trajectories are timestamp-aligned to form a structured prediction result set with consistent temporal order, forming a pre-fall risk judgment system.The three prediction results are combined with the original fused fall feature vector for joint decision analysis, comprehensively considering the current feature state and future risk trends, and implementing a multimodal joint judgment mechanism. This mechanism builds a classifier based on classification models such as support vector machines, logistic regression, and multi-layer perceptrons. The model takes the combined representation of the fused feature vector and the three prediction vectors as input, and determines whether a fall event is imminent as the output target. A decision threshold is set within the model to control the output sensitivity. When the model output of the fall probability exceeds a preset threshold and any modal prediction result indicates a clear behavioral risk trend, the system directly issues a fall alert, along with the current risk score, warning time point, and dominant modality identifier. If the current feature state does not reach the risk level but the predicted trajectory approaches the risk boundary, the system issues a low-level warning or a neutral observation status, indicating that the behavior has entered an unstable zone. The entire comprehensive judgment process constructs a dynamic risk trajectory on the time axis based on the fusion of current and future information.

[0043] The above describes the multimodal data fusion processing method in the embodiment of the present invention. The following describes the multimodal data fusion processing system in the embodiment of the present invention. Figure 2 In one embodiment of the present invention, a multimodal data fusion processing system includes: The collection module 201 is used to collect a multimodal dataset containing elderly people's activities through an intelligent elderly care robot; A correction module 202 is configured to construct a timestamp calibration matrix based on the multimodal data set and perform dynamic weight correction on the timestamp calibration matrix using a trimodal data envelope delay compensation algorithm to obtain a time-synchronized trimodal data stream; A feature extraction module 203 is configured to input the time-series synchronized trimodal data stream into a nested feature extractor for hierarchical feature extraction to obtain a trimodal fall feature vector group; A vector fusion module 204 is configured to perform vector fusion on the three-modal fall feature vector group to obtain a fall fusion feature vector; The output module 205 is used to input the fall fusion feature vector into the multimodal forward prediction model to predict the fall risk trajectory and output the fall detection result.

[0044] Through the collaborative efforts of these components, a trimodal data envelope delay compensation algorithm and a timestamp calibration matrix are employed to precisely synchronize heterogeneous sampling frequencies: 30fps for the RGB-D camera, 100Hz for the six-axis IMU sensor, and 16kHz for the microphone array. This ensures millisecond-level alignment of sensor data with diverse physical characteristics on a common time base, eliminating the feature mismatch problem inherent in traditional approaches due to timing deviations. A three-party competitive game analysis framework is employed to construct the three modalities—deep vision, navigation sensors, and speech recognition—as intelligent game participants. A Nash equilibrium algorithm is employed to dynamically and adaptively adjust the weights of each modality, overcoming the limitations of traditional fixed-weight fusion methods that are unable to adapt to changing environments. This significantly improves the system's adaptability and robustness across diverse scenarios. A hierarchical processing mechanism employs nested feature extractors to accurately extract and align features from deep image skeleton keypoints, six-axis IMU pose gradients, and sudden changes in speech sound pressure levels. This ensures that features from different modalities accurately describe the individual patient's state at the same moment, avoiding the timing inconsistencies inherent in traditional approaches that process features separately. Through a three-way parallel prediction mechanism, the system can predict and analyze changes in the elderly's body posture within the critical time window before they fall, breaking through the limitations of traditional methods that only judge at the current moment and having the ability to identify fall risks in advance. By adopting an asymmetric temporal weight compensation field mechanism, the system dynamically adjusts the weight distribution according to the different characteristic intensities of rapid falling movements, impact signals, and distress signals, effectively solving the problem of different contributions of the three sensors at different stages of a fall, and significantly improving the ability to distinguish between the elderly's slow rise and real fall behaviors. The changes in the body tilt angle of the depth image, the acceleration of the center of gravity displacement of the IMU, and the distress features of the voice spectrum are weighted and fused on a unified time axis to form a complete detection result including the probability of fall, confidence score, occurrence timestamp, and warning level, providing a reliable basis for the emergency response of the elderly care robot.

[0045] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0046] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0047] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fusion processing method for multimodal data, characterized in that: include: Collect multimodal datasets containing elderly activities through intelligent elderly care robots; A timestamp calibration matrix is ​​constructed based on the multimodal data set, and a trimodal data envelope delay compensation algorithm is used to perform dynamic weight correction on the timestamp calibration matrix to obtain a time-synchronized trimodal data stream; Inputting the time-series synchronized trimodal data stream into a nested feature extractor for hierarchical feature extraction to obtain a trimodal fall feature vector group; performing vector fusion on the three-modal fall feature vector group to obtain a fall fusion feature vector; The fall fusion feature vector is input into a multimodal forward prediction model to predict the fall risk trajectory and output a fall detection result.

2. The multimodal data fusion processing method according to claim 1, characterized in that: The multimodal dataset containing elderly activities collected by the intelligent elderly care robot includes: The RGB-D camera is used to collect depth visual information of the elderly's activity area to obtain raw depth image data. At the same time, the six-axis IMU sensor is used to collect inertial motion of the elderly's body posture to obtain raw IMU inertial data. The microphone array is used to collect omnidirectional audio of the ambient sound to obtain raw audio data. Performing depth restoration and color correction on the original depth image data to obtain corrected depth image data, performing zero bias correction and filtering on the original IMU inertial data to obtain corrected IMU inertial data, and performing background noise suppression and frequency domain analysis on the original audio data to obtain corrected audio data; The corrected depth image data, the corrected IMU inertial data, and the corrected audio data are time-series aligned to construct a multimodal dataset including a depth vision data stream, a navigation inertial data stream, and an environmental voice data stream.

3. The multimodal data fusion processing method according to claim 1, characterized in that: The method of constructing a timestamp calibration matrix based on the multimodal data set and dynamically weight-correcting the timestamp calibration matrix using a trimodal data envelope delay compensation algorithm to obtain a time-synchronized trimodal data stream includes: Organizing the depth vision reference timestamp, the navigation sensor reference timestamp, the voice sensor reference timestamp, and the time deviation values ​​between each modality in the multimodal dataset into a timestamp calibration matrix; The deviation of the heterogeneous sampling frequency differences of the RGB-D camera, the six-axis IMU sensor and the microphone array is calculated according to the timestamp calibration matrix to obtain the timing deviation analysis result; Using a trimodal data envelope delay compensation algorithm, dynamically weighting the timing deviation analysis results is performed to obtain a synchronous timestamp combination; The multimodal data set is time-rearranged and interpolated and aligned based on the synchronization timestamp combination to obtain a time-synchronized trimodal data stream.

4. The multimodal data fusion processing method according to claim 3, characterized in that: The three-modal data envelope delay compensation algorithm is used to dynamically assign weights to the timing deviation analysis results to obtain a synchronization timestamp combination, including: Based on the timing deviation analysis results, statistical analysis is performed on the real-time response delay characteristics of the depth vision modality, the navigation sensor modality, and the speech recognition modality to obtain response delay characteristic data; Dynamically allocating a depth vision weight coefficient, a navigation sensor weight coefficient, and a speech recognition weight coefficient according to the response delay characteristic data to obtain a dynamic weight coefficient combination; Inputting the dynamic weight coefficient combination into the trimodal data envelope delay compensation algorithm for weighted summation to obtain a timestamp compensation value; Deviation correction and synchronization alignment are performed based on the timestamp compensation value to obtain a synchronization timestamp combination including a depth vision synchronization timestamp, a navigation sensor synchronization timestamp, and a voice sensor synchronization timestamp.

5. The multimodal data fusion processing method according to claim 1, characterized in that: The step of inputting the time-synchronized trimodal data stream into a nested feature extractor for hierarchical feature extraction to obtain a trimodal fall feature vector group includes: Extracting a combination of separate data streams including an RGB-D image sequence, a six-axis IMU motion sequence, and a multi-channel audio sequence based on the time-series synchronized three-modal data stream; Using the deep image skeleton key point extraction layer in the nested feature extractor to perform human skeleton detection on the RGB-D image sequence to obtain a deep visual fall feature vector; The six-axis attitude change detection layer in the nested feature extractor is used to calculate the acceleration and angular velocity gradients of the six-axis IMU motion sequence to obtain the navigation inertial fall feature vector; Using the speech event mutation recognition layer in the nested feature extractor to perform sound pressure level detection on the multi-channel audio sequence to obtain an environmental speech fall feature vector; Feature alignment is performed on the depth vision fall feature vector, the navigation inertia fall feature vector, and the environmental voice fall feature vector to obtain a three-modal fall feature vector group including depth vision features, navigation inertia features, and environmental voice features.

6. The multimodal data fusion processing method according to claim 1, characterized in that: The performing vector fusion on the three-modal fall feature vector group to obtain a fall fusion feature vector includes: Based on the three-modal fall feature vector group, a strategy space including high-confidence detection, medium-confidence detection, low-confidence detection, and exit from competition is set for each participant in the deep vision game; a strategy space including dominant motion detection, auxiliary posture detection, baseline monitoring, and standby state is set for each participant in the navigation sensor game; and a strategy space including emergency call detection, abnormal sound monitoring, ambient sound analysis, and silent mode is set for each participant in the speech recognition game, thereby obtaining a three-party game model. According to the three-party game model, the accuracy benefit of the depth vision game participants minus the computational cost loss and the environmental interference loss, the motion detection accuracy benefit of the navigation sensor game participants minus the drift error loss and the vibration interference loss, and the acoustic event recognition rate benefit of the speech recognition game participants minus the noise impact loss and the false trigger penalty loss are calculated respectively to obtain a game benefit matrix; Performing a competition analysis on the game payoff matrix to obtain a competition analysis result, and performing an optimal weight solution based on the competition analysis result to obtain a dynamic fusion weight coefficient; The three-modal fall feature vector group is vector-fused based on the dynamic fusion weight coefficient to obtain a fall fusion feature vector.

7. The multimodal data fusion processing method according to claim 6, characterized in that: The competitive analysis is performed on the game payoff matrix to obtain a competitive analysis result, and the optimal weight is solved based on the competitive analysis result to obtain a dynamic fusion weight coefficient, including: Based on the game payoff matrix, a first comprehensive score of skeleton integrity, motion amplitude, and environment quality is calculated for participants in the depth vision game; a second comprehensive score of acceleration change rate, posture stability, and sensor accuracy is calculated for participants in the navigation sensor game; and a third comprehensive score of acoustic feature matching and signal-to-noise ratio is calculated for participants in the speech recognition game. executing a dominance competition rule process based on the first comprehensive score, the second comprehensive score, and the third comprehensive score to obtain a dominance allocation result; Inputting the dominant power allocation result into a Nash equilibrium algorithm for iterative optimization to obtain equilibrium solution data that meets the Nash equilibrium conditions; Optimal weight distribution is performed based on the equalization solution data to obtain a dynamic fusion weight coefficient including a depth vision weight coefficient, a navigation sensor weight coefficient, and a speech recognition weight coefficient.

8. The multimodal data fusion processing method according to claim 6, characterized in that: The performing vector fusion on the three-modal fall feature vector group based on the dynamic fusion weight coefficient to obtain a fall fusion feature vector includes: Performing time window processing based on the three-modal fall feature vector group to obtain time series matching window data; According to the time series matching window data, the vertical position change speed of the body center point and the body tilt angle data are extracted from the depth visual features, the center of gravity acceleration synthesis vector and the posture change angular velocity data are calculated from the navigation inertial features, and the distress keywords and abnormal sound data are detected from the environmental voice features through Mel frequency cepstral coefficient analysis to obtain the target fall feature data; Adopting an asymmetric time-series weight compensation field mechanism to adjust the weight of the target fall feature data, and obtaining a target weight coefficient combination including the weight parameters after compensation adjustment; The three-modal fall feature vector group is weightedly fused in combination with the target weight coefficient combination to obtain a fall fusion feature vector that fuses the body tilt angle change of the depth image, the IMU center of gravity displacement acceleration, and the voice spectrum call for help feature.

9. The multimodal data fusion processing method according to claim 8, characterized in that: The step of inputting the fall fusion feature vector into a multimodal forward prediction model to predict the fall risk trajectory and outputting the fall detection result includes: Inputting the fall fusion feature vector into a multimodal forward prediction model, starting a deep visual skeleton sequence prediction branch, a navigation sensor motion trajectory extrapolation branch, and a speech recognition emotional state assessment branch to perform three-way parallel prediction to obtain a parallel prediction data combination; Based on the parallel prediction data combination, the deep visual skeleton sequence prediction branch performs motion trajectory prediction, the navigation sensor motion trajectory extrapolation branch performs acceleration and angular velocity state estimation and trajectory extrapolation, and the speech recognition emotional state assessment branch performs speech emotion change trend analysis and psychological stress state prediction, thereby obtaining three-way prediction trajectory results; Comprehensive decision-making and fall judgment are performed based on the three-way predicted trajectory results and the fall fusion feature vector to obtain a fall detection result.

10. A multimodal data fusion processing system, characterized in that: A method for fusion processing of multimodal data according to any one of claims 1 to 9, wherein the multimodal data fusion processing system comprises: The acquisition module is used to collect multimodal datasets containing elderly people's activities through intelligent elderly care robots; a correction module, configured to construct a timestamp calibration matrix based on the multimodal data set, and perform dynamic weight correction on the timestamp calibration matrix using a trimodal data envelope delay compensation algorithm to obtain a time-synchronized trimodal data stream; a feature extraction module, configured to input the time-series synchronized trimodal data stream into a nested feature extractor for hierarchical feature extraction to obtain a trimodal fall feature vector group; a vector fusion module, configured to perform vector fusion on the three-modal fall feature vector group to obtain a fall fusion feature vector; The output module is used to input the fall fusion feature vector into the multimodal forward prediction model to predict the fall risk trajectory and output the fall detection result.

Images