Fall detection method and system based on multi-modal data fusion
Through multimodal data fusion technology, wav2vec 2.0 and LSTM models are used to extract features, and combined with Transformer model and Shapley value calculation, the false alarm and missed alarm problems in the existing fall detection methods are solved, and the accuracy of fall detection is improved and robustness, achieving more comprehensive testing results.
Patent Information
- Application Number
- CN202411006200.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-15
AI Technical Summary
The existing fall detection methods, including methods based on attitude sensors and computer vision, have problems with false alarms and missed reports, and the multimodal data fusion technology has shortcomings in capturing complex relationships between features, which affects the accuracy of detection.
The fall detection method based on multimodal data fusion is adopted, and the deep fusion features are finally generated by collecting audio and acceleration data, using wav2vec 2.0 and LSTM models. , design the loss function in combination with classification loss and consistency loss to improve the classification accuracy of the model.
Effectively capture detailed information and dynamic changes in audio and acceleration data, handle uncertainty and conflict information, improve the robustness and accuracy of detection results, reduce false alarms and missed reports, and enhance the reliability and real-time system sex.
Smart Images

Figure CN118965264A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of multimodal data fusion, and relates to a fall detection method and system based on multimodal data fusion. Background Art
[0002] As the global population aging problem intensifies, the health and safety of the elderly are receiving more and more attention. Due to the decline of physical functions, the elderly are prone to fall during daily activities, which not only causes physical injuries but may even endanger their lives. Therefore, how to effectively monitor the activity status of the elderly and detect and alarm falls in time has become a technical problem that needs to be solved urgently. At present, the market mainly includes fall detection methods based on posture sensors and fall detection methods based on computer vision.
[0003] The method based on posture sensors monitors the body posture of the elderly in real time by wearing posture sensors on certain parts of the body. When the sensor detects abnormal acceleration or posture changes, it will determine whether a fall has occurred and issue an alarm. The advantages of this method are strong real-time performance and easy deployment, but due to the single data collected, sudden acceleration or posture changes in daily activities, such as bending over to pick up something or turning around suddenly, will be mistaken for a fall. In addition, due to insufficient sensor sensitivity or the slow movements of the elderly when falling, missed reports may occur, resulting in failure to detect the fall event. Therefore, the fall detection method based on posture sensors is prone to false positives and missed reports.
[0004] Computer vision-based methods use cameras to capture video data, analyze the activity trajectory and posture changes of the elderly through deep learning algorithms, and identify fall events. The advantage of this method is that it does not require users to wear equipment and can cover a larger monitoring range. However, computer vision methods also face significant challenges. Camera deployment costs are high and coverage is limited. Camera monitoring may cause concerns about privacy protection, especially in public places such as homes and hospitals. In addition, video analysis has high requirements for lighting conditions, and changes in ambient light may affect the accuracy of detection. Video data processing requires strong computing power, which may become a bottleneck in real-time monitoring and analysis, affecting the real-time performance and reliability of the system.
[0005] Multimodal data fusion technology comprehensively processes and analyzes data from different modalities (such as vision, sound, acceleration, etc.), utilizing the complementary advantages of each data source to improve detection accuracy and robustness in order to obtain more comprehensive detection results.
[0006] However, existing multimodal data fusion technology is insufficient in capturing the complex relationships between features, which affects the accuracy of the final detection. Summary of the invention
[0007] In order to solve the above problems in the prior art, the present invention adopts a fall detection method based on multimodal data fusion, comprising: collecting multimodal data, inputting the multimodal data into a trained fall detection model, and obtaining a detection result;
[0008] The training process of the fall detection model includes:
[0009] S1, collecting multimodal data and extracting multimodal features based on the multimodal data;
[0010] S2, performing preliminary feature fusion on the multimodal features to obtain preliminary fusion features;
[0011] S3, calculating the weighted confidence Shapley value according to the preliminary fusion features, and performing weighted fusion on the preliminary fusion features according to the weighted confidence Shapley value to obtain the weighted fusion features;
[0012] S4. Use the Transformer model to process the weighted fusion features to obtain deep fusion features;
[0013] S5. Obtain classification results based on deep fusion features;
[0014] S6. Calculate the loss function value based on the classification results, the weighted preliminary fusion features, and the deep fusion features, and update the model parameters based on the loss function value. When the loss function value is minimized, the model training is completed.
[0015] The multimodal data includes: audio data and acceleration data; extracting multimodal features includes: extracting audio features based on the audio data using the wav2vec2.0 model, and extracting acceleration features based on the acceleration data using the LSTM model.
[0016] The preliminary feature fusion of multimodal features includes: aligning the multimodal features to the same feature space through linear transformation and adaptive normalization methods, and performing preliminary feature fusion on the aligned multimodal features to obtain preliminary fused features.
[0017] Calculating the weighted confidence Shapley value based on the preliminary fusion features includes:
[0018] Calculate the evidence confidence function of each modal feature in the preliminary fusion features, calculate the confidence Shapley value of each modal feature according to the evidence confidence function and the preliminary fusion features, calculate the divergence between the confidence Shapley values of each modal feature, calculate the Shapley similarity matrix according to the divergence between the confidence Shapley values, calculate the confidence Shapley reliability according to the Shapley similarity matrix, calculate the weighted confidence Shapley value according to the confidence Shapley reliability, weight the preliminary fusion features according to the weighted confidence Shapley value, and obtain the weighted fusion features.
[0019] The evidence confidence function for calculating each modality feature includes:
[0020]
[0021] Among them, i is the index of the modal feature in the preliminary fusion feature, m i (F) is the basic probability distribution, which indicates the support of modal feature i for subset F, E is the input set of the evidence confidence function, F is a subset of set E, Bel i (E) is the evidence confidence function of modal feature i.
[0022] Calculate the confidence Shapley value x for each modal feature i include:
[0023]
[0024] Where M is the number of all modal features, γ k is a combination coefficient, i represents the index of the modal feature, S represents the subset that does not contain the modal feature i, k represents the size of the subset S, TS i is the set of modal features i, TS represents the set of all modal features, Bel i () is the evidence confidence function of modal feature i.
[0025] Calculate the confidence Shapley reliability Rel i include:
[0026]
[0027] Among them, Rel T is the total reliability of all modal features, M is the number of all modal features, i and j are the indexes of modal features, η(x i ,x j ) is the Shapley similarity matrix, x i is the confidence Shapley value of modal feature i.
[0028] Calculate weighted confidence Shapley value Including: According to the confidence Shapley reliability Rel i Calculate the weight vector w i , according to the weight vector w i The confidence Shapley values of the modal features are weighted and combined to obtain the weighted confidence Shapley value
[0029] The loss function is:
[0030] L=L CE +λL consistency
[0031] Among them, L CE is the classification loss, L consistency is the consistency loss of weighted fusion features and deep fusion features, and λ is the adjustment parameter.
[0032] A system based on a fall detection method comprises: a lightweight feature extraction module, a feature alignment and adaptive fusion module, a collaborative Transformer module and a single-task optimization module; the lightweight feature extraction module is used to extract multimodal features from collected multimodal data; the feature alignment and adaptive fusion module is used to align and fuse the extracted multimodal features to generate preliminary fusion features; the collaborative Transformer module is used to further fuse the preliminary fusion features to generate deep fusion features; the single-task optimization module is used to classify the deep fusion features to generate classification results.
[0033] Beneficial effects:
[0034] 1. The present invention uses wav2vec 2.0 and LSTM models to extract audio and acceleration features respectively, which can effectively capture the detailed information in audio data and the dynamic change characteristics in acceleration data, and provide rich information for subsequent feature fusion; 2. The present invention introduces confidence and reliability calculations when calculating Shapley values, so that the fusion process can better handle uncertainty and conflicting information, thereby improving the robustness of the results; 3. The present invention introduces JS divergence and similarity matrix when calculating Shapley values, which can more accurately handle redundant information between features and reduce their negative impact, thereby improving the reliability and accuracy of the fusion results; 4. The present invention designs a loss function for a multimodal fusion classification task by combining classification loss and consistency loss, thereby improving the classification accuracy of the model. ; 5. The present invention effectively captures the relationship between the two modalities through the attention mechanism and cross attention mechanism of the transformer model, thereby obtaining a more accurate fusion result. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 A flow chart of a fall detection method based on multimodal data fusion provided by an embodiment of the present invention;
[0036] Figure 2 A structural diagram of a fall detection method based on multimodal data fusion provided by an embodiment of the present invention;
[0037] Figure 3 A system schematic diagram of a fall detection method based on multimodal data fusion provided by an embodiment of the present invention; DETAILED DESCRIPTION
[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0039] like Figure 1 , Figure 2 As shown, the embodiment of the present invention adopts a fall detection method based on multimodal data fusion, including: collecting multimodal data through a smart bracelet, the multimodal data including but not limited to: audio data and acceleration data; inputting the multimodal data into a trained fall detection model to obtain a detection result;
[0040] The training process of the fall detection model includes:
[0041] S1, collect multimodal data through the smart bracelet, and extract multimodal features based on the multimodal data;
[0042] S2, performing preliminary feature fusion on the multimodal features to obtain preliminary fusion features;
[0043] S3, calculating the weighted confidence Shapley value according to the preliminary fusion features, and performing weighted fusion on the preliminary fusion features according to the weighted confidence Shapley value to obtain the weighted fusion features;
[0044] S4. Use the Transformer model to process the weighted fusion features to obtain deep fusion features;
[0045] S5. Obtain classification results based on deep fusion features;
[0046] S6. Calculate the loss function value based on the classification results, the weighted preliminary fusion features, and the deep fusion features, and update the model parameters based on the loss function value. When the loss function value is minimized, the model training is completed.
[0047] Collecting multimodal data through smart bracelets includes: collecting audio data and acceleration data in real time through smart bracelets worn on the wrists of the elderly; the built-in microphone of the bracelet is used to collect audio data, and the nine-axis sensor is used to collect acceleration data.
[0048] Extracting multimodal features includes: using a wav2vec 2.0 model to perform feature extraction on the collected audio data to obtain an audio feature A; using an LSTM model to perform feature extraction on the collected acceleration data to obtain an acceleration feature S; the wav2vec 2.0 is an audio representation learning model.
[0049] The preliminary feature fusion of multimodal features includes: aligning audio features and acceleration features through linear transformation and adaptive normalization, and using Multi-Layer Perceptron (MLP) for preliminary feature fusion. The specific steps are as follows:
[0050] Step 1: Align the audio features and acceleration features to the same feature space through linear transformation and adaptive normalization; the specific steps are as follows:
[0051] Perform linear transformation on audio features and acceleration features respectively:
[0052] A * =W a A+b a
[0053] S * =W s S+b s
[0054] Among them, A * and S * is the result of linear transformation of audio features and acceleration features, W a and W s is the linear transformation matrix, b a and b s is the bias vector.
[0055] Normalize the transformed features:
[0056]
[0057]
[0058] Among them, A^ and S^ are the normalized results, μ a and σ a is the mean and standard deviation of the audio features, μ s and σ s are the mean and standard deviation of the acceleration feature.
[0059] Step 2: Perform a preliminary fusion of the aligned audio features and acceleration features. The specific steps are as follows:
[0060] Concatenate the aligned audio features and acceleration features into a comprehensive feature vector X i :
[0061] X i =concat(A^,S^)
[0062] The comprehensive feature vector is input into MLP for nonlinear transformation to obtain the preliminary fusion feature φ i :
[0063] φ i =MLP(X i )
[0064] Among them, the initial fusion feature φ i Contains n modal features, and concat is a concatenation operation.
[0065] Calculating the weighted confidence Shapley value based on the preliminary fusion features includes:
[0066] Step 1: Calculate the evidence confidence function Bel for each modal feature i in the preliminary fusion feature i (E):
[0067]
[0068] in, is the basic probability distribution, indicating the support of modal feature i for subset F, p i (B) is the probability of subset B, which is a subset of F, When E is the complete set of events or states, E is the input set of the evidence confidence function, and F is a subset of set E.
[0069] Step 2: According to the evidence confidence function Bel i (E) Calculate the confidence Shapley value x of each modality feature with the preliminary fusion feature i :
[0070]
[0071] Where M is the number of all modal features, is a combination coefficient, i represents the index of the modal feature, S represents the subset that does not contain the modal feature i, k represents the size of the subset S of the modal feature, TS i is the set of modal features i, and TS represents the set of all modal features.
[0072] Step 3: Calculate the divergence (Shapley Jensen-Shannon, JS)δ(x i ,x j ):
[0073]
[0074] where H(·) is the Shannon entropy, which measures the difference between the confidence Shapley values.
[0075] Step 4: According to the divergence δ(x i ,x j ) Calculate the Shapley similarity matrix η(x i ,x j ):
[0076] η(x i ,x j )=1-δ(x i ,x j )
[0077] Step 5: According to the Shapley similarity matrix η(x i ,x j ) Calculate the confidence Shapley reliability Rel i :
[0078]
[0079]
[0080] Among them, Rel T is the total reliability of all modal features, i and j are the indexes of the modal features.
[0081] Step 6: According to the confidence Shapley reliability Rel i Calculate the weight vector w i :
[0082]
[0083] Step 7: According to the weight vector w i Calculate weighted confidence Shapley value
[0084]
[0085] According to the weighted confidence Shapley value The initial fusion features are weighted to obtain the weighted fusion features x weighted include:
[0086]
[0087] Where n is the number of modal features.
[0088] The weighted fusion features are input into the Transformer model for further feature processing and enhancement including:
[0089] x'=Transformer(x weighted )
[0090] Among them, x' is the deep fusion feature after Transformer processing.
[0091] The Transformer model can capture the complex dependencies between features through the self-attention mechanism and multi-layer structure, and further enhance the feature representation. The specific steps are as follows:
[0092] Step 1: Calculate the attention weights between input features and dynamically adjust the importance of features:
[0093] Q = x weighted W Q
[0094] K=x weighted W K
[0095] V = x weighted W V
[0096]
[0097] Among them, Q, K, V are query vector, key vector and value vector, d k is the dimension of the weighted preliminary fusion feature.
[0098] Step 2: Use the multi-head self-attention mechanism to capture the multi-level dependencies between features:
[0099] MultiHead(Q,K,V)=
[0100] Concat(head1,head2,...,head h )W O
[0101]
[0102] head i represents the output of an attention head, W O is the output matrix; the multi-head attention mechanism performs parallel calculations through different heads, and each head has its own independent linear transformation matrix and
[0103] Step 3: Output the features of the self-attention mechanism Input into the feed-forward neural network FFN for further processing:
[0104]
[0105] Among them, W1, W1 are weights, and b1, b1 are biases.
[0106] The final fusion features are classified by the classifier to determine whether a fall event has occurred. If the classifier detects a fall event, an alarm is triggered immediately to notify relevant personnel.
[0107] The present invention designs a loss function for a multimodal fusion classification task, which needs to consider the accuracy of both audio features and acceleration features:
[0108] L=L CE +λL consistency
[0109]
[0110] L consistency =||x weighted -x'||
[0111] Among them, L CE is the classification loss, and the cross entropy function is used for corresponding calculation as the main loss function. is the predicted classification result, y i is the true label value; L consistency is the consistency loss of weighted fusion features and deep fusion features, and λ is an adjustment parameter used to adjust the main loss and feature consistency loss.
[0112] like Figure 3 As shown, the present invention adopts a system of a fall detection method based on multimodal data fusion, the system comprising: a lightweight feature extraction module, a feature alignment and adaptive fusion module, a collaborative Transformer module and a single-task optimization module, and the function and implementation of each module are as follows:
[0113] Lightweight feature extraction module: Use the wav2vec 2.0 model to extract audio features and use LSTM to extract acceleration features.
[0114] Feature alignment and adaptive fusion module: The audio features and acceleration features are aligned through linear transformation and adaptive normalization, so that the features of different modes are comparable in the same feature space; the aligned audio features and acceleration features are preliminarily fused using a multi-layer perceptron (MLP) to obtain preliminary fused features to capture the relationship and complementarity between features of different modes.
[0115] Collaborative Transformer module: Calculate the contribution of each modal feature in the fusion process to obtain the weighted confidence Shapley value, and weight the preliminary fusion features according to the weighted confidence Shapley value to ensure that the contribution of each feature is reasonably evaluated; use a lightweight Transformer model to further process and enhance the weighted preliminary fusion features to capture the complex dependencies between features, and finally generate deep fusion features for classification.
[0116] Single-task optimization module: Focus on the fall detection task to improve the focus and performance of the model; Implementation method: Design a single-task learning head specifically for fall detection to avoid the complexity brought by multi-task learning and focus on improving the accuracy and robustness of fall detection.
[0117] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A fall detection method based on multimodal data fusion, It is characterized in that include: Collect multimodal data, input the multimodal data into the trained fall detection model, and obtain the detection results; The training process of the fall detection model includes: S1, collecting multimodal data and extracting multimodal features based on the multimodal data; S2, performing preliminary feature fusion on the multimodal features to obtain preliminary fusion features; S3, calculating the weighted confidence Shapley value according to the preliminary fusion features, and performing weighted fusion on the preliminary fusion features according to the weighted confidence Shapley value to obtain the weighted fusion features; S4. Use the Transformer model to process the weighted fusion features to obtain deep fusion features; S5. Obtain classification results based on deep fusion features; S6. Calculate the loss function value based on the classification results, the weighted preliminary fusion features, and the deep fusion features, and update the model parameters based on the loss function value. When the loss function value is minimized, the model training is completed.
2. According to the fall detection method based on multimodal data fusion according to claim 1, It is characterized in that The multimodal data includes: audio data and acceleration data; extracting multimodal features includes: extracting audio features based on the audio data using a wav2vec 2.0 model, and extracting acceleration features based on the acceleration data using an LSTM model.
3. According to the fall detection method based on multimodal data fusion according to claim 1, It is characterized in that The preliminary feature fusion of multimodal features includes: aligning the multimodal features to the same feature space through linear transformation and adaptive normalization methods, and performing preliminary feature fusion on the aligned multimodal features to obtain preliminary fused features.
4. According to the fall detection method based on multimodal data fusion according to claim 1, It is characterized in that Calculating the weighted confidence Shapley value based on the preliminary fusion features includes: Calculate the evidence confidence function of each modal feature in the preliminary fusion features, calculate the confidence Shapley value of each modal feature according to the evidence confidence function and the preliminary fusion features, calculate the divergence between the confidence Shapley values of each modal feature, calculate the Shapley similarity matrix according to the divergence between the confidence Shapley values, calculate the confidence Shapley reliability according to the Shapley similarity matrix, calculate the weighted confidence Shapley value according to the confidence Shapley reliability, weight the preliminary fusion features according to the weighted confidence Shapley value, and obtain the weighted fusion features.
5. According to the fall detection method based on multimodal data fusion according to claim 4, It is characterized in that The evidence confidence function for calculating each modality feature includes: Among them, i is the index of the modal feature in the preliminary fusion feature, m i (F) is the basic probability distribution, which indicates the support of modal feature i for subset F, E is the input set of the evidence confidence function, F is a subset of set E, Bel i () is the evidence confidence function of modal feature i.
6. A fall detection method based on multimodal data fusion according to claim 4, It is characterized in that Calculate the confidence Shapley value x for each modal feature i include: Where M is the number of all modal features, γ k is a combination coefficient, i represents the index of the modal feature, S represents the subset that does not contain the modal feature i, k represents the size of the subset S, TS i is the set of modal features i, TS represents the set of all modal features, Bel i () is the evidence confidence function of modal feature i.
7. A fall detection method based on multimodal data fusion according to claim 4, It is characterized in that Calculate the confidence Shapley reliability Rel i include: Among them, Rel T is the total reliability of all modal features, M is the number of all modal features, i and j are the indexes of modal features, η(x i ,x j ) is the Shapley similarity matrix, x i is the confidence Shapley value of modal feature i.
8. According to claim 4, a fall detection method based on multimodal data fusion, It is characterized in that Calculate weighted confidence Shapley value Including: According to the confidence Shapley reliability Rel i Calculate the weight vector w i , according to the weight vector w i The confidence Shapley values of the modal features are weighted and combined to obtain the weighted confidence Shapley value 9. The fall detection method based on multimodal data fusion according to claim 1, It is characterized in that The loss function is: L=L CE +λL consistency Among them, L CE is the classification loss, L consistency is the consistency loss of weighted fusion features and deep fusion features, and λ is the adjustment parameter.
10. A system based on the fall detection method according to claims 1 to 9, It is characterized in that include: A lightweight feature extraction module, a feature alignment and adaptive fusion module, a collaborative Transformer module and a single-task optimization module; the lightweight feature extraction module is used to extract multimodal features from the collected multimodal data; the feature alignment and adaptive fusion module is used to align and fuse the extracted multimodal features to generate preliminary fusion features; the collaborative Transformer module is used to further fuse the preliminary fusion features to generate deep fusion features; the single-task optimization module is used to classify the deep fusion features to generate classification results.
Citation Information
Patent Citations
Fall detection method based on multi-sensor feature fusion
CN109740761A
Prediction method and system for postoperative early recurrence of hepatocellular carcinoma
CN116525113A
Analysis method for predicting pathogenicity of missense mutation
CN117393042A
Depth feature fusion and optimization method and system for multi-modal data
CN117909922A
Panoramic image-based road traffic carbon emission prediction method, apparatus and device, and medium
CN118153984A
Cited By
Three-dimensional human skeleton posture estimation method and model training method
CN119741422A
Fall monitoring optimization method and system based on multi-modal data fusion
CN120930059A
