Fall detection method based on multi-modal data fusion and double-flow graph learning
Through multimodal data fusion and dual-stream graph learning, combining acceleration, image and sound data, cross-modal features are generated, which solves the problem of insufficient single-modal data and improves the accuracy and reliability of fall detection, especially in Performance in complex environments.
Patent Information
- Application Number
- CN202411010611.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-26
AI Technical Summary
The existing fall detection technology relies on single-modal data, resulting in insufficient detection accuracy and reliability, especially in complex environments, and the effect is poor and the multimodal information cannot be fully processed.
Using a method based on multimodal data fusion and dual-stream graph learning, acceleration, image and sound data are collected, features are extracted through convolutional neural networks, and multimodal graph learning is used to combine different modal information to generate Cross-modal features to improve detection accuracy.
It improves the accuracy and reliability of the fall detection system, enhances the robustness in complex environments, and improves the performance of the detection system through the combination of multimodal information.
Smart Images

Figure CN118924284A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a fall detection method based on multimodal data fusion and dual-flow graph learning. Background Art
[0002] As the global population ages, health problems of the elderly are becoming increasingly prominent. It is estimated that the elderly population will reach 1.5 billion by 2050. In particular, in populous countries, the elderly population is not only high in proportion, but also faces a deep aging problem. Failure to detect and treat falls in time is the main factor causing death among the elderly. Therefore, various fall detection algorithms have emerged. However, how to improve the accuracy of fall detection algorithms has always been a difficult point in research. In the era of medical Internet of Things, fall detection algorithms can realize fall detection by analyzing data obtained from sensors such as wearable devices, environmental sensors, and cameras.
[0003] Some existing fall detection technologies mainly use image processing methods, which mainly use images to analyze human behavior for fall detection. However, images may be affected by factors such as lighting and occlusion, which will lead to a decrease in detection accuracy and have great defects in practical applications.
[0004] In addition, the fall detection method based on environmental equipment needs to rely on sensors. It mainly relies on devices such as radar or ground sensors to collect current or audio information to determine whether a fall has occurred. Since environmental equipment is very sensitive to interference such as external noise, the detection effect of this method is also poor in complex multi-person scenarios.
[0005] The above technologies all rely on unimodal data for fall detection. However, the real world is multimodal, and unimodal data only contains data on one aspect. In many cases, it is subject to many limitations, which will affect the reliability and accuracy of the fall detection system. Existing fall detection technologies rely on single-modal data such as images or sensor text data. Compared with systems based on multimodal data, unimodal data contains limited information, which affects the accuracy and reliability of fall detection.
[0006] Most existing fall detection algorithms use simple convolutional neural networks, but this approach cannot fully process complex multimodal information, nor can it effectively utilize data from different modalities. In contrast, the dual-stream graph neural network has obvious advantages in processing multimodal data due to its special structure. It can combine information flows from different modalities to more comprehensively capture complex relationships and features in the data, thereby improving the performance of the model. A single convolutional neural network cannot fully process multimodal information, reducing the accuracy of the detection system. Summary of the invention
[0007] In order to improve the accuracy and reliability of the detection system, the present invention proposes a fall detection method based on multimodal data fusion and dual-flow graph learning, which specifically includes the following steps:
[0008] Collect acceleration, image and sound data of the detection target;
[0009] The collected acceleration data is converted, that is, the one-dimensional data is converted into two-dimensional data, the converted data is spliced and fused with the image data, and a convolutional neural network is used to extract features to obtain fusion modal coding;
[0010] The sound data is processed using a fully connected layer to obtain the sound modal encoding;
[0011] A graph neural network is constructed for each modality. In the graph neural network of each modality, the node is the modality encoding corresponding to each sampling point, and the edge weight is the attention weight between nodes;
[0012] Single-modal graph learning is performed on the single-modal graph data obtained based on fusion modal coding and sound modal coding respectively to obtain single-modal learning features corresponding to fusion modal coding and sound modal coding;
[0013] Based on the multimodal graph data obtained by fusion modal coding and sound modal coding, multimodal graph learning is performed to obtain multimodal learning features corresponding to fusion modal coding and sound modal coding;
[0014] The single-modal and multi-modal codes of the same modality code are fused by addition and then fused with another modality code by splicing to obtain cross-modal features;
[0015] The cross-modal features are passed through a fully connected layer to obtain the fall prediction results.
[0016] Furthermore, the Gramian angular field method is used to encode the collected one-dimensional time series data into a two-dimensional image. The encoding process includes:
[0017]
[0018] in, Represents a one-dimensional data sequence, expressed as n is the number of sampling points of the one-dimensional data sequence, represents the i-th sampling point in a one-dimensional data sequence; φ i for Mapped to the angular cosine encoding in the polar coordinate system; GAF is based on the one-dimensional data sequence The converted two-dimensional data
[0019] Furthermore, when performing unimodal graph learning, the process of obtaining the corresponding unimodal learning features includes:
[0020]
[0021] in, Represents the features obtained after unimodal learning of the feature encoding of the jth node in the i-th unimodal graph; Represents the feature encoding of the jth node in the i-th unimodal graph; represents the learnable matrix for unimodal learning in the i-th unimodal graph; represents the aggregation module of the neighbor nodes of the jth node in the i-th unimodal graph; || represents the concatenation operation; ⊙ represents the element-wise product operation; σ() represents the activation function.
[0022] Furthermore, the aggregation module of the neighbor nodes of the jth node in the i-th unimodal graph It is expressed as:
[0023]
[0024] Among them, H is the number of heads of the multi-head attention mechanism; is the set of neighbor nodes of the jth node in the i-th unimodal graph; is the jth node in the i-th unimodal graph and its kth neighbor node The attention weight at the hth attention head; is the learnable matrix at the h-th attention head in the i-th unimodal graph; Represents the feature encoding of the kth node in the i-th unimodal graph.
[0025] Furthermore, the jth node in the i-th unimodal graph and its kth neighbor node Attention weight at the hth attention head It is expressed as:
[0026]
[0027] in, To use the projection matrix For the jth node in the i-th unimodal graph Feature encoding The attention value matrix obtained by projection is expressed as To use the projection matrix For the jth node in the i-th unimodal graph Neighbor nodes Feature encoding The attention key matrix obtained by projection is expressed as d k1 is the key matrix Dimension, SoftMax is a normalized exponential function.
[0028] Furthermore, the process of obtaining the corresponding multimodal learning features includes:
[0029]
[0030] in, Represents the i-th node in the multimodal graph Feature encoding Feature encoding after multimodal learning update; Represents the learnable matrix when performing multimodal learning; Represents the i-th node in the multimodal graph The aggregation module of neighbor nodes; || represents the concatenation operation; ⊙ represents the element-wise product operation; σ() represents the activation function.
[0031] Furthermore, the i-th node in the multimodal graph The aggregation module of neighbor nodes The calculation includes:
[0032]
[0033] in, It means to splice the 1st to the Hth attention heads together; Representation Node The set of neighbor nodes of represents the attention weight between the node corresponding to the i-th action clip modality a and the node corresponding to the j-th action clip modality t in the h-th head of the multi-head attention, for The corresponding learnable matrix, represents the feature vector of the node corresponding to the j-th action segment modality t, Represents the node set of mode a in the multimodal graph; represents the attention weight between the node corresponding to the i-th action clip modality t and the node corresponding to the j-th action clip modality a in the h-th head of the multi-head attention, for The corresponding learnable matrix, represents the feature vector of the node corresponding to the j-th action segment modality a, A set of nodes representing mode t in a multimodal graph.
[0034] Furthermore, the attention weight between the modality a of the i-th node and the modality t of the j-th node in the h-th head of the multi-head attention is It is expressed as:
[0035]
[0036] in, represents the influence of mode t on mode a in the multimodal graph; Q a is the query matrix corresponding to the feature encoding of mode a in the multimodal graph, expressed as K is the projection matrix corresponding to the value matrix of the h-th head modality a in multi-head attention; t The key matrix corresponding to the feature encoding of mode t in the multimodal graph is expressed as is the projection matrix corresponding to the key matrix of the h-th head modality t in multi-head attention; d k2 Denotes the key matrix K t Dimensions; Indicates the influence of mode a on mode t in the multimodal graph; Q t is the query matrix corresponding to the feature encoding of mode t in the multimodal graph, expressed as K is the projection matrix corresponding to the value matrix of the h-th head modality t in multi-head attention; a Represents the key matrix corresponding to the feature encoding of mode a in the multimodal graph, expressed as It is the projection matrix corresponding to the key matrix of the h-th head modality a in multi-head attention.
[0037] Furthermore, the process of obtaining prediction results using cross-modal features includes:
[0038]
[0039] p i =SoftMax(fc(x i ))
[0040] Among them, X represents the cross-modal feature, represents the output features of the sound unimodal graph, Represents the multimodal graph output features of the sound features to the fusion features, represents the output features of the fused feature single modal graph, Represents the multimodal graph output feature of the fusion feature to the sound feature; x i is the feature encoding of the action clip of the i-th sampling point in the cross-modal feature; fc(·) represents the fully connected layer; SoftMax(·) represents the normalized exponential function; p i It represents the probability that the action segment of the i-th sampling point is judged as a fall.
[0041] The present invention provides a fall detection method based on multimodal data fusion and dual-stream graph learning. It adopts a multimodal fusion approach to provide more comprehensive and richer data. Information of different modalities complements and verifies each other, thereby improving the accuracy of the system. The combination of multiple modal information can improve the robustness of the system in complex environments. On the other hand, the present invention uses the structure of a dual-stream graph neural network to more comprehensively capture the complex relationships and features in the graph, extracts and fuses features at different levels, and effectively improves the reliability and accuracy of the fall detection system. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 A flow chart of a fall detection method based on multimodal data fusion and dual-stream graph learning of the present invention;
[0043] Figure 2 It is a structural schematic diagram of a fall detection method based on multimodal data fusion and dual-flow graph learning of the present invention;
[0044] Figure 3 A schematic diagram of a single-mode network structure in a fall detection method based on multimodal data fusion and dual-flow graph learning of the present invention;
[0045] Figure 4 A schematic diagram of a multimodal network structure in a fall detection method based on multimodal data fusion and dual-stream graph learning in the present invention. DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0047] The present invention proposes a fall detection method based on multimodal data fusion and dual-flow graph learning, such as Figure 1 , specifically including the following steps:
[0048] Collect acceleration, image and sound data of the detection target;
[0049] The collected acceleration data is converted, that is, the one-dimensional data is converted into two-dimensional data, the converted data is spliced and fused with the image data, and a convolutional neural network is used to extract features to obtain fusion modal coding;
[0050] The sound data is processed using a fully connected layer to obtain the sound modal encoding;
[0051] A graph neural network is constructed for each modality. In the graph neural network of each modality, the node is the modality encoding corresponding to each sampling point, and the edge weight is the attention weight between nodes;
[0052] Single-modal graph learning is performed on the single-modal graph data obtained based on fusion modal coding and sound modal coding respectively to obtain single-modal learning features corresponding to fusion modal coding and sound modal coding;
[0053] Based on the multimodal graph data obtained by fusion modal coding and sound modal coding, multimodal graph learning is performed to obtain multimodal learning features corresponding to fusion modal coding and sound modal coding;
[0054] The single-modal and multi-modal codes of the same modality code are fused by addition and then fused with another modality code by splicing to obtain cross-modal features;
[0055] The cross-modal features are passed through a fully connected layer to obtain the fall prediction results.
[0056] As an optional implementation, this embodiment proposes a complete implementation process of a fall detection method based on multimodal data fusion and dual-flow graph learning, such as Figure 2 , specifically including the following steps:
[0057] Step 1. Collect the acceleration, image and sound data of the detection target. In this embodiment, the acceleration, image and sound data of the detection target obtained by sampling at the same sampling time are taken as an action segment, wherein the acceleration data can be collected by a wearable device, and the image and sound can be collected by a camera and a microphone.
[0058] Step 2. Perform modal conversion (Markov transform field, recursive graph, Gramian angle field) on the collected acceleration data, fuse the converted information with the collected image information, and perform feature encoding on the image information, acceleration fusion information and sound information.
[0059] In this embodiment, the focus is on the one-dimensional time series data recorded by the wearable sensor and the two-dimensional visual data recorded by the camera. Generally speaking, due to the heterogeneity of these two types of data, it is difficult to directly fuse their shapes. Therefore, in this embodiment, the Gramian Angular Field (GAF) method is used to encode the one-dimensional time series data into a two-dimensional image, and then the encoded image is superimposed with the visual data on the channel to achieve fusion. The specific process steps are as follows:
[0060] 1. Given the original time series X acc ={x 1 ,x 2 ,...,x n}, first we need to normalize it to the interval [-1,1], and get:
[0061]
[0062] Where n is the number of sampling points, X acc Represents the sampled acceleration data, x n For sequence X acc The nth sampling point in ; For sequence X acc The corresponding sequence after normalization is, For x n The corresponding value after normalization, in order to simplify the symbolic representation, the default value is X acc Normalization has been performed.
[0063] 2. Scale the time series Mapped to polar coordinates, the value Encoded as the angle cosine, the timestamp t i Encoded as radius, that is:
[0064]
[0065] Where N is a constant factor used to normalize the span of the polar coordinate system, which refers to the number of time points;
[0066] 3. Use trigonometric and functions to express the temporal correlation between different time points as GAF. The definition of GAF is as follows:
[0067]
[0068] Among them, GAF is based on the one-dimensional data sequence The converted two-dimensional data. GAF is a matrix of size n×n, that is, the length of the time series determines the encoded image X G size.
[0069] After the above processing, the GAF image X corresponding to the acceleration information is obtained. G ; Process the visual data, that is, process the collected image to obtain the corresponding grayscale image X I ; For the visual sensor, use a camera and connect two X G and X I Superposition, get two-channel fusion data, namely X F =[X G ,X I ].
[0070] In this embodiment, a unimodal flow graph learning module is designed to extract long-range contextual relationships and information from each modality using the powerful relationship modeling capabilities of the graph. In a unimodal graph, it includes a graph module, a node aggregation module, and a node update module. The graph module can be represented as in, represents graph data with mode i, represents the set of nodes in the graph data with mode i, ε i represents the edge set in the graph data with mode i, represents the edge weight set in the graph data with modality i. In the present invention, a graph node is an action segment, that is, acceleration, image and sound data collected at a certain moment. In the present invention, when i=a, it means that the acceleration and image data are fused to obtain the fused modality data X F , the data of the fusion modality is extracted through a convolutional neural network to obtain the corresponding fusion modality code; similarly, when i=t, it means that the sound data is processed through a fully connected network to obtain the corresponding sound modality code.
[0071] In this embodiment, a unimodal graph is established with fusion modal coding and sound modal coding as node features, and then the attention of the nodes in each modal graph is calculated, and the neighboring nodes are clustered using the attention, and the clustered data is subjected to unimodal learning to obtain the corresponding unimodal learning features. When performing unimodal graph learning, the process of obtaining the corresponding unimodal learning features includes:
[0072]
[0073] in, Represents the features obtained after unimodal learning of the feature encoding of the jth node in the i-th unimodal graph; Represents the feature encoding of the jth node in the i-th unimodal graph; represents the learnable matrix for unimodal learning in the i-th unimodal graph; represents the aggregation module of the neighbor nodes of the jth node in the i-th unimodal graph; || represents the concatenation operation; ⊙ represents the element-wise product operation; σ() represents the activation function.
[0074] As an optional implementation, the aggregation module of the neighbor nodes of the jth node in the i-th unimodal graph It is expressed as:
[0075]
[0076] Among them, H is the number of heads of the multi-head attention mechanism; is the set of neighbor nodes of the jth node in the i-th unimodal graph; is the jth node in the i-th unimodal graph and its kth neighbor node The attention weight at the hth attention head; is the learnable matrix at the h-th attention head in the i-th unimodal graph; Represents the feature encoding of the kth node in the i-th unimodal graph.
[0077] like Figure 3 In a unimodal graph structure, all nodes use the same modality of feature representation, and it is necessary to calculate the attention weights between a node and its neighboring nodes. and its kth neighbor node Attention weight at the hth attention head It is expressed as:
[0078]
[0079] in, To use the projection matrix For the jth node in the i-th unimodal graph Feature encoding The attention value matrix obtained by projection is expressed as To use the projection matrix For the jth node in the i-th unimodal graph Neighbor nodes Feature encoding The attention key matrix obtained by projection is expressed as d k1 is the key matrix Dimension, SoftMax is a normalized exponential function.
[0080] In the above way, the fusion modal coding and the sound modal coding are clustered respectively, and the clustered features corresponding to each node under the same mode are combined together to form the output features of the sound single modal graph. And the output features of the fused feature single modal graph
[0081] Before multimodal learning, a multimodal graph structure needs to be constructed. Unlike unimodal graphs, the nodes and edges of cross-modal graphs will change. Each cross-modal graph involves two modalities. Each node represents an action under a certain modality. Therefore, each graph contains information from two different modalities, making it heterogeneous. The edge only connects two nodes of different modalities, indicating the interaction between the two different modalities. In this way, a cross-modal graph is constructed, which is represented as in, The node set representing the sound mode and fusion mode of each action segment, that is, the sound mode and fusion mode of an action are taken as a node respectively, ε a∪t represents the set of edges between all nodes, represents the edge weight set between the nodes of fusion mode a and the nodes of sound mode t, Represents the set of edge weights between the nodes of fusion modality t and the nodes of sound modality a.
[0082] In the cross-modal graph G a∪t In the cross-modal graph, each action is regarded as two nodes, one representing the sound modality and the other representing the fusion modality. The edges in the cross-modal graph only connect different modality nodes. For example, a sound node is connected to all other modality nodes, which means that all text nodes have interactive relationships with sound nodes. These interactive relationships are assumed to be bidirectional and have differences in relationship strength. Therefore, an attention mechanism is introduced to measure this bidirectional interaction by calculating the weight of the edge.
[0083] Next, we use cross-modal flow graph learning to learn complementary information of different modalities. Cross-modal graphs focus on learning complementary information from multimodal data for fall recognition. Through edges across different modalities, features of one modality can be propagated to another modality. Figure 4 As shown in Figure 2, cross-modal graph learning also includes neighbor aggregation and node update. Neighbor aggregation is to collect features of other modalities, and node update is to fuse the features of two modalities. The process of node update is expressed as:
[0084]
[0085] in, Represents the i-th node in the multimodal graph The feature encoding of a certain mode Feature encoding after multimodal learning update; Represents the learnable matrix when performing multimodal learning; Represents the i-th node in the multimodal graph The aggregation module of neighbor nodes; || represents the concatenation operation; ⊙ represents the element-wise product operation; σ() represents the activation function.
[0086] In this embodiment, the i-th node in the multimodal graph The aggregation module of neighbor nodes The calculation includes:
[0087]
[0088] in, It means to splice the 1st to the Hth attention heads together; Representation Node The set of neighbor nodes of represents the attention weight between the node corresponding to the i-th action clip modality a and the node corresponding to the j-th action clip modality t in the h-th head of the multi-head attention, for The corresponding learnable matrix, represents the feature vector of the node corresponding to the j-th action segment modality t, Represents the node set of mode a in the multimodal graph; represents the attention weight between the node corresponding to the i-th action clip modality t and the node corresponding to the j-th action clip modality a in the h-th head of the multi-head attention, for The corresponding learnable matrix, represents the feature vector of the node corresponding to the j-th action segment modality a, A set of nodes representing mode t in a multimodal graph.
[0089] In this embodiment, the attention weight between the node corresponding to the i-th action clip modality a and the node corresponding to the j-th action clip modality t in the h-th head of the multi-head attention is It is expressed as:
[0090]
[0091] The attention weight between the node corresponding to the i-th action clip modality t and the node corresponding to the j-th action clip modality a in the h-th head of the multi-head attention It is expressed as:
[0092]
[0093] in, Indicates the influence of mode t on mode a in the multimodal graph. Similarly Indicates the influence of mode a on mode t in the multimodal graph; Q a is the query matrix corresponding to the feature encoding of mode a in the multimodal graph, expressed as K is the projection matrix corresponding to the value matrix of the h-th head modality a in multi-head attention; t The key matrix corresponding to the feature encoding of mode t in the multimodal graph is expressed as is the projection matrix corresponding to the key matrix of the h-th head modality t in multi-head attention; d k2 Denotes the key matrix K t Dimensions; Indicates the influence of mode a on mode t in the multimodal graph; Q t is the query matrix corresponding to the feature encoding of mode t in the multimodal graph, expressed as K is the projection matrix corresponding to the value matrix of the h-th head modality t in multi-head attention; a Represents the key matrix corresponding to the feature encoding of mode a in the multimodal graph, expressed as It is the projection matrix corresponding to the key matrix of the h-th head modality a in multi-head attention.
[0094] In the above way, the fusion modal coding and the sound modal coding in the multimodal graph are clustered respectively, and the clustered features corresponding to each node under the same modality are combined together to form the multimodal graph output features of the sound feature to the fusion feature. And the multi-modal graph output features of the fusion features to the sound features
[0095] The output features of the single-mode sound graph under an action clip and the output features of the multi-modal graph of the sound feature to the fusion feature are fused by adding, the output features of the single-mode fusion feature graph and the output features of the multi-modal graph of the fusion feature to the sound feature are fused by adding, and the fused features are connected by splicing to obtain the final feature representation, which is expressed as:
[0096]
[0097] Wherein, X represents a cross-modal feature, and the cross-modal feature X is input into the predictor for prediction. The present invention uses a fully connected layer for prediction, which is expressed as:
[0098] p = SoftMax(fc(X))
[0099] Among them, p represents the probability that the cross-modal feature X is judged as a fall.
[0100] As an optional implementation, the present invention can use the prediction results and the oscillator results to calculate the cross entropy loss, and use the cross entropy loss back propagation to update the parameters such as the weight matrix in the network.
[0101] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A fall detection method based on multimodal data fusion and dual-flow graph learning, characterized in that: The specific steps include: Collect acceleration, image and sound data of the detection target; The collected acceleration data is converted, that is, the one-dimensional data is converted into two-dimensional data, the converted data is spliced and fused with the image data, and a convolutional neural network is used to extract features to obtain fusion modal coding; The sound data is processed using a fully connected layer to obtain the sound modal encoding; A graph neural network is constructed for each modality. In the graph neural network of each modality, the node is the modality encoding corresponding to each sampling point, and the edge weight is the attention weight between nodes; Single-modal graph learning is performed on the single-modal graph data obtained based on fusion modal coding and sound modal coding respectively to obtain single-modal learning features corresponding to fusion modal coding and sound modal coding; Based on the multimodal graph data obtained by fusion modal coding and sound modal coding, multimodal graph learning is performed to obtain multimodal learning features corresponding to fusion modal coding and sound modal coding; The single-modal and multi-modal codes of the same modality code are fused by addition and then fused with another modality code by splicing to obtain cross-modal features; The cross-modal features are passed through a fully connected layer to obtain the fall prediction results.
2. According to claim 1, a fall detection method based on multimodal data fusion and dual-flow graph learning is characterized in that: The Gramian angular field method is used to encode the collected one-dimensional time series data into a two-dimensional image. The encoding process includes: in, Represents a one-dimensional data sequence, expressed as n is the number of sampling points of the one-dimensional data sequence, represents the i-th sampling point in a one-dimensional data sequence; φ i for Mapped to the angular cosine encoding in the polar coordinate system; GAF is based on the one-dimensional data sequence The converted two-dimensional data.
3. The fall detection method based on multimodal data fusion and dual-flow graph learning according to claim 1 is characterized in that: When performing unimodal graph learning, the process of obtaining the corresponding unimodal learning features includes: in, Represents the features obtained after unimodal learning of the feature encoding of the jth node in the i-th unimodal graph; Represents the feature encoding of the jth node in the i-th unimodal graph; represents the learnable matrix for unimodal learning in the i-th unimodal graph; represents the aggregation module of the neighbor nodes of the jth node in the i-th unimodal graph; || represents the concatenation operation; ⊙ represents the element-wise product operation; σ() represents the activation function.
4. According to claim 3, a fall detection method based on multimodal data fusion and dual-flow graph learning is characterized in that: Aggregation module for neighbor nodes of the jth node in the i-th unimodal graph It is expressed as: Among them, H is the number of heads of the multi-head attention mechanism; is the set of neighbor nodes of the jth node in the i-th unimodal graph; is the jth node in the i-th unimodal graph and its kth neighbor node The attention weight at the hth attention head; is the learnable matrix at the h-th attention head in the i-th unimodal graph; Represents the feature encoding of the kth node in the i-th unimodal graph.
5. The fall detection method based on multimodal data fusion and dual-flow graph learning according to claim 4 is characterized in that: The jth node in the i-th unimodal graph and its kth neighbor node Attention weight at the hth attention head It is expressed as: in, To use the projection matrix For the jth node in the i-th unimodal graph Feature encoding The attention value matrix obtained by projection is expressed as To use the projection matrix For the jth node in the i-th unimodal graph Neighbor nodes Feature encoding The attention key matrix obtained by projection is expressed as d k1 is the key matrix Dimension, SoftMax is a normalized exponential function.
6. The fall detection method based on multimodal data fusion and dual-flow graph learning according to claim 3 is characterized in that: The process of obtaining the corresponding multimodal learning features includes: in, Represents the i-th node in the multimodal graph The feature encoding of a certain mode Feature encoding after multimodal learning update; Represents the learnable matrix when performing multimodal learning; Represents the i-th node in the multimodal graph The aggregation module of neighbor nodes; || represents the concatenation operation; ⊙ represents the element-wise product operation; σ( ) represents the activation function.
7. The fall detection method based on multimodal data fusion and dual-flow graph learning according to claim 6 is characterized in that: The i-th node in the multimodal graph The neighbor node aggregation module The calculation includes: in, It means to splice the 1st to the Hth attention heads together; Representation Node The set of neighbor nodes of represents the attention weight between the node corresponding to the i-th action clip modality a and the node corresponding to the j-th action clip modality t in the h-th head of the multi-head attention, for The corresponding learnable matrix, represents the feature vector of the node corresponding to the j-th action segment modality t, Represents the node set of mode a in the multimodal graph; represents the attention weight between the node corresponding to the i-th action clip modality t and the node corresponding to the j-th action clip modality a in the h-th head of the multi-head attention, for The corresponding learnable matrix, represents the feature vector of the node corresponding to the j-th action segment modality a, A set of nodes representing mode t in a multimodal graph.
8. The fall detection method based on multimodal data fusion and dual-flow graph learning according to claim 7 is characterized in that: Attention weight between modality a of the i-th node and modality t of the j-th node in the h-th head of multi-head attention It is expressed as: in, represents the influence of mode t on mode a in the multimodal graph; Q a is the query matrix corresponding to the feature encoding of mode a in the multimodal graph, expressed as K is the projection matrix corresponding to the value matrix of the h-th head modality a in multi-head attention; t The key matrix corresponding to the feature encoding of mode t in the multimodal graph is expressed as is the projection matrix corresponding to the key matrix of the h-th head modality t in multi-head attention; d k2 Denotes the key matrix K t Dimensions; Indicates the influence of mode a on mode t in the multimodal graph; Q t is the query matrix corresponding to the feature encoding of mode t in the multimodal graph, expressed as K is the projection matrix corresponding to the value matrix of the h-th head modality t in multi-head attention; a Represents the key matrix corresponding to the feature encoding of mode a in the multimodal graph, expressed as It is the projection matrix corresponding to the key matrix of the h-th head modality a in multi-head attention.
9. The fall detection method based on multimodal data fusion and dual-flow graph learning according to claim 1, characterized in that: The process of using cross-modal features to obtain prediction results includes: p = SoftMax(fc(X)) Among them, X represents the cross-modal feature, represents the output features of the sound unimodal graph, Represents the multimodal graph output features of the sound features to the fusion features, represents the output features of the fused feature single modal graph, represents the multimodal graph output feature of the fusion feature to the sound feature; fc(·) represents the fully connected layer; SoftMax(·) represents the normalized exponential function; p represents the probability that the cross-modal feature X is judged as a fall.
Citation Information
Patent Citations
Tumble behavior analysis and detection method based on double-flow convolutional neural network
CN111626109A
Tumble detection algorithm based on double-flow network
CN112395994A
Graph convolutional network fall detection method based on human body key points
CN114387666A
Fall detection method based on self-attention double-current network
CN114463844A
Wearable old man falling detection system and method based on sound assistance
CN114898527A
Cited By
Fall behavior identification method based on hypergraph depth feature fusion
CN121236821A
