A fall detection method based on multimodal data fusion and dual-stream graph learning

Abstract

The present invention belongs to the field of artificial intelligence technology, and in particular relates to a fall detection method based on multimodal data fusion and dual-stream graph learning, comprising collecting acceleration, image and sound data of a detection target, obtaining a fusion modal code based on the acceleration and image fusion, and obtaining a sound modal code based on the sound data; performing single-modal graph learning on single-modal graph data obtained based on the fusion modal code and the sound modal code to obtain corresponding single-modal learning features; performing multimodal graph learning on multimodal graph data obtained based on the fusion modal code and the sound modal code to obtain multimodal learning features corresponding to the fusion modal code and the sound modal code; fusing the single-modal and multimodal codes of the same modal code by addition, and then fusing them with another modal code by splicing to obtain cross-modal features; and obtaining a fall prediction result by passing the cross-modal features through a fully connected layer. The present invention has higher accuracy and reliability.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a fall detection method based on multimodal data fusion and dual-stream graph learning. Background Art

[0002] As the global population ages, health issues facing the elderly are becoming increasingly prominent. The elderly population is projected to reach 1.5 billion by 2050. In populous countries in particular, the elderly population is not only disproportionately high but also faces a profound aging problem. Failure to promptly detect and treat falls is a major contributing factor to mortality among the elderly. Consequently, various fall detection algorithms have emerged. However, improving the accuracy of fall detection algorithms has been a research challenge. In the era of the medical Internet of Things, fall detection algorithms can analyze data from sensors such as wearable devices, environmental sensors, and cameras to detect falls.

[0003] Some existing fall detection technologies primarily rely on image processing methods, which primarily analyze human behavior using images to detect falls. However, images can be affected by factors such as lighting and occlusion, which can reduce detection accuracy and present significant drawbacks in practical applications.

[0004] Furthermore, fall detection methods based on environmental devices rely on sensors. They primarily rely on radar or ground sensors to collect current or audio information to determine if a fall has occurred. Because environmental devices are highly sensitive to interference such as external noise, this method's detection effectiveness is poor in complex multi-person scenarios.

[0005] All of the aforementioned technologies rely on unimodal data for fall detection. However, the real world is multimodal, and unimodal data only contains information on a specific aspect. In many cases, this data is subject to numerous limitations, which can affect the reliability and accuracy of fall detection systems. Existing fall detection technologies rely on single-modal data, such as images or sensor text data. Compared to systems based on multimodal data, unimodal data contains limited information, which affects the accuracy and reliability of fall detection.

[0006] Most existing fall detection algorithms use simple convolutional neural networks, but this approach cannot fully process complex multimodal information or effectively utilize data from different modalities. In contrast, the dual-stream graph neural network, due to its unique structure, offers significant advantages in processing multimodal data. It can combine information flows from different modalities to more comprehensively capture complex relationships and features in the data, thereby improving model performance. A single convolutional neural network cannot fully process multimodal information, reducing the accuracy of the detection system. Summary of the Invention

[0007] In order to improve the accuracy and reliability of the detection system, the present invention proposes a fall detection method based on multimodal data fusion and dual-stream graph learning, which specifically includes the following steps:

[0008] Collect acceleration, image and sound data of the detection target;

[0009] The collected acceleration data is converted, that is, the one-dimensional data is converted into two-dimensional data, the converted data is spliced ​​and fused with the image data, and a convolutional neural network is used to extract features to obtain fused modal coding;

[0010] The sound data is processed using a fully connected layer to obtain the sound modal code;

[0011] A graph neural network is constructed for each modality. In each modality’s graph neural network, the node is the modality code corresponding to each sampling point, and the edge weight is the attention weight between nodes.

[0012] Single-modal graph learning is performed on the single-modal graph data obtained based on fusion modal coding and sound modal coding respectively to obtain the single-modal learning features corresponding to fusion modal coding and sound modal coding;

[0013] Multimodal graph learning is performed based on the multimodal graph data obtained by fusion modal coding and sound modal coding to obtain multimodal learning features corresponding to fusion modal coding and sound modal coding;

[0014] The single modality and multimodality codes of the same modality code are fused by addition and then fused with another modality code by splicing to obtain cross-modal features;

[0015] The cross-modal features are passed through a fully connected layer to obtain the fall prediction results.

[0016] Furthermore, the Gramian angular field method is used to encode the collected one-dimensional time series data into a two-dimensional image. The encoding process includes:

[0017]

[0018] in, Represents a one-dimensional data sequence, expressed as n is the number of sampling points of the one-dimensional data sequence, represents the i-th sampling point in the one-dimensional data sequence; φ i for Mapped to the angular cosine encoding in the polar coordinate system; GAF is based on the one-dimensional data sequence Converted two-dimensional data

[0019] Furthermore, when performing unimodal graph learning, the process of obtaining the corresponding unimodal learning features includes:

[0020]

[0021] in, Represents the feature obtained after unimodal learning of the feature encoding of the jth node in the i-th unimodal graph; Represents the feature encoding of the jth node in the i-th unimodal graph; represents the learnable matrix for unimodal learning in the i-th unimodal graph; represents the aggregation module of the neighbor nodes of the jth node in the i-th unimodal graph; || represents the splicing operation; ⊙ represents the element-wise product operation; σ() represents the activation function.

[0022] Furthermore, the aggregation module of the neighbor nodes of the jth node in the i-th unimodal graph Expressed as:

[0023]

[0024] Where H is the number of heads in the multi-head attention mechanism; is the set of neighbor nodes of the jth node in the i-th unimodal graph; is the jth node in the i-th unimodal graph and its kth neighbor node The attention weight at the hth attention head; is the learnable matrix at the h-th attention head in the i-th unimodal graph; represents the feature encoding of the kth node in the i-th unimodal graph.

[0025] Furthermore, the jth node in the i-th unimodal graph and its kth neighbor node Attention weight at the hth attention head Expressed as:

[0026]

[0027] in, To use the projection matrix For the jth node in the i-th unimodal graph Feature encoding The attention value matrix obtained by projection is expressed as To use the projection matrix For the jth node in the i-th unimodal graph Neighbor nodes Feature encoding The attention key matrix obtained by projection is expressed as d k1 is the bond matrix Dimension, SoftMax is a normalized exponential function.

[0028] Furthermore, the process of obtaining the corresponding multimodal learning features includes:

[0029]

[0030] in, Represents the i-th node in the multimodal graph Feature encoding Feature encoding after multimodal learning update; Represents the learnable matrix when performing multimodal learning; Represents the i-th node in the multimodal graph The aggregation module of neighbor nodes; || represents the splicing operation; ⊙ represents the element-wise product operation; σ() represents the activation function.

[0031] Furthermore, the i-th node in the multimodal graph Neighbor node aggregation module The calculation includes:

[0032]

[0033] in, It means to splice the 1st to Hth attention heads together; Representation node The set of neighbor nodes of represents the attention weight between the node corresponding to the i-th action clip modality a and the node corresponding to the j-th action clip modality t in the h-th head of the multi-head attention, for The corresponding learnable matrix, Represents the feature vector of the node corresponding to the j-th action segment modality t, Represents the node set of mode a in the multimodal graph; represents the attention weight between the node corresponding to the i-th action clip modality t and the node corresponding to the j-th action clip modality a in the h-th head of the multi-head attention, for The corresponding learnable matrix, Represents the feature vector of the node corresponding to the j-th action segment modality a, A set of nodes representing mode t in a multimodal graph.

[0034] Furthermore, the attention weight between the modality a of the i-th node and the modality t of the j-th node in the h-th head of the multi-head attention is Expressed as:

[0035]

[0036] in, Indicates the influence of mode t on mode a in the multimodal graph; Q a is the query matrix corresponding to the feature encoding of mode a in the multimodal graph, expressed as K is the projection matrix corresponding to the value matrix of the h-th head modality a in multi-head attention; t The key matrix corresponding to the feature encoding of mode t in the multimodal graph is expressed as is the projection matrix corresponding to the key matrix of the h-th head modality t in the multi-head attention; d k2 Denotes the bond matrix K t Dimensions; Indicates the influence of mode a on mode t in the multimodal graph; Q t is the query matrix corresponding to the feature encoding of mode t in the multimodal graph, expressed as K is the projection matrix corresponding to the value matrix of the h-th head modality t in the multi-head attention; a Represents the key matrix corresponding to the feature encoding of mode a in the multimodal graph, expressed as is the projection matrix corresponding to the key matrix of the h-th head modality a in multi-head attention.

[0037] Furthermore, the process of obtaining prediction results using cross-modal features includes:

[0038]

[0039] p i =SoftMax(fc(x i ))

[0040] Among them, X represents the cross-modal feature, represents the output features of the sound unimodal graph, Represents the multimodal graph output features of the sound feature to the fusion feature, represents the output features of the fused feature single modal graph, Represents the multimodal graph output features of the fusion features to the sound features; x i is the feature encoding of the action clip of the i-th sampling point in the cross-modal feature; fc(·) represents the fully connected layer; SoftMax(·) represents the normalized exponential function; p i represents the probability that the action segment at the i-th sampling point is judged as a fall.

[0041] The present invention provides a fall detection method based on multimodal data fusion and dual-stream graph learning. It adopts a multimodal fusion approach to provide more comprehensive and richer data. Information from different modalities complements and verifies each other, thereby improving the accuracy of the system. The combination of multiple modal information can improve the robustness of the system in complex environments. On the other hand, the present invention uses the structure of a dual-stream graph neural network to more comprehensively capture the complex relationships and features in the graph, extract and fuse features at different levels, and effectively improve the reliability and accuracy of the fall detection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of a fall detection method based on multimodal data fusion and dual-stream graph learning of the present invention;

[0043] Figure 2 Schematic diagram of a fall detection method based on multimodal data fusion and dual-stream graph learning according to the present invention;

[0044] Figure 3 Schematic diagram of a single-modal network structure in a fall detection method based on multimodal data fusion and dual-stream graph learning according to the present invention;

[0045] Figure 4 This is a schematic diagram of the multimodal network structure in a fall detection method based on multimodal data fusion and dual-stream graph learning in the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] This paper proposes a fall detection method based on multimodal data fusion and dual-flow graph learning, such as Figure 1 , specifically including the following steps:

[0048] Collect acceleration, image and sound data of the detection target;

[0049] The collected acceleration data is converted, that is, the one-dimensional data is converted into two-dimensional data, the converted data is spliced ​​and fused with the image data, and a convolutional neural network is used to extract features to obtain fused modal coding;

[0050] The sound data is processed using a fully connected layer to obtain the sound modal code;

[0051] A graph neural network is constructed for each modality. In each modality’s graph neural network, the node is the modality code corresponding to each sampling point, and the edge weight is the attention weight between nodes.

[0052] Single-modal graph learning is performed on the single-modal graph data obtained based on fusion modal coding and sound modal coding respectively to obtain the single-modal learning features corresponding to fusion modal coding and sound modal coding;

[0053] Multimodal graph learning is performed based on the multimodal graph data obtained by fusion modal coding and sound modal coding to obtain multimodal learning features corresponding to fusion modal coding and sound modal coding;

[0054] The single modality and multimodality codes of the same modality code are fused by addition and then fused with another modality code by splicing to obtain cross-modal features;

[0055] The cross-modal features are passed through a fully connected layer to obtain the fall prediction results.

[0056] As an optional implementation method, this embodiment proposes a complete implementation process of a fall detection method based on multimodal data fusion and dual-flow graph learning, such as Figure 2 , specifically including the following steps:

[0057] Step 1. Collect the acceleration, image, and sound data of the detection target. In this embodiment, the acceleration, image, and sound data of the detection target obtained by sampling at the same sampling time are regarded as an action segment. The acceleration data can be collected by the wearable device, and the image and sound can be collected by the camera and microphone.

[0058] Step 2. Perform modal conversion (Markov transform field, recursive graph, Gramian angle field) on the collected acceleration data, fuse the converted information with the collected image information, and perform feature encoding on the image information, acceleration fusion information, and sound information.

[0059] In this example, we focus on the one-dimensional time series data recorded by wearable sensors and the two-dimensional visual data recorded by cameras. Generally speaking, due to the heterogeneity of these two types of data, it is difficult to directly fuse their shapes. Therefore, in this example, we use the Gramian Angular Field (GAF) method to encode the one-dimensional time series data into a two-dimensional image, and then superimpose the encoded image with the visual data on the channel to achieve fusion. The specific process steps are as follows:

[0060] 1. Given the original time series X acc ={x1,x2,...,x n}, first we need to normalize it to the interval [-1,1], and get:

[0061]

[0062] Among them, n is the number of sampling points, X acc Represents the sampled acceleration data, x n For sequence X acc The nth sampling point in ; For sequence X acc The corresponding sequence after normalization is, is x n The corresponding value after normalization, in order to simplify the symbolic representation, the default value is X acc Normalization has been performed.

[0063] 2. The scaled time series Map to polar coordinates, and convert the value Encoded as angle cosine, timestamp t i Encoded as radius, that is:

[0064]

[0065] Where N is a constant factor used to normalize the span of the polar coordinate system, which refers to the number of time points;

[0066] 3. Use the trigonometric sum function to express the temporal correlation between different time points as GAF. The definition of GAF is as follows:

[0067]

[0068] Among them, GAF is based on the one-dimensional data sequence The converted two-dimensional data. GAF is a matrix of size n×n, that is, the length of the time series determines the encoded image X G size.

[0069] After the above processing, the GAF image X corresponding to the acceleration information is obtained. G ; Process the visual data, that is, process the collected image to obtain the corresponding grayscale image X I ; For the visual sensor, use a camera and connect two X G and X I Superposition, get two-channel fusion data, namely X F =[X G ,X I ].

[0070] In this embodiment, a single-modal flow graph learning module is designed to extract long-range contextual relationships and information from each modality by leveraging the powerful relationship modeling capabilities of the graph. In a single-modal graph, it includes a graph module, a node aggregation module, and a node update module. The graph module can be represented as in, Represents graph data with mode i, represents the set of nodes in the graph data with mode i, ε i represents the set of edges in the graph data with mode i, represents the edge weight set in the graph data with modality i. In the present invention, a graph node is an action segment, i.e., acceleration, image, and sound data collected at a certain moment. In the present invention, when i=a, it means that the acceleration and image data are fused to obtain the fused modality data X F , the fusion modal data is extracted through a convolutional neural network to obtain the corresponding fusion modal code; similarly, when i=t, it means that the sound data is processed through a fully connected network to obtain the corresponding sound modal code.

[0071] In this embodiment, a unimodal graph is established using the fusion modal code and the sound modal code as node features. The attention of the nodes in each modal graph is then calculated, and the attention is used to cluster its neighboring nodes. The clustered data is then subjected to unimodal learning to obtain the corresponding unimodal learning features. When performing unimodal graph learning, the process of obtaining the corresponding unimodal learning features includes:

[0072]

[0073] in, Represents the feature obtained after unimodal learning of the feature encoding of the jth node in the i-th unimodal graph; Represents the feature encoding of the jth node in the i-th unimodal graph; represents the learnable matrix for unimodal learning in the i-th unimodal graph; represents the aggregation module of the neighbor nodes of the jth node in the i-th unimodal graph; || represents the splicing operation; ⊙ represents the element-wise product operation; σ() represents the activation function.

[0074] As an optional implementation, the aggregation module of the neighbor nodes of the jth node in the i-th unimodal graph Expressed as:

[0075]

[0076] Where H is the number of heads in the multi-head attention mechanism; is the set of neighbor nodes of the jth node in the i-th unimodal graph; is the jth node in the i-th unimodal graph and its kth neighbor node The attention weight at the hth attention head; is the learnable matrix at the h-th attention head in the i-th unimodal graph; represents the feature encoding of the kth node in the i-th unimodal graph.

[0077] like Figure 3 In a unimodal graph structure, all nodes use the same modality of feature representation, and it is necessary to calculate the attention weight between the node and its neighboring nodes. The jth node in the i-th unimodal graph and its kth neighbor node Attention weight at the hth attention head Expressed as:

[0078]

[0079] in, To use the projection matrix For the jth node in the i-th unimodal graph Feature encoding The attention value matrix obtained by projection is expressed as To use the projection matrix For the jth node in the i-th unimodal graph Neighbor nodes Feature encoding The attention key matrix obtained by projection is expressed as d k1 is the bond matrix Dimension, SoftMax is a normalized exponential function.

[0080] By clustering the fusion modal coding and the sound modal coding respectively in the above way, the clustered features corresponding to each node under the same modality are combined together to form the output features of the sound single modal graph. And the output features of the fused feature single modal graph

[0081] Before multimodal learning, a multimodal graph structure needs to be constructed. Unlike unimodal graphs, the nodes and edges of cross-modal graphs will change. Each cross-modal graph involves two modalities. Each node represents an action under a certain modality. Therefore, each graph contains information from two different modalities, thus having heterogeneity. The edge only connects two nodes of different modalities, representing the interaction between the two different modalities. In this way, a cross-modal graph is constructed, which is represented as in, The node set representing the sound mode and fusion mode of each action segment, that is, the sound mode and fusion mode of an action are regarded as a node respectively, ε a∪t represents the set of edges between all nodes, represents the edge weight set between the nodes of fusion modality a and the nodes of sound modality t, Represents the set of edge weights between the nodes of the fusion modality t and the nodes of the sound modality a.

[0082] In the cross-modal graph G a∪t In the cross-modal graph, each action is treated as two nodes, one representing the audio modality and the other the fusion modality. Edges in the cross-modal graph only connect nodes of different modalities. For example, connecting a sound node to all other modal nodes means that all text nodes have interactions with the audio node. These interactions are assumed to be bidirectional and vary in strength. Therefore, an attention mechanism is introduced to measure this bidirectional interaction by calculating the weight of the edges.

[0083] Next, we use cross-modal flow graph learning to learn complementary information from different modalities. Cross-modal graphs focus on learning complementary information from multimodal data for fall recognition. By crossing edges across different modalities, features from one modality can be propagated to another. Figure 4 As shown in Figure 2, cross-modal graph learning also includes neighbor aggregation and node updating. Neighbor aggregation is to collect features of other modalities, and node updating is to fuse the features of the two modalities. The node updating process is expressed as:

[0084]

[0085] in, Represents the i-th node in the multimodal graph The feature encoding of a certain mode Feature encoding after multimodal learning update; Represents the learnable matrix when performing multimodal learning; Represents the i-th node in the multimodal graph The aggregation module of neighbor nodes; || represents the splicing operation; ⊙ represents the element-wise product operation; σ() represents the activation function.

[0086] In this embodiment, the i-th node in the multimodal graph Neighbor node aggregation module The calculation includes:

[0087]

[0088] in, It means to splice the 1st to Hth attention heads together; Representation node The set of neighbor nodes of represents the attention weight between the node corresponding to the i-th action clip modality a and the node corresponding to the j-th action clip modality t in the h-th head of the multi-head attention, for The corresponding learnable matrix, Represents the feature vector of the node corresponding to the j-th action segment modality t, Represents the node set of mode a in the multimodal graph; represents the attention weight between the node corresponding to the i-th action clip modality t and the node corresponding to the j-th action clip modality a in the h-th head of the multi-head attention, for The corresponding learnable matrix, Represents the feature vector of the node corresponding to the j-th action segment modality a, A set of nodes representing mode t in a multimodal graph.

[0089] In this embodiment, the attention weight between the node corresponding to the i-th action segment modality a and the node corresponding to the j-th action segment modality t in the h-th head of the multi-head attention is Expressed as:

[0090]

[0091] The attention weight between the node corresponding to the i-th action clip modality t and the node corresponding to the j-th action clip modality a in the h-th head of the multi-head attention Expressed as:

[0092]

[0093] in, Indicates the influence of mode t on mode a in the multimodal graph. Similarly, Indicates the influence of mode a on mode t in the multimodal graph; Q a is the query matrix corresponding to the feature encoding of mode a in the multimodal graph, expressed as K is the projection matrix corresponding to the value matrix of the h-th head modality a in multi-head attention; t The key matrix corresponding to the feature encoding of mode t in the multimodal graph is expressed as is the projection matrix corresponding to the key matrix of the h-th head modality t in the multi-head attention; d k2 Denotes the bond matrix K t Dimensions; Indicates the influence of mode a on mode t in the multimodal graph; Q t is the query matrix corresponding to the feature encoding of mode t in the multimodal graph, expressed as K is the projection matrix corresponding to the value matrix of the h-th head modality t in the multi-head attention; a Represents the key matrix corresponding to the feature encoding of mode a in the multimodal graph, expressed as is the projection matrix corresponding to the key matrix of the h-th head modality a in multi-head attention.

[0094] Through the above methods, the fusion modal coding and sound modal coding in the multimodal graph are clustered respectively, and the clustered features corresponding to each node under the same modality are combined together to form the multimodal graph output features of the sound feature to the fusion feature. And the multimodal graph output features of the fusion features to the sound features

[0095] The output features of the sound unimodal graph under an action clip and the output features of the multimodal graph of the sound feature fusion feature are fused by adding them together. The output features of the unimodal graph of the fusion feature and the output features of the multimodal graph of the fusion feature fusion feature are fused by adding them together. The fused features are connected in a splicing manner to obtain the final feature representation, which is expressed as:

[0096]

[0097] Wherein, X represents the cross-modal feature. The cross-modal feature X is input into the predictor for prediction. The present invention uses a fully connected layer for prediction, which is expressed as:

[0098] p = SoftMax(fc(X))

[0099] Where p represents the probability that the cross-modal feature X is judged as a fall.

[0100] As an optional implementation, the present invention can use the prediction results and the oscillator results to calculate the cross entropy loss, and use the cross entropy loss back propagation to update the parameters such as the weight matrix in the network.

[0101] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A fall detection method based on multimodal data fusion and dual-stream graph learning, characterized in that: The specific steps include: Collect acceleration, image and sound data of the detection target; The collected acceleration data is converted, that is, the one-dimensional data is converted into two-dimensional data, the converted data is spliced ​​and fused with the image data, and a convolutional neural network is used to extract features to obtain fused modal coding; The sound data is processed using a fully connected layer to obtain the sound modal code; A graph neural network is constructed for each modality. In each modality’s graph neural network, the node is the modality code corresponding to each sampling point, and the edge weight is the attention weight between nodes. Single-modal graph learning is performed on the single-modal graph data obtained based on fusion modal coding and sound modal coding respectively to obtain the single-modal learning features corresponding to fusion modal coding and sound modal coding; When performing unimodal graph learning, the process of obtaining the corresponding unimodal learning features includes: ； in, Represents the feature obtained after unimodal learning of the feature encoding of the jth node in the i-th unimodal graph; Represents the feature encoding of the jth node in the i-th unimodal graph; 、 represents the learnable matrix for unimodal learning in the i-th unimodal graph; The aggregation module represents the neighbor nodes of the jth node in the i-th unimodal graph; Represents a splicing operation; Represents element-wise product operation; represents the activation function; The process of obtaining the corresponding multimodal learning features includes: ； in, Represents the i-th node in the multimodal graph The feature encoding of a certain mode Feature encoding after multimodal learning update; 、 Represents the learnable matrix when performing multimodal learning; Represents the i-th node in the multimodal graph Aggregation module for neighbor nodes; Multimodal graph learning is performed based on the multimodal graph data obtained by fusion modal coding and sound modal coding to obtain multimodal learning features corresponding to fusion modal coding and sound modal coding; The single modality and multimodality codes of the same modality code are fused by addition and then fused with another modality code by splicing to obtain cross-modal features; The cross-modal features are passed through a fully connected layer to obtain the fall prediction results, including: ； ； in, represents cross-modal features, represents the output features of the sound unimodal graph, Represents the multimodal graph output features of the sound feature to the fusion feature, represents the output features of the fused feature single modal graph, Represents the multimodal graph output features of the fusion features to the sound features; represents a fully connected layer; represents the normalized exponential function; p represents the cross-modal feature The probability of being classified as a fall.

2. A fall detection method based on multimodal data fusion and dual-stream graph learning according to claim 1, characterized in that: The Gramian angular field method is used to encode the collected one-dimensional time series data into a two-dimensional image. The encoding process includes: ； ； in, Represents a one-dimensional data sequence, expressed as , n is the number of sampling points of the one-dimensional data sequence, Represents the i-th sampling point in a one-dimensional data sequence; for Angular cosine encoding mapped to polar coordinates; Based on one-dimensional data series The converted two-dimensional data.

3. The fall detection method based on multimodal data fusion and dual-stream graph learning according to claim 1 is characterized in that: Aggregation module for neighbor nodes of the jth node in the i-th unimodal graph Expressed as: ； Where H is the number of heads in the multi-head attention mechanism; is the set of neighbor nodes of the jth node in the i-th unimodal graph; is the jth node in the i-th unimodal graph and its kth neighbor node The attention weight at the h-th attention head; is the learnable matrix at the h-th attention head in the i-th unimodal graph; represents the feature encoding of the kth node in the i-th unimodal graph.

4. A fall detection method based on multimodal data fusion and dual-stream graph learning according to claim 3, characterized in that: The jth node in the i-th unimodal graph and its kth neighbor node Attention weight at the hth attention head Expressed as: ； in, To use the projection matrix For the jth node in the i-th unimodal graph Feature encoding The attention value matrix obtained by projection is expressed as ; To use the projection matrix For the jth node in the i-th unimodal graph Neighbor nodes Feature encoding The attention key matrix obtained by projection is expressed as ;d k1 is the bond matrix Dimension, SoftMax is a normalized exponential function.

5. The fall detection method based on multimodal data fusion and dual-flow graph learning according to claim 1 is characterized in that: The i-th node in the multimodal graph Neighbor node aggregation module The calculation includes: ， in, It means to splice the 1st to Hth attention heads together; Representation node The set of neighbor nodes of represents the attention weight between the node corresponding to the i-th action clip modality a and the node corresponding to the j-th action clip modality t in the h-th head of the multi-head attention, for The corresponding learnable matrix, Represents the feature vector of the node corresponding to the j-th action segment modality t, Represents the node set of mode a in the multimodal graph; represents the attention weight between the node corresponding to the i-th action clip modality t and the node corresponding to the j-th action clip modality a in the h-th head of the multi-head attention, for The corresponding learnable matrix, Represents the feature vector of the node corresponding to the j-th action segment modality a, A set of nodes representing mode t in a multimodal graph.

6. A fall detection method based on multimodal data fusion and dual-flow graph learning according to claim 5, characterized in that: Attention weight between modality a of node i and modality t of node j in the h-th head of multi-head attention Expressed as: ； ； in, Represents the modes in a multimodal graph Modal influence; is the mode in the multimodal graph The query matrix corresponding to the feature encoding of is expressed as , is the projection matrix corresponding to the value matrix of the h-th head modality a in the multi-head attention; Represents the modes in a multimodal graph The key matrix corresponding to the feature encoding of is expressed as , is the projection matrix corresponding to the key matrix of the h-th head modality t in the multi-head attention; Represents the bond matrix Dimensions; Represents the modes in a multimodal graph Modal influence; is the mode in the multimodal graph The query matrix corresponding to the feature encoding of is expressed as , is the projection matrix corresponding to the value matrix of the h-th head modality t in the multi-head attention; Represents the modes in a multimodal graph The key matrix corresponding to the feature encoding of is expressed as , is the projection matrix corresponding to the key matrix of the h-th head modality a in multi-head attention.

Images