Multimodal information fusion recognition method and system based on attention mechanism

Abstract

The present invention relates to a multimodal information fusion recognition method and system based on an attention mechanism. The method and system first compress and represent the RGB and depth video frame sequences of human actions to generate a spatiotemporal information representation graph of the entire video; then, the RGB and depth spatiotemporal representation graphs are respectively input into a two-stream deep convolutional network to extract their high-level semantic features; then, the features of these two modalities are input into an attention information fusion module to obtain two different multimodal fusion feature representations; finally, the two multimodal feature vectors are added or concatenated to form a feature vector, which is classified through a fully connected layer and a softmax function to obtain the action class in the video to be tested. This method and system can effectively utilize the complementary information of RGB and depth data to generate a semantically rich multimodal feature representation, greatly improving the accuracy and anti-interference ability of human action recognition.

Description

Technical Field

[0001] The present invention belongs to the field of information recognition, and in particular relates to a multimodal information fusion recognition method and system based on an attention mechanism. Background Art

[0002] The primary challenge in action recognition is how to represent actions in videos. Extracting feature representations of human actions in videos involves traditional hand-crafted feature extraction methods (such as HOG, HOF, MBH, DT / iDT, etc.) and deep learning-based methods. The latter, which can easily extract deeper and more efficient feature representations, are becoming increasingly mainstream. In terms of network architecture, action features can be extracted using two-dimensional convolutional neural networks (2D CNN), three-dimensional convolutional neural networks (3D CNN), recurrent neural networks (RNN, LSTM), and graph neural networks (GCN). In terms of data modality, there are single-modality methods based on RGB frames, optical flow images, skeleton sequences, and depth images, as well as methods that combine two or more modalities. These methods use different network structures to extract features from different data modalities for human action recognition. Methods that utilize multi-modal data also include information fusion operations.

[0003] The basic idea behind traditional manual feature extraction methods, such as the DT algorithm, is to use optical flow to obtain trajectories in video sequences. Four features, namely, HOF, HOG, MBH, and trajectory, are then extracted along these trajectories. Finally, the features are encoded using the Fisher Vector (FV) method, and SVMs are trained based on the encoded results for classification. 2D CNNs and multimodal methods, such as two-stream networks, use two convolutional networks to model appearance and motion information in action videos: a spatial CNN and a temporal CNN. The former learns actions from static images, while the latter extracts motion information based on optical flow for recognition. Existing methods construct five input modalities using motion representation graphs and skeleton images and extract features using a five-stream convolutional network. Other existing methods also design a cross-modal compensation module to learn complementary information between RGB and depth dynamic images. The two networks then output predictions for the different modalities. However, these multimodal methods all employ a post-fusion approach, specifically fusing the predictions of each modality through methods such as averaging and multiplication.

[0004] With the explosive growth of video data, the demand for video data processing and applications has also increased significantly. Typical application scenarios include intelligent video surveillance systems, video retrieval, human-computer interaction, and gaming and entertainment. A crucial step in this process is human action recognition. Similar to how humans understand and perceive the external environment by processing information from multiple sensory organs, human action recognition systems and terminal devices can also increase their accuracy and robustness by utilizing multiple information sources or data modalities. However, while various visual sensors provide diverse video data modalities and enrich data sources, they also introduce challenges in processing and fusing data across modalities. Multimodal information can overcome the limitations of single-modal information, such as its strong susceptibility to external environmental influences and single features, and provide complementary information. However, different modalities often suffer from inconsistent data distribution, significant disparity in the semantic information of extracted features, and the inability to directly fuse features.

[0005] Most existing technologies rely solely on single-modal data, such as RGB images, depth images, and skeleton sequences, for action recognition. This is susceptible to external environmental influences and suffers from feature loss. For example, RGB image samples are easy to collect and are the most commonly used, but they lack three-dimensional structural information and are easily affected by external factors such as lighting changes. This results in low action recognition rates and poor robustness and generalization capabilities. Action recognition based on multimodal data utilizes independent networks to extract features from different modalities, followed by post-fusion operations at the decision-making level. This approach offers limited improvements and fails to fully utilize the complementary information between different modalities, making it impossible to obtain effective multimodal feature representations. Summary of the Invention

[0006] The embodiments of the present invention provide a multimodal information fusion recognition method and system based on an attention mechanism, so as to at least solve the technical problem that existing information fusion recognition methods cannot obtain effective multimodal feature representation.

[0007] According to one embodiment of the present invention, a multimodal information fusion recognition method based on an attention mechanism is provided, comprising the following steps:

[0008] Compress the RGB and depth video frame sequences of human body movements to generate a spatiotemporal information representation of the entire video;

[0009] The RGB and depth spatiotemporal information representation maps are respectively input into the two-stream deep convolutional network to extract their high-level semantic features;

[0010] The high-level semantic features of RGB and depth modalities are fused with attention information to obtain two different multimodal fusion feature representations;

[0011] The two multimodal fusion feature vectors are added or concatenated into one feature vector, which is then classified through a fully connected layer and a softmax function to obtain the action category in the video to be tested.

[0012] Furthermore, the RGB and depth video frame sequences of human body movements are compressed to generate a spatiotemporal information representation of the entire video, including:

[0013] Read the RGB and depth video sequences of human motion from the database in the terminal device. The RGB frames and depth images in the video sequence are one-to-one corresponding. Suppose the video sequence contains M frames. Perform sparse average sampling of the RGB video and depth sequence in the time dimension to obtain an image sequence of human motion.

[0014] Furthermore, for the sequence frames in the RGB video, the compressed spatiotemporal information is represented in three channels respectively, and the motion information obtained in the calculation process is rounded off, and finally a motion image similar to the RGB three-channel can be obtained;

[0015] For the depth sequence diagram, it is first converted into a grayscale image of [0, 255] by scaling, and then the motion spatiotemporal information of the single-channel grayscale image is represented, and finally the single-channel spatiotemporal information representation diagram is obtained by rounding.

[0016] Furthermore, the spatiotemporal information representations of RGB and depth are fed into a two-stream deep convolutional network to extract their high-level semantic features, including:

[0017] For the spatiotemporal information representation graph generated by RGB and depth, a network model is designed to jointly learn the features of RGB and depth information flows, and extract high-level semantic features of each modality through a deep convolutional network;

[0018] In terms of network architecture selection, Resnet, Inception, and VGG network models are selected. The two network structures are the same and optimized simultaneously. The extracted modal features are used for the next fusion operation.

[0019] Furthermore, in the generated spatiotemporal representation, RGB is a three-channel image and depth is a single-channel image. The spatiotemporal information representation of depth is repeated three times along its channel dimension to expand it into a three-channel image. The two spatiotemporal representation images are simultaneously input into the convolutional neural network to extract features. and Then, the feature map channel dimension is reduced by 1x1 convolution to obtain and Finally, it is flattened along the spatial dimension to obtain and Two two-dimensional feature vectors, used for subsequent feature fusion.

[0020] Furthermore, the high-level semantic features of the RGB and depth modalities are fused with attention information to obtain two different multimodal fusion feature representations including:

[0021] The high-level semantic features of RGB and depth modalities are input into the N-layer attention fusion module, and serve as different input roles in the fusion module of their own branch and the other branch, so that the two modalities are interactively fused through the attention mechanism, and the complementary information between the modalities is learned to obtain the multimodal feature X r and X d .

[0022] Furthermore, the input is the features of two modalities, where the modal features of the branch where the module is located are used as Q, and the modal features of the other branch are used as K and V. They are fused through the multi-head attention mechanism, and then the bottleneck activation module is used to further strengthen the learned multimodal features and enhance their expressive power.

[0023] Furthermore, the bottleneck activation module is configured as follows:

[0024] First, the input features Perform a global average pooling operation to compress the HW dimension of the feature to 1 to generate global context information Then, two layers of fully connected layers are used to reduce the dimension of the other dimension of the feature and restore it to the original dimension. The ReLu activation function is inserted in the middle to increase the nonlinearity, thus forming a bottleneck structure and obtaining the activation value. Finally, the features passed through the sigmoid function are used as activation values ​​and multiplied element-by-element with the original input features to strengthen or suppress the information in the original input features.

[0025] Furthermore, the two multimodal fusion feature vectors are added or concatenated into one feature vector, which is then classified through a fully connected layer and a softmax function to obtain the action categories in the video to be tested, including:

[0026] The two multimodal fusion feature vectors are added or concatenated into one feature vector to fully utilize their information and obtain the final fusion representation X fusion , action classification is performed through the softmax fully connected layer.

[0027] According to another embodiment of the present invention, a multimodal information fusion recognition system based on an attention mechanism is provided, comprising:

[0028] The spatiotemporal information representation graph generation module is used to compress the RGB and depth video frame sequences of human body movements and generate the spatiotemporal information representation graph of the entire video;

[0029] The high-level semantic feature extraction module is used to input the RGB and depth spatiotemporal information representation maps into the two-stream deep convolutional network to extract their high-level semantic features;

[0030] The multimodal fusion feature calculation module is used to fuse the high-level semantic features of RGB and depth modalities to obtain two different multimodal fusion feature representations;

[0031] The action category acquisition module is used to add or concatenate two multimodal fusion feature vectors into a feature vector, and then classify it through a fully connected layer and a softmax function to obtain the action category in the video to be tested.

[0032] In the multimodal information fusion recognition method and system based on the attention mechanism in the embodiment of the present invention, the RGB and depth video frame sequences of human movements are first compressed and represented to generate a spatiotemporal information representation graph of the entire video; then the spatiotemporal representation graphs of RGB and depth are respectively input into a dual-stream deep convolutional network to extract their high-level semantic features; then the features of these two modalities are input into the attention information fusion module to obtain two different multimodal fusion feature representations; finally, the two multimodal feature vectors are added or concatenated into a feature vector, and then classified through a fully connected layer and a softmax function to obtain the action class in the video to be tested. This method can effectively utilize the complementary information of RGB and depth data, and perform interactive fusion between the two modalities at the feature level, thereby generating a semantically rich multimodal feature representation, greatly improving the accuracy and anti-interference ability of human movement recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0034] Figure 1 This is an overview of the multimodal information fusion recognition method and system based on the attention mechanism of the present invention;

[0035] Figure 2 This is a diagram of the dual-stream multimodal feature extraction network structure in the multimodal information fusion recognition method and system based on the attention mechanism of the present invention;

[0036] Figure 3 Schematic diagram of multimodal information fusion and action classification in the multimodal information fusion recognition method and system based on the attention mechanism of the present invention;

[0037] Figure 4This is a structural diagram of the attention fusion module in the multimodal information fusion recognition method and system based on the attention mechanism of the present invention;

[0038] Figure 5 This is a structural diagram of the bottleneck activation module in the multimodal information fusion and recognition method based on the attention mechanism of the present invention and the system. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] This paper proposes a multimodal information fusion recognition method and system based on an attention mechanism. This method and system uses the attention mechanism to explore semantic connections between modalities, enabling beneficial combinations between them. This fully leverages the advantages of multimodal fusion and improves the accuracy and robustness of human action recognition in complex scenarios.

[0042] The present invention uses 2D CNN to learn motion features for both RGB and depth video sequences, and designs a fusion module based on the attention mechanism, which effectively combines the complementary information between different modal data, improves the accuracy and robustness of motion recognition, and reduces the impact of environmental changes and the lack of single modal features on the reduction of recognition accuracy. The present invention includes a multi-head cross-modal attention module and a bottleneck excitation module, and uses RGB and depth modal data for human motion recognition. These two types of data can be very conveniently collected by a depth camera. The present invention can effectively utilize the complementary information of RGB and depth data through the proposed fusion method, and perform interactive fusion between the two modalities at the feature level, thereby generating a semantically rich multimodal feature representation, which greatly improves the accuracy and anti-interference ability of human motion recognition.

[0043] The present invention can better handle the collaborative relationship of multimodal information to improve the accuracy and stability of action recognition. The method and system use a standardized human action database pre-stored in the terminal device to train and infer the model to achieve automatic recognition of human actions. Its main process includes: (1) spatiotemporal information representation of action videos; (2) extraction of multimodal high-level semantic features; (3) multimodal information fusion representation based on the attention mechanism, as well as multimodal feature integration and action classification.

[0044] The present invention first compresses the RGB and depth video frame sequences of human body movements to generate a spatiotemporal information representation graph of the entire video; then the RGB and depth spatiotemporal representation graphs are respectively input into a dual-stream deep convolutional network to extract their high-level semantic features; then the features of these two modalities are input into the attention information fusion module to obtain two different multimodal fusion feature representations; finally, the two multimodal feature vectors are added or concatenated into a feature vector, which is then classified through a fully connected layer and a softmax function to obtain the action category in the video to be tested.

[0045] The present invention is generally summarized as follows Figure 1 As shown in the figure, according to the algorithm processing flow, it is elaborated in detail from the following aspects:

[0046] (1) Representation of spatiotemporal information in action videos

[0047] Read the RGB and depth video sequences of human body movements from the database in the terminal device. The RGB frames and depth images in the video sequence are one-to-one corresponding. Assume that the video sequence contains M frames. First, perform sparse average sampling of the RGB video and depth sequence in the time dimension to obtain the image sequence of human body movements. <I1,I2,I3,...,I T>, T represents the number of sampled video frames (assuming the action consists of 100 frames, T = 10 after coefficient averaging, that is, 10 frames are used to represent the entire action, and the RGB and depth frame labels are the same). The spatiotemporal information representation diagram of the video sequence is defined as MI:

[0048]

[0049] For the sequence frames in the RGB video, the compressed spatiotemporal information can be represented in three channels respectively, and the motion information obtained in the calculation process is rounded off to obtain a motion image similar to the RGB three channels. For the depth sequence diagram, it is first converted into a grayscale image of [0, 255] by scaling, and then the motion spatiotemporal information of the single-channel grayscale image is represented, and the single-channel spatiotemporal information representation diagram is finally obtained by rounding off.

[0050] (2) Two-stream network model training and feature extraction

[0051] For the spatiotemporal information representation graph generated by RGB and depth, a network model is designed to jointly learn the features of RGB and depth information streams, and extract the high-level semantic features of each modality information through a deep convolutional network. In terms of network architecture selection, mature network models such as Resnet, Inception, and VGG can be selected. The two networks have the same structure and are optimized simultaneously, but do not share parameters. The extracted modal features are used for the next fusion operation. The dual-stream multimodal feature extraction network structure is as follows: Figure 2 shown.

[0052] In the spatiotemporal representation generated in step (1), RGB is a three-channel image and depth is a single-channel image. In order to adapt to the network model, the spatiotemporal information representation of depth is repeated three times along its channel dimension, thereby expanding it into a three-channel image. The two spatiotemporal representation images are simultaneously input into the convolutional neural network to extract features. and Then, the feature map channel dimension is reduced by 1x1 convolution to obtain and Finally, it is flattened along the spatial dimension to obtain and Two two-dimensional feature vectors, used for subsequent feature fusion.

[0053] (3) Multimodal information fusion representation and action classification based on attention mechanism

[0054] The two features are input into the N-layer attention fusion module, and serve as different input roles in the fusion module of the branch to which they belong and the other branch, namely Figure 3Q, K, and V in the image are combined to make the two modalities interactively integrated through the attention mechanism, learn the complementary information between the modalities, and obtain the multimodal feature X. r and X d Finally, these two multimodal features are added or concatenated to fully utilize their information and obtain the final fusion representation X fusion , action classification is performed through the softmax fully connected layer.

[0055] exist Figure 3 The specific structure of the attention fusion module shown in Figure 4 As shown. Its input is the features of two modalities, where the modal features of the branch where the module is located are used as Q, and the modal features of the other branch are used as K and V. They are fused through the multi-head attention mechanism, and then the bottleneck activation module (BEF) further strengthens the learned multimodal features to enhance their expressive power. Each step in the attention fusion module is interspersed with cross-layer connections and regularization operations to ensure the stability of deep network training. Since multi-head attention cannot distinguish the position information of elements in the input features, position encoding is added to each input feature. The entire process can be expressed as follows:

[0056]

[0057]

[0058] MultiHead(Q,K,V)=Concat(h1,...,h n )W O , (4)

[0059]

[0060] in, is the parameter matrix, d k It is the embedding dimension of the key. is the positional encoding of Q, is the position encoding of K and V, h is the number of heads of multi-head attention. h = 8, d m =1024,d k =d v =d m / h=128 is the default setting.

[0061] The bottleneck activation module (BEF) structure is as follows Figure 5 As shown, it first converts the input features Perform a global average pooling operation to compress the HW dimension of the feature to 1 to generate global context information As shown in Equation 6. Then, two layers of fully connected layers are used to reduce the dimension of the other dimension of the feature and restore it to the original dimension. The ReLu activation function is inserted in the middle to increase the nonlinearity, thus forming a bottleneck structure and obtaining the activation value As shown in Equation 7. Finally, the features passed through the sigmoid function are used as activation values ​​and multiplied element-by-element with the original input features to strengthen or suppress the information in the original input features, as shown in Equation 8. The dimensionality reduction ratio r limits the number of model parameters and enhances generalization performance (r = 8 in the application of the present invention).

[0062]

[0063] s=Sigmoid(W2ReLU(W1z)), (7)

[0064]

[0065] in, is the input of the bottleneck excitation module, is the parameter matrix, is the output of the bottleneck excitation module.

[0066] This paper addresses the problem of modal semantic fusion in multimodal human action recognition tasks by proposing an information fusion recognition method and system based on an attention mechanism. This method and system uses a dual-path deep convolutional network to simultaneously extract high-level semantic features from each modality by performing spatiotemporal information representation on RGB and depth video sequences. The extracted features are then subjected to an attention-based cross-modal fusion operation, enabling effective multimodal feature interaction and fusion. This fully utilizes the complementary information between different modalities and improves the accuracy and robustness of human action recognition. Furthermore, the bottleneck excitation module embedded in the proposed attention fusion module enhances the model's expressiveness and generalization capabilities while introducing very few parameters and computational complexity, thereby improving the accuracy of action recognition. Compared with existing technologies, the proposed information fusion method based on an attention mechanism performs efficient multimodal semantic fusion at the feature level, fully exploring the connections between information from different modalities. The fusion operation in this method is performed using a modular structure, making it easily applicable to other models and systems, such as visual question-answering models or robotic systems. Experiments have proven the feasibility of this invention, with good results.

[0067] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0068] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0069] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the system embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0070] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0071] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0072] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0073] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A multimodal information fusion recognition method based on attention mechanism, characterized in that: The following steps are involved: Compress the RGB and depth video frame sequences of human body movements to generate a spatiotemporal information representation of the entire video; The RGB and depth spatiotemporal information representation maps are respectively input into the two-stream deep convolutional network to extract their high-level semantic features; The high-level semantic features of RGB and depth modalities are fused with attention information to obtain two different multimodal fusion feature representations; The two multimodal fusion feature vectors are added or concatenated into one feature vector, which is then classified through a fully connected layer and a softmax function to obtain the action category in the video to be tested. The two different multimodal fusion feature representations obtained by fusing the high-level semantic features of the RGB and depth modalities with attention information include: The high-level semantic features of RGB and depth modalities are input into the N-layer attention fusion module, and serve as different input roles in the fusion module of their own branch and the other branch, so that the two modalities are interactively fused through the attention mechanism, and the complementary information between the modalities is learned to obtain the multimodal feature X r and X d ; The input is the features of two modalities, where the modal features of the branch where the module is located are used as Q, and the modal features of the other branch are used as K and V. They are fused through the multi-head attention mechanism, and then the bottleneck activation module is used to further strengthen the learned multimodal features and enhance their expressive power. The bottleneck activation module configuration is: First, the input features Perform a global average pooling operation to compress the HW dimension of the feature to 1 to generate global context information Then, two layers of fully connected layers are used to reduce the dimension of the other dimension of the feature and restore it to the original dimension. The ReLu activation function is inserted in the middle to increase the nonlinearity, thus forming a bottleneck structure and obtaining the activation value. Finally, the features passed through the sigmoid function are used as activation values ​​and multiplied element-by-element with the original input features to strengthen or suppress the information in the original input features.

2. The multimodal information fusion recognition method based on the attention mechanism according to claim 1 is characterized in that: The process of compressing the RGB and depth video frame sequences of the human body motion to generate a spatiotemporal information representation graph of the entire video includes: Read the RGB and depth video sequences of human motion from the database in the terminal device. The RGB frames and depth images in the video sequence are one-to-one corresponding. Suppose the video sequence contains M frames. Perform sparse average sampling of the RGB video and depth sequence in the time dimension to obtain an image sequence of human motion.

3. The multimodal information fusion recognition method based on the attention mechanism according to claim 2 is characterized in that: For the sequence frames in the RGB video, the temporal and spatial information is compressed in three channels respectively, and the motion information obtained in the calculation process is rounded off to obtain a motion image similar to the RGB three channels. For the depth sequence diagram, it is first converted into a grayscale image of [0, 255] by scaling, and then the motion spatiotemporal information of the single-channel grayscale image is represented, and finally the single-channel spatiotemporal information representation diagram is obtained by rounding.

4. The multimodal information fusion recognition method based on the attention mechanism according to claim 1 is characterized in that: The step of inputting the RGB and depth spatiotemporal information representation graphs into a two-stream deep convolutional network to extract high-level semantic features includes: For the spatiotemporal information representation graph generated by RGB and depth, a network model is designed to jointly learn the features of RGB and depth information flows, and extract high-level semantic features of each modality through a deep convolutional network; In terms of network architecture selection, Resnet, Inception, and VGG network models are selected. The two network structures are the same and optimized simultaneously. The extracted modal features are used for the next fusion operation.

5. The multimodal information fusion recognition method based on the attention mechanism according to claim 4 is characterized in that: The generated spatiotemporal representation graph has three channels of RGB and a single channel of depth. The spatiotemporal information representation graph of depth is repeated three times along its channel dimension to expand it into a three-channel image. The two spatiotemporal representation graphs are simultaneously input into the convolutional neural network to extract features. and Then, the feature map channel dimension is reduced by 1x1 convolution to obtain and Finally, it is flattened along the spatial dimension to obtain and Two two-dimensional feature vectors, used for subsequent feature fusion.

6. The multimodal information fusion and recognition method based on the attention mechanism according to claim 1 is characterized in that: The two multimodal fusion feature vectors are added or concatenated into one feature vector, and then classified through a fully connected layer and a softmax function to obtain the action category in the video to be tested, including: The two multimodal fusion feature vectors are added or concatenated into one feature vector to fully utilize their information and obtain the final fusion representation X fusion , action classification is performed through the softmax fully connected layer.

7. A multimodal information fusion and recognition system based on an attention mechanism according to the multimodal information fusion and recognition method based on an attention mechanism as claimed in claim 1, characterized in that: include: The spatiotemporal information representation graph generation module is used to compress the RGB and depth video frame sequences of human body movements and generate the spatiotemporal information representation graph of the entire video; The high-level semantic feature extraction module is used to input the RGB and depth spatiotemporal information representation maps into the two-stream deep convolutional network to extract their high-level semantic features; The multimodal fusion feature calculation module is used to fuse the high-level semantic features of RGB and depth modalities to obtain two different multimodal fusion feature representations; The action category acquisition module is used to add or concatenate two multimodal fusion feature vectors into a feature vector, and then classify it through a fully connected layer and a softmax function to obtain the action category in the video to be tested.

Images