Human body action recognition method based on multi-modal information fusion

Abstract

The invention provides a human body action recognition method based on multi-modal information fusion, and relates to the technical field of deep learning model construction. The core of the method is to construct an end-to-end deep learning model, and deep fusion is performed on multi-modal information such as skeleton key points, visual appearance and inertial sensors through a cross-modal attention fusion mechanism. According to the model, firstly, spatial-temporal characteristics of all modals are extracted by using a special sub-network; afterwards, the features are embedded into a unified space and input into a trans-modal attention fusion module based on a Transform architecture for interaction, and the module can dynamically calculate the importance weight between modals to realize adaptive feature enhancement and filtering; and finally, outputting an action category through a classifier. According to the method, through a complete model training process and optimization parameters, the problems of irrelevant information interference and difficulty in cooperative utilization of local and global features under a complex background are effectively solved, and the accuracy and robustness of action recognition are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of human motion recognition technology, and in particular to a human motion recognition method based on multimodal information fusion, which is used to improve the performance of human motion recognition in complex scenes. Background Technology

[0002] Human motion recognition is one of the core tasks of computer vision and is widely used in fields such as intelligent monitoring, human-computer interaction, and virtual reality. Existing technologies are mostly based on single-modality (such as RGB video) or simple multimodal fusion strategies (such as feature stitching or decision-level voting).

[0003] Single-modal methods are susceptible to environmental interference (illumination, occlusion, etc.) and have poor robustness. Traditional fusion methods fail to deeply explore the complementarity and intrinsic correlation between modalities, resulting in shallow fusion layers, poor model flexibility, and difficulty in dealing with uneven quality of modal data (such as missing modal signals or high noise). Furthermore, these methods typically treat feature extraction, fusion, and classification as isolated steps, lacking a unified, end-to-end learning model framework, which limits further performance improvement. Therefore, there is an urgent need for a multimodal fusion model that can perform deep feature interactions and adaptive weight allocation. Summary of the Invention

[0004] The purpose of this invention is to provide a human motion recognition method based on multimodal information fusion to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a human action recognition method based on multimodal information fusion, wherein the method is implemented by constructing an end-to-end deep learning model, and the construction and application of the model includes the following process:

[0006] (1) Capture three core types of data related to the target human body: RGB video sequence, image stream, and inertial sensor data. For RGB video, human target detection and continuous tracking technology are used to extract the temporal information of key points of skeletal joints between frames; background removal processing is performed on the image stream to enhance the feature recognition of the main human body area; noise reduction and purification are performed on the inertial sensor data, and sliding window segmentation technology is used to complete the time axis calibration with the video data to ensure that the multi-dimensional data are completely synchronized in time.

[0007] (2) In order to adapt to the feature extraction needs of different data types, three independent sub-networks are used. The first is a skeletal feature modeling network based on graph convolutional network (GCN), which focuses on mining the topological connection relationship and dynamic motion features between joints; the second is a visual feature extraction network using pre-trained 3D convolutional neural network (3D-CNN), which accurately captures the details of human appearance and the motion change rules of spatiotemporal dimensions; the third is to construct an inertial data parsing network of one-dimensional convolutional neural network (1D-CNN), which deeply mines the time-frequency domain feature information in IMU data and forms a highly discriminative temporal feature expression.

[0008] (3) First, perform precise time dimension calibration on the single-modal feature sequences output by each dedicated sub-network to eliminate the temporal offset between different modalities; then configure an independent fully connected transformation layer for each type of modal feature to uniformly map the modal features with different dimensions and distributions to the same high-dimensional embedding space, generating an embedding vector sequence with consistent dimensions that can directly perform cross-modal interaction.

[0009] (4) A cross-modal fusion system based on a Transformer encoder is constructed, with the input being the sequence of vectors from each modality after a unified embedding space. First, intra-modal self-attention calculation is used to strengthen the temporal correlation within a single modality and improve the coherence and integrity of intra-modal features. Then, a cross-modal attention interaction process is initiated, using the embedding sequence of any one modality as the query and the embedding sequences of other modalities as the key and value, respectively. The attention allocation weights between modalities are dynamically generated using a scaling dot product attention mechanism, realizing intelligent adaptive fusion of multimodal context information. Finally, the fused feature sequence after deep interaction integration is output.

[0010] (5) Perform global average pooling on the feature sequence after cross-modal fusion to compress the temporal dimension and generate a global comprehensive feature vector; input the vector into a fully connected network containing a Dropout layer to complete feature enhancement and dimension reduction; finally, send the processed feature vector into the Softmax classifier to output the probability distribution results of various action categories, providing a basis for subsequent decision-making.

[0011] (6) Determine the cross-entropy loss function as the core optimization objective for model training to accurately quantify the deviation between the prediction results and the true labels; adopt the Adam optimization algorithm and combine it with the backpropagation mechanism to iteratively update all learnable parameters of the model; reasonably configure hyperparameters such as initial learning rate, batch training size and total number of training iterations, and introduce real-time monitoring of the validation set and early stopping mechanism to effectively avoid overfitting problems and ensure that the model has good generalization ability.

[0012] (7) Load the model parameters that perform best on the validation set during training and complete the deployment of the inference model; input the new multi-dimensional data to be identified into the deployed model and perform forward propagation calculation; select the category with the highest probability value from the probability distribution output by Softmax as the final human action recognition result to achieve accurate determination of action category.

[0013] The first sub-network described above employs a graph convolutional network, calculating the attention weights of its neighboring nodes for each key node in the spatiotemporal graph structure to adaptively aggregate features; for node i and its neighboring node j, the attention coefficient e ij The calculation method is as follows:

[0014] e ij =LeakyReLU(a T [Wh i ||Wh j ])

[0015] Among them, h i and h i Let W be the node feature, W be the learnable weight matrix, a be the learnable attention vector, and || denote the concatenation operation; the normalized attention weights a ij =softmax j (e ij ), used for feature aggregation.

[0016] The cross-modal attention computation in process 4 adopts a multi-head attention mechanism, which linearly projects the query, key and value to multiple subspaces, executes the attention function in parallel and concatenates the results to capture the intermodal dependencies in different subspaces.

[0017] The cross-modal attention computation is performed in two stages:

[0018] The first stage uses the feature sequence E of the skeletal modality. s As query Q1, the feature sequence E of the visual modality u Using K1 as the key and V1 as the value, calculate the first-level cross-modal attention feature F. s-u =Attention(Q1, K1, V1);

[0019] In the second stage, the first-level cross-modal attention feature F s-u As query Q2, the characteristic sequence E of the inertial mode. i Using K2 as the key and V2 as the value, the deep fusion feature sequence F is calculated. fusion =Attention(Q2, K2, V2);

[0020] The attention calculation at each step uses the scaled dot product attention formula:

[0021]

[0022] In process 3, after mapping each modal feature to a unified feature space, a modal feature calibration module based on channel attention is introduced, which includes the following operations:

[0023] The first step is to apply the unified feature U∈R T*C Perform variance normalization:

[0024]

[0025] The second step is to calculate the channel attention weights Z∈R. C :

[0026] Z = σ(W2·RELU(W1·GAP(U)))

[0027] Where GAP is global averaging, W1 and W2 are learnable parameters, and σ is the Sigmoid function;

[0028] The third step is to output the calibrated features. This represents a multiplicative broadcast of the channel dimension.

[0029] In process 5, global average pooling is used to perform temporal aggregation on the fused feature sequence to obtain a global feature representation; the at least one fully connected layer includes a dropout layer, which randomly drops some neuron connections during forward propagation in the training phase to prevent model overfitting.

[0030] In process 6, an early stopping strategy is adopted. If the performance metrics on the validation set do not improve over multiple consecutive training epochs, training is terminated and the parameters of the currently optimal-performing model are saved. During model training, in addition to the cross-entropy loss function L... ce In addition, a modal alignment loss L based on contrastive learning is introduced. align The total loss function is:

[0031] L total =L ce +λL align

[0032] For samples within a batch, let the global feature set of the skeletal modality be {s}. i The global feature set of the visual modality is {U}. i}, then the modal alignment loss is calculated as follows:

[0033]

[0034] Among them, sim(u,u)=u Tv / (||u||||u||) represents the cosine similarity, T is the temperature hyperparameter, N is the batch size, and λ is the weighting coefficient that balances the two losses.

[0035] In the model inference stage of process 7, the forward computation of processes 1 to 5 is performed sequentially on the input multimodal data, and the category with the highest probability in the probability distribution output by the fully connected layer is taken as the final recognition result. When some modal data is missing, the existing modal features are mapped to a unified space through the fully connected layer in step 3, and attention fusion and subsequent computation in step 4 are performed using only the features of the existing modalities.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] 1. By using a cross-modal attention mechanism, dynamic weighted fusion is performed at the feature level, rather than using fixed rules. The model can learn the optimal fusion strategy autonomously, resulting in good deep fusion performance.

[0038] 2. The model can adaptively adjust its dependence on each modality based on the quality of the input data. When the signal-to-noise ratio of a certain modality is low (such as visual occlusion), it will automatically increase the weights of other reliable modalities to ensure the overall stability of the system.

[0039] 3. The entire model is differentiable from input to output, allowing for end-to-end optimization through gradient descent, thus avoiding the error accumulation and suboptimal problems caused by traditional step-by-step processing.

[0040] 4. The model filters out most background interference at the source through skeleton extraction and further focuses on action-related features through an attention mechanism, which significantly improves performance in complex scenes. Attached Figure Description

[0041] Figure 1 This is a flowchart of the overall architecture of the end-to-end model provided in this embodiment of the invention;

[0042] Figure 2 This is a schematic diagram illustrating the principle of the cross-modal attention fusion sub-model in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the multi-head attention mechanism used in an embodiment of the present invention;

[0044] Figure 4 This is a comparison chart of the model with other recognition accuracy results in this embodiment of the invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] It should be understood that the reference to "embodiment" in this application means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0047] To address the issues of single-modal methods in human motion recognition technology being susceptible to environmental interference and exhibiting poor robustness, this invention proposes... Figures 1 to 4 The human action recognition method based on multimodal information fusion shown includes:

[0048] Step 1: Processing multimodal data and aligning time series. Simultaneously, subjects wear wearable devices with integrated IMUs to synchronously acquire accelerometer and gyroscope data. Preprocessing includes: for RGB video sequences, using object detection algorithms to locate and track human bounding boxes, and cropping the target person region based on this; subsequently, using pose estimation algorithms to extract 2D or 3D human skeletal keypoint coordinates from the cropped RGB frames, forming a skeletal keypoint sequence; for depth image sequences, primarily performing background subtraction, leveraging the ease of foreground segmentation using depth information to extract depth map regions representing the human body to eliminate complex background interference; for IMU data, first performing low-pass filtering (such as a Butterworth filter) to remove high-frequency noise, then segmenting using fixed-length sliding windows with some overlap to ensure motion continuity; finally, using timestamp alignment or interpolation methods to strictly align the skeletal sequence, visual feature sequence, and IMU data segments on the time axis, laying the foundation for subsequent fusion processing.

[0049] Step 2: Three feature extraction sub-networks extract the relationships and features between skeletal points, extract spatiotemporal motion features of appearance, and extract temporal and frequency domain features, respectively. The first sub-network is dedicated to processing skeletal keypoint sequences, using a spatiotemporal graph convolutional network (ST-GCN) or its variant. This network models the human skeleton as a graph structure, with nodes representing joints and edges divided into spatial edges (connecting naturally adjacent joints in the human body) and temporal edges (connecting the same joint in adjacent frames). Through graph convolution operations on the spatiotemporal graph, it effectively captures the topological constraints and continuous motion patterns between joints. The second sub-network processes RGB video sequences, using a pre-trained 3D convolutional neural network (such as I3D or SlowFast). This network utilizes its general spatiotemporal representation capabilities learned on large video datasets (such as Kinetics) to simultaneously extract appearance and motion features from video clips. The third sub-network processes IMU data, using a one-dimensional convolutional neural network. This network can automatically extract meaningful temporal and frequency domain features, such as energy, periodicity, and motion patterns, from the raw acceleration and angular velocity signals by performing convolution operations in the temporal dimension.

[0050] Step 3: Align the features obtained in Step 2. This is typically done using linear interpolation or dynamic time warping to ensure that the feature sequences of all modalities have the same length. Then, the feature vectors of each modality are passed through an independent, trainable fully connected layer and projected into a shared, uniform-dimensional feature embedding space. This step aims to transform heterogeneous multimodal features to the same semantic scale, eliminating differences in dimensions and distribution between modalities, and facilitating subsequent cross-modal interactions.

[0051] Step 4: Construct a fusion module based on a Transformer encoder. Its internal processing flow is as follows: First, perform intra-modal self-attention calculation on the embedding sequence of each modality to strengthen the long-term temporal dependencies within each modality's sequence. Then, perform cross-modal attention calculation. In this stage, the embedding sequence of one modality is used as the query, and the embedding sequence of another modality is used as the key and value, respectively. For example, the skeletal modality can be used as the query, and the visual modality as the key and value. The attention weight of the skeletal features to the visual features is calculated by the scaling dot product attention formula (Attention(Q, K, V) = softmax(QK^T / √d_k)V), thereby fusing the relevant visual context information into the skeletal features. This process can be performed iteratively or in parallel to ensure sufficient interaction of information between modalities. Finally, the module outputs a deeply fused feature sequence, in which the features at each time step are rich in complementary information from multiple modalities.

[0052] Step 5: Perform feature aggregation on the deep fusion feature sequence (a two-dimensional tensor with dimensions of [time steps × feature dimension]) output from Step 4. Use global average pooling to pool the time dimension, compressing the variable-length temporal features into a fixed-length global feature vector. During the training phase, before inputting this global feature vector into the final classifier, pass it through a dropout layer to randomly drop some neurons with a preset probability. This is an effective regularization method to prevent the model from overfitting. Subsequently, the vector may pass through one or more fully connected layers for nonlinear transformation and feature dimensionality reduction. Finally, input the processed features into the Softmax classifier to calculate the probability of it belonging to each predefined action category, thus obtaining a probability distribution vector.

[0053] Step 6: Use a reasonable optimizer and appropriate hyperparameters to prevent problems such as gradient descent and overfitting during training. Supervised learning is used to train the entire end-to-end model. A cross-entropy loss function is defined as the optimization objective to measure the difference between the model's predicted probability distribution and the actual action labels. An adaptive matrix estimation optimizer is used, and the gradient of the loss function with respect to all model parameters is calculated using the backpropagation algorithm. The parameters are iteratively updated to minimize the loss. Before training, hyperparameters need to be set, including the initial learning rate, batch size, and maximum number of training epochs. To prevent overfitting, an early stopping strategy is adopted: after each training epoch, the model performance is evaluated on an independent validation set. When the loss metric on the validation set no longer decreases over several consecutive epochs, training is terminated early, and the model parameters saved from the epoch with the best performance on the validation set are rolled back as the final model.

[0054] Step 7: Application Deployment Phase. First, load the optimal model parameters obtained in Step 6. For new, multimodal data streams to be identified, perform the same preprocessing as in Step 1, and then input them into the loaded model for forward propagation. The model will execute the calculation process from Step 2 to Step 5 in sequence. Finally, read the output of the Softmax classifier, select the category with the highest probability value as the final action result output for this recognition, and complete a complete action recognition task.

[0055] Figure 4This is a bar chart comparing the recognition accuracy of the proposed multimodal information fusion action recognition method with existing single-modal dominant action recognition methods for human action categories in several scenarios. In the experiment, action data samples were selected, and the data samples were recognized separately, with the recognition accuracy for each action scenario calculated. As can be seen from the figure, the accuracy of the proposed multimodal information fusion action recognition method is significantly improved compared to existing single-modal dominant recognition methods. This experimental result demonstrates that the multimodal information fusion-based human action recognition method achieves efficient and accurate recognition of human actions, effectively addressing the technical limitations of existing single-modal or simple fusion action recognition methods, such as insufficient capture of temporal dependencies, poor modal information complementarity, and weak generalization ability in complex scenarios.

Claims

1. A human motion recognition method based on multimodal information fusion, characterized in that, The method is implemented by constructing an end-to-end deep learning model, the construction and application of which include the following steps: Step 1: First, perform multimodal data acquisition and preprocessing, simultaneously acquiring RGB video sequences, image sequences, and inertial measurement units (IMUs) containing the target human body; perform human body detection and tracking on the RGB video sequences, extracting continuous human skeletal key point sequences; perform background subtraction on the image sequences; and perform denoising and sliding window segmentation on the IMU data to achieve time alignment with the video data. Step 2: Next, perform modal feature extraction and sub-model construction, building three feature extraction sub-networks. The first sub-network processes the skeletal keypoint sequence based on a graph convolutional network to extract the topological relationships and motion features between joints; the second sub-network processes RGB video sequences based on a pre-trained 3D convolutional neural network to extract appearance and spatiotemporal motion features; the third sub-network processes IMU data based on a one-dimensional convolutional neural network to extract time-frequency domain features. Step 3: Then feature alignment and embedding are required. The modal feature sequences obtained in Step 2 are aligned in the time dimension and mapped to a unified feature embedding space through a fully connected layer to obtain an embedding vector sequence with the same dimension. Step 4: Construct a cross-modal attention fusion sub-model. The model is constructed based on the fusion module of the Transformer encoder. The embedded vector sequence output in Step 3 is used as input. First, intra-modal self-attention is calculated to enhance temporal dependencies. Then, cross-modal attention is calculated. The embedded sequence of one modality is used as the query, and the embedded sequences of other modalities are used as the key and value. Attention weights are calculated through the scaling dot product attention mechanism to achieve adaptive fusion of contextual information and output the deep fusion feature sequence. Step 5: Feature aggregation and classification sub-model construction. The fused feature sequence output from Step 4 is subjected to global average pooling to obtain a global feature vector. This vector is then passed through a fully connected network with a dropout layer for feature enhancement and dimensionality reduction. Finally, it is input into a Softmax classifier to obtain the probability distribution of the action category. Step 6: Model Training and Optimization. Define the cross-entropy loss function as the optimization objective for model training. Use the Adam optimizer to iteratively update all weight parameters of the model through the backpropagation algorithm; set the initial learning rate, batch size, and number of training epochs, and employ validation set monitoring and early stopping strategies to prevent overfitting. Step 7: Finally, perform model inference and action recognition. Load the optimal model parameters obtained from training in Step 6; input the new multimodal data into the model, perform forward propagation, and use the class with the highest probability in the Softmax output as the final recognition result.

2. The human action recognition method based on multimodal information fusion according to claim 1, characterized in that, In step 2, the first sub-network based on the graph convolutional network models the sequence of key points of the human skeleton as a spatiotemporal graph structure, where nodes represent joints and edges represent natural connections, including both spatial and temporal edges.

Images