A multi-modal human motion recognition method
By initializing the random pulse filter convolution matrix in the cascaded cross-attention network, the problem of fitting multimodal human action recognition on small datasets is solved, improving the model's fitting and generalization ability, and achieving better performance and adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2026-03-24
AI Technical Summary
Existing multimodal human motion recognition methods perform poorly on small datasets, making it difficult to fit and generalize effectively. In particular, multimodal networks based on VisionTransformer lack inductive bias during modality fusion, leading to performance degradation.
By improving the cascaded cross-attention network, the query (Q) and key (K) are initialized as random pulse filter convolution matrices, introducing the inductive bias of CNN, thereby improving the model's fitting and generalization ability on small datasets.
It improves the model's performance and versatility on small datasets, enhances the model's adaptability and generality, and is able to better capture subtle features and patterns in the dataset.
Smart Images
Figure CN119418401B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of human action recognition, and relates to a multi-modal human action recognition method. BACKGROUND
[0002] Human action recognition is a current research hotspot. Human action recognition is to analyze and process the input video or 3D action data to determine which category different actions belong to. Human action recognition technology has practical application value in various industries such as human-computer interaction scene, monitoring video, gesture recognition, rehabilitation training, robot and behavior understanding. In recent years, with the rise of multi-modal concept research, multi-modal action recognition has gradually become a research hotspot. Multi-modal action recognition aims to use multi-modal input to better recognize human activities. In multi-modal action recognition, according to the visibility data modal, it can be roughly divided into two categories, namely visual modal and non-visual modal. RGB, skeleton, depth, infrared sequence, point cloud and event stream visually "intuitively" represent human action and can be regarded as visual modal. In general, visual mode is very effective for human action recognition. Among them, RGB video data is the most common data type for human action recognition and has been widely used in monitoring systems. Skeleton data encodes the motion trajectory of human joints. When the action performed does not involve objects or scene background, skeleton data is concise and efficient for human action recognition.
[0003] The present application considers the fusion of RGB and skeletal modalities. RGB data and skeletal sequences can greatly improve the accuracy and efficiency of fine-grained activity analysis. RGB data captures appearance information of the activity scene, including color, texture, and other visual features, while skeletal sequences record detailed dynamic information of body posture and joint movement. This combination not only provides rich visual and motion information, but also helps to overcome the limitations of single modalities in dealing with similar backgrounds or subtle differences in action, and the two modalities complement each other and are very useful for fine-grained activity analysis. At present, multi-modal video Transformers have shown competitive performance, which usually reflects the remarkable success of Vision Transformers (ViT) in image recognition. Although multi-modal learning provides good performance, current methods tend to use a large amount of video data to learn ViT-based multi-modal networks, while small-scale data sets are limited by the limitations of the ViT attention architecture and are data inefficient, making it difficult to achieve fitting. Therefore, in the multi-modal fusion part, the present application takes the Cross-Attention-to- Concatenation network as the main body and improves it to make the data more efficient, so that it is easier to fit when facing small-scale data. Recent advances in CNN, especially ConvMixers, have made it increasingly similar to ViT in architecture. Specifically, ConvMixers divide multi-channel convolution into depth convolution and channel convolution, unlike previous CNN variants, ConvMixers use an equal resolution representation in all layers, without down-sampling the representation in consecutive layers. Although the transformer network also uses channel convolution, it uses multi-head attention to replace the depth convolution step. Recent studies have shown that random filter weights in the ConvMixer and ResNet frameworks can exhibit comparable performance to learned weights. The only parameters that need to be learned are those related to channel convolution, which are usually referred to as channel mixing in Transformer literature. This result is particularly intriguing because a large fraction of the parameters in a standard ConvMixer or ResNet are largely redundant, achieving effective performance without the need for learning. Due to the similarity of ConvMixer and ViT, the present application considers applying the unique characteristics of ConvMixer random filter weights to the multi-modal cross-attention network to ensure that there is the same inductive bias as in CNN networks, thereby facilitating easier fitting of the network.
[0004] Although the ViT shows great potential in processing large-scale data, its performance is poor compared with traditional convolutional neural networks (CNN) when facing small-scale data sets. Although the encoder of each modality can be initialized by pre-training, the lack of corresponding inductive bias of ViT compared with CNN will cause the network to be difficult to learn the context relationship between modalities from small-scale data sets when modalities are fused. SUMMARY
[0005] Therefore, the purpose of the present application is to provide a multi-modal human action recognition method. By improving the serial cross-attention network, the query (Q) and key (K) in the attention network are initialized to form a random pulse filter convolution matrix. This improvement initializes the inductive bias of CNN into the serial cross-attention network to improve the data effectiveness of the model and make it easier to fit on small-scale data sets.
[0006] To achieve the above purpose, the present application provides the following technical solutions:
[0007] A multi-modal human action recognition method, comprising the following steps:
[0008] Step one: extracting video spatial features through a convolutional neural network;
[0009] Step two: extracting time-dependent features of videos and skeleton points through a bidirectional long short-term memory network respectively;
[0010] Step three: encoding the skeleton features and video features through an encoder to unify the feature dimensions;
[0011] Step four: initializing the serial cross-attention network according to the determined initialization strategy;
[0012] Step five: inputting the encoded features of the skeleton modality and the video modality into the cross-attention network for feature fusion;
[0013] Step six: outputting the fusion features through a fully connected layer and a Softmax function to finally output the prediction results.
[0014] Further, in the step one, the convolutional neural network is a ResNet152 network.
[0015] Further, in the step four, the initialization strategy includes initializing the query Q and the key K into a random pulse filter convolution matrix.
[0016] Further, in the step five, the serial cross-attention network includes:
[0017] The first cross-attention module is used for cross-attention calculation on the skeleton modality feature and the RGB video modality feature.
[0018] The second cross-attention module is used for cross-attention calculation on the RGB video modality feature and the skeleton modality feature.
[0019] The full connection layer is used for splicing the outputs of the two cross-attention modules.
[0020] Further, the first cross-attention module and the second cross-attention module adopt a cross initialization strategy for initialization, and the full connection layer adopts an original initialization strategy for initialization.
[0021] Further, in step six, the full connection layer and the Softmax function constitute a classifier for mapping the fusion feature to different action categories.
[0022] The beneficial effects of the present application are that, compared with the prior art, the present application improves the series cross-attention network, forms a random pulse filter convolution matrix by initializing the query (Q) and the key (K) in the attention network. This improvement initializes the inductive bias of the CNN into the series cross-attention network to improve the data effectiveness of the model and make it easier to fit on small-scale data sets. Specifically, the introduction of the random pulse filter convolution matrix increases the randomness of the attention mechanism, which can better capture subtle features and patterns in the data set. This improvement not only considers the fitting ability of the model, but also takes into account the generalization ability of the model, because the introduction of a certain degree of randomness helps the model to adapt to various data distributions and noise conditions. Therefore, this improved method can improve the performance of the model on small-scale data sets and enhance its universality and versatility.
[0023] Other advantages, objects, and features of the present application will be in part apparent and in part pointed out hereinafter in the specification, and it is to be understood that the application can be carried out by practicing the application without reference to the above discussion of preferred embodiments of the application. The objects and other advantages of the present application will be realized and attained by the application described in the following description. BRIEF DESCRIPTION OF DRAWINGS
[0024] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, in which:
[0025] Figure 1 The overall block diagram for multi-modal human action recognition is shown in Figure 1.
[0026] Figure 2 The Vision Transformer encoder architecture block diagram is shown in Figure 2.
[0027] Figure 3 for a series cross attention network block diagram;
[0028] Figure 4 for an initialization strategy schematic block diagram. DETAILED DESCRIPTION
[0029] The present application is described and explained more fully with reference to the following detailed description. Other advantages of the present application will be more readily appreciated as the same becomes better understood by reference to the following detailed description. The following detailed description should be considered in connection with the accompanying drawings, wherein the understandings of the present application can be gained by those skilled in the art from the following detailed description, which describes and explains the present application, together with the accompanying drawings. The following embodiments are provided by way of illustration, and nothing therein should be taken as a limitation on the overall scope of the application. Such embodiments are combined with each other in their complementary relationships as appropriate.
[0030] The accompanying drawings are included to provide a further understanding of the application, and are incorporated in and constitute a part of this specification, illustrate embodiments of the application, and together with the description serve to explain the principles of the application. In the drawings:
[0031] The same or similar components in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "front", "back" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only for illustrative purposes, and cannot be understood as a limitation on the present application. For those skilled in the art, the specific meanings of the above terms can be understood according to the specific circumstances.
[0032] Referring to Figure 1 , the method comprises the following steps:
[0033] Step one: extracting video spatial features through ResNet152.
[0034] Step two: extracting time-dependent features of video and skeleton points respectively through Bi LSTM.
[0035] Step three: encoding the skeleton features and RGB video features respectively through the encoder to unify the feature dimensions.
[0036] Step four: initializing the series cross attention network according to the determined initialization strategy.
[0037] Step five: input the features of the bone modality and the RGB modality into a cross attention network for feature fusion.
[0038] Step six: the output fusion features are finally outputted after passing through a full connection layer and a Softmax function.
[0039] 1. RGB video feature extraction module
[0040] The present application uses a ResNet152 network to extract spatial features of RGB video data, and uses a bidirectional LSTM network to extract temporal features for more efficient extraction. The RGB image features extracted by the ResNet152 are input into the two-directional LSTM, and the relationship between each frame and the previous and next frames is extracted through the two-directional LSTM network, and a feature vector is outputted, and the expression is:
[0041]
[0042] In the formula: LSTM fw (·) is the calculation formula of the forward LSTM, LSTM bw (·) is the calculation formula of the reverse LSTM; t represents the t-th frame, i is the i-th sample, h is the hidden state vector after the LSTM network, X is the spatial feature extracted by the ResNet network, f i,t is the fused spatio-temporal feature.
[0043] 2. Bone feature extraction module
[0044] The data of the bone is mainly composed of bone points, so it is necessary to first normalize the bone features by equation (2.1), and then flatten them, so as to obtain the spatial features of the skeleton.
[0045]
[0046] The preprocessed skeleton features are input into the bidirectional LSTM, and the relationship between each part of the skeleton is extracted through the bidirectional LSTM network, and finally a vector is outputted. The expression is the same as above.
[0047] 3. Feature encoding module
[0048] The preprocessed features are transmitted through the position encoding layer. According to the architecture of the transformer, the dimension of each input feature must be a multiple of the number of heads (H) used by the multi-head self-attention (MSA) layer. The preliminarily extracted bone features and RGB video features are unified in dimension through the set Vision Transformer architecture, and the specific architecture diagram is shown in Figure 2 .
[0049] The expression formula is as follows:
[0050]
[0051] z′ l = MSA(LN(z l-1 ))+z l-1 , l = 1,..., L (6)
[0052] z l = MSA(LN(z′ l ))+z′, l = 1,..., L (7)
[0053] y = LN(z L ) (8)
[0054] The first formula is the aforementioned image block embedding, class vector appending and position encoding, the second formula is the MSA part, including multi-head self-attention, skip connection (Add) and layer normalization (Norm) three parts, which can repeat L MSA blocks. The third formula is the MLP part, including feedforward network (FFN), skip connection (Add) and layer normalization (Norm) three parts, which can also repeat L MSA blocks. The fourth formula is layer normalization.
[0055] 4. Serial cross-attention module
[0056] The serial cross-attention mechanism is evolved from the cross-attention mechanism. The cross-attention mechanism refers to that for the double-flow Transfomer, if the Q (query) embedding is exchanged / transformed in a cross-flow manner, the modal interaction can be perceived. If the two cross-attention flows can be further connected in series and processed by another transformer to simulate the global context, this way is called serial cross-attention, as shown in Figure 3 .
[0057] The overall formula of the serial cross-attention mechanism is as follows:
[0058] Z (A) ← MHSA(Q B , K A , V A ) (9)
[0059] Z B ← MHSA(Q A , K B , V B ) (10)
[0060] Z s ← SUM(Z (A) , Z (B) ) (11)
[0061] Z←MHSA(Q zs ,K zs V zs (12)
[0062] Z end ←T f (Z) (13)
[0063] In the formula Z (A) This indicates that the query Q is performed by inputting B. B And input A's key and value K A V A The calculated attention output. Z B This indicates that the query Q is performed by inputting A. A And the key and value K of input B B V B The calculated attention output. Z s Z represents the fusion feature of the input. (A) and fusion feature Z (B) Perform summation and concatenation. Z represents the input Z. s Query Q zs Key K zs Value V zs The attention output is calculated. MHSA represents multi-head attention mechanism. T f SUM represents the final transformation function, where SUM denotes the concatenation between two output features.
[0064] In this invention, the concatenated cross-attention mechanism is the main component for fusing skeletal modality features and RGB modality features. Its purpose is to learn the feature relationships between modalities and achieve feature fusion. First, the details of the cross-fusion of skeletal modalities are as follows:
[0065]
[0066] MHC f =Softmax([Z f 1; Z f 2; ...; Z f N])W f (15)
[0067] Z f W represents the score for each self-attention level. f MHC represents a trainable weight vector. f This represents the final score after all attention is combined.
[0068] Secondly, the fusion details of the RGB video modalities are shown in the following formula:
[0069]
[0070] MHC m = Softmax([Z m 1; Z m 2;...; Z m N])W m (17)
[0071] 5. Series cross attention module improvement strategy
[0072] Recent studies have shown that when Q and K are initialized in the traditional way (i.e. randomly), a random permutation matrix is generated when passing through the softmax operator, as shown in Figure 4 Although this is conducive to learning complex attention maps, it can cause problems when applied to smaller-scale learning tasks. However, by initializing Q and K to form a random pulse filter convolution matrix, we can incorporate the implicit bias of CNNs without resorting to more restrictive architectures.
[0073] Normally, the weights Q and K are randomly initialized, and after SoftMax, the attention map becomes a random permutation matrix, as shown by the yellow arrow. By first establishing a convolution matrix of random pulse filters and then initializing Q and K, the initial attention map becomes a random pulse convolution filter, as shown by the purple arrow. In series cross attention, the Q and K of the two paths are cross-multiplied, so the cross-initialization strategy is used. The part where the two-path cross-attention mechanism is serialized follows the original initialization strategy.
[0074] 6. Loss function
[0075] In the present application, the features of the final modality are fused to form high-level semantic features of the aggregated RGB modality and the skeletal modality, and the final evaluation class is obtained after flattening and mapping through the Softmax layer. Therefore, the cross-entropy loss is used as the loss function for training iteration. Cross-entropy loss is a commonly used loss function in deep learning, usually used for classification problems. It measures the gap between the model's predicted results and the actual results, and is one of the key indicators for optimizing model parameters. The formula of cross-entropy loss is as follows:
[0076]
[0077] where x i represents the i-th element of the true label, and y i represents the probability that the model predicts that x belongs to the i-th class.
[0078] Embodiment
[0079] 1. Evaluation index
[0080] Accuracy and F1 Score are used as evaluation indicators in the present application. Accuracy is one of the most common classification performance indicators, representing the proportion of correctly predicted samples in the total samples. Its formula is:
[0081]
[0082] F1 Score is the harmonic mean of Precision and Recall, especially suitable for class- imbalanced datasets. Its formula is:
[0083]
[0084] where Precision represents the proportion of positive samples that are actually positive, and the formula is:
[0085]
[0086] Recall represents the proportion of positive samples that are correctly predicted as positive, and the formula is:
[0087]
[0088] where TP (True Positives) represents true positives, i.e., the number of correctly predicted positive samples. FP (False Positives) represents false positives, i.e., the number of incorrectly predicted positive samples. FN (False Negatives) represents false negatives, i.e., the number of incorrectly predicted negative samples.
[0089] 2. Experimental data
[0090] The present application uses the public NTU_RGB+D dataset, which consists of 56880 action samples, covering RGB video, depth map sequence, 3D skeleton data, and infrared video for each sample. The dataset is captured simultaneously by 3 Microsoft Kinect v.2 cameras, with RGB video resolution of 1920x1080, depth map and infrared video of 512x424, and 3D skeleton data containing the three-dimensional positions of 25 body joints per frame.
[0091] Finally, it is to be explained that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the purpose and scope of the technical solutions, and all should be covered in the scope of the claims of the present application.
Claims
1. A multimodal human motion recognition method, characterized in that: The method includes the following steps: Step 1: Extract spatial features from the video using a convolutional neural network; Step 2: Extract the temporal dependency features of video and skeletal points respectively using a bidirectional long short-term memory network; Step 3: Encode the skeletal features and video features separately using an encoder to unify the feature dimensions; Step 4: Initialize the cascaded cross-attention network according to the predetermined initialization strategy; the initialization strategy includes initializing the query Q and key K as random pulse filter convolution matrices; Step 5: Input the features encoded by the skeletal modality and video modality into a cross-attention network for feature fusion; the cascaded cross-attention network includes: The first cross-attention module is used to perform cross-attention calculation on skeletal modal features and RGB video modal features; The second cross-attention module is used to perform cross-attention calculations on RGB video modal features and skeletal modal features; A fully connected layer is used to concatenate the outputs of two cross-attention modules; The first and second cross-attention modules are initialized using a cross-initialization strategy, while the fully connected layer is initialized using the original initialization strategy. Step 6: The output fused features are passed through a fully connected layer and a Softmax function to finally output the prediction result.
2. The multimodal human motion recognition method according to claim 1, characterized in that: In step one, the convolutional neural network is a ResNet152 network.
3. The multimodal human motion recognition method according to claim 1, characterized in that: In step six, the fully connected layer and the Softmax function constitute a classifier used to map the fused features to different action categories.