Multi-modal human body action recognition method

Abstract

The invention relates to a multi-modal human body action recognition method, and belongs to the field of human body action recognition. Video space features are extracted through a ResNet152 network, time-dependent features are extracted through a Bi LSTM network, and feature coding is carried out; afterwards, feature fusion is carried out by using an improved series cross attention network, a cross attention module is initialized by adopting a random pulse filtering convolution matrix, and the inductive deviation of the CNN is introduced into the cross attention module, so that the validity of model data is improved, and the model data is easier to fit on a small-scale data set; and finally, performing action category prediction on the fused features through a full connection layer and a Softmax function. According to the method, the performance of multi-modal human body action recognition on a small-scale data set is effectively improved, the universality and universality of the method are enhanced, and the method is suitable for various scenes such as human-computer interaction, monitoring videos and gesture recognition.

Description

Technical Field

[0001] The invention belongs to the field of human motion recognition and relates to a multi-modal human motion recognition method. Background Art

[0002] Human action recognition is a current research hotspot. Human action recognition is to analyze and process the input video or 3D action data to determine which category different actions belong to. Human action recognition technology has practical application value in various industries such as human-computer interaction scenarios, surveillance video, gesture recognition, rehabilitation training, robots and behavior understanding. In recent years, with the rise of multimodal concept research, multimodal action recognition has gradually become a research hotspot. Multimodal action recognition aims to use multimodal input to better identify human activities. In multimodal action recognition, according to the visibility data modality, it can be roughly divided into two categories, namely visual modality and non-visual modality. RGB, skeleton, depth, infrared sequence, point cloud and event stream visually "intuitively" express human actions and can be regarded as visual modalities. In general, visual modes are very effective for human action recognition. Among them, RGB video data is the most common data type for human action recognition and has been widely used in surveillance systems. Skeleton data encodes the movement trajectory of human joints. When the action performed does not involve objects or scene background, skeleton data is concise and efficient for human action recognition.

[0003] The present invention considers the fusion of RGB and skeleton modalities. RGB data and skeleton sequences can greatly improve the accuracy and efficiency of fine-grained activity analysis. RGB data captures the appearance information of the activity scene, including visual features such as color and texture, while the skeleton sequence records the dynamic information of body posture and joint movement in detail. This combination not only provides rich visual and motion information, but also helps to overcome the limitations of a single modality in dealing with similar backgrounds or subtle differences in movements. The two modalities complement each other and are very useful for fine-grained activity analysis. At present, as multimodal video Transformers have shown competitive performance, this generally reflects the remarkable success of Vision Transformer (ViT) in image recognition. Although multimodal learning provides good performance, current methods tend to use a large amount of video data to learn ViT-based multimodal networks, and for small-scale data sets, the data is limited by the limitations of ViT's own attention architecture. The data is inefficient and it is difficult to achieve fitting. Therefore, in the multimodal fusion part, the present invention takes the Cross-Attention to Concatenation network as the main body and improves it to make the data more efficient, so that it is easier to fit when facing small-scale data. Recent advances in CNNs, especially ConvMixers, have made them closer and closer to ViT in architecture. Specifically, ConvMixers splits multi-channel convolution into depthwise convolution and channelwise convolution. Unlike previous CNN variants, ConvMixers uses equal-resolution representations in all layers and does not downsample the representations in consecutive layers. Although Transformer networks also use channelwise convolution, multi-head attention is used to replace the depthwise convolution step. Recent studies have shown that random filter weights can perform comparable to learned weights in ConvMixer and ResNet frameworks. The only parameters that need to be learned are those related to channelwise convolution, which are usually referred to as channel mixing in Transformer literature. This result is particularly interesting because most of the parameters in standard ConvMixer or ResNet are largely redundant and do not need to be learned to achieve effective performance. Due to the similarities between ConvMixer and ViT, this paper considers applying the unique characteristics of ConvMixer random filter weights to multimodal cross-attention networks to ensure that the same inductive bias as CNN networks exists in the network, thereby facilitating the network to be easier to fit.

[0004] In the current research on multimodal human action recognition with ViT as the main body of the multimodal network, although it shows great potential when processing large-scale data, its performance becomes very poor compared to traditional convolutional neural networks (CNNs) when faced with small-scale datasets. Although it can be initialized by pre-training the encoder of each modality, since ViT lacks the corresponding inductive bias compared to CNN, it will make it difficult for the network to learn the contextual connection between modalities from a small-scale dataset when merging modalities. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a multimodal human action recognition method. By improving the tandem cross attention network, a random pulse filter convolution matrix is ​​formed by initializing the query (Q) and key (K) in the attention network. This improvement initializes the inductive bias of CNN into the tandem cross attention network to improve the data validity of the model and make it easier to fit on small-scale data sets.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A multimodal human action recognition method, the method comprising the following steps:

[0008] Step 1: Extract video spatial features through convolutional neural network;

[0009] Step 2: Extract the time-dependent features of video and skeleton points respectively through a bidirectional long short-term memory network;

[0010] Step 3: Encode the skeleton features and video features separately through the encoder to unify the feature dimensions;

[0011] Step 4: Initialize the tandem cross attention network according to the predetermined initialization strategy;

[0012] Step 5: Input the features encoded by the skeleton modality and video modality into the cross-attention network for feature fusion;

[0013] Step 6: The output fusion features pass through the fully connected layer and Softmax function to finally output the prediction result.

[0014] Furthermore, in step one, the convolutional neural network is a ResNet152 network.

[0015] Furthermore, in step 4, the initialization strategy includes initializing the query Q and the key K as a random pulse filter convolution matrix.

[0016] Further, in step 5, the tandem cross attention network includes:

[0017] The first cross attention module is used to perform cross attention calculation on the skeleton modality features and the RGB video modality features;

[0018] The second cross attention module is used to perform cross attention calculation on the RGB video modality features and the skeleton modality features;

[0019] A fully connected layer is used to concatenate the outputs of the two cross-attention modules.

[0020] Furthermore, the first cross-attention module and the second cross-attention module are initialized using a cross initialization strategy, and the fully connected layer is initialized using an original initialization strategy.

[0021] Furthermore, in step six, the fully connected layer and the Softmax function constitute a classifier for mapping the fused features to different action categories.

[0022] The beneficial effect of the present invention is that compared with the prior art, the present invention improves the serial cross attention network, and forms a random pulse filter convolution matrix by initializing the query (Q) and key (K) in the attention network. This improvement initializes the inductive bias of CNN into the serial cross attention network to improve the data validity of the model and make it easier to fit on small-scale data sets. Specifically, the introduction of the random pulse filter convolution matrix increases the randomness of the attention mechanism, enabling it to better capture subtle features and patterns in the data set. This improvement not only takes into account the fitting ability of the model, but also takes into account the generalization ability of the model, because the introduction of a certain degree of randomness helps the model adapt to various data distributions and noise conditions. Therefore, this improved method can improve the performance of the model on small-scale data sets and enhance its universality and versatility.

[0023] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0025] Figure 1 The overall framework diagram of multimodal human action recognition;

[0026] Figure 2 This is the Vision Transformer encoder architecture block diagram;

[0027] Figure 3 A block diagram of a tandem cross attention network;

[0028] Figure 4 This is a schematic diagram of the initialization strategy. DETAILED DESCRIPTION

[0029] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0030] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0031] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0032] See also Figure 1 , the method comprises the following steps:

[0033] Step 1: Extract video spatial features through ResNet152.

[0034] Step 2: Extract the time-dependent features of video and skeleton points respectively through Bi LSTM.

[0035] Step 3: Encode the skeleton features and RGB video features separately through the encoder to unify the feature dimensions.

[0036] Step 4: Initialize the tandem cross attention network according to the predetermined initialization strategy.

[0037] Step 5: Input the features encoded by the skeleton modality and RGB modality into the cross-attention network for feature fusion.

[0038] Step 6: The output fusion features pass through the fully connected layer and Softmax function to finally output the prediction result.

[0039] 1. RGB video feature extraction module

[0040] The present invention uses the ResNet152 network to extract the spatial features of RGB video data. In order to extract the temporal features more efficiently, a bidirectional LSTM network is used to extract the temporal features. The RGB image features extracted by ResNet152 are input into the LSTM in two directions. The relationship between each frame and the previous and next frames is extracted through the LSTM network in two directions, and the feature vector is output. The expression is:

[0041]

[0042] Where: LSTM fw (·) is the calculation formula of the forward LSTM, LSTM bw (·) is the calculation formula of reverse LSTM; t represents the tth frame, i represents the i-th sample, h represents the hidden state vector after the LSTM network, X represents the spatial feature extracted by the ResNet network, and f i,t The spatial and temporal characteristics of the fusion.

[0043] 2. Skeleton feature extraction module

[0044] The bone data is mainly composed of bone points, so it is necessary to first normalize the bone features using equation (2.1) and then flatten them to obtain the spatial features of the skeleton.

[0045]

[0046] The preprocessed skeleton features are input into the bidirectional LSTM, and the front-back relationship of each part of the skeleton is extracted through the bidirectional LSTM network, and finally the vector is output. The expression is the same as above.

[0047] 3. Feature encoding module

[0048] The preprocessed features are transmitted through the position encoding layer. According to the architecture of the transformer, each input feature dimension must be a multiple of the number of heads (H) used by the multi-head self-attention (MSA) layer. The initially extracted skeleton features and RGB video features are unified in dimension through the set Vision Transformer framework. The specific architecture diagram is shown in the figure. Figure 2 shown.

[0049] The expression formula is as follows:

[0050]

[0051] z′ l =MSA(LN(z l-1 ))+z l-1 ,l=1,...,L (6)

[0052] z l =MSA(LN(z′ l ))+z′,l=1,...,L (7)

[0053] y=LN(z L ) (8)

[0054] The first formula is the aforementioned image block embedding, category vector appending and position encoding. The second formula is the MSA part, including multi-head self-attention, jump connection (Add) and layer normalization (Norm). L MSA blocks can be repeated. The third formula is the MLP part, including feedforward network (FFN), jump connection (Add) and layer normalization (Norm). L MSA blocks can also be repeated. The fourth formula is layer normalization.

[0055] 4. Chaining Cross-Attention Modules

[0056] The tandem cross attention mechanism is an evolution of the cross attention mechanism. The cross attention mechanism refers to the fact that for a two-stream transformer, if the Q (query) embedding is exchanged / adjusted in a cross-stream manner, the modal interaction can be perceived. If the two cross attention streams can be further connected in series and processed by another transformer to simulate the global context, this method is called tandem cross attention, such as Figure 3 shown.

[0057] The overall formula of the cascaded cross attention mechanism is as follows:

[0058] Z (A) ←MHSA(Q B ,K A ,V A ) (9)

[0059] Z B ←MHSA(Q A ,K B ,V B ) (10)

[0060] Z s ←SUM(Z (A) ,Z (B) ) (11)

[0061] Z←MHSA(Q zs ,K zs ,V zs ) (12)

[0062] Z end ←T f (Z) (13)

[0063] Where Z (A) Represents the query Q through input B B and the key and value K of input A A ,V A The calculated attention output. Z B Represents the query Q through input A A and the key and value K of input B B ,V B The calculated attention output. Z s Represents the fusion feature Z of the input (A) And the fusion feature Z (B) Perform summation and concatenation. Z means by inputting Z s Query Q zs , key K zs , value V zs The attention output is calculated. MHSA stands for multi-head attention mechanism. f is the final transformation function, and SUM represents the concatenation between the two output features.

[0064] The serial cross attention mechanism in this invention is the main body of the fusion of skeleton modality features and RGB modality features, the purpose is to learn the feature relationship between the modalities and realize feature fusion. First, the cross fusion details of the skeleton modality are as follows:

[0065]

[0066] MHC f =Softmax([Z f 1; Z f 2;...;Z f N])W f (15)

[0067] Where Z f Represents the score of each self-attention, W f Represents a trainable weight vector, MHC f Represents the final score after all attentions are combined.

[0068] Next is the fusion details of RGB video modality. The formula is as follows:

[0069]

[0070] MHC m =Softmax([Z m 1; Z m 2;...;Z m N])W m (17)

[0071] 5. Improvement strategy of serial cross attention module

[0072] Recent studies have shown that when Q and K are initialized in the traditional way (i.e. randomly), a random permutation matrix is ​​produced when passing through the softmax operator, such as Figure 4 As shown in , while this is good for learning complex attention maps, it becomes problematic when applied to smaller-scale learning tasks. However, by initializing Q and K to form a random spike filter convolution matrix, we can incorporate the implicit bias of the CNN without adopting a more restrictive architecture.

[0073] Normally, weights Q and K are randomly initialized, and after SoftMax, the attention map becomes a randomly permuted matrix, as shown by the yellow arrow. First, the convolution matrix of the random pulse filter is established, and then Q and K are initialized, so that the initial attention map is a random pulse convolution filter, as shown by the purple arrow. In the series cross attention, the Q and K of the two paths are cross-multiplied, so the cross initialization strategy is used. The part where the two-way cross attention mechanism is connected in series follows the original initialization strategy.

[0074] 6. Loss Function

[0075] In the present invention, the features of the final modality are fused to form high-level semantic features of the aggregated RGB modality and the skeleton modality, and the final evaluation category is obtained after flattening and Softmax layer mapping. Therefore, the cross-entropy loss is used as the loss function for training iterations. Cross-entropy loss is a commonly used loss function in deep learning, usually used for classification problems. It measures the gap between the model prediction results and the actual results, and is one of the key indicators for optimizing model parameters. The formula for cross-entropy loss is as follows:

[0076]

[0077] Among them, x i Represents the i-th element of the true label, y i It represents the probability that the model predicts that x belongs to the i-th category.

[0078] Example

[0079] 1. Evaluation indicators

[0080] In this invention, accuracy and F1 score are used as evaluation indicators. Accuracy is one of the most common classification performance indicators, indicating the proportion of samples correctly predicted by the model to the total samples. Its formula is:

[0081]

[0082] The F1 score is the harmonic mean of precision and recall, and is particularly suitable for datasets with imbalanced categories. The formula is:

[0083]

[0084] Among them, precision represents the proportion of samples predicted to be positive that are actually positive, and the formula is:

[0085]

[0086] Recall refers to the proportion of samples that are actually positive that are correctly predicted as positive. The formula is:

[0087]

[0088] TP (True Positives) represents true positive examples, that is, the number of positive examples correctly predicted by the model. FP (False Positives) represents false positive examples, that is, the number of positive examples incorrectly predicted by the model. FN (False Negatives) represents false negative examples, that is, the number of negative examples incorrectly predicted by the model.

[0089] 2. Experimental Data

[0090] The present invention uses the public NTU_RGB+D dataset, which consists of 56,880 action samples, including RGB video, depth map sequence, 3D skeleton data, infrared video, etc. of each sample. The dataset is captured by 3 Microsoft Kinect v.2 cameras at the same time, with a resolution of 1920×1080 for RGB video, 512×424 for depth map and infrared video, and 3D skeleton data containing the 3D positions of 25 body joints per frame.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A multimodal human action recognition method, characterized in that: The method comprises the following steps: Step 1: Extract video spatial features through convolutional neural network; Step 2: Extract the time-dependent features of video and skeleton points respectively through a bidirectional long short-term memory network; Step 3: Encode the skeleton features and video features separately through the encoder to unify the feature dimensions; Step 4: Initialize the tandem cross attention network according to the predetermined initialization strategy; Step 5: Input the features encoded by the skeleton modality and video modality into the cross-attention network for feature fusion; Step 6: The output fusion features pass through the fully connected layer and Softmax function to finally output the prediction result.

2. A multimodal human motion recognition method according to claim 1, characterized in that: In step 1, the convolutional neural network is a ResNet152 network.

3. A multimodal human motion recognition method according to claim 1, characterized in that: In the step 4, the initialization strategy includes initializing the query Q and the key K as a random pulse filter convolution matrix.

4. The multimodal human motion recognition method according to claim 1, characterized in that: In step 5, the tandem cross attention network includes: The first cross attention module is used to perform cross attention calculation on the skeleton modality features and the RGB video modality features; The second cross attention module is used to perform cross attention calculation on the RGB video modality features and the skeleton modality features; A fully connected layer is used to concatenate the outputs of the two cross-attention modules.

5. A multimodal human motion recognition method according to claim 4, characterized in that: The first cross-attention module and the second cross-attention module are initialized using a cross initialization strategy, and the fully connected layer is initialized using the original initialization strategy.

6. A multimodal human motion recognition method according to claim 1, characterized in that: In step six, the fully connected layer and the Softmax function form a classifier for mapping the fused features to different action categories.

Images