A micro gesture recognition method based on multi-modal feature fusion and fine-tuning

Abstract

The application discloses a micro gesture recognition method based on multi-modal feature fusion and fine-tuning, and relates to the field of computer vision and action recognition. The application provides a general cross-modal knowledge fusion framework, which extracts multi-modal features by fine-tuning networks using video, skeleton and text modal information. Meanwhile, video-skeleton and text-skeleton fusion modules are introduced to enhance the interactivity between modalities. Contrastive learning is used to align feature distribution, and a freeze-fine-tuning strategy is combined to optimize model training, thereby reducing computational complexity and improving recognition efficiency. The method provided by the application can make up for the problem of single-modal information loss, enhance the perception ability of small gesture changes, improve the recognition accuracy and robustness, and is suitable for multiple application scenarios such as behavior monitoring, human-computer interaction and safety protection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and action recognition, and particularly relates to a micro gesture recognition method based on multi-modal feature fusion and fine-tuning. BACKGROUND

[0002] Action recognition is a key research direction in the field of computer vision, which aims to automatically recognize and analyze human gestures and actions through computer technology to infer the state and intention of the human body. By analyzing the overall posture of the human body, the position of the body joints and the action, more comprehensive information is obtained, the privacy of people is protected, and a more comprehensive and accurate emotion analysis method is provided for people. Action recognition is commonly used in action capture, human-computer interaction, virtual reality and other fields.

[0003] In the field of action recognition, most people are committed to the recognition of descriptive gestures. "Descriptive gestures" mainly refer to purposeful and more significant body movements, such as drinking water, running, etc. People can clearly express their emotions and attitudes through these actions. In some specific situations, such as interviews and competitions, people may deliberately hide or suppress their true feelings, which is not conducive to further analysis by computers. Therefore, micro gesture recognition is proposed. "Micro gestures" refer to spontaneous and unconscious subtle movements that are difficult to detect, but can reflect the true emotional state of individuals, especially tension, stress, etc., and can detect abnormal psychological states such as Alzheimer's disease and autism. It has important significance and research prospects in psychology, behavior analysis and communication research.

[0004] Due to the unique complexity and subtlety of micro gesture data, traditional deep learning methods based on single modalities are not capable of capturing the detailed information of these small movements. Single modal methods usually rely only on video frames or skeleton data, which is difficult to fully depict the hidden emotions and semantic information in micro gestures. Especially in the unconscious movements and ambiguous features contained in micro gestures, a single data source often lacks sufficient context and multi-dimensional support, resulting in limited performance of the model. In recent years, with the rapid development of visual-linguistic models, multi-modal methods have gradually become an important means of studying complex behavior patterns. Visual-linguistic models combine video data with text descriptions, not only making up for the shortcomings of single modal methods, but also giving the model stronger semantic understanding ability. Therefore, the present application proposes a micro gesture recognition method based on multi-modal fusion and fine-tuning, which further introduces skeleton information on the basis of visual-text, forming a multi-modal fusion method of video-skeleton-text. SUMMARY

[0005] To solve the above problems, the present application proposes a micro gesture recognition method based on multi-modal feature fusion and fine-tuning, proposes a general cross-membrane knowledge fusion framework, extracts the features of video, skeleton and text through fine-tuning network respectively, and performs late feature fusion and interaction, so as to make up for the missing information of single mode, and capture more fine-grained micro gesture details.

[0006] The technical scheme adopted by the present application is:

[0007] A micro gesture recognition method based on multi-modal feature fusion and fine-tuning, comprising the following steps:

[0008] Step 1, sampling and preprocessing multi-modal data.

[0009] Step 2, constructing a micro gesture recognition model, the micro gesture recognition model comprising a multi-modal encoder and a cross-membrane fusion module.

[0010] Step 3, constructing a loss function and training a fine-tuning model.

[0011] Step 4, the current human micro gesture behavior obtained by the camera is output to the trained model to obtain the micro gesture category to which it belongs.

[0012] Further, the specific process of step 1 is:

[0013] For video data, uniform frame sampling is performed on the original video, and the image is compressed, cropped and flipped 1:1.

[0014] For skeleton data, the skeleton joint data is obtained by the OpenPose pose estimation algorithm, the joint coordinates are normalized and centralized, and the time dimension is aligned to ensure the same frame rate as the video. For text data, different category names of actions are used as semantic text library, and the library is set to the format of CLIP text encoder pre-input.

[0015] Further, the specific process of step 2 is:

[0016] The multi-modal encoder includes a CLIP-based video encoder, a CLIP-based text encoder and a CTR-GCN-based skeleton encoder.

[0017] The video encoder of CLIP is a visual Transformer architecture with strong feature extraction capability. The text encoder of CLIP is a Transformer model that can capture the relationship between words through self-attention mechanism. CTR-GCN is a graph convolution model that refines the modeling of the skeleton by performing feature extraction and aggregation on each channel through convolution network, and improves the recognition performance.

[0018] The transmembrane fusion module includes a video-skeleton fusion module and a text-skeleton fusion module.

[0019] The video-skeleton fusion module is based on a single-layer Transformer architecture. For the video features encoded by the video encoder, the global class token Cls∈R T×C×1 where T represents time, and C represents the number of channels output by the video encoder. For the skeleton features encoded by the skeleton encoder, the features F where T represents time, C2 represents the number of channels output by the graph convolutional network, and N represents the dimension of the joint node. The skeleton features are linearly mapped in the channel dimension, the number of channels is unified, and the global class token Cls is concatenated with the skeleton features F P The third dimension is spliced to obtain the fusion features F1∈R T×C×(1+N) The fusion features are input into the Transformer encoder to encode the fusion features.

[0020] The text-skeleton fusion module is based on the K most likely classes in the skeleton classification results. For the skeleton features encoded by the skeleton encoder, the skeleton features F The features F P The joint node dimension N is averaged to obtain The pooled features are input into a fully connected layer to map to the class space to obtain where N2 represents the number of action classes, a softmax function is applied to the output of the fully connected layer to obtain the probability distribution of each class, and the K classes with the highest probability are selected as the most likely classes. According to the predicted K classes, the embedding features of the corresponding classes are loaded from the text modality and the K-class index is obtained These text features are obtained through a CLIP-based text encoder and can provide accurate semantic information related to the class. Then, F T The features are replicated in the time dimension and averaged along the K-class dimension to obtain the features This operation helps to integrate the information of the text and skeleton modalities into a single feature space.

[0021] Next, the skeleton features F P and the text features F T are merged through a concatenation operation to form the fusion features The fused features F2 are further extracted through a one-dimensional convolutional layer to obtain time-related feature representations, and the enhanced skeleton features are generated through a fully connected layer to obtain a classification probability distribution. The final prediction result is obtained by weighted fusion with the original skeleton classification result.

[0022] Further, the specific process of step 3 is:

[0023] For the result F1 of the fusion of the video modality and the skeleton modality, the features of F1 and the text are aligned through contrast learning. First, the features of F1 and the text F T The features are normalized, and then a similarity matrix is calculated The features of F1 and the text are aligned through a symmetric contrast loss function. For the result F2 of the fusion of the text modality and the skeleton modality, the predicted class is obtained through a fully connected layer, and a cross-entropy loss is used for training. The total loss of the model is the weighted sum of the two losses.

[0024] During the training of the model, a freeze-fine-tuning method is adopted. Specifically, for the video encoder and the text encoder, the parameters of the Transformer part are frozen, and an adapter module is added after each attention layer. For the video encoder, the adapter added is a space-time adapter. For the text encoder, since the text information does not have the features of time and space dimensions, the adapter added is based on a multi-layer perceptron. The adapter part in the video encoder and the text encoder is frozen, and the skeleton encoder is fine-tuned, and the remaining Transformer part is frozen, which reduces the computational complexity of the model.

[0025] Further, the specific process of step 4 is as follows:

[0026] For the video data to be processed obtained by the monitoring camera, video data and skeleton data are obtained through preprocessing, and text data are respectively processed through a video encoder, a skeleton encoder and a text encoder to obtain video features, skeleton features and text features. The video and skeleton features are fused through a fusion module to obtain fusion feature 1, and the fusion feature 1 and the text features are subjected to similarity calculation to obtain the micro gesture class with the highest similarity. The skeleton and text features are fused through a fusion module to obtain fusion feature 2, and the fusion feature 2 is classified through a fully connected layer to obtain the predicted micro gesture class. The two prediction results are added to obtain the final prediction result. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 FIG. 1 is a model diagram of the micro gesture recognition method based on multi-modal feature fusion and fine-tuning of the present application. DETAILED DESCRIPTION

[0028] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0029] The application provides a micro gesture recognition method based on multi-modal feature fusion and fine-tuning, which extracts features through multi-modal encoders of video, skeleton and text data, and realizes feature fusion through a cross-modal fusion module; the model is trained by contrast learning and cross-entropy loss, and the calculation complexity is reduced by freezing-fine-tuning strategy and adapter module; finally, the trained model is used to classify the human micro gesture behavior obtained by the camera, and the micro gesture category is output. The method improves the performance of micro gesture recognition through multi-modal feature fusion and efficient fine-tuning strategy.

[0030] As shown in Figure 1 A micro gesture recognition method based on multi-modal feature fusion and fine-tuning, specifically comprising the following steps:

[0031] Step 1, sample and preprocess multi-modal data. The specific process is as follows:

[0032] Step 1.1, select video frames according to fixed time interval for original video data, ensure that all video samples have consistent frame rate. Then, perform image compression on the sampled image, scale the image by 1:1; image cropping, center crop the image to remove excess background area; image flipping, horizontally flip the image.

[0033] Step 1.2, use OpenPose pose estimation algorithm to detect skeleton key points for original video data. First, input the video frame into the convolutional neural network to extract features, and after processing by the feedforward convolutional network, divide the feature map into two branches. Branch one is processed by the convolutional network to predict the key point confidence map, and each key point has an independent confidence heat map; branch two is processed by a set of two-dimensional vector fields to predict the limb connection vector field, which represents the connection relationship between key points. Finally, through the greedy matching algorithm, the optimal key point connection is calculated according to the limb connection vector field, and the skeleton information of the human body is parsed.

[0034] Step 1.3, organize the micro gesture category text, convert the text into the format required by CLIP pre-training, such as "a person is {action name}", to enhance the generality and generalization ability of the text description.

[0035] Step 2, build a micro gesture recognition model, which includes multi-modal encoders and cross-modal fusion modules. The specific process is as follows:

[0036] Step 2.1, for the input video, the video frames are input into the CLIP-based visual Transformer with adapter, the video frames are first cut into multiple fixed-size image blocks, each small image block has a size of 16x16, each small image block is mapped to a high-dimensional feature space through a linear projection layer, and position encoding is added to maintain spatial information, then the small image blocks are calculated through a 12-layer Transformer, each layer contains multi-head attention and feedforward neural network to capture global context information. The encoding output includes multiple tokens, among which the global class token Cls∈R T×C×1 represents the global feature of the whole frame, where T is the number of frames and C is the feature dimension.

[0037] Step 2.2, for the input text, the text vector is input into the CLIP-based Transformer with adapter. Specifically, the input text is first tokenized into a series of discrete tokens, and then mapped to a 512-dimensional vector space through a word embedding layer, then the embedded text is processed through the Transformer, and position encoding is added to each token to maintain sequence information. After encoding, the output of the text sequence includes multiple token-level features, among which the end token is used to extract the global semantic representation of the entire text to generate the final text-level feature vector where N2 is the number of categories and C2 is the feature dimension.

[0038] Step 2.3, for the input skeleton, the skeleton data is input into the CTR-GCN network for encoding. CTR-GCN adjusts the connection relationship between the joints through a graph topology learning mechanism, and processes the features of each joint through a channel attention mechanism to give different weights in different channels. The input skeleton is calculated through a multi-layer graph convolutional network to obtain the skeleton feature representation where T is the number of frames, C2 is the feature dimension, and N is the number of joints.

[0039] Step 2.4, in the video-skeleton fusion module, the processed video feature representation is Cls∈R T×C×1 , where T is the time frame and C is the video feature dimension. The processed skeleton feature representation is where T is the number of frames, C2 is the feature dimension, and N is the number of joints.

[0040] In order to fuse the skeleton features and video features, the channel alignment is performed on the skeleton features, and the skeleton features are mapped to the same dimension C as the video features through linear mapping:

[0041] F aligned =Linear(F P )∈R T×C×N (1)

[0042] The video feature Cls is aligned with the skeleton feature F aligned In the third dimension, the fusion feature is obtained by splicing:

[0043] F1 = [Cls, F aligned ] ∈ R T×C×(N+1) (2)

[0044] The fusion feature F1 is sent to a single-layer Transformer for further modeling, and the relationship between the video and the skeleton is captured through a multi-head self-attention mechanism. The modeled fusion feature is compared with the text feature to obtain the most likely micro gesture category.

[0045] Step 2.5, in the text-skeleton fusion module, the most likely K categories are selected through the skeleton classification result, and the text feature and the skeleton feature are fused. For the encoded skeleton feature, first average pooling is performed in the joint dimension:

[0046]

[0047] Then it is sent to a fully connected layer to map to the category space to obtain the category prediction result:

[0048] P = Softmax (FC (F pool )) (4)

[0049] The K categories with the highest probability are selected as candidate categories. For the encoded text feature, the embedding features of the K categories are obtained from the text modality, and are copied in the time dimension and averaged in the category dimension to obtain the fused text feature The skeleton feature before pooling and the fused text feature are combined through splicing operation:

[0050]

[0051] The fusion feature is sent to a convolution layer to extract timing information, and finally a fully connected layer is used to generate the final classification probability distribution.

[0052] Finally, through a weighted fusion strategy, the results predicted by the two fusion modules are added to improve the performance of micro gesture recognition.

[0053] Step 3, construct the loss function and train the fine-tuned model. The specific process is as follows:

[0054] Step 3.1, for the video-skeleton fusion feature and the text feature, a similarity loss is constructed through contrastive learning. First, the fusion feature F1 and the text feature F T are normalized. Then F1 and F TCosine similarity between them:

[0055]

[0056] Then, the alignment of fusion features and text features is performed through contrastive learning, using a symmetric contrastive loss function:

[0057] Loss of video-skeleton fusion modality to text modality:

[0058]

[0059] Loss of text modality to video-skeleton fusion modality:

[0060]

[0061] where t is the temperature parameter and B is the batch size. The two are added together:

[0062]

[0063] This loss function promotes the alignment between modalities by maximizing the similarity between fusion features and text and minimizing the difference between them.

[0064] Step 3.2, for the result F2 of the text modality and skeleton modality fusion, the predicted class is obtained through a fully connected layer, and the cross-entropy loss is used for training:

[0065]

[0066] where p[i][labels[i]] is the predicted probability corresponding to the true class of the i-th sample.

[0067] The total loss of the model is:

[0068] L = L1 + a x L2 (11)

[0069] where a is the weight coefficient of the loss function.

[0070] Step 3.3, during the model training process, a freeze-fine-tuning strategy is adopted to optimize the training efficiency and reduce the computational complexity. In the video encoder and text encoder, all parameters of the Transformer part are frozen, keeping their pre-training state unchanged, avoiding updating these parameters during training. The role of freezing the Transformer is to maintain its strong feature extraction capability. After the frozen Transformer part, an adapter module is added to enhance the model's ability. The purpose of the adapter module is to increase a small number of parameters for fine-tuning while keeping most of the model parameters unchanged.

[0071] For the video encoder, a space-time adapter is added. First, the dimension of the input feature is halved by the dimension reduction convolution; the convolution operation is performed in the time dimension using the time domain convolution, and the convolution kernel size of (3, 1) is adopted to capture the timing information; finally, the dimension of the feature is restored by the dimension increasing convolution. The output of the adapter is added to the original feature to obtain the enhanced feature.

[0072] Since the text information does not have the characteristics of time and space dimensions, the text adapter is built based on a multi-layer perception mechanism. First, the text feature is input into the multi-layer perception mechanism to obtain the enhanced feature. Then, through the adapter module, including two linear layers, first reduced to 1 / 4 of the original dimension, then enhanced the non-linear ability through the GELU activation function, and finally restored to the original dimension.

[0073] During the training process, only the parameters of the adapter part are updated, and the parameters of the Transformer part are frozen. This approach can reduce the number of model parameters during training, reduce the computational complexity, and effectively fine-tune the adapter module to adapt to specific tasks.

[0074] Step 4, the current human micro-gesture behavior obtained by the camera is output to the trained model to obtain the micro-gesture category it belongs to.

[0075] The present application makes full use of the multi-modal information of video, skeleton and text, extracts the features of each modality through fine-tuning the network, and performs feature fusion and interaction in the subsequent stage, so as to effectively make up for the lack of single modal information, improve the accuracy and robustness of micro-gesture recognition. In addition, the cross-modal knowledge fusion framework proposed by the present application can more deeply capture the fine-grained features of micro-gesture, improve the perception ability of the system to subtle actions, and still maintain high recognition performance in complex environments. The present application has strong adaptability and generalization ability, and is suitable for behavior recognition, security monitoring, human-computer interaction and other application scenarios, and has high practical application value.

Claims

1. A micro-pose recognition method based on multimodal feature fusion and fine-tuning, comprising the following steps: Step 1: Sampling and preprocessing the multimodal data; Step 2: Construct a micro-pose recognition model, which includes a multimodal encoder and a cross-membrane fusion module; The specific process of step 2 is as follows: Step 2.1: Construct the branch network of the multimodal encoder; A video encoder based on CLIP's visual Transformer, a text encoder based on CLIP's Transformer, and a skeleton encoder based on CTR-GCN; Step 2.2: Construct the video-skeleton fusion module; The video-skeleton fusion module adopts a single-layer Transformer architecture. For video features obtained by the video encoder, global tokens are extracted. The token dimension is (T, C), where T represents the time step and C is the number of channels output by the video encoder. For skeleton features processed by the skeleton encoder, the feature dimension is (T, C', N), where T represents the time step, C' is the number of channels output by the graph convolutional network, and N is the dimension of the joints. The skeleton features are adjusted in the channel dimension through linear mapping to ensure that the number of channels of the video features and skeleton features are consistent. The global tokens in the video features and the skeleton features after linear mapping are concatenated along the third dimension to form fused features. The fused features are input into the Transformer encoder for further encoding to generate the final fused feature representation. Step 2.3: Construct the text-skeleton fusion module; For the skeleton features obtained by the skeleton encoder, average pooling is performed along the keypoint dimension. The pooled features are then mapped to the category space through a fully connected layer. The Softmax function is applied to the output of the fully connected layer to calculate the probability distribution of each category. The K categories with the highest predicted probabilities are used as indices. The text features of the text encoder are used as queries to retrieve the text features of the K categories through the indices. The retrieved text features are copied along the time dimension and averaged along the K category dimensions to obtain the enhanced text feature representation. The skeleton features before pooling and the enhanced text features are merged through a concatenation operation to form a fused feature. The fused feature is then passed through a one-dimensional convolutional layer and a fully connected layer to generate new classification probability distributions, which are then weighted and fused with the original skeleton classification results to obtain the final prediction result. Step 3: Construct the loss function and train and fine-tune the model; Step 4: The current human micro-pose behavior acquired through the camera is output to the trained model to obtain the corresponding micro-pose category.

2. The micro-pose recognition method based on multimodal feature fusion and fine-tuning according to claim 1, characterized in that, Step 1 processes the data of video, skeleton, and text modalities respectively. The video data is sampled to ensure that all video samples have a consistent frame rate. The images are compressed, cropped, and flipped. The OpenPose pose estimation algorithm is used to detect skeleton key points in the processed images. Features are extracted through a convolutional neural network to parse human skeleton information. The micro-pose category text is formatted.

3. The micro-pose recognition method based on multimodal feature fusion and fine-tuning according to claim 1, characterized in that, In step 3, for the fused feature F of the video modality and skeleton modality, a contrastive learning method is used to align F with the encoded text features. The similarity matrix between the normalized fused feature F and the text features is calculated, and a symmetric contrastive loss function is constructed. The symmetric contrastive loss function consists of the contrastive loss from the fused feature F to the text features and the contrastive loss from the text features to the fused feature F. The two are added together to obtain the contrastive learning loss. For the fused feature F' of the text modality and skeleton modality, a loss function is constructed using cross-entropy classification loss. The contrastive loss and cross-entropy loss are weighted and summed to obtain the total loss during the training process.

4. The micro-pose recognition method based on multimodal feature fusion and fine-tuning according to claim 1, characterized in that, In step 3, a freeze-fine-tuning strategy is adopted. In the video encoder and text encoder, all parameters of the Transformer part are frozen to keep their pre-trained state unchanged. An adapter module is added after the frozen Transformer part and fine-tuned with a small number of parameters. The video adapter is based on the spatiotemporal adapter. The dimension of the input features is halved by dimensionality reduction convolution. Temporal convolution operation is used to capture temporal information, and the feature dimension is restored by dimensionality increase convolution. The adapter output is added to the original video features to obtain the enhanced video features. The text adapter uses an adapter module based on a multilayer perceptron. The feature dimension is reduced to 1 / 4 of the original dimension through two linear layers. Then, the nonlinearity is enhanced by the GELU activation function. Finally, the original dimension is restored. During training, the model only fine-tunes the skeleton encoder, video adapter, and text adapter.

Images