Large-model-driven spatial-temporal feature and text enhancement few-sample motion capture method

Abstract

The invention discloses a spatial-temporal feature and text enhancement few-sample motion capture method driven by a large model, which belongs to the technical field of motion capture, is used for video motion capture, and comprises the following steps: acquiring and preprocessing video data, the video data comprises query video data to be subjected to motion capture and support set video data with motion tags, and the query video data is subjected to motion capture; inputting the preprocessed video data into a visual encoder to obtain visual features of the video data; and integrating the two category probability distributions to obtain a motion capture result of the queried video data. According to the method, efficient spatio-temporal feature extraction is realized through the time sequence enhancement adapter and the spatio-temporal fusion adapter, and the spatio-temporal modeling capability of video features is enhanced; a multi-level attention mechanism is utilized, the fusion ability of text and video features is improved, and a class prototype with strong generalization ability is constructed; the capture accuracy of the model is remarkably improved in few sample learning tasks, the number of trainable parameters is small, and the calculation cost is reduced.

Description

Technical Field

[0001] The invention discloses a large model driven spatiotemporal feature and text enhanced few-sample motion capture method, belonging to the technical field of motion capture. Background Art

[0002] In the field of motion capture, few-shot learning is of great significance when dealing with limited annotated data of new categories. Traditional methods usually rely on large-scale data for training, resulting in high computational costs and limited generalization capabilities. Existing technologies still face challenges in efficiently extracting spatiotemporal features of videos and constructing class prototypes with strong generalization capabilities. Existing video motion capture methods usually adopt a two-stage strategy. The first is the feature extraction stage, which uses pre-trained convolutional neural networks to extract spatial features of video frames; the second is the temporal modeling stage, which models sequence information through recurrent neural networks or long short-term memory networks. Although these methods have achieved good results on standard datasets, they often perform poorly in few-shot scenarios due to data size limitations. In addition, the training of these methods usually requires a lot of computing resources and has significant time complexity problems when processing long videos. Some methods attempt to combine meta-learning and prototype learning to enhance the generalization ability of the model by building class prototypes. These methods use a small number of samples to learn feature prototypes that can represent the entire category, thereby improving the ability to capture motion on new categories. However, when building class prototypes, existing methods usually ignore the fusion of multimodal information, especially the potential value of text information. Text information, as a rich semantic clue, can provide powerful assistance for video motion capture. Summary of the invention

[0003] The purpose of the present invention is to provide a large model driven spatiotemporal feature and text enhanced few-sample motion capture method to solve the problem in the prior art that the motion capture method does not integrate text information resulting in insufficient capture accuracy.

[0004] A large model driven spatiotemporal feature and text enhanced few-shot motion capture method, comprising obtaining and preprocessing video data, wherein the video data comprises query video data to be motion captured and support set video data with action labels, and inputting the preprocessed video data into a visual encoder to obtain visual features of the video data;

[0005] Input the action labels of the support set into the CLIP text encoder to obtain the text features of the action labels of the support set, perform similarity calculation on the text features and the visual features of the query video data, and obtain the first category probability distribution; input the visual features and text features of the support set video data into the prototype construction module to obtain the prototype features of the support set video data, input the visual features of the query video data into the prototype construction module to obtain the prototype features of the query video data, perform time series similarity calculation on the prototype features of the support set video data and the prototype features of the query video data, and obtain the second category probability distribution;

[0006] Combining the two category probability distributions, the motion capture results of the query video data are obtained.

[0007] Obtaining the visual features of video data includes designing a temporal enhancement adapter and a spatiotemporal fusion adapter, and alternately embedding the two adapters into the CLIP-ViT visual encoder.

[0008] The CLIP-ViT visual encoder consists of N VIT Block layers. In the even-numbered layers, the temporal enhancement adapter is embedded before and after the multi-head attention layer. In the odd-numbered layers, the spatiotemporal fusion adapter is embedded only after the multi-head attention layer. VITBlock is the visual block.

[0009] The even-numbered layers sequentially send input information to the TE Adapter layer, the Layer Norm layer, the Multi-HeadAttention layer, the feature fusion layer, the TE Adapter layer, the Layer Norm layer, the MLP layer, the feature fusion layer, and then output; the output of the first TE Adapter layer is provided with a shortcut connection to the first feature fusion layer, and the output of the second TE Adapter layer is provided with a shortcut connection to the second feature fusion layer;

[0010] The TE Adapter layer includes an FC Down layer, a Gelu layer, an FC Up layer and a feature fusion layer in sequence, and the input of the TE Adapter layer is provided with a shortcut connection to the feature fusion layer;

[0011] The TE Adapter is a timing enhancement adapter, Layer Norm is layer normalization, Multi-HeadAttention is multi-head attention, MLP is a multi-layer perceptron, FC Down is a fully connected downsampling, FC Up is a fully connected upsampling, and Gelu is an activation function.

[0012] The odd-numbered layers send the input information to the Layer Norm layer, the Multi-Head Attention layer, the feature fusion layer, the STF Adapter layer, the Layer Norm layer, the MLP layer, the feature fusion layer in sequence, and then output it; the input of the spatiotemporal fusion adapter is provided with a shortcut connection to the first feature fusion layer, and the output of the STF Adapter layer is provided with a shortcut connection to the second feature fusion layer;

[0013] The STF Adapter layer includes an FC Down layer, a Max-Pool layer, a 3D-Conv layer, a Sigmiod layer, a neuron layer, an FC Up layer and a feature fusion layer in sequence. The input of the STF Adapter layer is provided with a quick connection connected to the feature fusion layer, and the output of the FCDown layer is provided with a quick connection connected to the neuron layer.

[0014] The STF Adapter is a spatiotemporal fusion adapter, Max-Pool is maximum pooling, 3D-Conv is a three-dimensional convolution, and Sigmiod is an activation function.

[0015] The similarity calculation includes performing a maximum pooling operation on the visual features of the query video data, and then performing a cosine similarity calculation on the visual features and text features of the query video data to obtain a first category probability distribution.

[0016] The prototype construction module includes performing a Concat operation on the visual feature Video Features and the text feature Txext Features as a query q, wherein the Concat is a fusion, inputting the text feature into the repeating layer Repeat, and then performing the first feature fusion with the visual feature as the key k and the value v, inputting q, k, and v into the Multi-Head Attention layer, and then performing a second feature fusion with the result of the Concat operation, and then inputting the feedforward neural network FFN, and then performing a third feature fusion with the output of the second feature fusion, and the result obtained is used as the second query q, and the result of the first feature fusion is used as the second key k and the second value v, and the second query q, the second key k, and the second value v are input into the second Multi-Head Attention layer, and then performing a fourth feature fusion with the result of the third feature fusion, and then inputting the feedforward neural network FFN, and then performing a fifth feature fusion with the output of the fourth feature fusion, and then inputting the MLP layer, and then performing a sixth feature fusion with the output of the fifth feature fusion, and then outputting.

[0017] The motion capture results of the query video data include:

[0018] ;

[0019] ;

[0020] In the formula, is the motion capture result of querying video data, is the first category probability distribution, is the second category probability distribution, is a tunable hyperparameter.

[0021] Compared with the prior art, the present invention has the following beneficial effects: the present invention realizes efficient spatiotemporal feature extraction through a timing enhancement adapter and a spatiotemporal fusion adapter, thereby enhancing the spatiotemporal modeling capability of video features; utilizes a multi-level attention mechanism to improve the fusion capability of text and video features, and constructs a class prototype with strong generalization capability; significantly improves the capture accuracy of the model in few-sample learning tasks, reduces the number of trainable parameters, and reduces the computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Schematic diagram of the CLIP-ViT visual encoder structure;

[0023] Figure 2 It is a schematic diagram of the structure of even-numbered layers;

[0024] Figure 3 It is a schematic diagram of the odd-numbered layer structure;

[0025] Figure 4 Build a schematic diagram of the module structure for the prototype;

[0026] Figure 5 The effect of hyperparameters on the accuracy of 5-way 1-shot tasks on the Kinetics-400 (K400) dataset;

[0027] Figure 6 The effect of hyperparameters on the accuracy of the 5-way 1-shot task on the SSv2-Small dataset. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] A large model driven spatiotemporal feature and text enhanced few-shot motion capture method, comprising obtaining and preprocessing video data, wherein the video data comprises query video data to be motion captured and support set video data with action labels, and inputting the preprocessed video data into a visual encoder to obtain visual features of the video data;

[0030] Input the action labels of the support set into the CLIP text encoder to obtain the text features of the action labels of the support set, perform similarity calculation on the text features and the visual features of the query video data, and obtain the first category probability distribution; input the visual features and text features of the support set video data into the prototype construction module to obtain the prototype features of the support set video data, input the visual features of the query video data into the prototype construction module to obtain the prototype features of the query video data, perform time series similarity calculation on the prototype features of the support set video data and the prototype features of the query video data, and obtain the second category probability distribution;

[0031] Combining the two category probability distributions, the motion capture results of the query video data are obtained.

[0032] Obtaining the visual features of video data includes designing a temporal enhancement adapter and a spatiotemporal fusion adapter, and alternately embedding the two adapters into the CLIP-ViT visual encoder.

[0033] CLIP-ViT visual encoder Figure 1 , including N VIT Block layers. In the even-numbered layers, the temporal enhancement adapter is embedded before and after the multi-head attention layer. In the odd-numbered layers, the spatiotemporal fusion adapter is embedded only after the multi-head attention layer. VIT Block is a visual block.

[0034] The even-numbered layers are Figure 2 , the input information is sent to the TE Adapter layer, Layer Norm layer, Multi-Head Attention layer, feature fusion layer, TE Adapter layer, Layer Norm layer, MLP layer, feature fusion layer in sequence, and then output; the output of the first TE Adapter layer is provided with a shortcut connection to the first feature fusion layer, and the output of the second TE Adapter layer is provided with a shortcut connection to the second feature fusion layer;

[0035] The TE Adapter layer includes an FC Down layer, a Gelu layer, an FC Up layer and a feature fusion layer in sequence, and the input of the TE Adapter layer is provided with a shortcut connection to the feature fusion layer;

[0036] The TE Adapter is a timing enhancement adapter, Layer Norm is layer normalization, Multi-HeadAttention is multi-head attention, MLP is a multi-layer perceptron, FC Down is a fully connected downsampling, FC Up is a fully connected upsampling, and Gelu is an activation function.

[0037] The odd-numbered layers are Figure 3 , the input information is sent to the Layer Norm layer, Multi-Head Attention layer, feature fusion layer, STF Adapter layer, Layer Norm layer, MLP layer, feature fusion layer in sequence, and then output; the input of the spatiotemporal fusion adapter is provided with a shortcut connection to the first feature fusion layer, and the output of the STF Adapter layer is provided with a shortcut connection to the second feature fusion layer;

[0038] The STF Adapter layer includes an FC Down layer, a Max-Pool layer, a 3D-Conv layer, a Sigmiod layer, a neuron layer, an FC Up layer and a feature fusion layer in sequence. The input of the STF Adapter layer is provided with a quick connection connected to the feature fusion layer, and the output of the FCDown layer is provided with a quick connection connected to the neuron layer.

[0039] The STF Adapter is a spatiotemporal fusion adapter, Max-Pool is maximum pooling, 3D-Conv is a three-dimensional convolution, and Sigmiod is an activation function.

[0040] The similarity calculation includes performing a maximum pooling operation on the visual features of the query video data, and then performing a cosine similarity calculation on the visual features and text features of the query video data to obtain a first category probability distribution.

[0041] Prototype building blocks such as Figure 4, including performing a Concat operation on the visual feature Video Features and the text feature TxextFeatures as a query q, wherein the Concat is a fusion, inputting the text feature into the repeating layer Repeat, and then performing a first feature fusion with the visual feature as a key k and a value v, inputting q, k, and v into the Multi-HeadAttention layer, and then performing a second feature fusion with the result of the Concat operation, and then inputting the feedforward neural network FFN, and then performing a third feature fusion with the output of the second feature fusion, and the result obtained is used as the second query q, and the result of the first feature fusion is used as the second key k and the second value v, and the second query q, the second key k, and the second value v are input into the second Multi-Head Attention layer, and then performing a fourth feature fusion with the result of the third feature fusion, and then inputting the feedforward neural network FFN, and then performing a fifth feature fusion with the output of the fourth feature fusion, and then inputting the MLP layer, and then performing a sixth feature fusion with the output of the fifth feature fusion, and then outputting.

[0042] The motion capture results of the query video data include:

[0043] ;

[0044] ;

[0045] In the formula, is the motion capture result of querying video data, is the first category probability distribution, is the second category probability distribution, is a tunable hyperparameter.

[0046] Tables 1 and 2 provide detailed comparative experiments between the present invention and the few-shot motion capture method FSAR, which mainly uses the pre-trained ViT-B / 16 model, and are fairly compared with other methods using large-scale pre-trained models to comprehensively evaluate the performance and advantages of the present invention. In Table 1, the overall fine-tuning of the trained visual backbone network is performed, and the parameters of the visual backbone network are efficiently fine-tuned. In Table 2, the overall fine-tuning of the trained visual backbone network is performed, and the parameters of the visual backbone network are efficiently fine-tuned.

[0047] Table 1. Effects of various methods on time series related datasets (unit: %)

[0048] ;

[0049] In Table 1, the method column lists a variety of existing motion capture methods. The pre-training model uses the existing INet-RN50 and the CLIP ViT-B / 16 of the present invention. SSv2-Small and SSv2-Full are two data sets, and 1-shot and 5-shot are two training tasks.

[0050] Table 2. Effects of various methods on spatially related datasets (unit: %)

[0051] ;

[0052] In Table 2, the method column lists a variety of existing motion capture methods. The pre-training model uses the existing INet-RN50 and CLIPViT-B / 16 of the present invention. HMDB51, UCF101 and Kinetics are three data sets, and 1-shot and 5-shot are two training tasks.

[0053] From the results of the temporal modeling related datasets shown in Table 1, the present invention far exceeds other methods on the SSv2-Small dataset. Specifically, in the 1-shot task, the present invention is 1.3% higher than MA-CLIP and 2.2% higher in the 5-shot task. This shows that the present invention not only ensures the high efficiency of parameters, but also surpasses other methods based on parameter efficient fine-tuning (PEFT) in performance. For the SSv2-Full dataset, the performance of the present invention in the 1-shot task is comparable to that of the CLIP-FSAR method with full fine-tuning. In addition, in the 5-shot task, the present invention also shows competitiveness. For datasets dominated by spatial information, such as HMDB51, UCF101, and Kinetics, the results shown in Table 2 show that the present invention can achieve performance comparable to that of the full fine-tuning method. Although the present invention is slightly inferior to the full fine-tuning method in some tasks, its advantage lies in parameter efficiency. Different from the full fine-tuning method that requires optimizing the entire model, the present invention only fine-tunes a small number of parameters, thereby effectively reducing the consumption of computing resources while maintaining performance comparable to that of the full fine-tuning model.

[0054] To evaluate the impact of each module in the framework, ablation experiments were conducted on the 5-way1-shot task of the SSv2-Small dataset. The experimental results are summarized in Table 3. Starting from the baseline model with a frozen backbone network and no learnable modules, its accuracy is 38.0%. When the three modules, Temporal Enhancement Adaptation Module (TEA), Spatiotemporal Fusion Adaptation Module (STFA), and Text Enhanced Prototype Module (TEPM), are introduced respectively, the performance is improved by 15.4%, 14.2%, and 14.1%, respectively. After combining any two modules, the performance is further improved, with the accuracy of TEA and TEPM reaching 59.2%, which shows that temporal modeling and prototype construction have complementary advantages in capturing tasks. Finally, when all three modules are introduced, the model achieves the best performance of 61.4%, which is 23.4% higher than the baseline. These results verify the effectiveness of each module and show that their synergy plays a key role in the overall performance improvement.

[0055] Table 3. Performance comparison of 5-way 1-shot tasks under different settings on the SSv2-Small dataset

[0056] ;

[0057] Tune hyperparameters to control and and analyze their impact on the final classification accuracy. Figure 5 As shown in the figure, in the SSv2-Small dataset dominated by time information, as the hyperparameter increases from 0, the accuracy rises rapidly and reaches a peak of 61.5% when the hyperparameter is 0.5 and the hyperparameter is 0.6. When the hyperparameter is further increased to 1, the accuracy drops sharply. This shows that in tasks dominated by time features, the balanced contribution of the two probabilities is optimal, and over-emphasizing one component will lead to performance degradation. In contrast, in Figure 6 In the K400 dataset dominated by spatial information, the accuracy gradually increases with the increase of hyperparameters, reaching a maximum value of 93.8% when the hyperparameter is 0.7, and then slightly decreases. This shows that in tasks with spatial information as the main feature, increasing the weight of the video-text matching probability significantly helps improve performance, but excessive attention to a certain probability may lead to a slight performance degradation. For consistency, all experiments in this invention use a hyperparameter of 0.6.

[0058] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large model-driven spatiotemporal feature and text-enhanced few-sample motion capture method, characterized in that: The method includes obtaining and preprocessing video data, wherein the video data includes query video data to be motion captured and support set video data with action labels, and inputting the preprocessed video data into a visual encoder to obtain visual features of the video data; Input the action labels of the support set into the CLIP text encoder to obtain the text features of the action labels of the support set, perform similarity calculation on the text features and the visual features of the query video data, and obtain the first category probability distribution; input the visual features and text features of the support set video data into the prototype construction module to obtain the prototype features of the support set video data, input the visual features of the query video data into the prototype construction module to obtain the prototype features of the query video data, perform time series similarity calculation on the prototype features of the support set video data and the prototype features of the query video data, and obtain the second category probability distribution; Combining the two category probability distributions, the motion capture results of the query video data are obtained.

2. The large model driven spatiotemporal feature and text enhanced few-sample motion capture method according to claim 1, characterized in that: Obtaining the visual features of video data includes designing a temporal enhancement adapter and a spatiotemporal fusion adapter, and alternately embedding the two adapters into the CLIP-ViT visual encoder.

3. The large model driven spatiotemporal feature and text enhanced few-sample motion capture method according to claim 2, characterized in that: The CLIP-ViT visual encoder consists of N VIT Block layers. In the even-numbered layers, the temporal enhancement adapter is embedded before and after the multi-head attention layer. In the odd-numbered layers, the spatiotemporal fusion adapter is embedded only after the multi-head attention layer. VITBlock is the visual block.

4. The large model driven spatiotemporal feature and text enhanced few-sample motion capture method according to claim 3, characterized in that: The even-numbered layers sequentially send input information to the TE Adapter layer, the Layer Norm layer, the Multi-HeadAttention layer, the feature fusion layer, the TE Adapter layer, the Layer Norm layer, the MLP layer, the feature fusion layer, and then output; the output of the first TE Adapter layer is provided with a shortcut connection to the first feature fusion layer, and the output of the second TE Adapter layer is provided with a shortcut connection to the second feature fusion layer; The TE Adapter layer includes an FC Down layer, a Gelu layer, an FC Up layer and a feature fusion layer in sequence, and the input of the TE Adapter layer is provided with a shortcut connection to the feature fusion layer; The TE Adapter is a timing enhancement adapter, Layer Norm is layer normalization, Multi-Head Attention is multi-head attention, MLP is a multi-layer perceptron, FC Down is a fully connected downsampling, FC Up is a fully connected upsampling, and Gelu is an activation function.

5. The large model driven spatiotemporal feature and text enhanced few-sample motion capture method according to claim 4, characterized in that: The odd-numbered layers send the input information to the Layer Norm layer, the Multi-Head Attention layer, the feature fusion layer, the STF Adapter layer, the Layer Norm layer, the MLP layer, the feature fusion layer in sequence, and then output it; the input of the spatiotemporal fusion adapter is provided with a shortcut connection to the first feature fusion layer, and the output of the STF Adapter layer is provided with a shortcut connection to the second feature fusion layer; The STF Adapter layer includes an FC Down layer, a Max-Pool layer, a 3D-Conv layer, a Sigmiod layer, a neuron layer, an FC Up layer and a feature fusion layer in sequence. The input of the STF Adapter layer is provided with a quick connection connected to the feature fusion layer, and the output of the FCDown layer is provided with a quick connection connected to the neuron layer. The STF Adapter is a spatiotemporal fusion adapter, Max-Pool is maximum pooling, 3D-Conv is a three-dimensional convolution, and Sigmiod is an activation function.

6. The large model driven spatiotemporal feature and text enhanced few-sample motion capture method according to claim 5, characterized in that: The similarity calculation includes performing a maximum pooling operation on the visual features of the query video data, and then performing a cosine similarity calculation on the visual features and text features of the query video data to obtain a first category probability distribution.

7. The large model driven spatiotemporal feature and text enhanced few-sample motion capture method according to claim 6, characterized in that: The prototype construction module includes performing a Concat operation on the visual feature Video Features and the text feature Txext Features as a query q, wherein the Concat is a fusion, inputting the text feature into the repeating layer Repeat, and then performing the first feature fusion with the visual feature as the key k and the value v, inputting q, k, and v into the Multi-Head Attention layer, and then performing a second feature fusion with the result of the Concat operation, and then inputting the feedforward neural network FFN, and then performing a third feature fusion with the output of the second feature fusion, and the result obtained is used as the second query q, and the result of the first feature fusion is used as the second key k and the second value v, and the second query q, the second key k, and the second value v are input into the second Multi-Head Attention layer, and then performing a fourth feature fusion with the result of the third feature fusion, and then inputting the feedforward neural network FFN, and then performing a fifth feature fusion with the output of the fourth feature fusion, and then inputting the MLP layer, and then performing a sixth feature fusion with the output of the fifth feature fusion, and then outputting.

8. The large model driven spatiotemporal feature and text enhanced few-sample motion capture method according to claim 7, characterized in that: The motion capture results of the query video data include: ； ； In the formula, is the motion capture result of querying video data, is the first category probability distribution, is the second category probability distribution, is a tunable hyperparameter.

Images