A method for video action recognition based on efficient parameter fine-tuning of DTSAdapter

Abstract

This invention discloses a method for video action recognition based on parameter-efficient fine-tuning using a DTSAdapter. The method includes: first, capturing video action data using a camera; then, labeling the video action data to obtain a video action recognition dataset; next, constructing a parameter-efficient fine-tuning video action recognition model; inputting the obtained video action recognition dataset into the parameter-efficient fine-tuning video action recognition model for training; and finally, inputting the video action recognition data to be tested into the trained parameter-efficient fine-tuning video action recognition model to obtain the video action recognition result. This invention inserts a time adapter module, an improved linear fast attention mechanism module, and a spatial adapter module at appropriate locations, and adds residual connections at appropriate locations. Furthermore, it employs a freezing process for some modules during backpropagation in training, thereby improving the model's accuracy while reducing computational cost and efficiency.

Description

Technical Field

[0001] This invention relates to a method for fine-tuning in downstream action recognition tasks, specifically to a method for DTSAdapter video action recognition based on efficient parameter fine-tuning. Background Technology

[0002] Video action recognition primarily relies on deep learning methods, largely thanks to the support of large video datasets such as Kinetics, UCF101, and Something-Something. The model architectures used in mainstream video action recognition methods have evolved from Convolutional Neural Networks (CNNs) to Transformers. Compared to image action recognition, the importance of temporal information for dynamic modeling has led to the introduction of various motion change learning techniques. Furthermore, different training methods have been developed, such as unsupervised learning and video-text contrastive learning. The introduction of large pre-trained models has created new opportunities for more powerful video action recognition models. For example, equipping CLIP with a temporal module and fine-tuning the model on video datasets can achieve good performance; learning video-specific cues and using the CLIP model for video action recognition tasks is also possible. However, with the significant increase in computational costs and resources for training in video action recognition, effectively applying large pre-trained image models to downstream video tasks (transfer learning) has become a mainstream research direction. Considering that training video models is much more expensive in terms of computational resources and time than training image models, this problem becomes particularly valuable in practice. On the other hand, overcoming the significant gap between images and videos in transfer learning is extremely challenging. Pre-trained image models lack the ability to infer temporal structured information, which is crucial in video understanding. Typically, a key design element of state-of-the-art video models is based on contemporary image models to learn the temporal dimension. While model initialization remains important, it largely transcends fine-tuning strategies, as architectural modifications are often forced beyond full model training or fine-tuning for each downstream task. In contrast, parameter-efficient adapter strategies hold great potential for large pre-trained image models.

[0003] With the widespread application of large-scale pre-trained language models in various downstream tasks, the popularity of parameter-efficient tuning strategies has surged. Existing parameter-efficient tuning methods can be broadly categorized into three types: The first type involves introducing downstream task-specific adapters. An adapter is a lightweight module inserted into a specific location within the pre-trained model. During downstream task-specific fine-tuning, only the parameters of those adapter modules inserted into the model need to be updated, while the parameters of the majority of the backbone model remain frozen, thus achieving parameter efficiency. The second type is cue-based tuning. This method does not change the network architecture but pre-adds a set of learnable tokens to the input of the model or intermediate layers. Only these newly added tokens need to be optimized and updated for each downstream task. The third type is learning weight approximations. This typically uses a low-rank approximation method to fit the original parameter matrix, thereby reducing the number of parameters.

[0004] This paper proposes an efficient fine-tuning method for video action recognition that utilizes the parameters of a pre-trained image model. This method can infer dynamic video content at a lower cost than the parameters required for each task. Compared with previous work, this method significantly reduces the number of parameters that need to be updated, and its performance is on par with or better than state-of-the-art video models with fully fine-tuned strategies. Summary of the Invention

[0005] To address the significant increase in computational costs and resources for training in the field of video action recognition, as well as the lack of temporal structured information in transfer learning, this invention proposes a method for video action recognition based on DTSAdapter with efficient parameter fine-tuning.

[0006] The technical solution of the present invention is as follows:

[0007] A method for video action recognition based on efficient parameter fine-tuning of a DTSAdapter includes the following steps:

[0008] S1. Video motion data is captured by a camera, and the video motion data is labeled to obtain a video motion recognition dataset;

[0009] S2. Construct a parameter-efficient fine-tuning video action recognition model. Input the obtained video action recognition dataset into the parameter-efficient fine-tuning video action recognition model for training to obtain a trained parameter-efficient fine-tuning video action recognition model. The parameter-efficient fine-tuning video action recognition model includes a preprocessing module, a patch embedding module, twelve image encoder modules, and a feature classification module connected in series.

[0010] S3. Input the video action recognition data to be tested into the trained parameter fine-tuning video action recognition model to obtain the video action recognition result.

[0011] The preprocessing module includes unified frame sampling, image size standardization, image rotation processing, image flipping processing, and image color transformation.

[0012] In step S2, the patch embedding module includes a convolutional layer, the video action recognition dataset is input to the convolutional layer, and the output of the convolutional layer is connected to the input of the first image encoder module.

[0013] In step S2, each image encoder module includes a time adapter module, a first normalization layer, an improved linear fast attention mechanism module, a spatial adapter module, a second normalization layer, and a forward propagation module.

[0014] The time adapter module, the first normalization layer, the improved linear fast attention mechanism module, and the spatial adapter module are connected in series. The input of the time adapter module serves as the input of the image encoder module. The output of the time adapter module is also connected to the output of the spatial adapter module via a first residual connection to obtain a spatiotemporal feature map. The output of the spatiotemporal feature map is connected to the input of the second normalization layer. The output of the second normalization layer is connected to the input of the forward propagation module. The output of the result of the first residual connection is also connected to the output of the forward propagation module via a second residual connection to obtain an encoded feature map. The output of the encoded feature map serves as the output of the image encoder module.

[0015] The time adapter module includes a first-channel dimensionality-reducing fully connected layer, a time-depth-separable convolutional layer, and a first-channel dimensionality-upgrading fully connected layer. The first-channel dimensionality-reducing fully connected layer, the time-depth-separable convolutional layer, and the first-channel dimensionality-upgrading fully connected layer are connected in series. The input of the first-channel dimensionality-reducing fully connected layer serves as the input of the time adapter module. The input of the first-channel dimensionality-reducing fully connected layer is also residually connected to the output of the first-channel dimensionality-upgrading fully connected layer. The output of the result of the residual connection serves as the output of the time adapter module.

[0016] The improved linear fast attention mechanism module includes a query input mapping fully connected layer, a key-value input mapping fully connected layer, a matrix hybrid operation module, a first output mapping fully connected layer, and a second output mapping fully connected layer. The input terminals of the query input mapping fully connected layer and the key-value input mapping fully connected layer serve as the input terminals of the improved linear fast attention mechanism module. The output terminal of the query input mapping fully connected layer is connected to the input terminal of the matrix hybrid operation module. The output terminal of the matrix hybrid operation module is connected to the output terminal of the key-value input mapping fully connected layer via a dot product connection. The result of the dot product connection is connected to the input terminal of the first output mapping fully connected layer. The output terminal of the query input mapping fully connected layer is also residually connected to the output terminal of the first output mapping fully connected layer. The result of the residual connection is connected to the input terminal of the second output mapping fully connected layer. The output terminal connected to the input terminal of the second output mapping fully connected layer serves as the output terminal of the improved linear fast attention mechanism module.

[0017] The spatial adapter module includes a second-channel dimensionality-reducing fully connected layer, a spatially depth-separable convolutional layer, and a second-channel dimensionality-upgrading fully connected layer. The second-channel dimensionality-reducing fully connected layer, the spatially depth-separable convolutional layer, and the second-channel dimensionality-upgrading fully connected layer are connected in series. The input of the second-channel dimensionality-reducing fully connected layer serves as the input of the spatial adapter module. The input of the second-channel dimensionality-reducing fully connected layer is also residually connected to the output of the second-channel dimensionality-upgrading fully connected layer. The output of the result of the residual connection serves as the output of the spatial adapter module.

[0018] The forward propagation module includes a forward first fully connected layer, an activation function, a first Dropout layer, a forward second fully connected layer, and a second Dropout layer connected in series. The input of the forward first fully connected layer serves as the input of the forward propagation module, and the output of the second Dropout layer serves as the output of the forward propagation module.

[0019] The time adapter module is configured according to the following formula:

[0020]

[0021] Up Proj =xW up +b up

[0022] Down Proj =xW down +b down

[0023] Where o is the output of the time adapter module; X is the input of the time adapter module; Down Proj (*) represents a channel-reduced fully connected layer; W down To reduce the dimensionality of the channel, the weights of the fully connected layer, b downThe bias value of the fully connected layer for channel dimensionality reduction; Up Proj (*) represents a fully connected layer for channel dimensionality enhancement; W up b is the weight of the channel-upgraded fully connected layer. up The bias value for the channel-upgraded fully connected layer; This is a time-depth separable convolution function with a kernel size of 3×1×1 and an output channel of c.

[0024] The space adapter module is configured according to the following formula:

[0025]

[0026] Up Proj =xW up +b up

[0027] Down Proj =xW dow n+b down

[0028] Where o is the output of the space adapter module; X is the input of the space adapter module; Down Proj (*) represents a channel-reduced fully connected layer; W down To reduce the dimensionality of the channel, the weights of the fully connected layer, b down The bias value of the fully connected layer for channel dimensionality reduction; Up Proj (*) represents a fully connected layer for channel dimensionality enhancement; W up b is the weight of the channel-upgraded fully connected layer. up The bias value for the channel-upgraded fully connected layer; This is a spatially depth-separable convolution function with a kernel size of 1×3×3 and an output channel of c.

[0029] The improved linear fast attention mechanism module is configured according to the following formula:

[0030] output = out_2 pro j(Q+out_1 proj (K*q))

[0031] Q = to_query(x)

[0032] K = to_key(x)

[0033]

[0034] Where x is the input of the improved linear fast attention mechanism module, to_query() is the query input mapping fully connected layer, to_key() is the key input mapping fully connected layer, and out_1 projThe first output is mapped to a fully connected layer, out_2 proj The second output mapping fully connected layer, Q is the output of the query input mapping fully connected layer, Q is an m×n×p three-dimensional matrix, k is the output of the key-value input mapping fully connected layer, and W is the output of the key-value input mapping fully connected layer. α The learnable parameters are randomly initialized, d is the number of channels in the input x, α is the query augmentation feature matrix, and α is an m×n×p three-dimensional matrix. i To query the two-dimensional matrix in the i-th row of the enhanced feature matrix α, Q i To query the output Q of the fully connected layer, we find the i-th row of the two-dimensional matrix, where i is the index, n is the total number of rows in the three-dimensional matrix, and q is the output of the matrix mixing operation module.

[0035] The feature classification module includes a global average pooling layer, a classification dropout layer, and a classification fully connected layer connected in series. The input of the global average pooling layer serves as the input of the feature classification module, and the output of the classification fully connected layer serves as the output of the feature classification module.

[0036] The model training in step S3 requires the following settings:

[0037] During the forward propagation of model training, no modules are frozen;

[0038] During the backpropagation process of model training, only the patch embedding module, the first normalized layer of the image encoder module, the improved linear fast attention mechanism module, the second normalized layer, and the forward propagation module are frozen.

[0039] The present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the above-described method.

[0040] The innovation of this invention lies in the structure of inserting a time adapter module, an improved linear fast attention mechanism module and a spatial adapter module at appropriate positions in the Transformer model, and adding residual connections at appropriate positions. This results in the extraction of more temporal and spatial information features, as well as the beneficial effect of reducing the number of parameters, and achieves the advantages of improved model accuracy and reduced energy consumption.

[0041] The innovation of this invention lies in freezing some modules during the backpropagation of training. This achieves better results than full fine-tuning by updating only a small portion of the parameters in the entire network framework, while also achieving higher accuracy. At the same time, it reduces the amount of parameter computation during model training, thereby reducing the energy consumption of the model.

[0042] The beneficial effects of this invention are:

[0043] 1. This invention inserts a time adapter module, an improved linear fast attention mechanism module, and a spatial adapter module at appropriate positions in the original Transformer model, and adds residual connections at appropriate positions, enabling the model to extract more temporal and spatial information, thereby improving model performance.

[0044] 2. The time adapter module and spatial adapter module designed in this invention decompose 3D convolution into 2D spatial convolution and 1D temporal convolution, respectively, which are used to adjust spatial information and learn temporal information. Furthermore, a bottleneck structure is adopted to reduce the amount of computation while avoiding redundant information, highlighting important spatiotemporal features, and improving the accuracy of action recognition.

[0045] 3. Compared with the traditional attention module, the improved linear fast attention mechanism module designed in this invention extracts more feature information, further improving the accuracy of the model.

[0046] 4. The backpropagation freeze process designed in this invention achieves better results than full fine-tuning by updating only a small part of the parameters in the entire network framework by updating only the parameters of the time adapter and spatial adapter modules. It can also achieve higher accuracy and reduce the parameter calculation of model training, thereby reducing the energy consumption of the model. Attached Figure Description

[0047] Figure 1 This is a diagram illustrating the overall network framework of the method of the present invention;

[0048] Figure 2 This is a schematic diagram of the structure of the time adapter module used in this invention;

[0049] Figure 3 This is a schematic diagram of the structure of the space adapter module used in this invention;

[0050] Figure 4 This is a schematic diagram of the structure of the improved linear fast attention mechanism module used in this invention. Detailed Implementation

[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0052] like Figure 1 As shown, an embodiment of the DTSAdapter video action recognition method based on efficient parameter fine-tuning of the present invention is as follows:

[0053] S1. Video motion data is captured by a camera, and the video motion data is labeled to obtain a video motion recognition dataset.

[0054] In practice, the collected video action recognition dataset contains 13,320 videos in 101 categories, which include camera movement and cluttered backgrounds.

[0055] S2. Construct a parameter-efficient fine-tuning video action recognition model. Input the obtained video action recognition dataset into the parameter-efficient fine-tuning video action recognition model for training to obtain a trained parameter-efficient fine-tuning video action recognition model. The parameter-efficient fine-tuning video action recognition model includes a preprocessing module, a patch embedding module, twelve image encoder modules, and a feature classification module connected in sequence.

[0056] In this embodiment, the input size of the parameter-efficient fine-tuning video action recognition model is 8×196×768, which is also the standard input of the standard Transformer architecture.

[0057] The preprocessing module includes unified frame sampling, image size standardization, image rotation processing, image flipping processing, and image color transformation.

[0058] In this embodiment, the image encoder module pre-trained using the contrastive learning model CLIP is used as the backbone model. The pre-trained weight parameters of the image encoder module are loaded into the model to initialize the weights of all convolutional and linear layers, and these weight parameters are frozen to ensure they are not updated during subsequent backpropagation. Specifically, the CLIP (Contrastive Language-Image Pretraining) model proposed by OpenAI is used. This model uses a contrastive learning method and is pre-trained on a large amount of image-text paired data, possessing the ability to extract features that capture the correlation between images and language. ViT-B / 16 is selected as the pre-trained CLIP model from a public resource repository (such as PyTorch Hub or Hugging Face Model Hub). In the CLIP model, the image encoder module is extracted, which is responsible for converting the input image into a vectorized feature tensor representation. The pre-trained weight file is downloaded and loaded, and the weights are applied to the image encoder module as the backbone model. Furthermore, all weight parameters of the model are frozen to maintain the pre-trained feature representation and ensure it is not updated during subsequent backpropagation.

[0059] In step S2, the patch embedding module includes a convolutional layer, the video action recognition dataset is input into the convolutional layer, and the output of the convolutional layer is connected to the input of the first image encoder module.

[0060] In step S2, each image encoder module includes a temporal adapter module, a first normalization layer, an improved linear fast attention mechanism module, a spatial adapter module, a second normalization layer, and a forward propagation module.

[0061] The temporal adapter module, the first normalization layer, the improved linear fast attention mechanism module, and the spatial adapter module are connected in series. The input of the temporal adapter module serves as the input of the image encoder module. The output of the temporal adapter module is also connected to the output of the spatial adapter module via a first residual connection to obtain a spatiotemporal feature map. The output of the spatiotemporal feature map is connected to the input of the second normalization layer. The output of the second normalization layer is connected to the input of the forward propagation module. The output of the result of the first residual connection is also connected to the output of the forward propagation module via a second residual connection to obtain an encoded feature map. The output of the encoded feature map serves as the output of the image encoder module.

[0062] In this embodiment, the designed temporal and spatial adapter modules are used to extract the temporal and spatial information of the feature tensors during training and insert them into the backbone model to form a parametrically efficient fine-tuning video action recognition model. The input size of the parametrically efficient fine-tuning video action recognition model is 768×8×14×14, where 768 is the channel dimension, 8 is the number of input video frames, and 14 is the size of the feature map. This arrangement of the feature tensors facilitates the extraction of spatiotemporal information through three-dimensional convolution.

[0063] like Figure 2 As shown, the time adapter module includes a first-channel dimensionality-reducing fully connected layer, a time-depth-separable convolutional layer, and a first-channel dimensionality-upgrading fully connected layer. The first-channel dimensionality-reducing fully connected layer, the time-depth-separable convolutional layer, and the first-channel dimensionality-upgrading fully connected layer are connected in series. The input of the first-channel dimensionality-reducing fully connected layer serves as the input of the time adapter module. The input of the first-channel dimensionality-reducing fully connected layer is also residually connected to the output of the first-channel dimensionality-upgrading fully connected layer. The output of the result of the residual connection serves as the output of the time adapter module.

[0064] The time adapter module should be configured using the following formula:

[0065]

[0066] Up Proj =xW up +b up

[0067] Down Proj =xW down +b down

[0068] Where o is the output of the time adapter module; x is the input of the time adapter module; Down Proj () represents a channel-reduced fully connected layer; W down To reduce the dimensionality of the channel, the weights of the fully connected layer, b down The bias value of the fully connected layer for channel dimensionality reduction; UpProj () represents a fully connected layer for channel dimensionality enhancement; W up b is the weight of the channel-upgraded fully connected layer. up The bias value for the channel-upgraded fully connected layer; This is a time-depth separable convolution function with a kernel size of 3×1×1 and an output channel of c.

[0069] like Figure 4 As shown, the improved linear fast attention mechanism module includes a query input mapping fully connected layer, a key-value input mapping fully connected layer, a matrix mixing operation module, a first output mapping fully connected layer, and a second output mapping fully connected layer. The input terminals of the query input mapping fully connected layer and the key-value input mapping fully connected layer serve as the input terminals of the improved linear fast attention mechanism module. The output terminal of the query input mapping fully connected layer is connected to the input terminal of the matrix mixing operation module. The output terminal of the matrix mixing operation module is connected to the output terminal of the key-value input mapping fully connected layer via a dot product connection. The result of the dot product connection is connected to the input terminal of the first output mapping fully connected layer. The output terminal of the query input mapping fully connected layer is also residually connected to the output terminal of the first output mapping fully connected layer. The result of the residual connection is connected to the input terminal of the second output mapping fully connected layer. The output terminal connected to the input terminal of the second output mapping fully connected layer serves as the output terminal of the improved linear fast attention mechanism module.

[0070] The improved linear fast attention mechanism module is configured according to the following formula:

[0071] output = out_2 proj (Q+out_1 proj (K*q))

[0072] Q = to_query(x)

[0073] K = to_key(x)

[0074]

[0075] Where x is the input of the improved linear fast attention mechanism module, to_query() is the query input mapping fully connected layer, to_key() is the key input mapping fully connected layer, and out_1 proj The first output is mapped to a fully connected layer, out_2 proj The second output mapping fully connected layer, Q is the output of the query input mapping fully connected layer, Q is an m×n×p three-dimensional matrix, K is the output of the key-value input mapping fully connected layer, and w α The learnable parameters are randomly initialized, d is the number of channels in the input x, α is the query augmentation feature matrix, and α is an m×n×p three-dimensional matrix. iTo query the two-dimensional matrix in the i-th row of the enhanced feature matrix α, Q i To query the output Q of the fully connected layer, we find the i-th row of the two-dimensional matrix, where i is the index, n is the total number of rows in the three-dimensional matrix, and q is the output of the matrix mixing operation module.

[0076] like Figure 3 As shown, the spatial adapter module includes a second-channel dimensionality reduction fully connected layer, a spatial depth separable convolutional layer, and a second-channel dimensionality increase fully connected layer. The second-channel dimensionality reduction fully connected layer, the spatial depth separable convolutional layer, and the second-channel dimensionality increase fully connected layer are connected in series. The input of the second-channel dimensionality reduction fully connected layer serves as the input of the spatial adapter module. The input of the second-channel dimensionality reduction fully connected layer is also residually connected to the output of the second-channel dimensionality increase fully connected layer. The output of the result of the residual connection serves as the output of the spatial adapter module.

[0077] The space adapter module is configured according to the following formula:

[0078]

[0079] Up Proj =xW up +b up

[0080] Down Proj =xW down +d bown

[0081] Where o is the output of the space adapter module; x is the input of the space adapter module; Down Proj (*) represents a channel-reduced fully connected layer; W down To reduce the dimensionality of the channel, the weights of the fully connected layer, b down The bias value of the fully connected layer for channel dimensionality reduction; Up Proj (*) represents a fully connected layer for channel dimensionality enhancement; W up b is the weight of the channel-upgraded fully connected layer. up The bias value for the channel-upgraded fully connected layer; This is a spatially depth-separable convolution function with a kernel size of 1×3×3 and an output channel of c.

[0082] The placement of the temporal and spatial adapters in this embodiment helps to aggregate temporal and spatial information. The temporal adapter module is connected in series at the input position of the image encoder to simulate the location of the temporal attention mechanism in the video ViT architecture, where temporal information can be captured more effectively without interference from other information. The spatial adapter module is connected in series after the improved linear fast attention mechanism module because the main function of the improved linear fast attention mechanism module is to learn the correlation between different spaces within a frame, i.e., spatial information. The spatial adapter can precisely adjust the spatial information, better integrating beneficial spatial information from the context.

[0083] The forward propagation module includes a forward first fully connected layer, an activation function, a first Dropout layer, a forward second fully connected layer, and a second Dropout layer connected in series. The input of the forward first fully connected layer serves as the input of the forward propagation module, and the output of the second Dropout layer serves as the output of the forward propagation module.

[0084] The forward propagation module is configured using the following formula:

[0085]

[0086] o = F2(F1(x))

[0087] Where x is the input of the forward propagation module, o is the output of the forward propagation module, F1(*) is the first lookahead transformation function, and F2(*) is the second linear transformation function. This is a first fully connected forward layer with input dimension c and output dimension c / 4. ReLU(*) is the activation function, and Dropout(*) is the function of the first or second Dropout layer. It is a feedforward second fully connected layer with an input dimension of c / 4 and an output dimension of c.

[0088] The feature classification module consists of a global average pooling layer, a classification dropout layer, and a classification fully connected layer connected in series. The input of the global average pooling layer serves as the input of the feature classification module, and the output of the classification fully connected layer serves as the output of the feature classification module.

[0089] The feature classification module is set according to the following formula:

[0090]

[0091] Where x is the input of the feature classification module, and o is the output of the feature classification module. is a function for a fully connected classification layer with input dimension c and output dimension 101; Dropout(*) is a function for a classification Dropout layer that randomly drops neurons with a given probability; GAP(*) is a function for global average pooling.

[0092] The probability distribution of action categories output by the video action recognition model with efficient parameter fine-tuning and the true labels of the video action recognition data are input into the loss function. The error between the model's current prediction result and the target label is calculated to obtain the loss value.

[0093] In this embodiment, the loss function of the video motion recognition model for efficient parameter fine-tuning is set according to the following formula:

[0094]

[0095] Where N is the total number of samples, which is 101 in this example, y i These are the actual label values. The model predicts the class probability, and L is the cross-entropy loss value.

[0096] After calculating the loss value, the gradient is calculated using the backpropagation algorithm.

[0097] During the forward propagation of model training, no modules are frozen.

[0098] During the backpropagation process of model training, only the patch embedding module, the first normalized layer of the image encoder module, the improved linear fast attention mechanism module, the second normalized layer, and the forward propagation module are frozen.

[0099] In this embodiment of the invention, the freezing process means that the module does not participate in parameter updates. Specifically, in traditional backpropagation, each parameter in the model adjusts itself based on the calculated gradient value. In the backpropagation process of the parameter-efficient fine-tuning video action recognition model of this invention, only the parameters of the temporal adapter module and the spatial adapter module are kept "updatable," i.e., the parameters of these modules are set to `requires_grad = True`, while the parameters of all other modules are marked as "non-updatable" (usually by setting `requires_grad = False`). This ensures that the weights of these layers are not updated during backpropagation, allowing them to be optimized based on the gradient of the loss function during backpropagation.

[0100] S3. Input the video action recognition data to be tested into the trained parameter fine-tuning video action recognition model to obtain the video action recognition result.

[0101] This invention also provides a storage medium storing a computer program that, when executed by a processor, implements a method for DTSAdapter video motion recognition based on efficient parameter fine-tuning. The computer program consists of instructions corresponding to the implementation of this method.

[0102] This invention employs a Transformer architecture to extract feature information and utilizes a decoupling approach to design two adapter modules. The 3D convolution is decomposed into 2D spatial convolution and 1D temporal convolution, which are embedded into the traditional adapter module to model temporal and spatial information. The temporal adapter module uses feature dimensionality reduction, depthwise separable convolution in the temporal dimension, and feature dimensionality increase. This aims to reduce computation and parameters while mimicking the temporal self-attention module of traditional video ViT, modeling the temporal information between different frames before the multi-head self-attention module of ViT. The spatial adapter module also uses feature dimensionality reduction, depthwise separable convolution in the spatial dimension, and feature dimensionality increase. While reducing computation and parameters, it is introduced after the improved linear fast attention mechanism module models spatial information, adjusting the model's spatial information modeling. This invention utilizes a parameter-efficient training method, freezing the VisionTransformer model derived from the CLIP pre-trained image encoder part. While maintaining its forward propagation parameter calculations, the parameters are not updated during backpropagation; only the parameters of the two inserted adapter modules are updated. This achieves better results than comprehensive fine-tuning by updating only a small portion of the parameters in the entire network framework. This parameter-efficient adapter method is not only efficient but also performs well. It can extract and utilize the pre-trained knowledge of large image models to achieve excellent video understanding with relatively low parameter cost.

[0103] The innovation of this invention lies in the structure of inserting a time adapter module, an improved linear fast attention mechanism module and a spatial adapter module at appropriate positions in the Transformer model, and adding residual connections at appropriate positions. This results in the extraction of more temporal and spatial information features, as well as the beneficial effect of reducing the number of parameters, and achieves the advantages of improved model accuracy and reduced energy consumption.

[0104] The innovation of this invention lies in freezing some modules during the backpropagation of training. This achieves better results than full fine-tuning by updating only a small portion of the parameters in the entire network framework, while also achieving higher accuracy. At the same time, it reduces the amount of parameter computation during model training, thereby reducing the energy consumption of the model.

[0105] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for video action recognition based on efficient parameter fine-tuning of a DTSAdapter, characterized in that, Includes the following steps: S1. Collect video motion data by shooting with a camera, and obtain a video motion recognition dataset after labeling the video motion data; S2. Construct a parameter-efficient fine-tuned video action recognition model. Input the obtained video action recognition dataset into the parameter-efficient fine-tuned video action recognition model for training to obtain a trained parameter-efficient fine-tuned video action recognition model. The parameter-efficient fine-tuned video action recognition model includes a preprocessing module, a patch embedding module, twelve image encoder modules, and a feature classification module, which are connected in series. Each image encoder module includes a temporal adapter module, a first normalization layer, an improved linear fast attention mechanism module, a spatial adapter module, a second normalization layer, and a forward propagation module. The temporal adapter module, the first normalization layer, the improved linear fast attention mechanism module, and the spatial adapter module are connected in series. The input of the temporal adapter module serves as the input of the image encoder module. The output of the temporal adapter module is also first residually concatenated with the output of the spatial adapter module to obtain a spatiotemporal feature map. The output of the spatiotemporal feature map is connected to the input of the second normalization layer. The output of the second normalization layer is connected to the input of the forward propagation module. The output of the result of the first residual connection is second residually concatenated with the output of the forward propagation module to obtain an encoded feature map. The output of the encoded feature map serves as the output of the image encoder module. The time adapter module is configured according to the following formula: Where o is the output of the time adapter module; x is the input of the time adapter module; Down Proj (*) represents a channel-reduced fully connected layer; W down To reduce the dimensionality of the channel, the weights of the fully connected layer, b down The bias value of the fully connected layer for channel dimensionality reduction; Up Proj (*) represents a fully connected layer for channel dimensionality enhancement; W up b is the weight of the channel-upgraded fully connected layer. up The bias value for the channel-upgraded fully connected layer; This is a time-depth-separable convolution function with a kernel size of 3×1×1 and c output channels. The space adapter module is configured according to the following formula: Where o is the output of the space adapter module; x is the input of the space adapter module; Down Proj (*) represents a channel-reduced fully connected layer; W down To reduce the dimensionality of the channel, the weights of the fully connected layer, b down The bias value of the fully connected layer for channel dimensionality reduction; Up Proj (*) represents a fully connected layer for channel dimensionality enhancement; W up b is the weight of the channel-upgraded fully connected layer. up The bias value for the channel-upgraded fully connected layer; A spatially depth-separable convolution function with a kernel size of 1×3×3 and an output channel of c; S3. Input the video action recognition data to be tested into the trained parameter fine-tuning video action recognition model to obtain the video action recognition result.

2. The method for DTSAdapter video action recognition based on efficient parameter fine-tuning according to claim 1, characterized in that: In step S2, the patch embedding module includes a convolutional layer, the video action recognition dataset is input to the convolutional layer, and the output of the convolutional layer is connected to the input of the first image encoder module.

3. The method for DTSAdapter video action recognition based on efficient parameter fine-tuning according to claim 1, characterized in that: The time adapter module includes a first-channel dimensionality-reducing fully connected layer, a time-depth-separable convolutional layer, and a first-channel dimensionality-upgrading fully connected layer. The first-channel dimensionality-reducing fully connected layer, the time-depth-separable convolutional layer, and the first-channel dimensionality-upgrading fully connected layer are connected in series. The input of the first-channel dimensionality-reducing fully connected layer serves as the input of the time adapter module. The input of the first-channel dimensionality-reducing fully connected layer is also residually connected to the output of the first-channel dimensionality-upgrading fully connected layer. The output of the result of the residual connection serves as the output of the time adapter module. The improved linear fast attention mechanism module includes a query input mapping fully connected layer, a key-value input mapping fully connected layer, a matrix hybrid operation module, a first output mapping fully connected layer, and a second output mapping fully connected layer. The input terminals of the query input mapping fully connected layer and the key-value input mapping fully connected layer serve as the input terminals of the improved linear fast attention mechanism module. The output terminal of the query input mapping fully connected layer is connected to the input terminal of the matrix mixing operation module. The output terminal of the matrix mixing operation module is connected to the output terminal of the key-value input mapping fully connected layer via dot product. The result of the dot product connection is connected to the input terminal of the first output mapping fully connected layer. The output terminal of the query input mapping fully connected layer is also connected to the output terminal of the first output mapping fully connected layer via residual connection. The result of the residual connection is connected to the input terminal of the second output mapping fully connected layer. The output terminal connected to the input terminal of the second output mapping fully connected layer serves as the output terminal of the improved linear fast attention mechanism module. The spatial adapter module includes a second-channel dimensionality-reducing fully connected layer, a spatially depth-separable convolutional layer, and a second-channel dimensionality-upgrading fully connected layer. The second-channel dimensionality-reducing fully connected layer, the spatially depth-separable convolutional layer, and the second-channel dimensionality-upgrading fully connected layer are connected in series. The input of the second-channel dimensionality-reducing fully connected layer serves as the input of the spatial adapter module. The input of the second-channel dimensionality-reducing fully connected layer is also residually connected to the output of the second-channel dimensionality-upgrading fully connected layer. The output of the result of the residual connection serves as the output of the spatial adapter module. The forward propagation module includes a forward first fully connected layer, an activation function, a first Dropout layer, a forward second fully connected layer, and a second Dropout layer connected in series. The input of the forward first fully connected layer serves as the input of the forward propagation module, and the output of the second Dropout layer serves as the output of the forward propagation module.

4. The method for DTSAdapter video action recognition based on efficient parameter fine-tuning according to claim 3, characterized in that: The improved linear fast attention mechanism module is configured according to the following formula: Where x is the input of the improved linear fast attention mechanism module, to_query() is the query input mapping fully connected layer, to_key() is the key input mapping fully connected layer, and out_1 proj The first output is mapped to a fully connected layer, out_2 proj For the second output mapping fully connected layer, Q is the output of the query input mapping fully connected layer, K is the output of the key-value input mapping fully connected layer, and w α The learnable parameters are randomly initialized, d is the number of channels in the input x, and α is the query augmentation feature matrix. i To query the two-dimensional matrix in the i-th row of the enhanced feature matrix α, Q i To query the output Q of the fully connected layer, we find the i-th row of the two-dimensional matrix, where i is the index, n is the total number of rows in the three-dimensional matrix, and q is the output of the matrix mixing operation module.

5. The method for DTSAdapter video action recognition based on efficient parameter fine-tuning according to claim 1, characterized in that: The feature classification module includes a global average pooling layer, a classification dropout layer, and a classification fully connected layer connected in series. The input of the global average pooling layer serves as the input of the feature classification module, and the output of the classification fully connected layer serves as the output of the feature classification module.

6. The method for DTSAdapter video action recognition based on efficient parameter fine-tuning according to claim 1, characterized in that, The model training in step S3 requires the following settings: During the forward propagation of model training, no modules are frozen; During the backpropagation process of model training, only the patch embedding module, the first normalized layer of the image encoder module, the improved linear fast attention mechanism module, the second normalized layer, and the forward propagation module are frozen.

7. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

Images