DTSADTER video action recognition method based on efficient fine tuning of parameters

Abstract

The invention discloses a DTSAdapter video action recognition method based on efficient fine tuning of parameters. The method comprises the following steps: firstly, shooting and collecting video action data through a camera, marking the video action data to obtain a video action recognition data set, then constructing a parameter efficient fine-tuning video action recognition model, and inputting the obtained video action recognition data set into the parameter efficient fine-tuning video action recognition model for training; and finally, inputting to-be-detected video action recognition data into the trained parameter efficient fine-tuning video action recognition model to obtain a video action recognition result. According to the invention, a time adapter module, an improved linear fast attention mechanism module and a space adapter module are inserted at proper positions, residual connection is added at proper positions, and freezing processing is carried out on part of modules in back propagation of training. The calculation cost and the calculation efficiency are reduced while the accuracy of the model is improved.

Description

Technical Field

[0001] The present invention relates to a method for fine-tuning in a downstream action recognition task, and in particular to a method for DTSAdapter video action recognition based on efficient parameter fine-tuning. Background Art

[0002] Video action recognition is mainly based on deep learning methods, thanks in large part to the support of large video datasets, such as Kinetics, UCF101, Something-Something, etc. The model architecture used by mainstream video action recognition methods has evolved from convolutional neural networks (CNNs) to Transformers. Compared with action recognition based on images, various learning techniques for motion changes have been introduced due to the importance of temporal information for dynamic modeling. In addition, different training methods have been developed, such as unsupervised learning and video-text contrastive learning. With the introduction of large pre-trained models, new opportunities have been created for video action recognition models with stronger performance. For example, CLIP can achieve good performance after sufficient fine-tuning of the model on video datasets by equipping it with a temporal module; CLIP models are used for video action recognition tasks by learning video-specific cues. However, as the computational cost and resources of training in the field of video action recognition have increased significantly, how to effectively use large pre-trained image models for downstream video tasks (transfer learning) has become a mainstream research direction. Considering that training video models is much more expensive than image models in terms of both computational resources and time, this problem becomes particularly valuable in practice. On the other hand, it is very challenging to overcome the huge gap between images and videos in transfer learning. Pre-trained image models lack the ability to infer temporal structured information, which is crucial in video understanding. Generally speaking, the key design of state-of-the-art video models is to learn the temporal dimension based on contemporary image models. While model initializations are still important, they are largely out of reach for fine-tuning strategies as architectural modifications are usually forced in addition to full model training or fine-tuning for each downstream task. In contrast, parameter-efficient adapter strategies have high potential for large pre-trained image models.

[0003] With the widespread application of large pre-trained language models in a variety of downstream tasks, the popularity of efficient parameter tuning strategies has increased. Existing efficient parameter tuning methods can be roughly divided into three categories: the first category is to introduce adapters specific to downstream tasks. An adapter is a lightweight module that is inserted at a specific location in a pre-trained model. During fine-tuning specific to downstream tasks, only the parameters of the adapter modules inserted in the model need to be updated, while the parameters that account for most of the entire backbone model are frozen, thereby achieving parameter efficiency; the second category is prompt tuning. This method does not change the network architecture, but pre-adds a set of learnable tokens to the input of the model or intermediate layer. Only these newly added tokens need to be optimized and updated for each downstream task; the third category is learning weight approximation. Usually, a low-rank approximation method is used to fit the original parameter matrix, thereby achieving the effect of reducing the number of parameters.

[0004] Our method for video action recognition that efficiently fine-tunes the parameters of a pre-trained image model is able to infer dynamic video content at a smaller cost than the parameters required for each task. Compared to previous work, this method significantly reduces the parameters that need to be updated and performs on par with or better than the state-of-the-art video models with a fully fine-tuned strategy. Summary of the invention

[0005] In order to solve the problems of significantly increased computational cost and resources for training in the field of video action recognition and missing temporal structured information in transfer learning, the present invention proposes a DTSAdapter video action recognition method based on efficient parameter fine-tuning.

[0006] The technical solution of the present invention is as follows:

[0007] A method for DTSAdapter video action recognition based on efficient parameter fine-tuning includes the following steps:

[0008] S1, collect video action data through camera shooting, and obtain video action recognition data set after marking the video action data;

[0009] S2, constructing a parameter-efficient fine-tuning video action recognition model, inputting the obtained video action recognition data set into the parameter-efficient fine-tuning video action recognition model for training, and obtaining a trained parameter-efficient fine-tuning video action recognition model; the parameter-efficient fine-tuning video action recognition model comprises a preprocessing module, a patch embedding module, twelve image encoder modules, and a feature classification module connected in series in sequence;

[0010] S3. Input the video action recognition data to be tested into the trained parameter efficient fine-tuning video action recognition model to obtain the video action recognition result.

[0011] The preprocessing module includes unified frame sampling, picture size standardization, picture rotation processing, picture flip processing and picture color change.

[0012] In step S2, the patch embedding module includes a convolutional layer, the video action recognition dataset is input into the convolutional layer, and the output end of the convolutional layer is connected to the input end of the first image encoder module.

[0013] In step S2, each of the image encoder modules includes a temporal adapter module, a first normalization layer, an improved linear fast attention mechanism module, a spatial adapter module, a second normalization layer and a forward propagation module;

[0014] The temporal adapter module, the first normalization layer, the improved linear fast attention mechanism module and the spatial adapter module are connected in series in sequence, the input end of the temporal adapter module is used as the input end of the image encoder module, the output end of the temporal adapter module is also residually connected with the output end of the spatial adapter module for the first time to obtain a spatiotemporal feature map, the output end of the spatiotemporal feature map is connected to the input end of the second normalization layer, the output end of the second normalization layer is connected to the input end of the forward propagation module, the output end of the result of the first residual connection is also residually connected with the output end of the forward propagation module for the second time to obtain a coding feature map, and the output end of the coding feature map is used as the output end of the image encoder module.

[0015] The time adapter module includes a first channel dimension reduction fully connected layer, a time depth separable convolutional layer and a first channel dimension increase fully connected layer; the first channel dimension reduction fully connected layer, the time depth separable convolutional layer and the first channel dimension increase fully connected layer are sequentially connected in series, the input end of the first channel dimension reduction fully connected layer serves as the input end of the time adapter module, the input end of the first channel dimension reduction fully connected layer is also residually connected to the output end of the first channel dimension increase fully connected layer, and the output end of the residual connection result serves as the output end of the time adapter module;

[0016] The improved linear fast attention mechanism module includes a query input mapping fully connected layer, a key-value input mapping fully connected layer, a matrix mixing operation module, a first output mapping fully connected layer and a second output mapping fully connected layer; the input end of the query input mapping fully connected layer and the input end of the key-value input mapping fully connected layer are used as the input end of the improved linear fast attention mechanism module, the output end of the query input mapping fully connected layer is connected to the input end of the matrix mixing operation module, the output end of the matrix mixing operation module is connected to the output end of the key-value input mapping fully connected layer by dot multiplication, the result of the dot multiplication connection is connected to the input end of the first output mapping fully connected layer, the output end of the query input mapping fully connected layer is also residually connected to the output end of the first output mapping fully connected layer, the result of the residual connection is connected to the input end of the second output mapping fully connected layer, and the output end of the second output mapping fully connected layer is connected to the input end of the improved linear fast attention mechanism module;

[0017] The spatial adapter module includes a second channel dimension reduction fully connected layer, a spatial depth separable convolutional layer, and a second channel dimension increase fully connected layer; the second channel dimension reduction fully connected layer, the spatial depth separable convolutional layer, and the second channel dimension increase fully connected layer are sequentially connected in series, the input end of the second channel dimension reduction fully connected layer serves as the input end of the spatial adapter module, the input end of the second channel dimension reduction fully connected layer is also residually connected to the output end of the second channel dimension increase fully connected layer, and the output end of the residual connection result serves as the output end of the spatial adapter module;

[0018] The forward propagation module includes a forward first fully connected layer, an activation function, a first Dropout layer, a forward second fully connected layer and a second Dropout layer connected in series in sequence; the input end of the forward first fully connected layer serves as the input end of the forward propagation module, and the output end of the second Dropout layer serves as the output end of the forward propagation module.

[0019] The time adapter module is set according to the following formula:

[0020]

[0021] Up Proj =xW up +b up

[0022] Down Proj =xW down +b down

[0023] Among them, o is the output of the time adapter module; X is the input of the time adapter module; Down Proj (*) is the channel dimension reduction fully connected layer; W down is the weight of the channel dimension reduction fully connected layer, b downis the bias value of the channel dimension reduction fully connected layer; Up Proj (*) is the channel dimension-enhanced fully connected layer; W up is the weight of the channel dimension-enhanced fully connected layer, b up is the bias value of the channel dimension-upgraded fully connected layer; It is a time-depth separable convolution function with a kernel size of 3×1×1 and c output channels.

[0024] The space adapter module is set according to the following formula:

[0025]

[0026] Up Proj =xW up +b up

[0027] Down Proj =xW dow n+b down

[0028] Among them, o is the output of the space adapter module; X is the input of the space adapter module; Down Proj (*) is the channel dimension reduction fully connected layer; W down is the weight of the channel dimension reduction fully connected layer, b down is the bias value of the channel dimension reduction fully connected layer; Up Proj (*) is the channel dimension-enhanced fully connected layer; W up is the weight of the channel dimension-enhanced fully connected layer, b up is the bias value of the channel dimension-upgraded fully connected layer; It is a spatial depth-wise separable convolution function with a kernel size of 1×3×3 and output channels of c.

[0029] The improved linear fast attention mechanism module is set according to the following formula:

[0030] output = out_2 pro j(Q+out_1 proj (K*q))

[0031] Q = to_query(x)

[0032] K = to_key(x)

[0033]

[0034] Among them, x is the input of the improved linear fast attention mechanism module, to_query() is the query input mapping fully connected layer, to_key() is the key value input mapping fully connected layer, out_1 projFully connected layer for the first output map, out_2 proj is the second output mapping fully connected layer, Q is the output of the query input mapping fully connected layer, Q is a three-dimensional matrix of m×n×p, k is the output of the key-value input mapping fully connected layer, W α is a randomly initialized learnable parameter, d is the channel dimension of input x, α is the query enhancement feature matrix, α is a three-dimensional matrix of m×n×p, α i is the two-dimensional matrix of the i-th row of the query enhancement feature matrix α, Q i The two-dimensional matrix of the i-th row of the output Q of the fully connected layer that maps the query input, i is the index, n is the total number of rows of the three-dimensional matrix, and q is the output of the matrix mixing operation module.

[0035] The feature classification module includes a global average pooling layer, a classification Dropout layer and a classification fully connected layer connected in series in sequence; the input end of the global average pooling layer serves as the input end of the feature classification module, and the output end of the classification fully connected layer serves as the output end of the feature classification module.

[0036] The model training in step S3 is set up with the following requirements:

[0037] During the forward propagation of model training, all modules are not frozen;

[0038] During the back-propagation process of model training, only the patch embedding module, the first normalization layer of the image encoder module, the improved linear fast attention mechanism module, the second normalization layer, and the forward propagation module are frozen.

[0039] The present invention also provides a storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0040] The innovation of the present invention lies in the structure of inserting a time adapter module at an appropriate position in the Transformer model, improving the linear fast attention mechanism module and the spatial adapter module, and adding a residual connection at an appropriate position, which brings about the feature extraction of more time and spatial information and the beneficial effect of reducing the number of parameters, thereby achieving the advantages of improved model accuracy and reduced energy consumption.

[0041] The innovation of the present invention lies in the method of freezing some modules during the back propagation of training, so that only a small part of the parameters in the entire network framework is updated to achieve better results than comprehensive fine-tuning, and a higher accuracy can be achieved, while reducing the amount of parameter calculations in model training, thereby reducing the energy consumption of the model.

[0042] Beneficial effects of the present invention:

[0043] 1. The present invention inserts a time adapter module at an appropriate position in the original Transformer model, improves the linear fast attention mechanism module and the spatial adapter module, and adds a residual connection at an appropriate position, so that the model can extract more time and space information, thereby improving the model performance.

[0044] 2. The temporal adapter module and spatial adapter module designed in the present invention decompose the 3D convolution into 2D spatial convolution and 1D temporal convolution, respectively, which are used to adjust spatial information and learn temporal information, and further adopt a bottleneck structure to reduce the amount of calculation while avoiding redundant information, highlighting important spatiotemporal features, and improving the accuracy of action recognition.

[0045] 3. Compared with the traditional attention module, the improved linear fast attention mechanism module designed by the present invention extracts more feature information and further improves the accuracy of the model.

[0046] 4. The freezing process of back propagation in training designed by the present invention only updates the parameters of the time adapter and space adapter modules, thereby achieving better results than comprehensive fine-tuning by only updating a small part of the parameters in the entire network framework, and can achieve a higher accuracy rate. At the same time, it reduces the amount of parameter calculation for model training, thereby reducing the energy consumption of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is the overall network framework diagram of the method of the present invention;

[0048] Figure 2 A schematic diagram of the structure of the time adapter module used in the present invention;

[0049] Figure 3 A schematic diagram of the structure of a space adapter module used in the present invention;

[0050] Figure 4 Schematic diagram of the structure of the improved linear fast attention mechanism module used in the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0052] like Figure 1 As shown, an embodiment of a method for DTSAdapter video action recognition based on efficient parameter fine-tuning of the present invention is as follows:

[0053] S1. Collect video action data through camera shooting, and obtain a video action recognition dataset after annotating the video action data.

[0054] In the specific implementation, the collected video action recognition dataset contains 13,320 videos in 101 categories, and the videos contain camera motion and cluttered background.

[0055] S2. Construct a parameter-efficient fine-tuning video action recognition model, input the obtained video action recognition data set into the parameter-efficient fine-tuning video action recognition model for training, and obtain a trained parameter-efficient fine-tuning video action recognition model; the parameter-efficient fine-tuning video action recognition model includes a preprocessing module, a patch embedding module, twelve image encoder modules and a feature classification module connected in series in sequence.

[0056] In this embodiment, the input size of the parameter efficient fine-tuning video action recognition model is 8×196×768, which is also the standard input of the standard Transformer architecture.

[0057] The preprocessing module includes unified frame sampling, picture size standardization, picture rotation processing, picture flip processing, and picture color change.

[0058] In this embodiment, the image encoder module pre-trained by the contrastive learning model CLIP is used as the backbone model, and the pre-trained weight parameters of the image encoder module are loaded into the model to initialize the weights of all convolutional layers and linear layers of the model, and these weight parameters are frozen to ensure that they will not be updated during subsequent back propagation; specifically, the CLIP (Contrastive Language-Image Pretraining) model proposed by OpenAI is obtained. The model uses a large amount of image-text pairing data for pre-training through a contrastive learning method, and has the feature extraction capability of capturing the association between image and language. From a public resource library (such as PyTorch Hub or Hugging Face Model Hub), ViT-B / 16 is selected as the pre-trained CLIP model. In the CLIP model, the image encoder module is extracted, which is responsible for converting the input image into a vectorized feature tensor representation. The pre-trained weight file is downloaded and loaded, and the weights are applied to the image encoder module and used as the backbone model. Further, all weight parameters of the model are frozen to keep the pre-trained feature representation to ensure that it will not be updated during subsequent back propagation.

[0059] In step S2, the patch embedding module includes a convolutional layer, the video action recognition dataset is input to the convolutional layer, and the output of the convolutional layer is connected to the input of the first image encoder module.

[0060] In step S2, each image encoder module includes a temporal adapter module, a first normalization layer, an improved linear fast attention mechanism module, a spatial adapter module, a second normalization layer and a forward propagation module.

[0061] The temporal adapter module, the first normalization layer, the improved linear fast attention mechanism module and the spatial adapter module are connected in series in sequence. The input end of the temporal adapter module is used as the input end of the image encoder module. The output end of the temporal adapter module is also residually connected with the output end of the spatial adapter module for the first time to obtain a spatiotemporal feature map. The output end of the spatiotemporal feature map is connected to the input end of the second normalization layer. The output end of the second normalization layer is connected to the input end of the forward propagation module. The output end of the result of the first residual connection is also residually connected with the output end of the forward propagation module for the second time to obtain a coding feature map. The output end of the coding feature map is used as the output end of the image encoder module.

[0062] In this embodiment, the designed time adapter and space adapter modules are used to train the time and space information of the feature tensor during training, and are inserted into the backbone model to become a parameter-efficient fine-tuning video action recognition model. The input size of the parameter-efficient fine-tuning video action recognition model is 768×8×14×14, where 768 is the channel dimension, 8 is the number of input video frames, and 14 is the size of the feature map. This arrangement of the feature tensor facilitates the three-dimensional convolution to extract spatiotemporal information.

[0063] like Figure 2 As shown, the time adapter module includes a first channel dimensionality reduction fully connected layer, a time depth separable convolutional layer and a first channel dimensionality increase fully connected layer; the first channel dimensionality reduction fully connected layer, the time depth separable convolutional layer and the first channel dimensionality increase fully connected layer are connected in series in sequence, the input end of the first channel dimensionality reduction fully connected layer serves as the input end of the time adapter module, the input end of the first channel dimensionality reduction fully connected layer is also residually connected to the output end of the first channel dimensionality increase fully connected layer, and the output end of the residual connection result serves as the output end of the time adapter module.

[0064] The time adapter module is set according to the following formula:

[0065]

[0066] Up Proj =xW up +b up

[0067] Down Proj =xW down +b down

[0068] Among them, o is the output of the time adapter module; x is the input of the time adapter module; Down Proj () is the channel dimension reduction fully connected layer; W down is the weight of the channel dimension reduction fully connected layer, b down is the bias value of the channel dimension reduction fully connected layer; UpProj () is the channel dimension-enhancing fully connected layer; W up is the weight of the channel dimension-enhanced fully connected layer, b up is the bias value of the channel dimension-upgraded fully connected layer; It is a time-depth separable convolution function with a kernel size of 3×1×1 and c output channels.

[0069] like Figure 4 As shown, the improved linear fast attention mechanism module includes a query input mapping fully connected layer, a key-value input mapping fully connected layer, a matrix mixing operation module, a first output mapping fully connected layer and a second output mapping fully connected layer; the input end of the query input mapping fully connected layer and the input end of the key-value input mapping fully connected layer serve as the input end of the improved linear fast attention mechanism module, the output end of the query input mapping fully connected layer is connected to the input end of the matrix mixing operation module, the output end of the matrix mixing operation module and the output end of the key-value input mapping fully connected layer are dot-connected, the result of the dot-product connection is connected to the input end of the first output mapping fully connected layer, the output end of the query input mapping fully connected layer is also residually connected to the output end of the first output mapping fully connected layer, the result of the residual connection is connected to the input end of the second output mapping fully connected layer, and the output end of the second output mapping fully connected layer is connected to the input end of the improved linear fast attention mechanism module.

[0070] The improved linear fast attention mechanism module is set according to the following formula:

[0071] output = out_2 proj (Q+out_1 proj (K*q))

[0072] Q = to_query(x)

[0073] K = to_key(x)

[0074]

[0075] Among them, x is the input of the improved linear fast attention mechanism module, to_query() is the query input mapping fully connected layer, to_key() is the key value input mapping fully connected layer, out_1 proj Fully connected layer for the first output map, out_2 proj is the second output mapping fully connected layer, Q is the output of the query input mapping fully connected layer, Q is a three-dimensional matrix of m×n×p, K is the output of the key-value input mapping fully connected layer, w α is a randomly initialized learnable parameter, d is the channel dimension of input x, α is the query enhancement feature matrix, α is a three-dimensional matrix of m×n×p, α iis the two-dimensional matrix of the i-th row of the query enhancement feature matrix α, Q i The two-dimensional matrix of the i-th row of the output Q of the fully connected layer that maps the query input, i is the index, n is the total number of rows of the three-dimensional matrix, and q is the output of the matrix mixing operation module.

[0076] like Figure 3 As shown, the spatial adapter module includes a second channel dimensionality reduction fully connected layer, a spatial depth separable convolutional layer and a second channel dimensionality increase fully connected layer; the second channel dimensionality reduction fully connected layer, the spatial depth separable convolutional layer and the second channel dimensionality increase fully connected layer are connected in series in sequence, the input end of the second channel dimensionality reduction fully connected layer serves as the input end of the spatial adapter module, the input end of the second channel dimensionality reduction fully connected layer is also residually connected to the output end of the second channel dimensionality increase fully connected layer, and the output end of the residual connection result serves as the output end of the spatial adapter module.

[0077] The space adapter module is set according to the following formula:

[0078]

[0079] Up Proj =xW up +b up

[0080] Down Proj =xW down +d bown

[0081] Among them, o is the output of the space adapter module; x is the input of the space adapter module; Down Proj (*) is the channel dimension reduction fully connected layer; W down is the weight of the channel dimension reduction fully connected layer, b down is the bias value of the channel dimension reduction fully connected layer; Up Proj (*) is the channel dimension-enhanced fully connected layer; W up is the weight of the channel dimension-enhanced fully connected layer, b up is the bias value of the channel dimension-upgraded fully connected layer; It is a spatial depth-wise separable convolution function with a kernel size of 1×3×3 and output channels of c.

[0082] The position selection of the temporal adapter and the spatial adapter designed in this embodiment helps to aggregate temporal and spatial information. The temporal adapter module is connected in series at the input position of the image encoder to simulate the position of the temporal attention mechanism in the video ViT architecture, where temporal information can be better captured without interference from other information. The spatial adapter module is connected in series after the improved linear fast attention mechanism module because the function of the improved linear fast attention mechanism module is mainly to learn the association between different spaces within the frame, that is, spatial information, and the spatial adapter can just play the role of adjusting the spatial information and better integrate the beneficial spatial information in the context.

[0083] The forward propagation module includes a forward first fully connected layer, an activation function, a first Dropout layer, a forward second fully connected layer, and a second Dropout layer connected in series in sequence; the input end of the forward first fully connected layer serves as the input end of the forward propagation module, and the output end of the second Dropout layer serves as the output end of the forward propagation module.

[0084] The forward propagation module is set up according to the following formula:

[0085]

[0086] o=F2(F1(x))

[0087] Where x is the input of the forward propagation module, o is the output of the forward propagation module, F1(*) is the first forward transformation function, F2(*) is the second linear transformation function, is the first forward fully connected layer with input dimension c and output dimension c / 4, RELU(*) is the activation function, Dropout(*) is the function of the first Dropout layer or the second Dropout layer, It is the second forward fully connected layer with input dimension c / 4 and output dimension c.

[0088] The feature classification module includes a global average pooling layer, a classification Dropout layer and a classification fully connected layer connected in series in sequence; the input end of the global average pooling layer serves as the input end of the feature classification module, and the output end of the classification fully connected layer serves as the output end of the feature classification module.

[0089] The feature classification module is set according to the following formula:

[0090]

[0091] Among them, x is the input of the feature classification module, o is the output of the feature classification module, is the function of the classification fully connected layer with input dimension c and output dimension 101; Dropout(*) is the function of the classification Dropout layer, which randomly drops neurons with a given probability; GAP(*) is the function of the global average pooling.

[0092] The action category probability distribution output by the parameter-efficient fine-tuning video action recognition model and the true label of the video recognition action data are input into the loss function, and the error between the current prediction result of the model and the target label is calculated to obtain the loss value.

[0093] The loss function of the parameter efficient fine-tuning video action recognition model of this embodiment is set according to the following formula:

[0094]

[0095] Where N is the total number of samples, in this case 101, y i is the actual label value, The category probability predicted by the model, L is the cross entropy loss value.

[0096] After calculating the loss value, the back-propagation algorithm is used to calculate the gradient.

[0097] During the forward propagation of model training, all modules are not frozen.

[0098] During the back-propagation process of model training, only the patch embedding module, the first normalization layer of the image encoder module, the improved linear fast attention mechanism module, the second normalization layer, and the forward propagation module are frozen.

[0099] In an embodiment of the present invention, the freezing process means that the module does not participate in parameter updating. In a specific implementation, in the traditional back propagation process, each parameter in the model will adjust itself according to the calculated gradient value. In the back propagation process of the parameter efficient fine-tuning video action recognition model of the present invention, only the parameters of the time adapter module and the space adapter module are kept "updatable", that is, the parameters of these modules are set to requires_grad = True, and the parameters of all other modules are marked as "non-updatable" (usually by setting requires_grad = False) to ensure that the weights of these layers will not be updated during back propagation, so that they can be optimized according to the gradient of the loss function during back propagation.

[0100] S3. Input the video action recognition data to be tested into the trained parameter efficient fine-tuning video action recognition model to obtain the video action recognition result.

[0101] The present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements a method for DTS Adapter video action recognition based on efficient parameter fine-tuning, wherein the computer program is an instruction corresponding to the method for implementing DTS Adapter video action recognition based on efficient parameter fine-tuning.

[0102] The present invention adopts the Transformer architecture to extract feature information, and designs two adapter modules using the decoupling idea to decompose the 3D convolution into 2D spatial convolution and 1D temporal convolution, which are embedded in the traditional adapter module for modeling temporal information and spatial information. The temporal adapter module adopts feature dimensionality reduction, deep separable convolution of the temporal dimension and feature dimensionality increase, aiming to reduce the amount of calculation and parameters, and imitates the temporal self-attention module of the traditional video ViT, and models the temporal information between different frames in the feature before the multi-head self-attention module of ViT. The spatial adapter module adopts feature dimensionality reduction, deep separable convolution of the spatial dimension and feature dimensionality increase, while reducing the amount of calculation and parameters, and introduces the spatial adapter module after improving the linear fast attention mechanism module to model spatial information, and adjusts the modeling of spatial information. The present invention uses a parameter-efficient training method to freeze the VisionTransformer model, which comes from the image encoder part of CLIP pre-training, keeps its forward propagation parameter calculation, and does not update the parameters during back propagation, only updates the parameters of the two inserted adapter modules, and achieves a better effect than comprehensive fine-tuning by updating a small part of the parameters in the entire network framework. This parameter-efficient adapter approach is not only efficient but also performs well. It can extract and utilize pre-trained knowledge of large image models to achieve superior video understanding with small parameter cost.

[0103] The innovation of the present invention lies in the structure of inserting a time adapter module at an appropriate position in the Transformer model, improving the linear fast attention mechanism module and the spatial adapter module, and adding a residual connection at an appropriate position, which brings about the feature extraction of more time and spatial information and the beneficial effect of reducing the number of parameters, thereby achieving the advantages of improved model accuracy and reduced energy consumption.

[0104] The innovation of the present invention lies in the method of freezing some modules during the back propagation of training, so that only a small part of the parameters in the entire network framework is updated to achieve better results than comprehensive fine-tuning, and a higher accuracy can be achieved, while reducing the amount of parameter calculations in model training, thereby reducing the energy consumption of the model.

[0105] Finally, it should be noted that the above embodiments and explanations are only used to illustrate the technical solution of the present invention rather than to limit it. Those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope disclosed in the technical solution of the present invention, which should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for DTSAdapter video action recognition based on efficient parameter fine-tuning, characterized in that: The steps include: S1, collect video action data through camera shooting, and obtain video action recognition data set after marking the video action data; S2, constructing a parameter-efficient fine-tuning video action recognition model, inputting the obtained video action recognition data set into the parameter-efficient fine-tuning video action recognition model for training, and obtaining a trained parameter-efficient fine-tuning video action recognition model; the parameter-efficient fine-tuning video action recognition model comprises a preprocessing module, a patch embedding module, twelve image encoder modules, and a feature classification module connected in series in sequence; S3. Input the video action recognition data to be tested into the trained parameter efficient fine-tuning video action recognition model to obtain the video action recognition result.

2. The method of DTSAdapter video action recognition based on efficient parameter fine-tuning according to claim 1, characterized in that: In step S2, the patch embedding module includes a convolutional layer, the video action recognition dataset is input into the convolutional layer, and the output end of the convolutional layer is connected to the input end of the first image encoder module.

3. The method of DTSAdapter video action recognition based on efficient parameter fine-tuning according to claim 2, characterized in that: In the step S2, each of the image encoder modules includes a time adapter module, a first normalization layer, an improved linear fast attention mechanism module, a space adapter module, a second normalization layer and a forward propagation module; the time adapter module, the first normalization layer, the improved linear fast attention mechanism module and the space adapter module are connected in series in sequence, the input end of the time adapter module is used as the input end of the image encoder module, the output end of the time adapter module is also residually connected with the output end of the space adapter module for the first time to obtain a spatiotemporal feature map, the output end of the spatiotemporal feature map is connected to the input end of the second normalization layer, the output end of the second normalization layer is connected to the input end of the forward propagation module, the output end of the result of the first residual connection is also residually connected with the output end of the forward propagation module for the second time to obtain a coding feature map, and the output end of the coding feature map is used as the output end of the image encoder module.

4. The method of DTSAdapter video action recognition based on efficient parameter fine-tuning according to claim 3 is characterized in that: The time adapter module includes a first channel dimension reduction fully connected layer, a time depth separable convolutional layer and a first channel dimension increase fully connected layer; the first channel dimension reduction fully connected layer, the time depth separable convolutional layer and the first channel dimension increase fully connected layer are sequentially connected in series, the input end of the first channel dimension reduction fully connected layer serves as the input end of the time adapter module, the input end of the first channel dimension reduction fully connected layer is also residually connected to the output end of the first channel dimension increase fully connected layer, and the output end of the residual connection result serves as the output end of the time adapter module; The improved linear fast attention mechanism module includes a query input mapping fully connected layer, a key-value input mapping fully connected layer, a matrix mixing operation module, a first output mapping fully connected layer and a second output mapping fully connected layer; The input end of the query input mapping fully connected layer and the input end of the key value input mapping fully connected layer are used as the input end of the improved linear fast attention mechanism module, the output end of the query input mapping fully connected layer is connected to the input end of the matrix mixing operation module, the output end of the matrix mixing operation module is connected to the output end of the key value input mapping fully connected layer by dot multiplication, the result of the dot multiplication connection is connected to the input end of the first output mapping fully connected layer, the output end of the query input mapping fully connected layer is also residually connected to the output end of the first output mapping fully connected layer, the result of the residual connection is connected to the input end of the second output mapping fully connected layer, and the output end of the second output mapping fully connected layer is used as the output end of the improved linear fast attention mechanism module; The spatial adapter module includes a second channel dimension reduction fully connected layer, a spatial depth separable convolutional layer, and a second channel dimension increase fully connected layer; the second channel dimension reduction fully connected layer, the spatial depth separable convolutional layer, and the second channel dimension increase fully connected layer are sequentially connected in series, the input end of the second channel dimension reduction fully connected layer serves as the input end of the spatial adapter module, the input end of the second channel dimension reduction fully connected layer is also residually connected to the output end of the second channel dimension increase fully connected layer, and the output end of the residual connection result serves as the output end of the spatial adapter module; The forward propagation module includes a forward first fully connected layer, an activation function, a first Dropout layer, a forward second fully connected layer and a second Dropout layer connected in series in sequence; the input end of the forward first fully connected layer serves as the input end of the forward propagation module, and the output end of the second Dropout layer serves as the output end of the forward propagation module.

5. The method of DTSAdapter video action recognition based on efficient parameter fine-tuning according to claim 4 is characterized in that: The time adapter module is set according to the following formula: Up Proj =xW up +b up Down Proj =xW down +b down Among them, o is the output of the time adapter module; x is the input of the time adapter module; Down Proj (*) is the channel dimension reduction fully connected layer; W down is the weight of the channel dimension reduction fully connected layer, b down is the bias value of the channel dimension reduction fully connected layer; Up Proj (*) is the channel dimension-enhanced fully connected layer; W up is the weight of the channel dimension-enhanced fully connected layer, b up is the bias value of the channel dimension-upgraded fully connected layer; It is a time-depth separable convolution function with a kernel size of 3×1×1 and c output channels.

6. The method of DTSAdapter video action recognition based on efficient parameter fine-tuning according to claim 4, characterized in that: The space adapter module is set according to the following formula: Up Proj =xW up +b up Down Proj =xW down +b down Among them, o is the output of the space adapter module; x is the input of the space adapter module; Down Proj (*) is the channel dimension reduction fully connected layer; W down is the weight of the channel dimension reduction fully connected layer, b down is the bias value of the channel dimension reduction fully connected layer; Up Proj (*) is the channel dimension-enhanced fully connected layer; W up is the weight of the channel dimension-enhanced fully connected layer, b up is the bias value of the channel dimension-upgraded fully connected layer; It is a spatial depth-wise separable convolution function with a kernel size of 1×3×3 and output channels of c.

7. The method of DTSAdapter video action recognition based on efficient parameter fine-tuning according to claim 3, characterized in that: The improved linear fast attention mechanism module is set according to the following formula: output=out_2pro j (Q + out_1pro j (K*q)) Q = to_query(x) K = to_key(x) Among them, x is the input of the improved linear fast attention mechanism module, to_query() is the query input mapping fully connected layer, to_key() is the key value input mapping fully connected layer, out_1pro j Fully connected layer for the first output map, out_2pro j is the second output mapping fully connected layer, Q is the output of the query input mapping fully connected layer, K is the output of the key-value input mapping fully connected layer, and w α is a randomly initialized learnable parameter, d is the channel dimension of input x, α is the query enhancement feature matrix, and α i is the two-dimensional matrix of the i-th row of the query enhancement feature matrix α, Q i The two-dimensional matrix of the i-th row of the output Q of the fully connected layer that maps the query input, i is the index, n is the total number of rows of the three-dimensional matrix, and q is the output of the matrix mixing operation module.

8. The method of DTSAdapter video action recognition based on efficient parameter fine-tuning according to claim 1, characterized in that: The feature classification module includes a global average pooling layer, a classification Dropout layer and a classification fully connected layer connected in series in sequence; the input end of the global average pooling layer serves as the input end of the feature classification module, and the output end of the classification fully connected layer serves as the output end of the feature classification module.

9. The method of DTSAdapter video action recognition based on efficient parameter fine-tuning according to claim 1, characterized in that: The model training in step S3 is set up with the following requirements: During the forward propagation of model training, all modules are not frozen; During the back-propagation process of model training, only the patch embedding module, the first normalization layer of the image encoder module, the improved linear fast attention mechanism module, the second normalization layer, and the forward propagation module are frozen.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images