Video action recognition method based on auxiliary time modeling module

Abstract

The invention discloses a video action recognition method based on an auxiliary time modeling module, and belongs to the field of mode recognition. According to the method, a series of time modeling modules (ECTLs) are designed, and the advantages of Transform and LSTM in the aspect of processing sequence information are efficiently combined, so that the potential of CLIP in the aspect of improving the efficiency and universality of an action recognition technology is explored. The ECTL module can extract local and global features from the sequential sequence, thereby improving the efficiency of sequential modeling. Experiments carried out on three data sets (HMDB-51, UCF-101 and Someting-Someting V2) show that the ECTL series time modeling module provided by the invention has effectiveness, and has universality in various video recognition environments. According to the method, the extremely competitive performance is kept under complete supervision, and the accuracy which is equivalent to or even higher than that of the most advanced method is realized on three public reference data sets.

Description

TECHNICAL FIELD

[0001] The application provides a video action recognition method based on an auxiliary time modeling module, and belongs to the field of pattern recognition. BACKGROUND

[0002] Video action recognition is a core technology for understanding human behavior, human-computer interaction and scene understanding, and has important academic value and engineering practical significance in various applications such as intelligent monitoring, medical auxiliary diagnosis and sports competition analysis. At the same time, compared with the language image pre-training (CLIP), it has achieved remarkable success in various visual tasks recently. How to effectively extend the function of CLIP to the video field by using powerful time modeling technology is still a key problem that has not been solved. When extending CLIP to the field of video action recognition to perform fully supervised recognition tasks, there are two key considerations that are crucial: 1) guiding the model to pay more attention to the dynamic change area of object movement, so as to more effectively link the action with the text; 2) in view of the fact that previous technologies usually ignore local information. How to design a model that contains both local and global information?

[0003] Previous video action recognition methods categorize human actions in videos into pre-determined classes. This approach ignores the semantic information contained in videos by fixing the label classes. Therefore, it limits the generality and generalization ability of these methods. Due to the lack of rationality in design, the applicability of these methods in the real world is greatly discounted. The emergence of large-scale datasets consisting of image-text pairs has made significant progress in the pre-training of visual language models, such as CLIP, CoCa, ALIGN, and Florence. These models are trained on a wide range of datasets containing a large number of images and their corresponding text annotations. This process helps to establish a solid link between visual elements and natural language, enabling the model to effectively learn semantic representations from images. Therefore, these models have shown advanced capabilities in transfer learning and generalization across different application domains. On the basis of this success, researchers have made efforts to apply these image-based visual language models, such as CLIP (Contrastive Language-Image Pre-training), to videos, extending their generality and generalization ability to the video domain. However, extending this pre-training strategy to videos presents significant challenges, as follows: 1) Unlike image-text pairs that are readily available over the internet, aligned video-text data is very scarce and challenging to curate. 2) Videos themselves are complex, containing not only static information present in each video frame but also dynamic temporal information between video frames. Therefore, it requires a large computational cost, while capturing appearance cues of videos through image-text pairs is much less computationally budgeted. Therefore, compared to developing video language pre-training models, adjusting image language models for video-based tasks to apply to recognition scenarios including zero-shot, few-shot, and fully supervised, while retaining their versatile multi-modal learning representations, is a promising research direction.

[0004] Due to the powerful representation ability and excellent generalization characteristics of CLIP, it has great potential to transfer its powerful knowledge to the video field. The key challenge faced by transfer learning lies in the modal difference between images and videos, and many research teams in the video field have proposed corresponding solutions. For example, optimizing some "continuous prompt vectors" to adapt CLIP to the video understanding task; through an end-to-end method, the knowledge in the CLIP model is transferred to video language retrieval; introduce a decoder module dedicated to video to simulate long-range inter-frame dependencies in video sequences; encode motion information through frame alignment to replace the commonly used temporal attention operation, and so on. The most intuitive and easy-to-implement strategy is to use a learnable module designed for spatiotemporal modeling. However, these designed modules have limited effectiveness in capturing temporal clues in video data, so there is still much room for improvement. In addition, most of these methods mainly study the adaptability of CLIP in the zero-shot and few-shot video action recognition scenarios, while the adaptability of CLIP in the fully supervised recognition scenario in the video task field is relatively limited. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art, and to propose a simple and effective method: design a series of time modeling modules, namely the efficient combination of Transformer and LSTM (ECTL), to explore the potential of CLIP in improving the efficiency and universality of action recognition technology. ECTL modules connect LSTM and Transformer in a cascading manner, making time modeling more efficient by extracting local features and global dependencies in time series respectively. Through sufficient experiments, it is proved that the present application can be generalized to different video recognition scenarios and can remain competitive under full supervision, with top-1 accuracy on several public benchmark datasets comparable to or even higher than the latest technology.

[0006] A video action recognition method based on auxiliary time modeling module, the overall structure diagram is as shown in Figure 1 The method comprises the following steps:

[0007] S1. Preparation. Extract video frames from the video, and obtain text descriptions by filling the labels with video text prompts;

[0008] S2. Input the multiple video frames obtained in S1 into the image encoder of the CLIP pre-training to obtain multiple image features, and input the text description into the text encoder to obtain text features;

[0009] S3. Input the image features of the multiple video frames obtained in S2 into the key time modeling module in the present application for time modeling to capture time information and obtain video features;

[0010] S4. Calculate the similarity between the video features obtained in S3 and the text features obtained in S2. For each video sample, the class with the highest similarity score is selected as the predicted class, and the recognition accuracy is finally calculated.

[0011] Preferably, the preparation work in S1 includes pre-processing the original video, including uniformly sampling video frames according to a set frame number, and filling a text description about the label through a video text prompt.

[0012] Preferably, the step of generating image features using a pre-trained image encoder in S2 is as follows: given a video segment composed of T sampled frames where HxW represents the spatial resolution, and C represents the number of channels, which is 3 here. For each frame in the T sampled frames, take the t-th frame as an example, divide it into N non-overlapping PxP pixel patches t∈{1,…,T}, Then use linear projection to embed into patch embeddings. The calculation formula of the input embedding vector of the t-th frame is as follows:

[0013]

[0014] where the superscript of z represents the layer index, (0) represents the input layer, ε represents the matrix, n∈{1,…,N}, e spa_pos represents the learnable spatial position embedding. Finally, the original pre-trained image encoder E image will process the T frames of images in turn to generate frame-level visual representations (image features) I={I t |1≤t≤T}.

[0015] Preferably, the process of capturing time information between video frames in the time modeling module ECTL in S3 is as follows: for ECTL-A time modeling module, the frame-level representation is first input into LSTM, then the time position information is added on the output of LSTM, and finally it is input into Transformer to obtain video-level representation. The calculation formula is as follows:

[0016] H=[LSTM(I1,(h0,c0)),LSTM(I2,(h1,c1)),…,LSTM(I T ,(h T-1 ,c T-1 ))]#(2)

[0017]

[0018] where H is the output tensor of LSTM, e tem_posdenotes the learnable temporal position embedding, MSA is the multi-head self-attention layer of the Transformer, MLP is the multi-layer perceptron of the Transformer, and LN is the layer normalization layer of the Transformer.

[0019] For the ECTL-B temporal modeling module, the frame-level representation is first added with the temporal position information, and finally the frame-level representation is input into the module composed of LSTM and Transformer together to obtain the video-level representation. The calculation formula is as follows:

[0020]

[0021] H α = [LSTM(H α-1 , (h, c))] # (7)

[0022] wherein K represents the input of the module, W is a learnable matrix, H represents the hidden state matrix output by the LSTM layer, the subscript is the number of layers of the LSTM, h and c are the hidden state and cell state respectively. The output H α As the input, the output of the Transformer part is finally averaged and pooled to obtain the video-level representation.

[0023] For the ECTL-C temporal modeling module, the frame-level representation is first added with the temporal position information, and then input into the Transformer layer to extract the global temporal information, and finally the output of the Transformer part is input into the LSTM to perform local temporal modeling.

[0024] Preferably, the video-to-text and text-to-video similarity scores of S4 are calculated according to the following formula:

[0025]

[0026] wherein sim(x, y) and sim(y, x) represent the symmetric similarity between the two modalities, i.e. the cosine distance; τ is a learnable temperature parameter, and N is the number of training pairs. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is the network framework diagram of the application

[0028] Figure 2 is the module structure diagram of ECTL-A of the application

[0029] Figure 3 is the module structure diagram of ECTL-B of the application

[0030] Figure 4 is a module structure diagram of the ECTL-C of the present application DETAILED DESCRIPTION

[0031] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0032] The present application will be further described below in combination with the drawings.

[0033] A video action recognition method based on an auxiliary time modeling module, comprising the following steps:

[0034] S1. Preparation. Extract video frames from the video, and fill in the text description by labeling the video text prompt;

[0035] S2. Input the plurality of video frames obtained in S1 into the image encoder of the CLIP pre-training to obtain a plurality of image features, and input the text description into the text encoder to obtain text features;

[0036] S3. Input the image features of the plurality of video frames obtained in S2 into the key time modeling module in the present application to perform time modeling and capture time information, so as to obtain video features;

[0037] S4. Calculate the similarity between the video features obtained in S3 and the text features obtained in S2, and for the video sample, the class with the highest similarity score obtained is taken as the predicted class, and finally the recognition accuracy is calculated.

[0038] Now some English meanings in the present application will be explained. CLIP: Contrastive Language-Image Pre-training Model; embedding: embedding; patch: patch; MLP: Multi-Layer Perceptron; MSA: Multi-Head Self-Attention; Transformer: Transformer; LSTM: Long Short-Term Memory Network; ECTL: Time Modeling Module of Efficient Combination of Transformer and LSTM in the Present Application.

[0039] Dataset preparation:

[0040] The HMDB-51 dataset contains 6,849 video clips of 51 action categories, mainly from movies, established public databases, and online video platforms such as YouTube. The actions in the dataset include facial expressions such as smiling and laughing, as well as body movements such as climbing and diving. The UCF-101 dataset consists of 13320 video clips, which are samples from various sports samples collected from BBC / ESPN broadcast channels, as well as samples from the video website YouTube, a total of 101 categories, including makeup, musical instruments, sports, etc. The Something-Something V2 dataset is a large annotated video clip dataset that records basic actions between humans and some objects in daily life. The dataset contains 220,847 videos, with a total of 174 action categories labeled. In addition, this dataset places particular emphasis on temporal relationships, making it more challenging.

[0041] Parameters and experiments:

[0042] The optimizer used in the present application is the AdamW optimizer, the number of model training rounds is 50 epochs, and the weight decay is 0.2. The learning rate is preheated for the first 10% of the total training period, and the cosine strategy is used for decay to zero in the remaining training. In the present application, the CLIP model based on ViT-B / 16 and ViT-B / 32 is used to evaluate the universality of the present application, and the sparse sampling frame of each video is 8 frames to ensure computational efficiency. In addition, on the UCF-101 and HMDB-51 datasets, the basic learning rate of the pre-trained CLIP framework is 5x10 -6 , and the basic learning rate of the ECTL module with learnable parameters is 5x10 -5 , and the learning rate decay step is 15; on the Something-Something V2 dataset, the basic learning rate of the pre-trained CLIP framework is 5.5x10 -6 , and the basic learning rate of the ECTL module with learnable parameters is 25 times that of the pre-trained parameters, and the learning rate is decayed by the multistep strategy every 20 epochs. On the UCF-101 and HMDB-51 datasets, all the comparative methods and the method of the present application are trained using 1 NVIDIA V100 TensorCore GPU, and on the Something-Something V2 dataset, a NVIDIA GeForce RTX 4090 GPU is used for training. The Top-1 and Top-5 accuracies are used to evaluate the performance of the model

[0043] The action recognition method based on the ECTL time modeling module of the application is tested and evaluated together with various advanced methods on the HMDB-51, UCF-101 and Something-Something V2 three data sets, and the final results are shown in Tables 1-3.

[0044] As can be seen from Tables 1, 2 and 3, the ECTL-A, ECTL-B and ECTL-C of the application all achieve performance comparable to or even superior to the current most advanced methods of adjusting CLIP for video action recognition. In the case of the same type of image encoder, the top1 accuracy can be improved by an average of about 2% to 3%. The effect is also very significant on the UCF-101 data set. Although the ECTL series modules of the application are less effective than these most advanced excellent works on the Something-Something V2 data set, they are more excellent on the HMDB-51 and UCF-101 data sets. On the Something-Something V2 data set, the performance is significantly improved compared to the method of using only Transformer or LSTM for time modeling, which shows the progressiveness of our method.

[0045] Table 1 Recognition results of different recognition methods on the HMDB-51 data set

[0046]

[0047] Table 2 Recognition results of different recognition methods on the UCF-101 data set

[0048]

[0049]

[0050] Table 3 Recognition results of different recognition methods on the Something-Something V2 data set

[0051]

[0052] Table 4 Ablation experiment results on CLIP with ViT-B / 32 as the backbone

[0053]

[0054] A series of ablation experiments are performed to verify the effectiveness of the LSTM and the Transformer part in the video action recognition task based on the auxiliary time modeling module. The experimental results are shown in Table 4. Compared with only using the Transformer to model the relationship between video frames in time, the ECTL-A module designed by us increases the top1 accuracy by nearly 3%, 0.36% and 0.02% respectively on the HMDB-51, UCF-101 and Something-Something V2 data sets based on the CLIP framework with ViT-B / 32 as the backbone, and the parameter amount is only increased by 2.1M. Compared with only using LSTM to obtain the motion information in the video, the top1 accuracy is increased by 2.15%, 0.95% and 6.85% respectively. Compared with the baseline method, the performance is improved more, which is 1.7%, 0.58% and 20.21% respectively. The performance of the ECTL-B, ECTL-C and ECTL-A modules is similar. After using superposition, the performance of the model is improved, and the accuracy of the model is higher.

[0055] The comparative experiment results from the three data sets show that the ECTL time modeling module proposed in the application has better performance in recognition accuracy, and is better than several other classification networks.

[0056] The above embodiments are only used to illustrate the technical solutions of the application, rather than limit them. Although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some or all of the technical features can be replaced by equivalents, and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A video action recognition method based on an auxiliary time modeling module, characterized in that, The identification method includes: S1. Preparation. Extract video frames from the video, and fill in the text descriptions of the tags using video text prompts; S2. Input the multiple video frames obtained in S1 into the multiple image features of the CLIP pre-trained image encoder, and at the same time input the text description into the text encoder to obtain text features; S3. Input the image features of multiple video frames obtained in S2 into the key time modeling module of this invention to perform time modeling, capture time information, and obtain video features; S4. Calculate the similarity between the video features obtained in S3 and the text features obtained in S2. For the video sample, take the category with the highest similarity score as the predicted category, and finally calculate the recognition accuracy.

2. The video action recognition method based on an auxiliary time modeling module according to claim 1, characterized in that, The preparation work described in S1 is as follows: preprocessing the original video, including sampling and extracting video frames evenly according to a set number of frames, and filling the tags with video text prompts to obtain a text description about the tags.

3. The video action recognition method based on an auxiliary time modeling module according to claim 1, characterized in that, The step in S2, which uses a pre-trained image encoder to generate image features, is as follows: given a video segment consisting of T sampled frames. Where H×W represents the spatial resolution, and C represents the number of channels, which is 3 here. For each of the T sampled frames, taking the t-th frame as an example, it is divided into N non-overlapping P×P pixel groups. Then use linear projection Will Embedded into patch embeddings. The formula for calculating the input embedding vector for frame t is as follows: Where the superscript of z represents the layer index, (0) represents the input layer, ε represents the matrix, n∈{1,…,N}, and e spa_pos. This represents the learnable spatial location embedding. Finally, the original pre-trained image encoder E... image The system will process T frames of images sequentially to generate a frame-level visual representation (image features) I = {I t |1≤t≤T}.

4. The video action recognition method based on an auxiliary time modeling module according to claim 1, characterized in that, The process by which the ECTL time modeling module in S3 captures inter-frame temporal information in video is as follows: For the ECTL-A time modeling module, the frame-level representation is first input into the LSTM, then time position information is added to the LSTM output, and finally input into the Transformer to obtain the video-level representation. The calculation formula is as follows: H=[LSTM(I1,(h0,c0)),LSTM(I2,(h1,c1)),…,LSTM(I T ,(h T-1 ,c T-1 ))] Where H is the output tensor of the LSTM, e tem_pos. This represents the learnable temporal location embedding, MSA is the multi-head self-attention layer of the Transformer, MLP is the multilayer perceptron of the Transformer, and LN is the layer normalization layer of the Transformer. For the ECTL-B temporal modeling module, temporal location information is first added to the frame-level representation. Finally, the frame-level representation and the temporal location information are input together into a module composed of LSTM and Transformer to obtain the video-level representation. The calculation formula is as follows: Where K represents the module input, W is the learnable matrix, H represents the hidden state matrix output by the LSTM layer, the subscripts indicate the LSTM layer number, and h and c represent the hidden state and cell state, respectively. Output H α The input is passed through β transformer blocks, and the output of the transformer part is finally averaged and pooled to obtain the video-level representation. For the ECTL-C time modeling module, time location information is first added to the frame-level representation, and then input into the Transformer layer to extract global time information. Finally, the output of the Transformer part is input into the LSTM for local time modeling.

5. The video action recognition method based on an auxiliary time modeling module according to claim 1, characterized in that, The similarity scores between video and text and between text and video, as described in S4, are calculated using the following formula: Where sim(x,y) and sim(y,x) represent the symmetric similarity between two modes, i.e., the cosine distance; τ is the learnable temperature parameter; and N is the number of training pairs.

Images