T-CLIP video classification method, system and device based on time residual attention mechanism and storage medium

Abstract

The invention discloses a T-CLIP video classification method, system and device based on a time residual attention mechanism and a storage medium, and belongs to the technical field of multi-modal video data analysis, and the classification method comprises the following steps: obtaining an original video data set as a training set, and carrying out the data enhancement of videos and labels of the training set; establishing a video classification network model based on a time residual attention mechanism; preprocessing the enhanced data set, inputting the preprocessed enhanced data set into a video classification network for training, and outputting and storing an optimal optimizer parameter and a network weight; analyzing character behaviors of the input video by using the trained model, and matching action labels for classification; the video classification network based on the time residual attention mechanism comprises a video sampling module, a video coding module, a text coding module and a discriminator module. The T-CLIP video classification method provided by the invention has strong zero sample video action recognition capability, and can efficiently and accurately classify according to the video content.

Description

Technical Field

[0001] The present invention relates to a T-CLIP video classification method, system, device and storage medium based on a temporal residual attention mechanism, and belongs to the technical field of multimodal video data analysis. Background Art

[0002] With the rapid growth of Internet video content, video classification technology has become increasingly important in various fields. For example, in the field of security monitoring, suspicious behaviors or abnormal activities can be automatically identified by analyzing surveillance videos, and alarms can be issued in a timely manner. In the field of intelligent transportation, traffic accidents can be quickly identified through video classification, and relevant departments can be automatically notified to improve the efficiency of accident handling. Illegal road behaviors such as running red lights and driving against traffic can be automatically recorded and evidence of violations can be generated. Traditional video classification methods often rely on manually extracted features, which makes it difficult to effectively capture the temporal information and semantic content in the video. In recent years, the rapid development of deep learning has provided new solutions for video classification, especially models based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

[0003] In the field of CNN-based video classification, D.Tran et al. proposed a 3D CNN that uses three-dimensional filters to perform convolution. The convolution kernel is applied to the three dimensions of the video to capture both spatial and temporal features, but its classification performance decreases when the video content contains complex actions. Karen Simonyan proposed to build two independent convolutional neural networks to process video data. One network processes RGB frames to extract spatial features, and the other network processes optical flow maps to capture motion information. By combining the outputs of the two, this method can achieve significant results in complex action classification. C.Feichtenhofer proposed to divide the video frames into two streams: slow stream and fast stream. The slow stream runs at a lower frame rate and is responsible for capturing spatial semantic information; while the fast stream runs at a higher frame rate and is responsible for capturing motion information. This method improves the performance of capturing video diversity while maintaining significant performance in complex action classification.

[0004] In the field of RNN-based video classification, Joe Yue-Hei uses a bidirectional long short-term memory network (LSTM) to encode all frame features of the video in sequence. The model adds an attention layer, and the hidden state output at each moment has an adaptive weight, which is then linearly weighted to obtain the final feature vector. Zhao proposed a network that is "double deep" in the spatial and temporal layers. In the feature extraction stage, the long-distance dependencies with deep convolutional neural networks are captured with improved p-non-local operations. In the class prediction stage, we proposed a hybrid unkeyed attention combined with forward and backward bidirectional LSTM, using a multi-period model fusion based on the confusion matrix to efficiently learn the sequential nature of the data.

[0005] However, existing methods face challenges in zero-shot learning, which requires deep neural networks to recognize new unseen classes during testing without seeing them during training. CNNs and RNNs often rely on task-specific training data and have poor generalization capabilities. The generalization ability to classify new classes without manual annotation makes it of great significance in practical applications, so it is necessary to build a video classification model with strong zero-shot learning capabilities. Summary of the invention

[0006] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned prior art and propose a T-CLIP video classification method, system, device and storage medium based on the temporal residual attention mechanism. By performing data enhancement processing on the original data set, the diversity of the video is increased and the problem of overfitting in training is solved. By improving the classic CLIP algorithm and utilizing the image-text knowledge space of CLIP, it not only learns the image visual information, but also learns the related text semantics, and has a strong generalization ability to improve the accuracy and generalization of video classification.

[0007] The technical solution adopted by the present invention to solve its technical problems is: a T-CLIP video classification system based on a temporal residual attention mechanism, the system comprising a temporal sampling module, a video encoding module, a text encoding module and a discriminator module, which is used to train a training set that has undergone image enhancement processing until the video and label classification results converge.

[0008] The present invention also proposes a T-CLIP video classification method based on a temporal residual attention mechanism, the method comprising the following steps:

[0009] Step 1: Get the original video dataset as the training set, perform data augmentation on the videos and labels of the training set, mix the videos in the spatial dimension, and mix the corresponding labels according to the voxel ratio of the video;

[0010] Step 2: Establish a video classification network model based on the temporal residual attention mechanism;

[0011] Step 3: Input the enhanced dataset into the video classification network for training, and output and save the optimal optimizer parameters and network weights;

[0012] Step 4: Use the trained model to analyze the behavior of the characters in the input video and classify the video by matching the action labels;

[0013] Furthermore, the data augmentation method in step 1 includes: randomly selecting two or more video samples in the video data set, cutting and pasting the selected video clips to form a new video cube, mixing the corresponding labels of each video clip in proportion according to the number of voxels, generating new labels as a new training set, and setting x∈R T*W*H is a video sequence, where T, W and H represent the number of frames, width and height of the video sequence respectively, let y be the video label, and the enhancement formula for two training samples (x1, y1) and (x2, y2) is:

[0014] x * =M⊙x1+(1-M)⊙x2

[0015] y * =λ M y1+(1-λ M )y2

[0016] where ⊙ is the element-wise dot product, is the voxel proportion, M∈{0,1} T*W*H A binary tensor mask.

[0017] Furthermore, the training method of the T-CLIP video classification network model in step 3 includes the following steps:

[0018] Step 3-1: Input the enhanced video data set into the network for preprocessing, and perform temporal downsampling of the video samples according to the sampling rate α to obtain a video frame sequence;

[0019] Step 3-2: Extract the video feature vector containing temporal features through the multi-layer improved Transformer block and residual attention mechanism of the video encoding module, denoted as VF;

[0020] Step 3-3: Input the enhanced video text label into the text encoding module to extract the text feature vector of the video label, denoted as TF;

[0021] Step 3-4: Input VF and TF into the discriminator for L2 normalization and calculate the loss function to determine the text label that best matches the training sample;

[0022] Step 3-5: Calculate the contrast loss function based on the classified text labels obtained by the discriminator and the true text labels, and use it to update the parameters of the video classification network. After the model parameters converge, save the optimal network model parameters at this time.

[0023] Furthermore, in step 3-2, the video encoding module divides each video frame into a plurality of frame blocks of size P*P*C, and flattens these frame blocks into P 2*C; each flattened frame block is mapped to a higher-dimensional feature space through a fully connected layer, and spatial encoding is added to maintain the spatial position information of the frame block. The mapped sequence features are input into T-ViT, and processed by multiple layers of Transformer blocks. The features of different layers are integrated, and the residual attention mechanism retains the low-level features. Finally, the fused video feature vector VF is obtained;

[0024] The residual attention mechanism adds the feature input of each layer to the output of the temporal self-attention layer in the Transformer block, thereby retaining the low-level feature information. The calculation formula of residual attention is:

[0025] R s ATT t，f (X i )=ATT t，f (X i )+X i

[0026] Among them ATT f，t is the attention output of the t-th frame block of the f-th frame, X i is the input feature of the lth layer.

[0027] Furthermore, the temporal self-attention layer expands the temporal attention view of the traditional self-attention layer, so that each frame block can obtain information from its own frame and temporal adjacent frames to aggregate global temporal information and inject temporal modeling capabilities into the network. The extended formula is:

[0028]

[0029] Among them ATT f，t is the attention output of the tth tag in the fth frame, d represents the dimension of the vector, Q t，f represents the query vector of the tth tag in the fth frame, K (f-1～f+1) and V (f-1～f+1) Represents a key value matrix composed of f-1 to f+1 frame key value vectors. Further, after the text encoding module in step 3-3 preprocesses the input text, it uses the pre-trained BPE (byte pair encoding) to divide the input label text into smaller text tokens, looks up the word embedding table to convert each token into a word embedding, and adds position encoding; the embedding sequence is encoded through a multi-layer Transformer, and finally the output of the first token is used as the global representation of the text label, that is, the text feature vector TF.

[0030] Further, the step 3-4 includes: the discriminator module uses VF and TF to calculate the loss function, L2 normalization is performed when the feature input is processed, the loss function is cosine similarity, let v1 be the video feature vector, v2 be the text feature vector, since the feature vector has a modulus length of 1 after L2 normalization, the cosine similarity calculation formula can be simplified to:

[0031] cosin_simalarity(v1,v2)=v1·v2.

[0032] The present invention also proposes an electronic device, which is loaded with a processor and a memory storing a plurality of instructions, and the processor is capable of reading the instructions and being used to execute a T-CLIP video classification method based on a temporal residual attention mechanism of the present invention.

[0033] The present invention also proposes a readable storage medium, which stores a computer program internally. When the stored program is executed by a processor, a T-CLIP video classification method based on a temporal residual attention mechanism of the present invention can be implemented.

[0034] Beneficial effects:

[0035] 1. The present invention discloses a T-CLIP video classification method based on a temporal residual attention mechanism. By performing data enhancement processing on the original data set, the diversity of the video is increased and the problem of overfitting in training is solved. By improving the classic CLIP algorithm and utilizing the image-text knowledge space of CLIP, the method not only learns the image visual information but also learns the related text semantics and has a strong generalization ability. The self-attention layer of the Transformer block in the CLIP algorithm is expanded in the time domain, the temporal modeling capability is injected, the temporal features are aggregated, and the migration from image classification to video classification is realized. By introducing the residual attention mechanism, the input features are added to the features after the self-attention processing, which helps to retain the diversity of the input features and avoids the problem of feature collapse caused by the decrease in the ability of the model to extract and retain diversified features as the network depth increases. By using the BPE algorithm in the text encoding module, the model's ability to process unfamiliar words is enhanced, while the vocabulary of training is reduced, and the computational and memory burdens are reduced.

[0036] 2. The optimal T-CLIP video classification model after training in the present invention can classify videos with high accuracy, and has strong practicality and huge potential for development in the fields of security monitoring and intelligent transportation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the T-CLIP video classification method based on the temporal residual attention mechanism of the present invention.

[0038] Figure 2 This is a schematic diagram of the data enhancement algorithm used in the present invention.

[0039] Figure 3 It is a schematic diagram of the structure of the video sampling module of the present invention.

[0040] Figure 4 It is a schematic diagram of the structure of the video encoding module of the present invention.

[0041] Figure 5 It is a structural schematic diagram of the text encoding module of the present invention.

[0042] Specific implementation

[0043] In order to fully understand the design purpose, technical solution and beneficial effects of the present invention, the invention will be further described in detail below in conjunction with the accompanying drawings. All technical and scientific terms used in this specification have the meanings commonly understood by ordinary practitioners in the relevant field.

[0044] like Figures 1 to 5 As shown, the present invention proposes a T-CLIP video classification method, system, device and storage medium based on the temporal residual attention mechanism, which makes full use of the powerful zero-sample classification ability of the CLIP basic model, and injects temporal modeling ability into the model by improving the video encoding module T-ViT, using the Transformer block extended in the time domain, and migrating the CLIP basic model from image classification to video classification. The residual attention mechanism is introduced to add the input features to the features after self-attention processing, which retains the diversity of the input features and avoids the problem of feature collapse caused by the decrease in the ability of the model to extract and retain diversified features as the network depth increases. In addition, the present invention uses the public video datasets Kinetics-400 and UCF101, and improves the diversity of the training set through data enhancement, thereby improving the generalization performance of the model and solving the overfitting problem. The principle is as follows: Figure 2 The specific design process of the T-CLIP video classification method based on the temporal residual attention mechanism of the present invention includes the following steps:

[0045] Step 1: Obtain a public video dataset as a training set, which contains the corresponding video content and text labels, as follows Figure 2 As shown in the figure, two video samples are randomly selected from the video dataset, and the selected video clips are cut and mixed to form a new video cube. The corresponding labels of each video clip are mixed in proportion according to the number of voxels, and new labels are generated as the new training set. Let x∈R T*W*H is a video sequence, where T, W and H represent the number of frames, width and height of the video sequence respectively, let y be the video label, and the enhancement formula for two training samples (x1, y1) and (x2, y2) is:

[0046] x * =M⊙x1+(1-M)⊙x2

[0047] y * =λ M y1+(1-λ M )y2

[0048] where ⊙ is the element-wise dot product, is the voxel proportion, M∈{0,1} T*W*H A binary tensor mask.

[0049] Step 2: Establish a T-CLIP video classification network model based on the temporal residual attention mechanism. The video classification network model structure is as follows: Figure 1 As shown, it includes a video sampling module, a video encoding module, a text encoding module and a discriminator module, which are used to train the training set after image enhancement processing until the video and label classification results converge.

[0050] The video sampling module performs temporal sampling on the video samples after data enhancement. The module structure is as follows: Figure 3 As shown, the enhanced video data set is input into the network for preprocessing, and the video samples are temporally downsampled according to the sampling rate α to obtain a video frame sequence;

[0051] The improvements to the video encoding module include the temporal expansion of the self-attention layer and the introduction of the residual attention mechanism. Figure 4 As shown in the figure, the specific process is as follows: First, each video frame is divided into multiple frame blocks of size P*P*C, and these frame blocks are flattened into P 2 *C vector; each flattened frame block is mapped to a higher-dimensional feature space through a trainable fully connected layer. In this process, spatial encoding is added to maintain the spatial position information of the frame block, and finally the image block embedding is obtained. The mapped sequence features are input to T-ViT and processed by a multi-layer Transformer block. In this process, the residual attention mechanism adds the feature input a of each layer and the output b of the temporal self-attention layer in the Transformer block, and outputs the mixed feature c, so that the low-level feature information can still be retained after each Transformer block, preventing the attention map from becoming gradually similar as the model depth increases, and solving the limitation of feature collapse. The calculation formula of the residual attention mechanism is:

[0052] ResATT t，f (X i )=ATT t，f (X i )+X i

[0053] Among them ATT f，t is the attention output of the t-th frame block of the f-th frame, X l is the input feature of the lth layer. The present invention performs time domain expansion on the self-attention layer of the Transformer block, so that each frame block can obtain information from the video frame to which it belongs and the adjacent frames in the time domain. The time domain expansion formula is:

[0054]

[0055] Where d represents the dimension of the vector, Q t，f represents the query vector of the tth frame block of the fth frame, K (f-1～f+1) and V (f-1～f+1) They represent the key matrix and value matrix composed of the key-value vectors of frames f-1 to f+1 respectively. The video frame is processed by the video encoding module and finally the fused video feature vector VF is obtained;

[0056] After preprocessing the input text, the text encoding module uses the pre-trained BPE (byte pair encoding) to split the input label text into smaller text tokens, looks up the word embedding table to convert each token into a word embedding, and adds position encoding; the embedding sequence is encoded through a multi-layer Transformer, and finally the output of the first token is used as the global representation of the text label, that is, the text feature vector TF, with the following structure: Figure 5 shown.

[0057] The discriminator module uses the output VF of the video encoding module and the output TF of the text encoding module to calculate the loss function. The feature input is L2 normalized, and the loss function is calculated by cosine similarity. Let v1 be the video feature vector and v2 be the text feature vector. Since the feature vector has a modulus length of 1 after L2 normalization, the calculation formula is simplified to:

[0058] cosin_simalarity(v1, v2)=v1·v2, where cosin_simalarity(v1, v2) is the cosine similarity. The closer the value is to 1, the higher the matching degree between the video and the label.

[0059] Step 3: The original data set is used as the training set, and after being processed by the image enhancement module, it is sent to the T-CLIP video classification network for training. The network calculates the gradient value of the model parameters with respect to the loss in each batch through back propagation, and then uses the Adam optimizer to update the model parameters according to the gradient information of the network parameters. The loss function value is continuously reduced through iteration until convergence, and the optimal T-CLIP video classification network model and its parameters are saved at this time.

[0060] Step 4: After the training results converge, the optimal network model parameters are imported into the T-CLIP video classification network, and a new video dataset (such as UCF101) is selected as the test set to verify the zero-sample classification performance of the model.

[0061] The present invention also discloses a T-CLIP video classification system based on a temporal residual attention mechanism, comprising: a data enhancement module for performing data enhancement on a video; a T-CLIP video classification network model, wherein the T-CLIP video classification network model comprises a video sampling module, a video encoding module, a text encoding module and a discriminator module;

[0062] The training method of the T-CLIP video classification network model of the present invention comprises:

[0063] Step 3-1: Input the enhanced video data set into the network for preprocessing, and perform temporal downsampling of the video samples according to the sampling rate α to obtain a video frame sequence;

[0064] Step 3-2: Extract the video feature vector containing temporal features through the multi-layer improved Transformer block and residual attention mechanism of the video encoding module, denoted as VF;

[0065] Step 3-3: Input the enhanced video text label into the text encoding module to extract the text feature vector of the video label, denoted as TF;

[0066] Step 3-4: Input VF and TF into the discriminator for L2 normalization and calculate the loss function to determine the text label that best matches the training sample;

[0067] Step 3-5: Calculate the contrast loss function based on the classified text labels obtained by the discriminator and the true text labels, and use it to update the parameters of the video classification network. After the model parameters converge, save the optimal network model parameters at this time.

[0068] The present invention also proposes an electronic device, which is loaded with a processor and a memory storing a plurality of instructions. The processor can read the instructions and is used to execute a T-CLIP video classification method based on a temporal residual attention mechanism of the present invention.

[0069] The present invention also proposes a readable storage medium that stores a computer program internally. When the program is executed by a processor, it can implement a T-CLIP video classification method based on a temporal residual attention mechanism of the present invention.

[0070] The simulation experiment and data of the present invention are as follows:

[0071] (1) Experimental content

[0072] Zero-sample classification refers to the classification of unseen categories. In order to demonstrate the zero-sample video classification capability of the present invention, two classic video datasets are used as a training set (Kinetics-400) and a test set (UCF101).

[0073] Experiment 1: Video classification based on PS-GNN model of graph neural network;

[0074] Experiment 2: Video classification based on the E2E model of 3D convolutional neural network;

[0075] Experiment 3: Video classification based on the REST model of ResNet101;

[0076] Experiment 4: Video classification based on TEXT4VIS model of semantic transfer learning;

[0077] Experiment 5: Video classification based on T-CLIP model with temporal residual attention mechanism;

[0078] (2) Experimental results

[0079] The experiment gives the accuracy indicators of the classification performance of the above five methods on the test set UCF101. The closer the value is to 1, the stronger the zero-sample video classification performance of the model is. The specific results are shown in Table 1:

[0080] Table 1 Zero-shot classification performance of different algorithms on UCF101

[0081] Comparative experiment Classification Accuracy Experiment 1 36.1％ Experiment 2 48.0％ Experiment 3 58.7％ Experiment 4 85.8％ Experiment 5 (this method) 89.9％

[0082] From the data in Table 1, it can be seen that the T-CLIP model based on the temporal residual attention mechanism proposed in the present invention has the highest zero-sample classification accuracy and the best performance, which illustrates the improvement in generalization performance brought about by the introduction of the residual attention mechanism in the present invention.

[0083] In summary, the T-CLIP method and storage medium based on the temporal residual attention mechanism firstly uses the data augmentation algorithm to solve the problem of overfitting in model training; secondly, it realizes the migration of zero-shot learning ability from the image domain to the video domain, making full use of the powerful knowledge base of the basic CLIP model; in addition, the residual attention mechanism is introduced to retain the diversity of input features, avoid the problem of feature collapse, and enhance the generalization performance of the model.

[0084] The embodiments described above are only preferred examples of the present invention and are not intended to limit the use of other forms of the present invention. However, those skilled in the art should understand that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and scope of the present invention. The protection scope of the present invention is not limited to the above-mentioned embodiments. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention. All features disclosed in this specification, or all methods or steps in the process disclosed, except for mutually exclusive features, methods or process steps, can be combined in any way. The present invention is not limited to any specific embodiment, but should be understood to include the full scope defined by the claims and all equivalents.

Claims

1. A T-CLIP video classification system based on temporal residual attention mechanism, characterized in that: The system includes a time-series sampling module, a video encoding module, a text encoding module and a discriminator module, which are used to train a training set that has undergone image enhancement processing until the video and label classification results converge.

2. A T-CLIP video classification method based on temporal residual attention mechanism, characterized in that: The method comprises the following steps: Step 1: Get the original video dataset as the training set, perform data augmentation on the videos and labels of the training set, mix the videos in the spatial dimension, and mix the corresponding labels according to the voxel ratio of the video; Step 2: Establish a video classification network model based on the temporal residual attention mechanism; Step 3: Input the enhanced dataset into the video classification network for training, and output and save the optimal optimizer parameters and network weights; Step 4: Use the trained model to analyze the character behaviors in the input video and classify the video by matching action labels.

3. A T-CLIP video classification method based on temporal residual attention mechanism according to claim 2, characterized in that: The data augmentation method in step 1 includes: randomly selecting two or more video samples in the video data set, cutting and pasting the selected video clips to form a new video cube, mixing the corresponding labels of each video clip in proportion according to the number of voxels, generating new labels as a new training set, and setting x∈R T*W*H is a video sequence, where T, W and H represent the number of frames, width and height of the video sequence respectively, let y be the video label, and the enhancement formula for two training samples (x1, y1) and (x2, y2) is: x * =M⊙x1+(1-M)⊙x2 y * =λ M y1+(1-λ M )y2 where ⊙ is the element-wise dot product, is the voxel proportion, M∈{0,1} T*W*H A binary tensor mask.

4. A T-CLIP video classification method based on temporal residual attention mechanism according to claim 2, characterized in that: The training method of the T-CLIP video classification network model in step 3 comprises the following steps: Step 3-1: Input the enhanced video data set into the network for preprocessing, and perform temporal downsampling of the video samples according to the sampling rate α to obtain a video frame sequence; Step 3-2: Extract the video feature vector containing temporal features through the multi-layer improved Transformer block and residual attention mechanism of the video encoding module, denoted as VF; Step 3-3: Input the enhanced video text label into the text encoding module to extract the text feature vector of the video label, denoted as TF; Step 3-4: Input VF and TF into the discriminator for L2 normalization and calculate the loss function to determine the text label that best matches the training sample; Step 3-5: Calculate the contrast loss function based on the classified text labels obtained by the discriminator and the true text labels, and use it to update the parameters of the video classification network. After the model parameters converge, save the optimal network model parameters at this time.

5. A T-CLIP video classification method based on temporal residual attention mechanism according to claim 4, characterized in that: In step 3-2, the video encoding module divides each video frame into a plurality of frame blocks of size P*P*C, and flattens these frame blocks into P 2 *C, maps each flattened frame block to a higher-dimensional feature space through a fully connected layer, adds spatial encoding to maintain the spatial position information of the frame block, inputs the mapped sequence features into T-ViT, and is processed by multiple layers of Transformer blocks. The features of different layers are integrated, and the residual attention mechanism retains the low-level features. Finally, the fused video feature vector VF is obtained; The residual attention mechanism includes: adding the feature input of each layer and the output of the temporal self-attention layer in the Transformer block to retain the low-level feature information. The calculation formula of residual attention is: Travel t，f (X l )=AT t，f (X l )+X l Among them ATT f，t is the attention output of the t-th frame block of the f-th frame, X l is the input feature of the lth layer.

6. A T-CLIP video classification method based on temporal residual attention mechanism according to claim 5, characterized in that: The temporal self-attention layer expands the temporal attention view of the traditional self-attention layer, so that each frame block can obtain information from its own frame and adjacent frames in the temporal domain to aggregate global temporal information and inject temporal modeling capabilities into the network. The extended formula is: Among them ATT f，t is the attention output of the tth tag in the fth frame, d represents the dimension of the vector, Q t，f represents the query vector of the tth tag in the fth frame, K (f-1～f+1) and V (f-1～f+1) Represents the key value matrix composed of f-1 to f+1 frame key value vectors.

7. The T-CLIP video classification method based on the temporal residual attention mechanism according to claim 4, characterized in that: After the text encoding module in step 3-3 preprocesses the input text, it uses the pre-trained BPE to split the input label text into smaller text tokens, looks up the word embedding table to convert each token into a word embedding, and adds position encoding. The embedding sequence is encoded through a multi-layer Transformer, and finally the output of the first token is used as the global representation of the text label, that is, the text feature vector TF.

8. The T-CLIP video classification method based on the temporal residual attention mechanism according to claim 4, characterized in that: The step 3-4 includes: the discriminator module calculates the loss function using VF and TF, performs L2 normalization processing when the feature input is processed, and the loss function is calculated by cosine similarity. Let v1 be the video feature vector and v2 be the text feature vector. Since the modulus length of the feature vector is 1 after L2 normalization, the calculation formula of cosine similarity is simplified to: cosin_simalarity(v1,v2)=v1·v2.

9. An electronic device, characterized in that: The device is loaded with a processor and a memory storing a plurality of instructions, and the processor can read the instructions and be used to execute any one of the methods of claims 2-7.

10. A readable storage medium storing a computer program, characterized in that: When the stored program is executed by the processor, it can implement the method as described in any one of claims 2-7.

Images