Video identification method based on dynamic space-time full-scale feature fusion

Abstract

The invention discloses a video recognition method based on dynamic space-time full-scale feature fusion, which comprises the following steps: a video data set used for training a deep neural network model is acquired, the video data set comprises a plurality of video sequences, and each video sequence is composed of a plurality of time frames; training a pre-defined video recognition deep neural network model by using the video data set, wherein the video recognition deep neural network model comprises a parallel space-time adapter and a self-prompt generator; and applying the trained video recognition deep neural network model to a video recognition task, inputting a to-be-recognized video sequence, and outputting a classification prediction result of the video. The method has the advantages that through cooperative work of the parallel space-time adapter and the self-prompt generator, efficient modeling of dynamic changes in the video is achieved, the accuracy and robustness of video recognition are remarkably improved, and the method is suitable for various video recognition tasks such as supervised learning, few-sample learning and zero-sample recognition.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a video recognition method based on dynamic spatiotemporal full-scale feature fusion. Background Art

[0002] With the popularity of the Internet and smart devices, the amount of video data has exploded, and video recognition technology has become a core task in the field of computer vision. In recent years, vision-language models (VLMs) such as CLIP and ALIGN have shown excellent performance in image recognition and natural language processing tasks due to their ability to be pre-trained on large-scale datasets. However, there are several major limitations when directly applying these models to video recognition: 1. Mismatch between static images and dynamic videos: Existing vision-language models such as CLIP are mainly trained for pairing static images and texts, and they perform well in extracting spatial features. However, video data has a continuous time dimension, and the changes in spatial and temporal features need to be considered simultaneously. Since models such as CLIP lack temporal modeling capabilities, direct application to video recognition will result in insufficient tracking of dynamic objects and insufficient ability to capture inter-frame continuity. 2. The computational efficiency of existing spatiotemporal embedding methods is low: To address this problem, some studies have introduced spatiotemporal embedding models (such as ST-Adapter and VitaCLIP), which achieve video modeling by embedding temporal information into the spatial features of the model. Although this method can capture the motion and detail changes in the video, it requires stacking multiple spatiotemporal modules in the model, which has high computational complexity, resulting in low efficiency of model training and reasoning, and is not suitable for practical application scenarios with limited resources. 3. Insufficient spatial-temporal information fusion of side spatiotemporal models: Another type of method, such as EVL, uses side spatiotemporal modeling, that is, temporal modeling independent of spatial features. This method improves computational efficiency to a certain extent, but because it does not effectively integrate the impact of temporal dynamics on spatial details, it is easy to ignore the interaction between spatial and temporal information in the video, resulting in unsatisfactory recognition results. 4. Insufficient processing of dynamic spatiotemporal features: Existing models usually perform poorly in processing complex nonlinear spatiotemporal patterns. For example, when dealing with irregular motion or changes in spatial structure within a time frame, these models find it difficult to achieve a comprehensive understanding and efficient modeling of spatiotemporal features. Therefore, how to seamlessly integrate information at the spatial, temporal, and space-time scales has become an urgent problem to be solved.

[0003] In summary, existing video recognition methods have obvious limitations in processing the dynamic characteristics and complex spatiotemporal changes of video data, and cannot fully utilize the multi-scale information in the video for accurate recognition. Therefore, there is an urgent need for a new method that can efficiently integrate spatiotemporal information and improve the accuracy and robustness of video recognition. Summary of the invention

[0004] The present invention proposes a video recognition method based on dynamic spatiotemporal full-scale feature fusion, which realizes comprehensive modeling in space and time scales by designing parallel spatiotemporal adapter (PTA) and self-prompt generator (SPG) modules. PTA is used for efficient temporal modeling, and SPG is used for dynamically capturing spatial features. The two work together, so that the method of the present invention can effectively fuse spatiotemporal information in dynamic changes between frames.

[0005] A video recognition method based on dynamic spatiotemporal full-scale feature fusion, comprising the following steps:

[0006] S1. Obtain a video dataset for training a deep neural network model, wherein the video dataset includes multiple video sequences, and each video sequence consists of multiple time frames;

[0007] S2. Using the video data set to train a predefined video recognition deep neural network model, the video recognition deep neural network model includes a parallel spatiotemporal adapter and a self-prompt generator;

[0008] S3. Apply the trained video recognition deep neural network model to the video recognition task, input the video sequence to be recognized, and output the classification prediction result of the video.

[0009] Furthermore, in step S1, the video data set is (x1, y1, t1), (x2, y2, t2), ..., (x N ,y N ,t N ), N is the total number of videos in the video dataset, x i represents the i-th video, y i Indicates the class label corresponding to the i-th video, t i Indicates the text name corresponding to the class label, y i ∈{1,…,C}, C represents the total number of categories, and each category contains multiple videos.

[0010] Furthermore, the parallel spatiotemporal adapter is predefined as a self-attention layer wrapped by an adapter, which is used to model the temporal interaction ability of the same position between different frames; the self-hint generator is predefined as a module constructed by a pooling layer and a multi-layer linear layer, which is used to extract spatial information of different resolutions of video frames.

[0011] Furthermore, the parallel spatiotemporal adapter adopts a self-attention mechanism and uses the following formula to realize temporal information modeling:

[0012]

[0013] Among them, PTA stands for Parallel Spatiotemporal Adapter, which is used for temporal feature extraction; V i-1 is the visual token of the i-1th layer, N bs represents the batch size, T represents the number of frames, K represents the number of patches, D represents the feature dimension, Down(V i-1 ) indicates that V is converted to i-1 Projected to a low-dimensional space, Attn represents the self-attention layer, and Up represents the use of an upsampling projector to restore the refined temporal features. Represents the temporal feature output of the i-th layer.

[0014] Furthermore, the self-hint generator extracts multi-scale features from the video input using the following formula:

[0015]

[0016] in, represents the spatial feature map of the i-th layer, Avg represents the average pooling operation, and Projector represents the projection operation of the multi-layer linear layer;

[0017] The self-hint generator is also used to convert R i and Perform splicing operations to form new

[0018]

[0019] Furthermore, the video recognition deep neural network model realizes spatiotemporal feature fusion through the following formula:

[0020]

[0021] Among them, α is a learnable weight parameter used to balance spatial and temporal features.

[0022] Furthermore, step S2 specifically includes:

[0023] S21. Randomly sample a batch of video samples from the video dataset The batch size N bs given in advance;

[0024] S22, input the videos in the video sample set B into the video recognition deep neural network model to obtain the corresponding video representation output:

[0025]

[0026] in Represents video sample xi The output feature representation of is 1*D, where D represents the feature dimension;

[0027] S23, the text name corresponding to the class label t i Input into the preset text encoder to obtain text space representations for all categories:

[0028]

[0029] A textual spatial representation of each category;

[0030] S24, constructing a contrastive learning loss function based on the video representation output obtained in S22 and the text space representation obtained in S23, and using an Adam optimizer or a stochastic gradient descent optimizer with momentum, and a back propagation algorithm to train the video recognition deep neural network model;

[0031] S25. Repeat steps S21 to S24 until the model converges.

[0032] Furthermore, in step S24, the method for constructing the contrastive learning loss function is as follows:

[0033] Based on video sample x i Video representation output and the corresponding category y j The text space representation of , calculates the semantic similarity vector:

[0034]

[0035] in express and The inner product operation of Represents the output feature representation The L2 norm of Representing text space The L2 norm of ;

[0036] The entire video recognition deep neural network model is trained by optimizing the visual-textual semantic similarity when the video and text belong to the same category, and minimizing the semantic similarity when they belong to different categories; therefore, the contrastive learning loss function is defined as:

[0037]

[0038] Among them, SM represents the softmax function.

[0039] Further, in step S3, the video belongs to category y c The probability is expressed as:

[0040]

[0041] in, Represents the video feature output F V and the i-th text space representation The cosine similarity between ;

[0042] The similarity prediction label of the video and text, that is, the classification prediction result of the video, is:

[0043] y pred =argmax(P(y c |F V ))

[0044] Among them, argmax is the operation of finding the maximum value.

[0045] Compared with the existing video recognition methods, the video recognition method based on dynamic spatiotemporal full-scale feature fusion of the present invention has the following beneficial effects:

[0046] Firstly, the present invention designs an efficient video recognition method, so that the video model can make full use of the graphic model and the trained weight information, thereby achieving better results in video time series modeling;

[0047] Secondly, through the design of parallel spatiotemporal adapter (PTA) and self-prompt generator (SPG), the present invention can effectively extract features from different time frames and spatial resolutions; PTA can capture the temporal changes across frames and enhance the modeling ability of the model for temporal features; SPG improves the capture of spatial features at different resolutions through multi-scale pooling technology, ensuring the robustness to changes in object appearance and shape;

[0048] Finally, by comparing video representation and text representation, the present invention achieves efficient visual-text matching. During the training process, the model maximizes the video-text similarity of the correct category and minimizes the similarity of the wrong category, thereby improving the model's ability to distinguish videos of different categories. At the same time, the CLIP-based pre-trained text encoder further enhances the model's effect in multimodal information fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of the video recognition method based on dynamic spatiotemporal full-scale feature fusion of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0052] refer to Figure 1 In a preferred embodiment of the present invention, a video recognition method based on dynamic spatiotemporal full-scale feature fusion comprises the following steps:

[0053] The first step is to obtain a video dataset for training a deep neural network model. The dataset contains multiple video sequences, each of which consists of multiple time frames.

[0054] As a preferred implementation scheme, in this embodiment, the sample format of the video data set is (x1, y1, t1), (x2, y2, t2), …, (x N ,y N ,t N ), N is the total number of videos in the video dataset, x i represents the i-th video, y i Indicates the class label corresponding to the i-th video, t i Indicates the text name corresponding to the class label, y i ∈{1,…,C}, C represents the total number of categories, and each category contains multiple videos.

[0055] In the second step, the video dataset is used to train a pre-defined video recognition deep neural network model, which contains a parallel spatiotemporal adapter (PTA) and a self-prompt generator (SPG) to model the spatiotemporal features of the video content.

[0056] As a preferred implementation, in this embodiment, a parallel spatiotemporal adapter (PTA) is predefined as an adapter-wrapped self-attention layer and is encapsulated in a bottleneck structure comprising downsampling and upsampling projections to model the temporal interaction capabilities of the same position between different frames.

[0057] The Self-Prompt Generator (SPG) is pre-defined as a module built of a pooling layer and a multi-linear layer (MLP) to extract spatial information of different resolutions of video frames.

[0058] Specifically, the parallel spatiotemporal adapter adopts the self-attention mechanism and uses the following formula to realize temporal information modeling:

[0059]

[0060] Among them, PTA stands for Parallel Spatiotemporal Adapter, which is used for temporal feature extraction; V i-1 is the visual token of the i-1th layer, N bs represents the batch size, T represents the number of frames, K represents the number of patches, D represents the feature dimension, Down(V i-1 ) indicates that V is converted to i-1 Projected to a low-dimensional space, Attn represents the self-attention layer, and Up represents the use of an upsampling projector to restore the refined temporal features. Represents the temporal feature output of the i-th layer.

[0061] The self-hint generator extracts multi-scale features from the video input using the following formula:

[0062]

[0063] in, represents the spatial feature map of the i-th layer, Avg represents the average pooling operation, and Projector represents the projection operation of the multi-layer linear layer;

[0064] The self-prompting generator is also used to convert R i and Perform splicing operations to form new

[0065]

[0066] Therefore, the video recognition deep neural network model can achieve spatiotemporal feature fusion through the following formula:

[0067]

[0068] Among them, α is a learnable weight parameter used to balance spatial and temporal features.

[0069] As a preferred implementation scheme, in this embodiment, batch processing is adopted when training the video recognition deep neural network model.

[0070] First, randomly sample a batch of video samples from the video dataset The batch size N bs given in advance;

[0071] Then the videos in the video sample set B are input into the video recognition deep neural network model to obtain the corresponding video representation output:

[0072]

[0073] in Represents video sample x i The output feature representation of is 1*D, where D represents the feature dimension;

[0074] Then the text name corresponding to the class label t i Input into the preset text encoder to obtain text space representations for all categories:

[0075]

[0076] A textual spatial representation of each category;

[0077] Next, a contrastive learning loss function is constructed based on the video representation output and text space representation obtained in the above steps, and the video recognition deep neural network model is trained using an Adam optimizer or a stochastic gradient descent optimizer with momentum and a back propagation algorithm;

[0078] Finally, repeat the above steps until the model converges.

[0079] The third step is to apply the trained video recognition deep neural network model to the video recognition task, input the video sequence to be recognized, and output the classification prediction result of the video.

[0080] As a preferred implementation, in this embodiment, the video belongs to category y c The probability is expressed as:

[0081]

[0082] in, Represents the video feature output F V and the i-th text space representation The cosine similarity between ;

[0083] The similarity prediction label of the video and text, that is, the classification prediction result of the video, is:

[0084] y pred =argmax(P(y c |F V ))

[0085] Among them, argmax is the operation of finding the maximum value.

[0086] Through the above technical scheme, the embodiment of the present invention proposes a video recognition method based on dynamic spatiotemporal full-scale feature fusion. The present invention realizes comprehensive modeling of video spatiotemporal features by introducing a parallel spatiotemporal adapter (PTA) and a self-prompting generator (SPG). In this process, PTA effectively models the temporal interaction between frames, and SPG dynamically captures multi-resolution spatial features. By adaptively fusing spatiotemporal features and combining the contrastive learning loss function, the model can achieve better results in video classification tasks, improve the accuracy of video recognition, and alleviate the overfitting problem.

[0087] Obviously, it should be understood by those skilled in the art that the various steps of the present invention can be implemented by a general computing device, which can be concentrated on a single computing device or distributed on a network composed of multiple devices. Optionally, these steps can be implemented by executable program code, which can be stored in a storage device and executed by a computing device. In some cases, these steps can be performed in an order different from that described herein, or made into integrated circuit modules respectively, or multiple steps or modules can be integrated into a single circuit module for implementation. Therefore, the present invention is not limited to any specific hardware or software combination.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video recognition method based on dynamic spatiotemporal full-scale feature fusion, characterized in that: The following steps are involved: S1. Obtain a video dataset for training a deep neural network model, wherein the video dataset includes multiple video sequences, and each video sequence consists of multiple time frames; S2. Using the video data set to train a predefined video recognition deep neural network model, the video recognition deep neural network model includes a parallel spatiotemporal adapter and a self-prompt generator; S3. Apply the trained video recognition deep neural network model to the video recognition task, input the video sequence to be recognized, and output the classification prediction result of the video.

2. The video recognition method based on dynamic spatiotemporal full-scale feature fusion as claimed in claim 1, characterized in that: In step S1, the video data set is (x1, y1, t1), (x2, y2, t2), ..., (x N ,y N ,t N ), N is the total number of videos in the video dataset, x i represents the i-th video, y i Indicates the class label corresponding to the i-th video, t i Indicates the text name corresponding to the class label, y i ∈{1,…,C}, C represents the total number of categories, and each category contains multiple videos.

3. The video recognition method based on dynamic spatiotemporal full-scale feature fusion as claimed in claim 2, characterized in that: The parallel spatiotemporal adapter is predefined as a self-attention layer wrapped by an adapter, which is used to model the temporal interaction ability of the same position between different frames; the self-hint generator is predefined as a module constructed by a pooling layer and a multi-layer linear layer, which is used to extract spatial information of different resolutions of video frames.

4. The video recognition method based on dynamic spatiotemporal full-scale feature fusion as claimed in claim 3, characterized in that: The parallel spatiotemporal adapter adopts a self-attention mechanism and uses the following formula to implement temporal information modeling: Among them, PTA stands for Parallel Spatiotemporal Adapter, which is used for temporal feature extraction; V i-1 is the visual token of the i-1th layer, N bs represents the batch size, T represents the number of frames, K represents the number of patches, D represents the feature dimension, Down(V i-1 ) indicates that V is converted to i-1 Projected to a low-dimensional space, Attn represents the self-attention layer, and Up represents the use of an upsampling projector to restore the refined temporal features. Represents the temporal feature output of the i-th layer.

5. The video recognition method based on dynamic spatiotemporal full-scale feature fusion as claimed in claim 4, characterized in that: The self-hint generator extracts multi-scale features from the video input using the following formula: in, represents the spatial feature map of the i-th layer, Avg represents the average pooling operation, and Projector represents the projection operation of the multi-layer linear layer; The self-hint generator is also used to convert R i and Perform splicing operations to form new 6. The video recognition method based on dynamic spatiotemporal full-scale feature fusion as claimed in claim 5, characterized in that: The video recognition deep neural network model realizes spatiotemporal feature fusion through the following formula: Among them, α is a learnable weight parameter used to balance spatial and temporal features.

7. The video recognition method based on dynamic spatiotemporal full-scale feature fusion according to any one of claims 1 to 6, characterized in that: Step S2 specifically includes: S21. Randomly sample a batch of video samples from the video dataset The batch size N bs given in advance; S22, input the videos in the video sample set B into the video recognition deep neural network model to obtain the corresponding video representation output: in Represents video sample x i The output feature representation of is 1*D, where D represents the feature dimension; S23, the text name corresponding to the class label t i Input into the preset text encoder to obtain text space representations for all categories: A textual spatial representation of each category; S24, constructing a contrastive learning loss function based on the video representation output obtained in S22 and the text space representation obtained in S23, and using an Adam optimizer or a stochastic gradient descent optimizer with momentum, and a back propagation algorithm to train the video recognition deep neural network model; S25. Repeat steps S21 to S24 until the model converges.

8. The video recognition method based on dynamic spatiotemporal full-scale feature fusion as claimed in claim 7, characterized in that: In step S24, the method for constructing the contrastive learning loss function is as follows: Based on video sample x i Video representation output and the corresponding category y j The text space representation of , calculates the semantic similarity vector: in express and The inner product operation of Represents the output feature representation The L2 norm of Representing text space The L2 norm of ; The entire video recognition deep neural network model is trained by optimizing the visual-textual semantic similarity when the video and text belong to the same category, and minimizing the semantic similarity when they belong to different categories; therefore, the contrastive learning loss function is defined as: L contras =∑i-y j logSM~sim(x i ,y j ) Among them, SM represents the softmax function.

9. The video recognition method based on dynamic spatiotemporal full-scale feature fusion as claimed in claim 8, characterized in that: In step S3, the video belongs to category y c The probability is expressed as: in, Represents the video feature output F V and the i-th text space representation The cosine similarity between ; The similarity prediction label of the video and text, that is, the classification prediction result of the video, is: y pred =argmax(P(y c |F V )) Among them, argmax is the operation of finding the maximum value.

Images