A small sample behavior recognition method based on a cross-modal contrast learning network

Abstract

A few-shot action recognition method based on cross-modal contrastive learning networks, involving computer vision technology. A. Given a set of videos, randomly select n from each video. seg A. Input the sampled video frames into the spatiotemporal enhancement module to obtain enhanced video vectors. B. Input the visual vectors generated in step B into the semantic generation network to generate semantic vectors. C. Concatenate the visual vectors and semantic vectors to construct a hybrid feature vector, and input it into a nonlinear contrast mapping head to obtain a transformed hybrid feature vector. E. Use the visual vectors obtained in step B and Gaussian noise to generate a synthetic vector, and input it into the nonlinear contrast mapping head to obtain the final synthetic vector. F. Treat the transformed hybrid feature vector generated in step D and the synthetic vector generated in step E as class prototypes, and use cosine similarity to calculate the distance between class prototypes to obtain the prediction probability. Compared with current mainstream few-sample behavior recognition methods, the proposed method improves classification performance.

Description

Technical Field

[0001] This invention relates to computer vision technology, specifically to a few-shot behavior recognition method based on a cross-modal contrastive learning network. Background Technology

[0002] Few-shot action recognition is a crucial research area in computer vision, playing a vital role in intelligent surveillance, abnormal behavior detection, and human-computer interaction. It aims to identify the category of a given video using a limited number of samples. In recent years, deep learning-based methods have achieved significant success in computer vision. However, these powerful deep learning methods require a large number of labeled samples for training to obtain a good model. In reality, acquiring massive amounts of labeled data is very expensive or impractical, as data collection is a tedious process that consumes significant human and material resources. These issues may limit the application of these deep learning-based methods in real-world scenarios. Therefore, how to obtain a robust model using a small number of labeled samples under a few-shot learning setting is one of the key problems in solving video classification.

[0003] Furthermore, significant progress has been made in image classification. However, directly applying image classification to video classification can lead to a sharp decline in detection performance. This is because, compared to images, videos contain an additional temporal dimension, resulting in more complex inherent properties. Using existing image classification methods to solve video classification tasks typically yields unsatisfactory detection results. Therefore, effectively extracting temporal information from videos is one of the key issues in video classification. Summary of the Invention

[0004] The purpose of this invention is to address the problems of insufficient data and spatiotemporal information mining in small sample behavior recognition, and to provide a small sample behavior recognition method with high accuracy based on a cross-modal contrastive learning network.

[0005] This invention includes the following steps:

[0006] 1) Given a small sample behavior recognition dataset, the dataset contains several videos, each video consisting of several video frames. Use a sparse temporal sampling strategy to sample the video frames.

[0007] 2) Input the sampled video frames into the backbone network to obtain visual features F. v ={F1,F2,…,F i ,…,F seg}, F i Let F represent the visual features of the i-th video frame. v The visual vector is obtained by inputting the spatiotemporal enhancement module;

[0008] 3) The visual vector generated in step 2) is input into the semantic generation network to generate semantic vectors;

[0009] 4) Concatenate the visual vector generated in step 2) and the semantic vector generated in step 3) to construct a hybrid feature vector, input it into a nonlinear contrast mapping head, and obtain the transformed hybrid feature vector.

[0010] 5) Use the visual vector obtained in step 2) and Gaussian noise to generate a synthetic vector, and input it into the nonlinear contrast mapping head to obtain the final synthetic vector;

[0011] 6) Treat the mixed feature vector of the transformation generated in step 4) and the composite vector generated in step 5) as class prototypes, and use the cosine similarity metric ψ(·,·) to calculate the distance between class prototypes.

[0012] In step 1), the specific steps for sampling video frames can be as follows: Sample a video V... a Randomly divided into n seg There are n video segments, and one frame is randomly selected from each video segment. Therefore, the number of randomly selected n... seg The video frames constitute a new video segment V′ a ={I1,I2,…,I i ,…,I seg}; where I i This represents a video frame randomly selected from the i-th video segment.

[0013] In step 2), the specific steps for obtaining the visual vector can be as follows:

[0014] First, in visual feature F v The above uses three different 1×12D convolutional layers to obtain query, key, and value features, which are represented as follows: and exist and The weight W is obtained by using element-wise multiplication. j Therefore, W j Weighted The process is defined as follows:

[0015]

[0016] Here, ⊙ represents element-wise multiplication. This represents the weighted value features; in order to preserve the original visual features, in the features A residual operation is used to obtain visual feature X'; to obtain temporal information, a 3×3 2D convolutional layer is used to process visual feature X', which can be described as follows:

[0017] Y(t) = conv(X′) t+1 )-X′ t ,1≤t≤n seg -1

[0018] Where Y(t) represents the motion feature at time t, conv(·) represents a 3×32D convolution operation, and X′ t+1 Let X′ represent the visual features at time t+1. t Let Y represent the visual features at time t; specifically, the motion features at the last time step are set to zero; the motion features at different times are connected together to form the final motion features, which can be defined as Y = [Y(1), Y(2), ..., Y(n)]. seg [·]; where [·] represents a join operation; then, average pooling is used in the spatial and temporal dimensions to obtain the pooled motion feature Y. p Subsequently, in the pooling motion feature Y p The above uses three different fully connected layers to obtain query, key, and value embeddings. This process can be represented as follows:

[0019]

[0020] in, These represent the embedding of query, key, and value, respectively; through and Weighted product The weighted embeddings are obtained; finally, residual connections are used to preserve the original temporal context information. This process can be described as follows: in, This is the obtained visual vector.

[0021] In step 3), the specific steps for the generative network to generate semantic vectors can be as follows: based on the visual vectors generated in step 2), Using a semantic generative network g(·) to generate semantic vectors can be expressed as follows: Among them, v m The generated semantic vector is represented by g(·); g(·) is a non-linear neural network consisting of two fully connected layers and a ReLU activation function; a semantic generation loss is used to make the obtained semantic vector closer to the corresponding real class semantic vector, which is defined as follows:

[0022]

[0023] Where α represents the class label, ω(·) represents the Word2Vec network, and N represents the number of action classes.

[0024] In step 4), the nonlinear contrastive mapping head includes two fully connected layers and a ReLU activation function to better learn the transformed mixed feature vectors. Design a cross-modal contrastive learning loss to supervise network training; if samples in the support set have the same class label as samples in the query set, then samples in the support set are positive samples; if samples in the support set have different class labels than samples in the query set, then samples in the support set are negative samples; the cross-modal contrastive learning loss... Defined as:

[0025]

[0026] in, This represents the mixed feature vector of the query sample. This represents the mixed feature vector of positive samples. This represents the mixed feature vector of negative samples. ψ(·,·) represents the cosine similarity measure, and τ represents the temperature parameter.

[0027] In step 5), the specific steps for generating the synthesized vector using the visual vector obtained in step 2) and Gaussian noise can be as follows: Generator G uses the visual vector and Gaussian noise z ~ Γ(0,I) obtained in step 2) to generate the synthesized vector, and discriminator D is used to distinguish whether the synthesized vector is real or fake; generator G and discriminator D work together to optimize the parameters of the entire prototype generation and comparison network; in order to better optimize the prototype generation and comparison network, generator G and discriminator D learn using the objective function V(G,D), which is defined as follows:

[0028]

[0029] in, Represents the visual vector, v m Represents a semantic vector. Indicates synthetic features, Represents the distribution of training samples. This represents the distribution of the synthesized vector and the semantic vector. Furthermore, the synthesized vector is input into a non-linear contrast mapping head to obtain the final synthesized vector.

[0030] In step 6), the distance between class prototypes is calculated using the cosine similarity metric ψ(·,·), defined as follows:

[0031]

[0032] Among them, F q F represents the class prototype of the query sample q. β F represents the class prototype of the action class labeled β. kp(c) represents the prototype of the k-th class in the support set, where K represents the number of classes in the support set. pre =β|q) represents the predicted probability that the query sample q belongs to class label β.

[0033] Compared with the prior art, the present invention has the following outstanding technical effects and advantages:

[0034] This invention addresses the issues of insufficient data and spatiotemporal information mining in few-shot behavior recognition. By integrating comparative and adversarial branches, it obtains discriminative feature representations and achieves better performance. In the comparative branch, a spatiotemporal enhancement module is designed to obtain enhanced video spatial and temporal features. Furthermore, a semantic generation network is designed to generate semantic vectors. The visual vectors and semantic vectors are concatenated and input into a nonlinear contrastive mapping head, allowing the features of both modalities to be mapped to the same feature space. In the adversarial branch, a prototype generation network is used to synthesize more feature vectors to alleviate the data shortage problem. A few-shot classifier is used to obtain the prediction results. Experiments show that compared with current mainstream few-shot behavior recognition methods, the classification performance of this invention is improved. Attached Figure Description

[0035] Figure 1 This is an overall flowchart of an embodiment of the present invention.

[0036] Figure 2 This is a schematic diagram of the spatiotemporal enhancement module. Detailed Implementation

[0037] The method of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. This embodiment is implemented under the premise of the technical solution of the present invention, and provides implementation methods and specific operation processes. However, the protection scope of the present invention is not limited to the following embodiments.

[0038] like Figure 1 As shown, the implementation of this embodiment of the invention includes the following steps:

[0039] A. Given a small sample behavior recognition dataset, sample video frames using a sparse temporal sampling strategy. Specifically, first, sample a video V... a Randomly divided into n seg There are n video clips. Then, one frame is randomly selected from each video clip. Therefore, the number of randomly selected n... seg The video frames constitute a new video segment V′ a ={I1,I2,…,I i ,…,I seg}. Among them, I i This represents a video frame randomly selected from the i-th video segment. The number of video segments is set to n. seg =8.

[0040] B. Input the video frames generated in step A into the backbone network to obtain visual features F. v ={F1,F2,…,F i ,…,F seg}. Among them, F i Let F represent the visual features of the i-th video frame. Then, the video features F... v Input to the spatiotemporal enhancement module (e.g.) Figure 2 As shown), to obtain enhanced video vectors. Specifically, firstly, in visual feature F v The above uses three different 1×12D convolutional layers to obtain query, key, and value features, which are represented as follows: and Then, in and The weight W is obtained by using element-wise multiplication. j Therefore, W j weighted The process is defined as follows:

[0041]

[0042] Here, ⊙ represents element-wise multiplication. This represents the weighted value feature. To preserve the original visual features, the feature... A residual operation is used to obtain visual features X'. To obtain temporal information, a 3×3 2D convolutional layer is used to process visual features X', which can be described as follows:

[0043] Y(t) = conv(X′) t+1 )-X′ t ,1≤t≤n seg -1,

[0044] Where Y(t) represents the motion feature at time t, conv(·) represents a 3×32D convolution operation, and X′ t+1 Let X′ represent the visual features at time t+1. t This represents the visual features at time t. Specifically, the motion features at the last time step are set to zero. Then, the motion features from different time steps are concatenated to form the final motion features, which can be defined as Y = [Y(1), Y(2), ..., Y(n)]. seg [·] represents a join operation. Next, average pooling is used in both spatial and temporal dimensions to obtain the pooled motion features Y. p Then, in the pooling motion feature Y p The above uses three different fully connected layers to obtain query, key, and value embeddings. This process can be represented as follows:

[0045]

[0046] in, These represent the query, key, and value embeddings, respectively. (Through...) and Weighted product The weighted embeddings are obtained. Finally, residual connections are used to preserve the original temporal context information. This process can be described as follows: in, This is the obtained visual vector.

[0047] C. Based on the visual vectors generated in step B Using a semantic generative network g(·) to generate semantic vectors can be expressed as follows: Among them, v m This represents the generated semantic vector. g(·) is a non-linear neural network, mainly consisting of two fully connected layers and a ReLU activation function. Additionally, a semantic generation loss is used to make the obtained semantic vector more closely resemble the corresponding real class semantic vector, defined as follows:

[0048]

[0049] Where α represents the class label, ω(·) represents the Word2Vec network, and N represents the number of action classes.

[0050] D. Based on the semantic vector generated in step C, concatenate it with the visual vector generated in step B to construct a hybrid feature vector. Then, use a non-linear contrast mapping head to obtain the transformed hybrid feature vector. This maps features from different modalities to the same feature space. The nonlinear contrastive mapping head mainly consists of two fully connected layers and a ReLU activation function. This is to better learn the mixed feature vectors of the transformation. A cross-modal contrastive learning loss was designed to supervise network training. If samples in the support set have the same class label as samples in the query set, then samples in the support set are considered positive samples. If samples in the support set have different class labels than samples in the query set, then samples in the support set are considered negative samples. Cross-modal contrastive learning loss. Defined as:

[0051]

[0052] in, This represents the mixed feature vector of the query sample. This represents the mixed feature vector of positive samples. Let ψ(·,·) represent the mixed feature vector of negative samples. Let ψ(·,·) represent the cosine similarity measure, and let τ represent the temperature parameter, where τ = 1.

[0053] E. Generator G uses the visual vector obtained in step B and Gaussian noise z ~ Γ(0,I) to generate a synthetic vector. Discriminator D is used to distinguish whether the synthetic vector is real or fake. Generator G and discriminator D work together to optimize the parameters of the entire prototype generation and contrast network. To better optimize the prototype generation and contrast network, generator G and discriminator D learn using the objective function V(G,D), which is defined as follows:

[0054]

[0055] in, Represents the visual vector, v m Represents a semantic vector. Indicates synthetic features, Represents the distribution of training samples. This represents the distribution of the synthesized vector and the semantic vector. Furthermore, the synthesized vector is input into a non-linear contrast mapping head to obtain the final synthesized vector.

[0056] F. The blended feature vector generated in step D and the composite vector generated in step E are considered as class prototypes. The distance between class prototypes is calculated using the cosine similarity metric ψ(·,·), defined as follows:

[0057]

[0058] Among them, F q F represents the class prototype of the query sample q. β F represents the class prototype of the action class labeled β. k p(c) represents the prototype of the k-th class in the support set, where K represents the number of classes in the support set. pre =β|q) represents the predicted probability that the query sample q belongs to class label β.

[0059] Table 1 presents the experimental results of the present invention and existing methods on the Kinetics dataset. The experiment shows that the classification performance of the present invention is significantly improved compared with the current mainstream small sample behavior recognition methods.

[0060] Table 1 shows the experimental results on the Kinetics dataset.

[0061] method 1-shot 2-shot 3-shot 4-shot 5-shot MAML 54.2 - - - 75.3 CMN 60.5 70.0 75.6 77.3 78.9 TARN 64.8 72.9 76.2 78.0 78.5 CFA 69.9 - 80.5 - 83.1 Embodied Learning 67.8 77.8 81.1 82.6 85.0 ARN 63.7 - - - 82.4 OTAM 73.0 - - - 85.8 AMeFu-Net 74.1 81.1 84.3 85.6 86.8 TRX 63.6 76.2 81.8 83.4 85.9 STRM - - - - 86.7 MTFAN 74.6 - - - 87.4 VIM 73.8 - - - 87.1 Method of the present invention 75.8 82.1 85.0 86.1 87.5

[0062] "-" indicates that the method did not provide experimental results.

[0063] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A few-shot behavior recognition method based on a cross-modal contrastive learning network, characterized in that... Includes the following steps: 1) Given a small sample behavior recognition dataset, the dataset contains several videos, each video consisting of several video frames. Use a sparse temporal sampling strategy to sample the video frames. 2) Input the sampled video frames into the backbone network and obtain visual features. , Indicates the first Visual features of each video frame, video features The visual vector is obtained by inputting the spatiotemporal enhancement module; 3) The visual vector generated in step 2) is input into the semantic generation network to generate semantic vectors; 4) Concatenate the visual vector generated in step 2) and the semantic vector generated in step 3) to construct a hybrid feature vector, input it into a non-linear contrast mapping head, and obtain the transformed hybrid feature vector. ; The nonlinear contrastive mapping head includes two fully connected layers and a ReLU activation function to better learn the transformed mixed feature vectors. A cross-modal contrastive learning loss is designed to supervise network training. If samples in the support set have the same class label as samples in the query set, then samples in the support set are considered positive samples; if samples in the support set have different class labels than samples in the query set, then samples in the support set are considered negative samples. The cross-modal contrastive learning loss... Defined as: ， in, This represents the mixed feature vector of the query sample. This represents the mixed feature vector of positive samples. Represents the mixed feature vector of negative samples; Represents the cosine similarity measure. Indicates temperature parameter; 5) Use the visual vector obtained in step 2) and Gaussian noise to generate a synthetic vector, and input it into the nonlinear contrast mapping head to obtain the final synthetic vector; 6) Treat the transformed hybrid feature vector generated in step 4) and the synthesized vector generated in step 5) as class prototypes, and use cosine similarity measurement. Calculate the distance between class prototypes; The use of cosine similarity measurement The distance between class prototypes is calculated as follows: ， in, Indicates query sample class prototype, Indicates that the label is The class prototype of the action class, Indicates support for the centralized first class prototype Indicates the number of classes supported in the set. Indicates query sample Belongs to class tags The predicted probability.

2. The few-shot behavior recognition method based on a cross-modal contrastive learning network as described in claim 1, characterized in that... In step 1), sampling the video frames specifically involves: taking a video... Randomly divided into From each video segment, one frame is randomly selected; the randomly selected frame... Each video frame constitutes a new video segment; ； in, Indicates from the first A video frame randomly selected from a video segment.

3. The few-shot behavior recognition method based on a cross-modal contrastive learning network as described in claim 2, characterized in that... In step 2), the specific steps for obtaining the visual vector are as follows: First, in terms of visual features The above utilizes three different 2D convolutional layers are used to obtain query, key, and value features, which are represented as follows: , and ;exist and Weights are obtained by using element-wise multiplication. Thus using Weighted The process is defined as follows: ； ， in, Indicates element-wise multiplication; This represents the weighted value features; in order to preserve the original visual features, in the features Visual features are obtained using a residual operation. To obtain timing information, a 2D convolutional layers process visual features The process is described as follows: , in, This represents the motion characteristics over time t. express 2D convolution operations, Indicates in Visual characteristics of a moment Indicates in Visual features at each moment; motion features at the last moment are set to zero; motion features from different moments are connected together to form the final motion feature, defined as... ,in, The connection operation is represented; then, average pooling is used in the spatial and temporal dimensions to obtain pooling motion features. Subsequently, in the pooling motion characteristics The above uses three different fully connected layers to obtain query, key, and value embeddings, which can be represented as follows: ， ， ， in, , , These represent the embedding of query, key, and value, respectively; through and Weighted product The weighted embeddings are obtained; finally, residual connections are used to preserve the original temporal context information. This process is described as follows: ;in, This is the obtained visual vector.

4. The few-shot behavior recognition method based on a cross-modal contrastive learning network as described in claim 3, characterized in that... In step 3), the specific steps for the generative network to generate semantic vectors are as follows: based on the visual vectors generated in step 2), Using semantic generative networks To generate semantic vectors, expressed as: ;in, This represents the generated semantic vector; It is a non-linear neural network consisting of two fully connected layers and a ReLU activation function; it utilizes a semantic generation loss to make the obtained semantic vectors more closely resemble the corresponding real class semantic vectors, as defined below: in, Indicates class tags, This represents a Word2Vec network. Indicates the number of action classes.

5. The few-shot behavior recognition method based on a cross-modal contrastive learning network as described in claim 4, characterized in that... In step 5), the specific steps for generating a synthesized vector using the visual vector obtained in step 2) and Gaussian noise are as follows: Generator Using the visual vectors and Gaussian noise obtained in step 2) Generate composite vectors, discriminator Used to distinguish whether a synthesized vector is real or fake; generator and discriminator Collaborative optimization of the entire prototype generation and comparison network parameters; to better optimize the prototype generation and comparison network, the generator... and discriminator Using the objective function Learning is defined as follows: ， in, Represents visual vectors. Represents a semantic vector. Indicates synthetic features, Represents the distribution of training samples. It represents the distribution of the synthesized vector and semantic vector; the synthesized vector is input into the nonlinear contrast mapping head to obtain the final synthesized vector.

Images