Adaptive method for action recognition field based on comparative learning and feature mixing

By comparing the learning and feature mixing methods, the time-consistent features of the video are extracted and cross-domain mixed features are generated, which solves the problem of performance degradation of the action recognition model on the unseen data set, and improves the robustness and generalization of the model performance.

CN119007290APending Publication Date: 2024-11-22UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411097540.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing action recognition models perform poorly on unseen datasets, especially due to performance degradation due to domain differences between video datasets, especially the diversity of time information and the lack of labeling.

Method used

Using a method based on contrast learning and feature mixing, time information is extracted through graph convolution neural networks, the feature similarity of the front and back segments in the same video is maximized, and cross-domain mixed features are generated to reduce the domain difference between the source domain and the target domain. And use pseudo-label enhancement model learning in the target domain.

Benefits of technology

It improves the robustness and generalization performance of the action recognition model on the unseen data set, effectively reduces the feature differences between the source domain and the target domain, and improves the accuracy of the target domain.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007290A_ABST
    Figure CN119007290A_ABST
Patent Text Reader

Abstract

The invention discloses an adaptive method in the field of action recognition based on comparative learning and feature mixing, and belongs to the technical field of deep learning. According to the novel unsupervised video field adaptive method for the action recognition task, cross-domain learning is carried out by using comparative learning and feature mixing, so that the learning ability, obtained on source domain data, of the model is migrated to target domain data. The basic structure of the action recognition network is composed of a three-dimensional convolutional network and a graph neural network, firstly, overall features in a video are extracted through the three-dimensional convolutional network, then the features are input into the graph neural network, and the time relation in the video is obtained. In the field self-adaption part, the invention provides a novel field self-adaption method based on comparative learning. Firstly, first n fragments and last n fragments are extracted from the same video, then the feature similarity of the first n fragments and the last n fragments is maximized, the time consistency of each video in a source domain and a target domain is achieved, and therefore feature alignment between the source domain and the target domain is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0003] The present invention belongs to the technical field of deep learning, and specifically designs an adaptive method in the field of action recognition based on contrastive learning and feature mixing. Background Art

[0005] In recent years, action recognition tasks have received widespread attention and made significant progress, mainly due to the advancement of deep learning technology and the introduction of large-scale annotated datasets. However, a common problem is that when a trained action recognition model is tested on an unseen dataset, its performance will drop significantly. Unseen datasets and existing datasets often show different characteristics in terms of background, lighting, and human activities. Therefore, improving the robustness and generalization performance of action recognition models on unseen datasets has become an important research direction.

[0006] To solve this problem, some scholars have begun to study domain adaptation methods suitable for action recognition tasks. Typically, domain adaptation methods take data from two domains as input: labeled source domain data and unlabeled target domain data. Then, in order to alleviate the performance degradation of the model on the target domain, various methods are used to transfer the knowledge obtained from the source domain to the target domain. Video data contains rich temporal information, which means that the inter-domain differences of different video datasets are not only reflected in the different actions performed by different individuals in diverse scenes, but also in the different time points or durations of the actions. Therefore, domain alignment methods that consider temporal information perform better in unsupervised video domain adaptation tasks.

[0007] At the same time, some unsupervised video domain adaptation methods use contrastive learning to extract temporal information. For example, contrastive learning is used to extract video features from RGB and optical flow patterns respectively, or to obtain temporal information by changing the speed of video changes, so as to achieve feature consistency of samples obtained from the same video but at different time speeds. Most of the above methods process temporal or spatial features based on the entire video as the basic unit. However, since human actions are continuous, in action recognition tasks, there is usually a substantial temporal correlation between the first n frames and the last n frames of the same video, which requires studying temporal information from the perspective of the correlation between video clips. Summary of the invention

[0009] In this paper, we propose a domain adaptation method for action recognition based on contrastive learning and feature mixing. First, we use graph convolutional neural networks to extract temporal information from videos. Then, we apply contrastive learning on both the source and target domains to achieve feature consistency between video clips. Specifically, we sample two clips of the same length from the same video - one from the first n frames of the video and the other from the last n frames of the video. Then, we maximize the feature similarity between these clips in the same video and minimize the feature similarity between clips extracted from different videos through contrastive learning. At the same time, since the lack of true labels in the target domain will affect the accuracy of the model, we use pseudo labels on the target domain. In this way, the model can further learn consistent features of video clips belonging to the same video or with the same pseudo labels. Although the above method helps to achieve domain alignment, it does not directly reduce the domain difference between the source and target domains at the feature level. Therefore, we generate cross-domain mixed features from the two domains and then input the mixed features into the network to obtain the results, which effectively alleviates the performance degradation caused by the domain difference between the source domain video and the target domain video features.

[0010] Specific content:

[0011] A method for action recognition domain adaptation based on contrastive learning and feature mixing, comprising the following steps:

[0012] Step 1: Obtain source domain and target domain data and preprocess the video data;

[0013] Step 2: extract features from the preprocessed data obtained in step 1 through a three-dimensional convolutional network and a graph neural network;

[0014] In step 3, two segments are obtained from the front and back parts of each video respectively, and then contrastive learning is applied on both the source and target domains to maximize the feature similarity between the front and back segments of each video and learn the temporal consistency features of each video.

[0015] Step 4: Obtain pseudo labels of the target domain data, and add pseudo labels based on the video temporal consistency contrast learning method described in step 3 to introduce category information.

[0016] Step 5: Mix the target domain data with the source domain data based on its pseudo-label, and input the mixed features into the classifier network for training to align the feature expressions of the source domain and the target domain.

[0017] Step 6: Use the methods proposed in steps 3, 4, and 5 to train the network on the source domain and target domain data, transfer the knowledge learned on the labeled source domain data to the unlabeled target domain data, and improve the accuracy of the network on the target domain data set.

[0018] Step 1 of the present invention comprises:

[0019] Step 1-1: Use imageio and opencv libraries to read video files. According to the provided file name, path, size of each frame image and number of frames L, read the specified video file and return an array containing L frame images.

[0020] Step 1-2: Use the shuffle function and the video file name to batch read the data obtained in step 1-1. According to the number of data files, the index is randomly shuffled, and then the files are read in the order of these shuffled indexes to complete the data extraction.

[0021] Step 1-3: Use the methods of the transforms module to process the data read in step 1-2. Normalize the data to ensure that it meets the requirements of subsequent processing.

[0022] Step 2 of the present invention comprises:

[0023] Step 2-1, obtain the feature vector of each frame of video data through the three-dimensional convolution network. The calculation formula of the three-dimensional convolution is as follows:

[0024]

[0025] Among them, R i represents the size of the three-dimensional kernel in the time dimension, is the value of the previous layer connected to the feature map at (p, q, r). In the present invention, we adopt the basic structure of three-dimensional convolution based on the I3D (Inflated 3D) framework. The I3D (Inflated 3D) network captures the spatiotemporal features of the video by expanding the 2D convolution kernel to the 3D convolution kernel. Its basic structure includes an input layer, a 3D convolution layer, a 3D maximum pooling layer, multi-layer 3D convolution and pooling layers, a flattening operation, a fully connected layer, and a Softmax classifier.

[0026] Step 2-2, model the temporal relationship within the video data through a graph neural network. First, the feature vector of each frame obtained in step 2-1 is input into the graph neural network, a graph is constructed based on the time sequence, each feature vector is used as a node of the graph, and edges are constructed according to the relationship between frames; then, a graph convolutional network (GCN) is used to perform convolution operations on the graph, and the neighbor information of each node is aggregated through a message passing mechanism to update the node features; next, after multiple layers of graph convolution are superimposed, complex spatiotemporal relationships are gradually captured; finally, the updated node features are fused. In the present invention, we obtain node features through three layers of graph convolution operations.

[0027] Step 2-3: Flatten and fuse the node features that have been aggregated and updated in step 2-2 to form a unified feature representation. Then, this feature representation is input into the fully connected layer, through which the features are further processed and transformed. The fully connected layer will integrate these features and finally output the video classification result.

[0028] Step 3 of the present invention comprises:

[0029] Step 3-1: In the source domain and the target domain, the video V is divided into V o and V n Two parts. Among them, V o is the continuous n frames taken from the first frame of the video, V n is the last n frames of the video. V is obtained through the network structure constructed in step 2 o and V n characteristic expression.

[0030] Step 3-2: Apply contrastive learning on the source domain and the target domain respectively. o and V n is the positive sample pair in contrastive learning, V obtained from other videos o and V n is a negative sample pair. By contrasting the loss, we maximize V from the same video. o and V n The feature similarity of V from different videos is minimized. o and V n similarity of features.

[0031] Step 3-3, the specific loss function of contrastive learning is as follows:

[0032]

[0033] in, is the exponent of cosine similarity and τ is a hyperparameter.

[0034] Step 4 of the present invention comprises:

[0035] Step 4-1: Through step 3, we obtain the temporal consistency features of the source domain and target domain videos. However, for the target domain data that lacks labels, the category information of the video is still missing. Therefore, we first generate pseudo-label information for the target domain video through the network structure constructed in step 2.

[0036] Step 4-2, after obtaining the pseudo-labels of the target domain video, we compare the V of the video with the same pseudo-label based on the contrastive learning described in step 3-2. o and V nAs positive sample pairs. By adding pseudo labels, we learn the temporal consistency features of the same video while also learning the temporal consistency features between different videos of the same category.

[0037] Step 4-3, the loss function of contrastive learning after adding pseudo labels is as follows:

[0038]

[0039]

[0040] in, is the positive sample set, Yes V i 's pseudo-labels.

[0041] Step 5 of the present invention comprises:

[0042] Step 5-1: For labeled source domain data, we obtain the feature expression of each video through the network structure constructed in step 2.

[0043] Step 5-2: For the unlabeled target domain data, we obtain the feature expression f of each video through the network structure constructed in step 2 t j And pseudo label information

[0044] Step 5-3, perform feature mixing operation on source domain and target domain videos with the same labels and pseudo labels. The calculation formula is as follows:

[0045]

[0046] in, That is, the label of the source domain video is the same as the pseudo label corresponding to the target domain video.

[0047] Step 5-4: Input the mixed features obtained in step 5-3 into the classifier network for training. The loss function is as follows:

[0048]

[0049] The present invention proposes an action recognition domain adaptation method based on contrastive learning and feature mixing, which has the following advantages and beneficial effects compared with the existing methods:

[0050] The present invention proposes a new video domain adaptation method: first, the temporal consistency of the previous and next video segments is achieved in the source domain and the target domain respectively through contrastive learning; then, the domain difference of feature representation between the source domain and the target domain is reduced by generating cross-domain hybrid features; compared with the existing methods, this paper proposes a new application method of contrastive learning in video domain adaptation. Specifically, we sample two segments from the first n frames and the last n frames of the same video respectively, and then maximize the feature similarity between them to obtain the temporal consistency features of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is the overall flow chart of the method of the present invention;

[0053] Figure 2 It is a schematic diagram of sampling the original video;

[0054] Figure 3 It is the overall network framework diagram;

[0055] Figure 4 It is a schematic diagram of video temporal consistency contrast learning;

[0056] Figure 5 is a schematic diagram of feature mixing;

[0057] Figure 6 This is an example diagram of the action recognition dataset. DETAILED DESCRIPTION

[0059] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0060] The present invention is implemented including:

[0061] The overall process of the method proposed by the present invention is as follows Figure 1 As shown, the specific construction steps of the action recognition domain adaptation method in the embodiment of the present invention are as follows:

[0062] Step 1: Obtain source domain and target domain data and preprocess the video data;

[0063] Step 2: extract features from the preprocessed data obtained in step 1 through a three-dimensional convolutional network and a graph neural network;

[0064] In step 3, two segments are obtained from the front and back parts of each video respectively, and then contrastive learning is applied simultaneously on the source and target domains to maximize the feature similarity between the front and back segments of each video and learn the temporal consistency features of the front and back features of each video.

[0065] Step 4: Obtain pseudo labels of the target domain data, and add pseudo labels based on the video temporal consistency contrast learning method described in step 3 to introduce category information.

[0066] Step 5: Mix the target domain data with the source domain data based on its pseudo-label, and input the mixed features into the classifier network for training to align the feature expressions of the source domain and the target domain.

[0067] Step 6: Use the methods proposed in steps 3, 4, and 5 to train the network on the source domain and target domain data, transfer the knowledge learned on the labeled source domain data to the unlabeled target domain data, and improve the accuracy of the network on the target domain data set.

[0068] Step 1 of the present invention comprises:

[0069] Step 1-1: Download the public action recognition dataset, where the source domain is the UCF101 dataset, which contains 2009 videos, and the target domain is the HMDB51 dataset, which contains 1200 videos. Use the imageio and opencv libraries to read the video file. According to the provided file name, path, size of each frame image, and number of frames L, read the specified video file and return an array containing L frames of images.

[0070] Step 1-2: Use the shuffle function and the video file name to batch read the data obtained in step 1-1. According to the number of data files, the index is randomly shuffled, and then the files are read in the order of these shuffled indexes to complete the data extraction.

[0071] Step 1-3: Use the methods of the transforms module to process the data read in step 1-2. Normalize the data to ensure that it meets the requirements of subsequent processing.

[0072] Step 2 of the present invention comprises:

[0073] Step 2-1, obtain the feature vector of each frame of video data through the three-dimensional convolution network. The calculation formula of the three-dimensional convolution is as follows:

[0074]

[0075] Among them, R i represents the size of the three-dimensional kernel in the time dimension, is the value of the previous layer connected to the feature map at (p, q, r). In the present invention, we adopt the basic structure of three-dimensional convolution based on the I3D (Inflated 3D) framework. The I3D (Inflated 3D) network captures the spatiotemporal features of the video by expanding the 2D convolution kernel to the 3D convolution kernel. Its basic structure includes an input layer, a 3D convolution layer, a 3D maximum pooling layer, multi-layer 3D convolution and pooling layers, a flattening operation, a fully connected layer, and a Softmax classifier.

[0076] Step 2-2, model the temporal relationship within the video data through a graph neural network. First, the feature vector of each frame obtained in step 2-1 is input into the graph neural network, a graph is constructed based on the time sequence, each feature vector is used as a node of the graph, and edges are constructed according to the relationship between frames; then, a graph convolutional network (GCN) is used to perform convolution operations on the graph, and the neighbor information of each node is aggregated through a message passing mechanism to update the node features; next, after multiple layers of graph convolution are superimposed, complex spatiotemporal relationships are gradually captured; finally, the updated node features are fused. In the present invention, we obtain node features through three layers of graph convolution operations.

[0077] Step 2-3: Flatten and fuse the node features that have been aggregated and updated in step 2-2 to form a unified feature representation. Then, this feature representation is input into the fully connected layer, through which the features are further processed and transformed. The fully connected layer will integrate these features and finally output the video classification result.

[0078] Step 3 of the present invention comprises:

[0079] Step 3-1: In the source domain and the target domain, the video V is divided into V o and V n Two parts. Among them, V o is the continuous n frames taken from the first frame of the video, V n is the last n frames of the video. V is obtained through the network structure constructed in step 2 o and V n characteristic expression.

[0080] Step 3-2: Apply contrastive learning on the source domain and the target domain respectively. o and V n is the positive sample pair in contrastive learning, V obtained from other videos o and V n is a negative sample pair. By contrasting the loss, we maximize V from the same video. o and V n The feature similarity of V from different videos is minimized.o and V n similarity of features.

[0081] Step 3-3, the specific loss function of contrastive learning is as follows:

[0082]

[0083] in, is the exponent of cosine similarity and τ is a hyperparameter.

[0084] Step 4 of the present invention comprises:

[0085] Step 4-1: Through step 3, we obtain the temporal consistency features of the source domain and target domain videos. However, for the target domain data that lacks labels, the category information of the video is still missing. Therefore, we first generate pseudo-label information for the target domain video through the network structure constructed in step 2.

[0086] Step 4-2, after obtaining the pseudo-labels of the target domain video, we compare the V of the video with the same pseudo-label based on the contrastive learning described in step 3-2. o and V n As positive sample pairs. By adding pseudo labels, we learn the temporal consistency features of the same video while also learning the temporal consistency features between different videos of the same category.

[0087] Step 4-3, the loss function of contrastive learning after adding pseudo labels is as follows:

[0088]

[0089]

[0090] in, is the positive sample set, Yes V i 's pseudo-labels.

[0091] Step 5 of the present invention comprises:

[0092] Step 5-1: For labeled source domain data, we obtain the feature expression of each video through the network structure constructed in step 2.

[0093] Step 5-2: For the unlabeled target domain data, we obtain the feature expression f of each video through the network structure constructed in step 2 t j And pseudo label information

[0094] Step 5-3, perform feature mixing operation on source domain and target domain videos with the same labels and pseudo labels. The calculation formula is as follows:

[0095]

[0096] in, That is, the label of the source domain video is the same as the pseudo label corresponding to the target domain video.

[0097] Step 5-4: Input the mixed features obtained in step 5-3 into the classifier network for training. The loss function is as follows:

[0098]

[0099] The present invention proposes an action recognition domain adaptation method based on contrastive learning and feature mixing, which has the following advantages over the prior art:

[0100] The present invention proposes a new video domain adaptation method: first, the temporal consistency of the previous and next video segments is achieved in the source domain and the target domain respectively through contrastive learning; then, the domain difference of feature representation between the source domain and the target domain is reduced by generating cross-domain hybrid features; compared with the existing methods, this paper proposes a new application method of contrastive learning in video domain adaptation. Specifically, we sample two segments from the first n frames and the last n frames of the same video respectively, and then maximize the feature similarity between them to obtain the temporal consistency features of the video.

[0101] The present invention provides an adaptive method for the field of action recognition based on contrastive learning and feature mixing. The above description is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A domain adaptation method for action recognition based on contrastive learning and feature mixing, characterized in that: The following steps are involved: Step 1: Obtain source domain and target domain data and preprocess the video data; Step 2: extract features from the preprocessed data obtained in step 1 through a three-dimensional convolutional network and a graph neural network; Step 3: Obtain two clips from the front and back parts of each video, and then apply contrastive learning on both the source and target domains to maximize the feature similarity between the front and back clips of each video and learn the temporal consistency features of each video. Step 4, obtain pseudo labels of the target domain data, and add pseudo labels based on the contrastive learning method described in step 3 to introduce category information; Step 5: Mix the target domain data with the source domain data based on the pseudo-label of the target domain data, and input the obtained mixed features into the classifier network for training to align the feature expressions of the source domain and the target domain; Step 6: Use the methods proposed in steps 3, 4, and 5 to train the network on the source domain and target domain data, transfer the knowledge learned on the labeled source domain data to the unlabeled target domain data, and improve the accuracy of the network on the target domain data set.

2. The method according to claim 1, characterized in that Step 1 includes: Step 1-1: Use imageio and opencv libraries to read video files. According to the provided file name, path, size of each frame image and number of frames L, read the specified video file and return an array containing L frame images. Step 1-2: Use the shuffle function and the video file name to batch read the data obtained in step 1-1, randomly shuffle the indexes according to the number of data files, and then read the files in the order of the shuffled indexes to complete data extraction; Step 1-3: Use the method of the transforms module to process the data read in step 1-2, normalize the data, and ensure that it meets the requirements of subsequent processing.

3. The method according to claim 1, characterized in that Step 2 includes: Step 2-1, obtain the feature vector of each frame of video data through a three-dimensional convolutional network. The calculation formula of three-dimensional convolution is: Among them, R i represents the size of the three-dimensional kernel in the time dimension, is the value of the previous layer connected to the feature map at (p, q, r). In the present invention, we adopt the basic structure of three-dimensional convolution based on the I3D (Inflated 3D) framework. The I3D (Inflated 3D) network captures the spatiotemporal characteristics of the video by expanding the 2D convolution kernel to the 3D convolution kernel. Its basic structure includes an input layer, a 3D convolution layer, a 3D maximum pooling layer, multi-layer 3D convolution and pooling layers, a flattening operation, a fully connected layer, and a Softmax classifier; Step 2-2, model the temporal relationship within the video data through a graph neural network. First, input the feature vector of each frame obtained in step 2-1 into the graph neural network, build a graph based on the time sequence, use each feature vector as a node of the graph, and build edges based on the relationship between frames; then, use the graph convolutional network (GCN) to perform convolution operations on the graph, aggregate the neighbor information of each node through the message passing mechanism, and update the node features; next, through the superposition of multiple layers of graph convolution, gradually capture the complex spatiotemporal relationship; finally, fuse the updated node features. In the present invention, we obtain node features through three layers of graph convolution operations; Step 2-3: Flatten and fuse the node features that have been aggregated and updated in step 2-2 to form a unified feature representation. Then, this feature representation is input into the fully connected layer, through which the features are further processed and transformed. The fully connected layer will integrate these features and finally output the video classification result.

4. The method according to claim 1, characterized in that Step 3 includes: Step 3-1: In the source domain and the target domain, the video V is divided into V o and V n Two parts. Among them, V o is the continuous n frames taken from the first frame of the video, V n is the last n frames of the video. V is obtained through the network structure constructed in step 2 o and V n characteristic expression; Step 3-2: Apply contrastive learning on the source domain and the target domain respectively. o and V n V is a positive sample pair in contrastive learning, obtained from other videos o and V n is a negative sample pair. By contrasting the loss, we maximize V from the same video. o and V n feature similarity, minimizing V from different videos o and V n similarity of characteristics; Step 3-3, the specific loss function of contrastive learning is: in, is the exponent of cosine similarity and τ is a hyperparameter.

5. The method according to claim 1, characterized in that Step 4 includes: Step 4-1: Through step 3, we obtain the temporal consistency features of the source domain and target domain videos. However, for the target domain data that lacks labels, the category information of the video is still missing. Therefore, we first generate pseudo-label information of the target domain video through the network structure constructed in step 2; Step 4-2, after obtaining the pseudo-labels of the target domain video, we compare the V of the video with the same pseudo-label based on the contrastive learning described in step 3-2. o and V n As positive sample pairs. By adding pseudo labels, we learn the temporal consistency features of the same video while also learning the temporal consistency features between different videos of the same category; Step 4-3, the loss function of contrastive learning after adding pseudo labels is: in, is the positive sample set, Yes V i 's pseudo-labels.

6. The method according to claim 1, characterized in that Step 5 includes: Step 5-1: For labeled source domain data, we obtain the feature expression of each video through the network structure constructed in step 2. Step 5-2: For the unlabeled target domain data, we obtain the feature expression f of each video through the network structure constructed in step 2 t j And pseudo-label information Step 5-3, perform feature mixing operation on source domain and target domain videos with the same labels and pseudo labels. The calculation formula is: in, That is, the label of the source domain video is the same as the pseudo label corresponding to the target domain video; Step 5-4: Input the mixed features obtained in step 5-3 into the classifier network for training. The loss function is calculated as:

Citation Information

Patent Citations

  • Semi-supervised video classification method and system based on neighbor consistency and comparative learning

    CN115311605A

  • Unsupervised field adaptive image recognition method based on prototype network contrast learning

    CN116883751A

  • Unsupervised domain adaptation target re-identification method

    WO2022001489A1