A self-supervised action recognition method based on cross-modal temporal contrast learning

Abstract

The present application relates to the technical field of video processing, and particularly relates to a self-supervised action recognition method based on cross-modal time sequence contrast learning, wherein RGB frames and optical flow data are generated from unlabeled video samples, different data enhancement methods are used to obtain different augmented views of input data, encoding features of different views are obtained through an encoder, and a double-path self-supervised action recognition network is established; further, global semantic dependencies of a video sequence are captured through contrast learning of instance discrimination, time sequence motion characteristics between local segments are modeled through a local time sequence contrast learning task, and a preliminary model of two branches independently trained is obtained after initialization training; the final model is obtained through collaborative training based on the cross-modal self-supervised action recognition and the local time sequence contrast learning method; finally, the parameters of the final model are fine-tuned using labeled data, and the effect is evaluated using test data to obtain the recognition performance of the model.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and specifically to a self-supervised action recognition method based on cross-modal temporal contrastive learning. Background Technology

[0002] With the rapid development of internet technology, multimedia information data is constantly being generated, and the massive amount of video data presents enormous challenges to data analysis and understanding. Computer vision technology can quickly and in real-time capture effective information from data, enabling tasks such as target detection, tracking, and behavior recognition in videos. Human action recognition is a research hotspot in the field of computer vision, primarily determining human behavior by analyzing the correlation and visual appearance features of image frames in a video sequence. It has broad application prospects in multiple fields such as intelligent surveillance, intelligent security, and video retrieval.

[0003] Currently, supervised learning-based video representation methods have achieved superior performance in many visual recognition tasks, relying on a large amount of manually labeled training sample data. Obtaining these datasets requires significant human and time investment. Self-supervised learning, due to its efficiency in utilizing data information and its generalization ability, has attracted widespread attention from researchers. Self-supervised learning mainly utilizes pre-tasks to mine the supervision information of unlabeled data itself, learns generalizable representations through network training, and transfers these representations to downstream tasks to improve model performance. Among current self-supervised learning algorithms, instance-based contrastive learning methods have been extensively studied. The core idea of ​​contrastive learning is to construct representations by encoding the similarity or dissimilarity of two things. Although instance-based contrastive learning alone can achieve significant results in many video understanding tasks, this method encourages the model to learn similar features within the same instance, ignoring the temporal variation characteristics of videos, and the features extracted based on segments are relatively coarse, which has significant limitations in some fine-grained scenarios. In early contrastive learning methods, cross-modal semantic information interaction was largely ignored, and single-modal learning, due to semantic limitations, is not conducive to the model's clustering effect.

[0004] Existing patents related to action recognition include the following two aspects:

[0005] In the field of supervised action recognition: In 2023, the 28th Research Institute of China Electronics Technology Group Corporation published an invention patent entitled "A Video Target Behavior Recognition Method Based on Spatial Grids." This invention uses spatial grid positioning, combined with the surrounding grid conditions, to analyze the behavior of targets in video frames through target detection and motion detection. Also in 2023, Changzhou University published an invention patent entitled "A Human Behavior Recognition Method Based on a Dual-Channel Hybrid Graph Convolutional Network." This invention, based on graph convolutional networks, constructs feature maps and individual characteristic maps from action data, and improves behavior recognition performance by fusing feature similarity and individual characteristics into a dual-channel hybrid graph convolutional network. In 2023, Xi'an Jiaotong University published an invention patent entitled "A Real-Time Behavior Recognition Method Based on Temporal Attention Mechanism and Dual-Stream Network." This invention improves network recognition accuracy by extracting video features at different time frame rates and applying temporal attention weighting to images that contribute more to the network. Finally, in 2023, Tianjin University published an invention patent entitled "A Video Behavior Recognition Method Based on an Attention Mechanism." This invention more fully preserves key information in video feature representation by finely distinguishing the importance of different channels of frame-level features.

[0006] In the field of self-supervised action recognition: In 2022, Peking University Shenzhen Graduate School published an invention patent entitled "A Comparative Self-Supervised Human Behavior Recognition Method and System Based on Spatiotemporal Information Aggregation." This invention achieves more reliable representation by effectively aggregating video spatiotemporal information through intra-stream fusion and inter-stream voting of skeleton action sequences, motion information, and skeletal information. Also in 2022, Hefei University of Technology published an invention patent entitled "Human Action Recognition Method and System Based on Graph Neural Networks." This invention utilizes downsampling with short connections and corresponding upsampling layers to achieve 2D feature extraction and joint point recognition of data, and then uses the obtained 2D joint information... Input graph neural networks to improve the performance of 3D action recognition; in 2022, Peking University published an invention patent entitled "An Action Recognition Method and Electronic Device Based on Multi-Task Self-Supervised Learning". This invention improves the accuracy of model action recognition by designing multiple self-supervised tasks such as action prediction, puzzle-making and contrastive learning; in 2023, Shandong Artificial Intelligence Research Institute published an invention patent entitled "An Unsupervised Cross-Domain Video Action Recognition Method Based on Multi-Discriminator Collaboration and Strong-Weak Sharing Mechanism". This invention uses a dual-domain discriminator network to discriminate video action samples and conducts adversarial training between the network and the feature extractor to obtain features with discriminative and domain-invariant properties. Summary of the Invention

[0007] The purpose of this invention is to provide a self-supervised action recognition method based on cross-modal temporal contrastive learning, which aims to solve the problem that existing human action recognition methods obtain limited information about human behavior on unlabeled data.

[0008] To achieve the above objectives, this invention provides a self-supervised action recognition method based on cross-modal temporal contrastive learning, comprising the following steps:

[0009] Step 1: Generate RGB frames and optical flow data from unlabeled video samples;

[0010] Step 2: Use different data augmentation methods to obtain different augmented views of the input data;

[0011] Step 3: Obtain the encoded features of different views through the encoder and establish a dual-path self-supervised action recognition network;

[0012] Step 4: Capture global semantic dependencies of video sequences through instance-based contrastive learning;

[0013] Step 5: Model the temporal motion characteristics between local segments through a local temporal comparison learning task;

[0014] Step 6: Combine Step 4 and Step 5 to initialize and train the dual-path self-supervised action recognition network, and obtain a preliminary model with two branches trained independently.

[0015] Step 7: Perform information exchange between multiple modalities using cross-modal consistency mining methods;

[0016] Step 8: Based on cross-modal self-supervised action recognition and local temporal contrastive learning methods, co-train the preliminary models of the two modalities to obtain the final model;

[0017] Step 9: Fine-tune the parameters of the final model using labeled data, and evaluate the model's performance using test data to obtain the model's recognition performance.

[0018] Preferably, the process of generating RGB frames and optical flow data from unlabeled video samples specifically involves extracting frame-level video sequences from the unlabeled video samples and using the unsupervised TV-LI algorithm to extract the corresponding optical flow maps from the frame sequences.

[0019] Preferably, the process of obtaining different augmented views of the input data using different data augmentation methods specifically involves randomly cropping and sampling the input data, and augmenting the RGB frames and optical flow sampling segments using random cropping, horizontal flipping, Gaussian blurring, and color dithering data augmentation strategies to obtain different augmented views of the same instance as positive sample pairs.

[0020] Preferably, in the process of obtaining the encoded features of different views through the encoder and establishing a dual-path self-supervised action recognition network, the encoder is a deep convolutional neural network S3D, and the dual-path self-supervised action recognition network has two branches: RGB and optical flow. For each branch, the data-enhanced sample data is input into the encoder to obtain a visual representation of the features, and then the features are projected into a low-dimensional embedding space through an MLP layer.

[0021] Preferably, in the process of capturing the global semantic dependencies of a video sequence through contrastive learning of instance discrimination, the output features of the dual-path self-supervised action recognition network are used to perform instance discrimination by maximizing the semantic consistency of different views of the same video, thereby capturing the global semantic dependencies of the video.

[0022] Preferably, the implementation process of modeling the temporal motion characteristics between local segments through a local temporal comparison learning task is as follows:

[0023] Global contrastive learning features struggle to model local motion and temporal variations in video actions, exhibiting poor feature representation in fine-grained scenarios. To fully capture the differences between actions within the same video and enable the model to learn temporal information between frames, a local temporal contrastive learning module is designed, comprising two contrastive learning tasks: a local contrastive task, which learns the similarity between different local segments of the same video, distinguishing features from different instances and increasing the fine-grainedness of the representation; and a local temporal contrastive task, which learns the differences between non-overlapping local segments of the same video, increasing the temporality of the representation.

[0024] Preferably, the process of obtaining the preliminary model with independent training of the two branches specifically involves training the RGB and optical flow networks independently, and iteratively optimizing the model through instance comparison learning and local temporal comparison learning on the same instance to obtain preliminary training models for the two modalities.

[0025] Preferably, the implementation process of information interaction between multiple modalities through cross-modal consistency mining is as follows:

[0026] Cross-modal consistency mining methods utilize one network to collect positive samples for another network to facilitate intermodal information exchange. Specifically, optical flow samples are input into the encoder to extract features; the similarity of these features with other features in the repository is compared, and the top k most similar instances in the optical flow embedding space are selected as positive samples for the RGB network. Similarly, the RGB network can be used to select positive samples for the optical flow network.

[0027] Preferably, in the process of co-training preliminary models of two modalities based on cross-modal self-supervised action recognition and local temporal contrastive learning methods to obtain the final model, the preliminary training models of the two modalities are trained alternately. By combining the cross-modal global consistency mining network and the local temporal contrastive learning method, intramodal data association and intermodal semantic collaborative interaction are achieved.

[0028] Preferably, the process of fine-tuning the parameters of the final model using labeled data and evaluating the model's performance using test data to obtain the model's recognition performance involves verifying the model's performance through fine-tuning training on an action recognition task. A linear classifier, namely a fully connected layer and a softmax layer, is added after the self-supervised pre-trained encoder. Then, the entire model is trained on the action recognition task under supervision, and the trained model is evaluated using test data to obtain the model's recognition performance.

[0029] This invention provides a self-supervised action recognition method based on cross-modal temporal contrastive learning. It generates RGB frames and optical flow data from unlabeled video samples, then uses different data augmentation methods to obtain different augmented views of the input data. An encoder is used to obtain the encoded features of these different views, and a dual-path self-supervised action recognition network is established. Further, instance-discriminatory contrastive learning captures the global semantic dependencies of the video sequence, and a local temporal contrastive learning task models the temporal motion characteristics between local segments. This is used to initialize and train the network, resulting in a preliminary model with two independently trained branches. Cross-modal consistency mining is used to facilitate information interaction between multiple modalities. Based on cross-modal self-supervised action recognition and local temporal contrastive learning, the preliminary models of the two modalities are collaboratively trained to obtain the final model. Finally, labeled data is used to fine-tune the parameters of the final model, and test data is used to evaluate the model's performance. This invention improves the accuracy of human action recognition by obtaining deeper-level relevant information for the classification task through intra-modal data association and inter-modal semantic collaborative interaction. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating a self-supervised action recognition method based on cross-modal temporal contrastive learning according to the present invention. Detailed Implementation

[0032] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0033] Please see Figure 1 This invention provides a self-supervised action recognition method based on cross-modal temporal contrastive learning, and the specific steps are further explained below:

[0034] S1: Generate RGB frames and optical flow data from unlabeled video samples;

[0035] Specifically, given a training dataset containing N video instances, let V = {v1, v2, ..., v...} N Extract frame-level video sequences and use the unsupervised TV-LI algorithm to extract the corresponding optical flow maps M = {m1, m2, ..., m} from the frame sequences. N}

[0036] S2: Different data augmentation methods are used to obtain different augmented views of the input data;

[0037] Specifically, from a video sequence v i A segment is randomly sampled from the input data, and augmented using data augmentation strategies such as random cropping, horizontal flipping, Gaussian blurring, and color dithering to obtain different augmented views of the same instance as positive samples. The remaining video clips are negative samples N. - That is, v j ∈N - ,j≠i, and stored in the repository in the form of a queue, similarly an augmented view of optical flow data can be obtained.

[0038] S3: Obtain the encoded features of different views through the encoder and establish a dual-path self-supervised action recognition network;

[0039] Specifically, the feature encoder is a deep convolutional neural network S3D, and the dual-path self-supervised action recognition network has two branches: RGB and optical flow. For each branch, the data-augmented samples are input into the feature encoder f. q and f k This yields a visual representation of the features, which are then projected into a low-dimensional embedding space via an MLP layer g(·). For example, for the RGB branch, the view... The embedding features are represented as The embedding features of positive samples are The embedding features of negative samples are

[0040] S4: Capture global semantic dependencies of video sequences through instance-based contrastive learning;

[0041] Specifically, for the output features of the dual-path self-supervised action recognition network, instance discrimination is performed by maximizing the semantic consistency of different views of the same video. This requires minimizing the following loss function to constrain the model:

[0042]

[0043] Among them, z·z + It is the dot product between two vectors, and τ is the temperature coefficient. The instance contrast loss of the optical flow branch can be defined similarly. Through instance contrast learning, the model learns to bring similar instances closer together and push different instances further apart in the embedding space.

[0044] S5: Enhance the generalization of the representation through a local temporal contrastive learning task, and train the two branches independently to obtain a preliminary model;

[0045] Specifically, the local temporal contrastive learning module captures the differences between actions within the same video, enabling the model to learn temporal information between frames. Four non-overlapping local segments {s1, s2, s3, s4} are randomly sampled from a video sequence. In the local contrastive learning, s1 is the sample, and {s1...}... a If s1, s2, s3, s4 are considered as positive samples of this sample, then the local contrast loss between segments is:

[0046]

[0047] Among them, z i For sample s i The features (i = 1, ..., 4) are given, where a represents data augmentation.

[0048] Local contrastive learning enhances the fine-grained representation. To further learn temporal variation information in the video, local temporal contrastive learning is designed, using s1 as the input sample. a If {s2, s3, s4} are considered positive samples and {s2, s3, s4} are considered negative samples, then the local temporal contrast loss between segments is:

[0049]

[0050] The loss of the local timing comparison module is:

[0051] L LTCL =L local +L LT

[0052] S6: Combine steps 4 and 5 to perform constrained training on the network, and obtain a preliminary model with two branches trained independently.

[0053] Specifically, the model is iteratively optimized through instance-based contrastive learning and local temporal contrastive learning on the same instance to obtain preliminary trained models for two modalities. The initial training phase needs to minimize the following loss function:

[0054] L1=αL InfoNCE +βL LTMC

[0055] Here, α and β are coefficients for balancing the loss. Setting α = β = 1 makes the algorithm more general.

[0056] S7: Information exchange between multiple modalities is achieved through cross-modal consistency mining methods;

[0057] Specifically, cross-modal consistency mining methods utilize one network to collect positive samples for another network to facilitate intermodal information exchange. For example, to collect positive samples for an RGB network using the optical flow network, it is necessary to first process a batch of optical flow samples {m1, m2, ..., m...} B Input to encoder to extract features Where B is the batch size, this feature is compared with other features in the repository for similarity, and the k nearest neighbor samples in the optical flow embedding space are selected as supplementary positive samples for the RGB network. At this time, the positive sample set of the RGB network is the sample video sequence v. i Data augmentation plus v i The first k nearest neighbors in the optical flow feature space, the positive sample set P 1i It is expressed as follows:

[0058]

[0059] in, For RGB sample v i Data augmentation views It is the similarity between the i-th and j-th videos in the optical flow view. topK(·) means selecting the k most similar sample features from N samples and outputting the sample index value. Similarly, the RGB network can be used to select positive samples for the optical flow network.

[0060] In the cross-modal consistency mining module, the loss function of the RGB network assisted by the optical flow network is as follows:

[0061]

[0062] in, For video sample v i Features For positive sample features of RGB view, This is a negative sample.

[0063] S8: Based on cross-modal self-supervised action recognition and local temporal contrastive learning methods, the preliminary models of the two modalities are trained in a collaborative manner to obtain the final model;

[0064] Specifically, the two initial branches of the network are trained alternately, and combined with the cross-modal global consistency mining network and the local temporal contrastive learning method, intramodal data association and intermodal semantic collaborative interaction are achieved to obtain the final model. During the model training process, the following loss function needs to be minimized:

[0065] L2=αL RF +βL LTMC

[0066] Here, α and β are coefficients for balancing the loss. Setting α = β = 1 makes the algorithm more general.

[0067] S9: Fine-tune the parameters of the final model using labeled data, and evaluate the model's performance using test data to obtain the model's recognition performance.

[0068] Specifically, the model performance is verified by fine-tuning the training of the action recognition task. A linear classifier (i.e., a fully connected layer and a softmax layer) is added after the self-supervised pre-trained encoder. Then, the entire model is trained for the action recognition task under supervision. The trained model is evaluated using test data to obtain the model's recognition performance.

[0069] Supervised learning-based deep neural networks have been successfully applied to various computer vision tasks. These methods require training on labeled datasets, so the model performance depends to some extent on the quantity and quality of labeled data. Manual labeling of this data requires a lot of resources and is costly. Compared with supervised deep learning methods, self-supervised learning obtains generalized features by using a large amount of unlabeled data for self-supervision and then transferring them to downstream tasks. This can effectively improve the performance of human behavior recognition models and avoid the problem of using a large amount of labeled data.

[0070] While self-supervised deep learning methods have achieved significant results in the image domain, their development in the video domain has been relatively slow. This is partly because video is more difficult to process than images. Generally, image processing only requires attention to spatial information, while video processing, in order to more accurately analyze the content, requires attention to both spatial and temporal information. Another reason is that applying self-supervised deep learning methods to the video domain consumes substantial computational resources.

[0071] Contrastive learning methods have attracted research interest in recent years. They do not rely on labeled data and mainly use the data itself as supervision information to learn more valuable feature representations.

[0072] This invention discloses a self-supervised action recognition method based on cross-modal temporal contrastive learning. It generates RGB frames and optical flow data from unlabeled video sample data. Based on a dual-path temporal contrastive learning framework, it independently trains the RGB frames and optical flow data to obtain an initial model. The initialized model is then trained alternately, with one network collecting positive samples for the other to perform cross-modal information interaction, resulting in a self-supervised video representation model. The model's performance is evaluated using test data on an action recognition task to obtain its recognition performance. This invention utilizes contrastive learning to construct a self-supervised model to extract more task-related and valuable information from multimodal video data, solving the problem of limited human behavior information obtained from unlabeled data in existing human behavior recognition methods.

[0073] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A self-supervised action recognition method based on cross-modal temporal contrastive learning, characterized in that, Includes the following steps: Step 1: Generate RGB frames and optical flow data from unlabeled video samples; Step 2: Use different data augmentation methods to obtain different augmented views of the input data; Step 3: Obtain the encoded features of different views through the encoder and establish a dual-path self-supervised action recognition network; In the process of obtaining the encoded features of different views through the encoder and establishing a dual-path self-supervised action recognition network, the encoder is a deep convolutional neural network S3D, and the dual-path self-supervised action recognition network has two branches: RGB and optical flow. For each branch, the data-enhanced sample data is input into the encoder to obtain the visual representation of the features, and then the features are projected into the low-dimensional embedding space through the MLP layer. Step 4: Capture global semantic dependencies of video sequences through instance-based contrastive learning; In the process of capturing the global semantic dependency of video sequences through contrastive learning of instance discrimination, for the output features of the dual-path self-supervised action recognition network, instance discrimination is performed by maximizing the semantic consistency of different views of the same video to capture the global semantic dependency of the video. Step 5: Model the temporal motion characteristics between local segments through a local temporal comparison learning task; In the process of modeling the temporal motion characteristics between local segments through the local temporal contrast learning task, the local temporal contrast learning task includes two contrast learning tasks: local contrast task, which learns the similarity between different local segments of the same video, distinguishes features from different instances, and increases the fineness of the representation; and local temporal contrast task, which learns the differences between non-overlapping local segments of the same video, and increases the temporality of the representation. Step 6: Combine steps 4 and 5 to initialize and train the dual-path self-supervised action recognition network to obtain a preliminary model with two branches trained independently; The process of obtaining the preliminary model with independent training of the two branches is as follows: the RGB and optical flow networks are trained independently, and the models are iteratively optimized through instance comparison learning and local temporal comparison learning on the same instance to obtain the preliminary training models of the two modalities. Step 7: Perform information exchange between multiple modalities using cross-modal consistency mining methods; In the process of information interaction between multiple modalities through cross-modal consistency mining, optical flow samples are input into the encoder to extract features; the extracted features are compared with other features in the repository, and the top k similar instances in the optical flow embedding space are selected as positive samples for the RGB network; a similar operation method is used when selecting positive samples for the optical flow network using the RGB network. Step 8: Based on cross-modal self-supervised action recognition and local temporal contrastive learning methods, co-train the preliminary models of the two modalities to obtain the final model; Step 9: Fine-tune the parameters of the final model using labeled data, and evaluate the model's performance using test data to obtain the model's recognition performance.

2. The self-supervised action recognition method based on cross-modal temporal contrastive learning as described in claim 1, characterized in that, The process of generating RGB frames and optical flow data from unlabeled video samples involves extracting frame-level video sequences from the unlabeled video samples and using the unsupervised TV-LI algorithm to extract the corresponding optical flow maps from the frame sequences.

3. The self-supervised action recognition method based on cross-modal temporal contrastive learning as described in claim 2, characterized in that, The process of obtaining different augmented views of input data using different data augmentation methods involves randomly cropping and sampling the input data at different times, and then augmenting the RGB frames and optical flow sampling segments using random cropping, horizontal flipping, Gaussian blurring, and color dithering data augmentation strategies to obtain different augmented views of the same instance as positive sample pairs.

4. The self-supervised action recognition method based on cross-modal temporal contrastive learning as described in claim 1, characterized in that, In the process of co-training preliminary models of two modalities based on cross-modal self-supervised action recognition and local temporal contrastive learning methods to obtain the final model, the preliminary training models of the two modalities are trained alternately. By combining cross-modal global consistency mining network and local temporal contrastive learning methods, intramodal data association and intermodal semantic collaborative interaction are achieved.

5. The self-supervised action recognition method based on cross-modal temporal contrastive learning as described in claim 1, characterized in that, The process of fine-tuning the parameters of the final model using labeled data and evaluating the model's performance using test data to obtain the model's recognition performance involves verifying the model's performance through fine-tuning training on an action recognition task. A linear classifier, consisting of a fully connected layer and a softmax layer, is added after the self-supervised pre-trained encoder. Then, the entire model is trained on an action recognition task under supervision. The trained model is evaluated using test data to obtain the model's recognition performance.

Images