A temporal-spatial feature contrast learning method based on rotation angle constraint

Abstract

The present invention discloses a temporal-spatial feature contrast learning method based on rotation angle constraint. The method firstly performs temporal-spatial feature contrast learning by defining positive and negative sample pairs at the whole video level, so as to more effectively characterize the global context information of the whole video. At the same time, in order to overcome the shortcoming of contrastive learning in terms of weak description ability in temporal structure, the present invention also introduces a video rotation angle prediction task as a regularization term of contrast loss, further enhancing the temporal-spatial feature's ability to characterize the spatial appearance information and temporal motion clues and their evolution of the video, and improving the overall description ability and discrimination of the features. It overcomes the problem that the prior art only defines positive and negative sample pairs at the local frame level or clip level of the video for contrastive learning, and cannot effectively capture the global context information of the video in a long temporal-spatial range, resulting in problems such as weak video representation ability, weak semantic evolution characterization in the time dimension, and poor temporal structure description.

Description

Technical Field

[0001] The invention belongs to the technical field of relay protection of electric power systems, and in particular relates to a time-space feature comparison learning method based on rotation angle constraints. Background Art

[0002] With the rapid development of communication technology and the widespread popularity of video acquisition devices, more and more videos are being generated and shared, such as surveillance videos in smart cities, live videos in social media, and nursing videos in smart homes. How to understand and analyze these massive emerging video data has great theoretical and application value. The prerequisite for accurately understanding these videos is how to effectively represent their spatiotemporal features. At present, various spatiotemporal feature learning methods based on deep neural networks have been widely and successfully applied in many video understanding tasks, such as human action recognition and video retrieval, due to their powerful ability to capture spatial and temporal representations at the same time. In order to achieve good performance, supervised feature learning methods based on deep neural networks usually require millions of annotated videos for training. However, in practice, for new video understanding tasks, collecting such large-scale annotated video datasets is quite time-consuming and labor-intensive. To this end, the academic community has proposed feature learning techniques based on self-supervision.

[0003] In the past few years, self-supervised video representation learning represented by contrastive learning has received great attention. The outstanding advantage of this learning paradigm is that it can effectively define positive and negative sample pairs of training samples for contrastive training without the need for labeled training data, so that the feature representations of positive sample pairs have a closer distance in the embedding space, while the distance of negative sample pairs is increased, and finally a good data representation is learned. At present, contrastive learning has approached or even exceeded supervised feature learning methods in downstream tasks such as image classification, object detection and target segmentation. Recent works have extended contrastive learning methods in the image domain to the video domain. In summary, these methods mainly define positive and negative sample pairs at the local frame level or clip level of the video for contrastive learning, and therefore cannot effectively capture the global contextual information of the video in a long time-space range, resulting in weak video representation capabilities and inability to effectively characterize the evolving semantic information in the time dimension. Summary of the invention

[0004] The purpose of the present invention is to provide a time-space feature contrast learning method based on rotation angle constraints, which performs time-space feature contrast learning by defining positive and negative sample pairs at the entire video level, and uses video rotation angle prediction as a regularization term for contrast loss to improve the global description and discrimination of video data.

[0005] The technical solution adopted in the present invention is: a time-space feature contrast learning method based on rotation angle constraints, which performs time-space feature contrast learning by defining positive and negative sample pairs at the entire video level, and introduces a video rotation angle prediction task as a regularization term for the contrast loss, thereby further enhancing the time-space feature's ability to depict the video's spatial appearance information and temporal motion clues and their evolution, and improving the overall description ability and discrimination power of the features.

[0006] The present invention is also characterized in that:

[0007] The time-space feature comparison learning method based on rotation angle constraint, the specific operation steps are as follows:

[0008] Step 1: extract several frames of images from each training video sample in the training data set, perform rotation transformation on the image, and generate enhanced transformation video samples;

[0009] Step 2: construct a time-space feature contrast learning model, which includes a time-space feature extraction subnetwork, a latent space embedding projection transformation subnetwork and a rotation angle classifier subnetwork; input the enhanced transformation video sample generated in step 1 into the time-space feature extraction subnetwork to extract the time-space feature vector, and randomly initialize the time-space feature extraction subnetwork;

[0010] Step 3: Input the feature vector described in step 2 into the latent space embedding projection transformation subnetwork to perform feature dimensionality reduction and generate a low-dimensional embedding vector;

[0011] Step 4: Calculate the contrast loss using the low-dimensional embedding vector in the latent space;

[0012] Step 5: Input the feature vector described in step 2 into the rotation angle classifier subnetwork to obtain the corresponding rotation angle prediction probability;

[0013] Step 6: Calculate the angle prediction loss according to the rotation angle prediction probability described in step 5 and the rotation transformation angle applied to the image in step 1;

[0014] Step 7: Train the temporal-spatial feature contrast learning model based on rotation angle constraint according to the contrast loss of step 4 and the angle prediction loss of step 6;

[0015] Step 8: Use the weights of the feature extraction subnetwork in the above self-supervised temporal-spatial feature contrast learning model as the initial parameters or feature extractor, and fine-tune the model parameters of supervised training on the visual recognition task of the downstream small-scale dataset to obtain the final classification result.

[0016] Step 1 is as follows:

[0017] For the training data set Each training video sample v in k Extract no less than 8 frames of images at equal intervals in time sequence, and crop each frame of image according to 112x112 pixels. The video downsample generated by the cropped image is expressed as

[0018] Downsampling the video Each frame of the image undergoes two independent rotation transformations T σ (·), generating two enhanced transformed video samples and Where T σ The rotation parameter σ of (·) is a rotation angle randomly selected from the set {0°, 90°, 180°, 270°}; N video samples can generate a total of 2N enhanced transformed video samples; two enhanced transformed video samples generated from the same video are defined as positive sample pairs, and video sample pairs formed in other cases are all negative sample pairs.

[0019] Step 2: The specific process is as follows:

[0020] The enhanced transformed video samples are and Input to the spatiotemporal feature extraction subnetwork E(·,θ E ) to extract 512-dimensional time-space feature vectors: the extracted feature vectors are used and It means that E(·,θ E ) is a 3D residual network R3D-18 with 18 layers of depth, θ E are network parameters and are randomly initialized.

[0021] Step 3 is as follows:

[0022] The eigenvectors f 2k-1 and f 2k Input to the latent space embedding projection transformation subnetwork P(·,θ P ) to perform feature dimensionality reduction, and the low-dimensional embedding vector after dimensionality reduction is expressed as: h i =P(f 2k-1 ), h j =P(f 2k ), the dimension of the latent space is 128; P(·,θ P ) consists of a fully connected layer, with 512 and 128 input and output channels, respectively. P are network parameters and are randomly initialized.

[0023] Step 4 is as follows:

[0024] Using the low-dimensional embedding vector h in the latent space iand h j Calculate the contrast loss L ctr :

[0025]

[0026] in, Represents the similarity between positive sample pairs, i∈{1,2,…,2N}, j∈{1,2,…,2N}; 1 [k≠i] ∈{0, 1} is the indicator function of the positive and negative sample pairs. When k≠j, the value is 1. The similarity is expressed as h k It means that i 、h j Samples generated by different video enhancements, h i With h j Constitute a positive sample pair, h i With h k To form a negative sample pair, the temperature parameter τ is set to 0.07;

[0027] Step 5 is as follows:

[0028] The low-dimensional embedding vector from step 3 is input to the rotation angle classifier subnetwork D(·,θ D ), obtain the corresponding rotation angle prediction probability: i∈{1,2,…,2N},f i Represents the eigenvector f 2k-1 or 2k , D(·,θ D ) consists of two fully connected layers in series. The input and output channels of the first fully connected layer are 512 and 64, respectively, and the input and output channels of the second fully connected layer are 64 and 4, respectively. D are the parameters of the two fully connected layers and are randomly initialized.

[0029] Step 6 is as follows:

[0030] The angle prediction loss is defined as the angle prediction probability The sample was applied in step 1 The rotation angle σ on i The estimated error between: i∈{1, 2, …, 2N}, where d(·, ·) is the cross entropy function.

[0031] Step 7 is as follows:

[0032] Step 7.1: Calculate the training loss L = L ctr +λL cls , where the value of the regularization parameter λ is set to 0.3;

[0033] Step 7.2: Use the stochastic gradient descent algorithm combined with the training loss L to train the spatiotemporal feature extraction subnetwork E(·, θ E ), latent space embedding projection transformation subnetwork P(·,θ P ) and the rotation angle classifier subnetwork D(·,θ D ), and obtain the temporal-spatial feature contrast learning model.

[0034] The beneficial effect of the present invention is that the present invention is based on the time-space feature contrast learning method with rotation angle constraint, and performs time-space feature contrast learning by defining positive and negative sample pairs at the entire video level, so as to more effectively characterize the global context information of the entire video. At the same time, in order to overcome the disadvantage of contrast learning's weak description ability in temporal structure, the present invention also introduces a video rotation angle prediction task as a regularization term of contrast loss, further enhancing the time-space feature's ability to describe the spatial appearance information and temporal motion clues and their evolution of the video, and improving the overall description ability and discrimination of the feature. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of the time-space feature comparison learning method based on rotation angle constraint of the present invention; DETAILED DESCRIPTION

[0036] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] The present invention is based on the time-space feature comparison learning method of rotation angle constraint, such as Figure 1 The specific steps are as follows:

[0038] Step 1: extract several frames of images from each training video sample in the training data set, perform rotation transformation on the image, and generate enhanced transformation video samples;

[0039] Step 2: construct a time-space feature contrast learning model, which includes a time-space feature extraction subnetwork, a latent space embedding projection transformation subnetwork and a rotation angle classifier subnetwork; input the enhanced transformation video sample generated in step 1 into the time-space feature extraction subnetwork to extract the time-space feature vector, and randomly initialize the time-space feature extraction subnetwork;

[0040] Step 3: Input the feature vector described in step 2 into the latent space embedding projection transformation subnetwork to perform feature dimensionality reduction and generate a low-dimensional embedding vector;

[0041] Step 4: Calculate the contrast loss using the low-dimensional embedding vector in the latent space;

[0042] Step 5: Input the feature vector described in step 2 into the rotation angle classifier subnetwork to obtain the corresponding rotation angle prediction probability;

[0043] Step 6: Calculate the angle prediction loss according to the rotation angle prediction probability described in step 5 and the rotation transformation angle applied to the image in step 1;

[0044] Step 7: Train the temporal-spatial feature contrast learning model based on rotation angle constraint according to the contrast loss of step 4 and the angle prediction loss of step 6;

[0045] Step 8: Use the weights of the feature extraction subnetwork in the above self-supervised temporal-spatial feature contrast learning model as the initial parameters or feature extractor, and fine-tune the model parameters of supervised training on the visual recognition task of the downstream small-scale dataset to obtain the final classification result.

[0046] Step 1 is as follows:

[0047] For the training data set Each training video sample v in k Extract no less than 8 frames of images at equal intervals in time sequence, and crop each frame of image according to 112x112 pixels. The video downsample generated by the cropped image is expressed as

[0048] Downsampling the video Each frame of the image undergoes two independent rotation transformations T σ (·), generating two enhanced transformed video samples and Where T σ The rotation parameter σ of (·) is a rotation angle randomly selected from the set {0°, 90°, 180°, 270°}; N video samples can generate a total of 2N enhanced transformed video samples; two enhanced transformed video samples generated from the same video are defined as positive sample pairs, and video sample pairs formed in other cases are all negative sample pairs.

[0049] Step 2:

[0050] The enhanced transformed video samples are and Input to the spatiotemporal feature extraction subnetwork E(·,θ E ) to extract 512-dimensional time-space feature vectors: the extracted feature vectors are used and It means that E(·,θ E ) is a 3D residual network R3D-18 with 18 layers of depth, θ E are network parameters and are randomly initialized.

[0051] Step 3 is as follows:

[0052] The eigenvectors f 2k-1 and f 2k Input to the latent space embedding projection transformation subnetwork P(·,θ P ) to perform feature dimensionality reduction, and the low-dimensional embedding vector after dimensionality reduction is expressed as: h i =P(f 2k-1 ), h j =P(f 2k ), the dimension of the latent space is 128; P(·,θ P ) consists of a fully connected layer, with 512 and 128 input and output channels, respectively. P are network parameters and are randomly initialized.

[0053] Step 4 is as follows:

[0054] Using the low-dimensional embedding vector h in the latent space i and h j Calculate the contrast loss L ctr :

[0055]

[0056] in, Represents the similarity between positive sample pairs, i∈{1,2,…,2N}, j∈{1,2,…,2N}; 1 [k≠i] ∈{0, 1} is the indicator function of the positive and negative sample pairs. When k≠j, the value is 1. The similarity is expressed as h k It means that i 、h j Samples generated by different video enhancements, h i With h j Constitute a positive sample pair, h i With h k To form a negative sample pair, the temperature parameter τ is set to 0.07;

[0057] Step 5 is as follows:

[0058] The low-dimensional embedding vector from step 3 is input to the rotation angle classifier subnetwork D(·,θ D ), obtain the corresponding rotation angle prediction probability: i∈{1, 2, …, 2N}, f i Represents the eigenvector f 2k-1 or 2k , D(·,θ D) consists of two fully connected layers in series. The input and output channels of the first fully connected layer are 512 and 64, respectively, and the input and output channels of the second fully connected layer are 64 and 4, respectively. D are the parameters of the two fully connected layers and are randomly initialized.

[0059] Step 6 is as follows:

[0060] The angle prediction loss is defined as the angle prediction probability The sample was applied in step 1 The rotation angle σ on i The estimated error between: i∈{1, 2, …, 2N}, where d(·, ·) is the cross entropy function.

[0061] Step 7 is as follows:

[0062] Step 7.1: Calculate the training loss L = L ctr +λL cls , where the value of the regularization parameter λ is set to 0.3;

[0063] Step 7.2: Use the stochastic gradient descent algorithm combined with the training loss L to train the spatiotemporal feature extraction subnetwork E(·, θ E ), latent space embedding projection transformation subnetwork P(·,θ P ) and the rotation angle classifier subnetwork D(·,θ D ), and obtain the temporal-spatial feature contrast learning model.

[0064] Example

[0065] 1. Simulation conditions:

[0066] The simulation experiment of the present invention was carried out in a hardware environment of 4 NVIDIA Titan X GPUs and a software environment of the PyTorch deep learning framework.

[0067] 2. Simulation content:

[0068] The two data sets used in the simulation experiment of the present invention are the internationally disclosed human action recognition benchmark sets Kinectics-400 and UCF101, the former of which comes from the reference ("The Kinetics Human Action Video Dataset", arXiv preprint arXiv:1705.06950, 2017), and the latter of which comes from the reference ("UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild", arXiv preprintarXiv:1212.0402, 2012). Kinectics-400 with ultra-large-scale training samples is used as a pre-training data set for self-supervised contrastive learning of spatiotemporal features, while the smaller-scale UCF101 data set is used as a supervised transfer learning parameter fine-tuning and test data set for downstream action recognition tasks.

[0069] In the pre-training stage, the training subset of Kinectics-400 was used for training. The stochastic gradient descent algorithm was used to learn the network parameters. The number of training rounds was set to 50, and the batch size of each iteration on each GPU was 32. The initial learning rate was set to 0.01, and the learning rate was decayed by 10 times in the 15th and 30th rounds. At the same time, the training samples were randomly cropped, horizontally flipped, Gaussian blurred, and color jittered.

[0070] In the transfer learning parameter fine-tuning stage, the training subset of UCF101 is used for training. Since the UCF101 dataset contains 101 categories, the number of output channels of the fully connected layer in the classifier is 101. The network parameters are fine-tuned using the stochastic gradient descent algorithm, the number of training rounds is set to 300, and the batch size of each iteration on each GPU is 32. The initial learning rate is set to 0.001, and the learning rate is decayed by 10 times in the 150th and 250th rounds. At the same time, the training samples are randomly cropped, horizontally flipped, and color jittered for enhancement.

[0071] In the test phase, the UCF101 test subset is used to evaluate the recognition performance. First, five 16-frame clips are extracted from each test video at equal intervals, and then the mean of the predicted probabilities of these five clips is calculated as the final classification result of the test sample.

[0072] In order to compare the effect of the spatiotemporal feature learning method of the present invention, the classic R3D-18 network based on random parameter initialization is used as a comparison benchmark in the experiment. At the same time, a variant of the method proposed by the present invention is also implemented in the experiment, that is, the loss function after the rotation angle prediction regularization constraint is removed to train the network, which is used to examine the influence of the regularization constraint term on the spatiotemporal feature extraction effect. The classification accuracy comparison results obtained by the simulation method are shown in Table 1.

[0073] 3. Simulation effect analysis:

[0074] Table 1 is a comparison of the classification accuracy of the three methods in simulation. It can be seen from Table 1 that the present invention can effectively train the feature extractor to extract spatiotemporal features that can better characterize the essential attributes of the action category from the input data. At the same time, the rotation angle regularization term can further improve the recognition performance based on the single contrastive learning method, which fully demonstrates that the regularization constraint and contrastive learning have strong descriptive complementarity. Compared with the traditional R3D-18 method that initializes random parameters and trains from scratch on the UCF101 dataset, the present method effectively improves the spatiotemporal description ability of the feature for the action, enhances the category discrimination of the model, and improves the classification accuracy.

[0075] Table 1 List of classification accuracy obtained by three methods in simulation

[0076] Simulation Method Classification accuracy Classification method of the present invention (contrastive learning + rotation angle prediction) 77.1％ Classification method of the present invention (only comparative learning) 72.4％ R3D-18 (parameters are randomly initialized) 42.5％

[0077] In summary, the present invention designs a temporal-spatial feature contrast learning network architecture based on rotation angle constraints, which can effectively guide the 3D convolutional neural network to learn video-level temporal-spatial expressions with stronger representation and generalization capabilities, and use this as a feature extractor for downstream recognition tasks or the initial value of the network parameters for transfer learning, thereby further improving the classification performance of the downstream recognition model.

Claims

1. A temporal-spatial feature contrast learning method based on rotation angle constraints, characterized in that: By defining positive and negative sample pairs at the entire video level to perform temporal-spatial feature contrast learning, and introducing the video rotation angle prediction task as a regularization term for the contrast loss, the temporal-spatial feature is further enhanced to depict the spatial appearance information and temporal motion clues and their evolution of the video, thereby improving the overall description ability and discrimination of the feature. The specific steps are as follows: Step 1: extract several frames of images from each training video sample in the training data set, perform rotation transformation on the image, and generate enhanced transformation video samples; Step 2: construct a time-space feature contrast learning model, which includes a time-space feature extraction subnetwork, a latent space embedding projection transformation subnetwork and a rotation angle classifier subnetwork; input the enhanced transformation video sample generated in step 1 into the time-space feature extraction subnetwork to extract the time-space feature vector, and randomly initialize the time-space feature extraction subnetwork; Step 3: Input the feature vector described in step 2 into the latent space embedding projection transformation subnetwork to perform feature dimensionality reduction and generate a low-dimensional embedding vector; Step 4: Calculate the contrast loss using the low-dimensional embedding vector in the latent space; Step 5: Input the feature vector described in step 2 into the rotation angle classifier subnetwork to obtain the corresponding rotation angle prediction probability; Step 6: Calculate the angle prediction loss according to the rotation angle prediction probability described in step 5 and the rotation transformation angle applied to the image in step 1; Step 7: Train the temporal-spatial feature contrast learning model based on rotation angle constraint according to the contrast loss of step 4 and the angle prediction loss of step 6; Step 8: Use the weights of the feature extraction subnetwork in the above self-supervised temporal-spatial feature contrast learning model as the initial parameters or feature extractor, and fine-tune the model parameters of supervised training on the visual recognition task of the downstream small-scale dataset to obtain the final classification result.

2. The time-space feature contrast learning method based on rotation angle constraint according to claim 1 is characterized in that: Step 1 is as follows: For the training data set Each training video sample v in k Extract no less than 8 frames of images at equal intervals in time sequence, and crop each frame of image according to 112x112 pixels. The video downsample generated by the cropped image is expressed as Downsampling the video Each frame of the image is rotated twice independently to generate two enhanced transformed video samples and Where T σ The rotation parameter σ of (·) is a rotation angle randomly selected from the set {0°, 90°, 180°, 270°}; N video samples can generate a total of 2N enhanced transformed video samples; two enhanced transformed video samples generated from the same video are defined as positive sample pairs, and video sample pairs formed in other cases are all negative sample pairs.

3. The time-space feature contrast learning method based on rotation angle constraint according to claim 2 is characterized in that: Step 2: The enhanced transformed video samples are and Input to the spatiotemporal feature extraction subnetwork E(·,θ E ) to extract 512-dimensional time-space feature vectors: the extracted feature vectors are used and It means that E(·,θ E ) is a 3D residual network R3D-18 with 18 layers of depth, θ E are network parameters and are randomly initialized.

4. The time-space feature contrast learning method based on rotation angle constraint according to claim 3 is characterized in that: Step 3 is as follows: The eigenvectors f 2k-1 and f 2k Input to the latent space embedding projection transformation subnetwork P(·,θ P ) to perform feature dimensionality reduction, and the low-dimensional embedding vector after dimensionality reduction is expressed as: h i =P(f 2k-1 ), h j =P(f 2k ), the dimension of the latent space is 128; P(·,θ P ) consists of a fully connected layer, with 512 and 128 input and output channels, respectively. P are network parameters and are randomly initialized.

5. The time-space feature contrast learning method based on rotation angle constraint according to claim 4 is characterized in that: Step 4 is as follows: Using the low-dimensional embedding vector h in the latent space i and h j Calculate the contrast loss L ctr : in, Represents the similarity between positive sample pairs, i∈{1,2,…,2N}, j∈{1,2,…,2N}; 1 [k≠i] ∈{0,1} is the indicator function of the positive and negative sample pairs. When k≠j, the value is 1. The similarity is expressed as h k It means that i 、h j Samples generated by different video enhancements, h i With h j Constitute a positive sample pair, h i With h k To form a negative sample pair, the temperature parameter τ is set to 0.

07.

6. The time-space feature contrast learning method based on rotation angle constraint according to claim 3 is characterized in that: Step 5 is as follows: The low-dimensional embedding vector from step 3 is input to the rotation angle classifier subnetwork D(·,θ D ), obtain the corresponding rotation angle prediction probability: f i represents the eigenvector f 2k-1 or 2k , D(·,θ D ) consists of two fully connected layers in series. The input and output channels of the first fully connected layer are 512 and 64, respectively, and the input and output channels of the second fully connected layer are 64 and 4, respectively. D are the parameters of the two fully connected layers and are randomly initialized.

7. The time-space feature contrast learning method based on rotation angle constraint according to claim 3 is characterized in that: Step 6 is as follows: The angle prediction loss is defined as the angle prediction probability The sample was applied in step 1 The rotation angle σ on i The estimated error between: where d(·,·) is the cross entropy function.

8. The time-space feature contrast learning method based on rotation angle constraint according to claim 3 is characterized in that: Step 7 is as follows: Step 7.1: Calculate the training loss L = L ctr +λL cls , where the value of the regularization parameter λ is set to 0.3; Step 7.2: Use the stochastic gradient descent algorithm combined with the training loss L to train the spatiotemporal feature extraction subnetwork E(·,θ E ), latent space embedding projection transformation subnetwork P(·,θ P ) and the rotation angle classifier subnetwork D(·,θ D ), and obtain the trained spatiotemporal feature contrast learning model.

Images