A video clustering method based on self-supervised learning

Abstract

The present invention discloses a video clustering method based on self-supervision. The method uses a self-supervision method to learn and obtain the feature values ​​of a video in RGB and optical flow, and cluster the feature values. In order to achieve this goal, the nearest neighbor of the video in RGB and optical flow is calculated as a proxy task to realize label-free video feature extraction. In addition, each frame of the video is clustered, and the clustering label with the highest number of appearances in each video is used as the static clustering label of the video. Finally, the clustering features directly extracted from the video are used as the dynamic features of the video, and the final video clustering results are obtained in combination with the static clustering labels. The present invention can extract discriminative feature representations on the data set of video recognition, effectively improving the accuracy of the video alignment problem.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a video clustering method based on self-supervised learning. Background Art

[0002] Clustering is a technique widely used in fields such as machine learning, data mining, and statistical analysis. Its purpose is to group objects that are similar to each other into the same set, and different objects into different sets. The purpose of video clustering is to group videos of the same type into the same class, and videos of different classes into different classes. Unlike traditional video classification methods, video clustering methods do not need to consider the true labels of the videos. Traditional clustering methods, such as Kmeans and spectral clustering, require specific feature extraction methods when they are applied to video data, and there are problems such as disappearing cluster boundaries. For high-dimensional video data, traditional methods cannot guarantee that feature information is effectively utilized, resulting in poor performance.

[0003] Due to the development of deep neural networks in recent years, clustering methods based on deep learning have been improved. Many studies have combined deep neural networks with the loss function of deep clustering methods to learn video feature representations that are more suitable for clustering. Recently, self-supervised feature learning has performed well in video recognition tasks. In the process of extracting feature values ​​of video frames, the main challenge comes from the fact that classical feature extraction methods are more suitable for feature classification rather than clustering. Therefore, deep clustering methods can be combined with self-supervised video feature learning to obtain video features suitable for clustering. Summary of the invention

[0004] Purpose of the invention: In this paper, we do not directly classify the extracted video features to complete the recognition task, but calculate the nearest neighbors from different perspectives of RGB and optical flow, shorten the distance between the feature values ​​of the nearest neighbors, make the video features have clustering characteristics, and use clustering and reconstruction loss functions to achieve self-supervised video clustering, thereby completing the self-supervised video recognition task. The present invention provides a video clustering method based on self-supervised learning.

[0005] Technical solution: A video clustering method based on self-supervised learning, characterized in that it includes the following steps:

[0006] Step 1: Video V i Use two different sampling strategies to sample and get the sequence V containing global information i_l and a sequence V containing local information i_s ;

[0007] Step 2: Video V i Encode on RGB and optical flow respectively to get the eigenvalue f 1 (V i ), f2 (V i );

[0008] Step 3: Set the eigenvalue f in RGB 1 (V i ) in f 1 The k nearest neighbors in (V) are f 1 (V i1 ), ..., f 1 (V ik ), shorten the distance of the corresponding optical flow eigenvalue, i.e., f 2 (V i ) and f 2 (V i1 ), ..., f 2 (V ik ) distance;

[0009] Step 4: Repeat step 3 to narrow the distance between the RGB feature values ​​guided by the nearest neighbor of the optical flow;

[0010] Step 5: Since different clips of the same video belong to the same category, zoom in 1 (V i ) and f 1 (V i_s ) and f 1 (V i_l ) distance;

[0011] Step 6: For each video V in the dataset V i Extract video frame X ij ;

[0012] Step 7: Use conditional variational autoencoder to cluster all images of X to obtain video static labels;

[0013] Step 8: Use the video feature values ​​obtained in steps 3, 4, and 5, combined with the video static labels obtained in step 7, to perform video clustering to obtain video cluster labels.

[0014] Furthermore, in step 1, sequence V i_l The sampling strategy adopted is to divide the video into segments of equal length and select one frame from each segment to form a new sequence; the sequence V i_s The sampling strategy adopted is to randomly select a starting point and cut a fixed-length video segment as a new sequence.

[0015] Furthermore, in step 2, the encoding uses an S3D network to extract features.

[0016] Furthermore, in step 3, the normalized temperature-scale cross entropy loss is used to close f 2 (V i ) and f2 (V i1 ), ..., f 2 (V ik ) distance.

[0017] Furthermore, in step 5, relative entropy loss is used to bring f closer 1 (V i ) and f 1 (V i_s ) and f 1 (V i_l ) distance.

[0018] Furthermore, in step six, the picture frame under the video RGB is extracted.

[0019] Furthermore, in step seven, the conditional variational autoencoder clusters the images, where the video labels of the frames are used as conditional inputs of the autoencoder.

[0020] Furthermore, in step eight, Kmeans is used to cluster the video feature values, and the feature values ​​are updated through a clustering loss function.

[0021] Furthermore, in step eight, the clustering loss function is defined as the nearest neighbor log-likelihood loss plus the cross entropy loss to prevent clustering degradation.

[0022] Beneficial effects: For video clustering, the nearest neighbor calculation of the video in RGB and optical flow is used as a proxy task to obtain video feature values ​​that are more suitable for clustering. At the same time, the dynamic features of the video are combined with the static image clustering method to optimize the feature value extraction method, thereby improving the video clustering performance. The invention can extract discriminative feature representations on the video recognition data set, effectively improving the accuracy of the video alignment problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of the method of the present invention;

[0024] Figure 2 It is the algorithm framework diagram of the present invention;

[0025] Figure 3 This is a comparison diagram between the present invention and other methods. DETAILED DESCRIPTION

[0026] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0027] This embodiment provides a video clustering method based on self-supervision and a method for video recognition of the UCF101 action dataset. By extracting features from the video and performing deep clustering on the extracted feature values, good video recognition effects can be achieved.

[0028] The process of this method is as follows Figure 1 As shown in Figure 2 Shown

[0029] Step 1: Video V i Use two different sampling strategies to sample and get the sequence V containing global information i_l and a sequence V containing local information i_s , sequence V i_l The sampling strategy adopted is to divide the video into segments of equal length and select one frame from each segment to form a new sequence; the sequence V i_s The sampling strategy adopted is to randomly select a starting point and cut a fixed-length video segment as a new sequence;

[0030] Step 2: Video V i Encode on RGB and optical flow respectively to get the eigenvalue f 1 (V i ), f 2 (V i ), the encoding uses S3D network for feature extraction;

[0031] Step 3: Set the eigenvalue f in RGB 1 (V i ) in f 1 The k nearest neighbors in (V) are f 1 (V i1 ), ..., f 1 (V ik ), we narrow the distance corresponding to the optical flow eigenvalue, i.e., f 2 (V i ) and f 2 (V i1 ), ..., f 2 (V ik ), using the normalized temperature-scale cross entropy loss to get closer to f 2 (V i ) and f 2 (V i1 ), ..., f 2 (V ik ) distance;

[0032] Step 4: Repeat step 3, we also close the distance of RGB feature values ​​guided by the nearest neighbor of optical flow;

[0033] Step 5: Since different clips of the same video belong to the same category, relative entropy loss is used to bring f closer 1 (V i ) and f 1 (V i_s) and f 1 (V i_l ) distance;

[0034] Step 6: For each video V in the dataset V i Extract video frame X ij ;

[0035] Step 7: Use conditional variational autoencoder to cluster all images of X to obtain video static labels;

[0036] Step 8: Use the video feature values ​​obtained in steps 3, 4, and 5, combined with the video static labels obtained in step 7 to perform video clustering to obtain video clustering labels, use Kmeans to cluster the video feature values, and update the feature values ​​through a clustering loss function, which is defined as the nearest neighbor log-likelihood loss plus a cross entropy loss to prevent clustering degradation.

[0037] In this embodiment, the UCF101 action dataset consists of 101 types of human action videos, totaling 13,320 videos, covering common human action categories, including playing the piano, diving, riding a bicycle, playing baseball, etc. The video frame is a 320×240 color image.

[0038] During training, the learning rate of the Adam optimizer is set to 1×10 -4 The feature encoder loss is the weighted sum of the clustering loss and the self-supervised video feature loss, and the variance loss weight λ is set to 10 -2 In the clustering loss, the weight of the cross entropy loss function is set to 2. Before video feature extraction, the video frames are randomly cropped into 224×224 color images for S3D encoding. The model is trained 300 times with a batch size of 128.

[0039] We compare the state-of-the-art self-supervised video learning method (CoCLR) based on RGB and optical flow with our method on the UCF101 dataset. Figure 3 As shown in the figure, our method performs better than CoCLR. In terms of video clustering accuracy, our method is 7.0% higher than CoCLR+Kmeans, and the clustering evaluation index ARI is 0.0735 higher. On the UCF101 subset UCF10, our method is 18.9% higher than CoCLR+Kmeans, and the clustering evaluation index ARI is 0.1803 higher. The main reason is that the video features obtained by the current self-supervised video feature extraction algorithm do not have characteristics suitable for classification. This also means that the algorithm proposed in this invention has great advantages in practical applications.

Claims

1. A video clustering method based on self-supervised learning, characterized in that: The following steps are involved: Step 1: Video V i Two different sampling strategies are used to sample and obtain the sequence V containing global information. i_l and a sequence V containing local information i_s ; Step 2: Video V i Encode on RGB and optical flow respectively to get the feature value f1(V i ),f2(V i ); Step 3: The characteristic value f1(V i ) in f1(V) is the k nearest neighbor of f1(V i1 ),…,f1(V ik ), close the distance of the corresponding optical flow feature value, that is, f2(V i ) and f2(V i1 ),…,f2(V ik ) distance; Step 4: Repeat step 3 to narrow the distance between the RGB feature values ​​guided by the nearest neighbor of the optical flow; Step 5: Since different clips of the same video belong to the same category, zoom in on f1(V i ) and f1(V i_s ) and f1(V i_l ) distance; Step 6: For each video V in the dataset V i Extract video frame X ij ; Step 7: Use conditional variational autoencoder to cluster all images of X to obtain video static labels; Step 8: Use the video feature values ​​obtained in steps 3, 4, and 5, combined with the video static labels obtained in step 7, to perform video clustering to obtain video cluster labels.

2. The video clustering method based on self-supervised learning according to claim 1, characterized in that: In step 1, sequence V i_l The sampling strategy adopted is to divide the video into segments of equal length and select one frame from each segment to form a new sequence; the sequence V i_s The sampling strategy adopted is to randomly select a starting point and cut a fixed-length video segment as a new sequence.

3. The video clustering method based on self-supervised learning according to claim 1, characterized in that: In step 2, the encoding uses an S3D network for feature extraction.

4. The video clustering method based on self-supervised learning according to claim 1, characterized in that: In step 3, the normalized temperature-scale cross entropy loss is used to pull f2(V i ) and f2(V i1 ),…,f2(V ik )distance.

5. The video clustering method based on self-supervised learning according to claim 1, characterized in that: In step 5, relative entropy loss is used to close f1(V i ) and f1(V i_s ) and f1(V i_l ) distance.

6. The video clustering method based on self-supervised learning according to claim 1, characterized in that: In step six, the image frame under the video RGB is extracted.

7. The video clustering method based on self-supervised learning according to claim 1, characterized in that: In step 7, the conditional variational autoencoder clusters the images, where the video label of the frame is used as the conditional input of the autoencoder.

8. The video clustering method based on self-supervised learning according to claim 1, characterized in that: In step eight, Kmeans is used to cluster the video feature values, and the feature values ​​are updated through the clustering loss function.

9. The video clustering method based on self-supervised learning according to claim 8, characterized in that: The clustering loss function is defined as the nearest neighbor log-likelihood loss, coupled with a cross-entropy loss to prevent clustering degradation.

Images