An unsupervised representation learning method based on relation comparison

Abstract

The present invention discloses an unsupervised learning method based on relational comparison. Specifically, the present invention constructs a loss function by comparing the similarity between instances, guides the training of a neural network, enables the neural network model to extract its most essential features from unlabeled training data, and can achieve better results in downstream tasks such as video image classification and target recognition. The present invention utilizes the similarity comparison between samples instead of the positive and negative sample instance comparison, avoiding the performance bottleneck caused by inappropriate selection of positive and negative samples.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision, deep learning, etc., and relates to unsupervised representation learning technology, specifically an unsupervised representation learning method for relational comparison. Background Art

[0002] Deep learning has achieved great success in the fields of computer vision and natural language processing. Most deep learning models are trained in a supervised manner. However, supervised methods require a large amount of labeled data, which is time-consuming and expensive, which has promoted the exploration of unsupervised learning methods.

[0003] Unsupervised learning is to learn feature representations from a large amount of unlabeled data. Its starting point is to learn more effective feature representations from the data itself rather than labels. Early methods usually use pre-tasks to generate supervisory signals, such as jigsaw puzzles, coloring, and rotation prediction. However, these methods are difficult to generalize to different downstream tasks and are highly dependent on the design quality of the pre-tasks. Afterwards, contrastive methods based on instance discrimination were used for representation learning without pre-tasks, and significant progress was made. MoCo proposed a sample dictionary library with a momentum update strategy to store negative samples. In SimCLR, the calculation of feature embedding uses a larger batch size and introduces nonlinear projection to improve the quality of learned representations. However, these methods rely on a large batch size or sample dictionary library to retrieve negative sample pairs, which has high computational cost and large redundancy of operations. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an unsupervised representation learning method based on relationship comparison, which utilizes the similarity comparison between samples instead of the positive and negative sample instance comparison, thus avoiding the performance bottleneck caused by inappropriate selection of positive and negative samples.

[0005] The present invention first enhances the data according to the original data. Then, two identical networks are used to generate different enhanced embedding representations of the same sample. Then, an instance queue is constructed, and a loss function is constructed by comparing the similarity distribution of the instances in the queue with the current instance, and the network is trained by back propagation.

[0006] The following describes the method in detail:

[0007] First, the given samples are randomly transformed into different enhanced versions through the data enhancement module. The appropriate enhancement strategy can be selected according to the specific type of data and the model. For example, the video sequence can adopt time / space enhancement, the picture sequence can be randomly rotated, cropped, blurred, color changed, etc., and the bone data can adopt enhancement strategies such as rotation and cropping. Among them, rotation is a linear transformation that moves the bone joint points in a fixed direction as a spatial dimension enhancement; cropping is to fill multiple frames symmetrically onto the sequence, and then randomly crop it to the original length as a temporal dimension enhancement.

[0008] Then, two student networks f with trainable parameters and the same structure are used s and teacher network t As a feature encoder, the enhanced versions x and x′ are embedded in the hidden space respectively. You can choose a suitable feature encoder according to the data type. For example, video data can be but not limited to I3D, image data can be but not limited to Resnet, and the skeleton sequence representing human body movements can use a graph convolutional neural network as an encoder. Finally, the output of the encoder is expanded to obtain two feature vectors h and h′ of length M. Then use two projection layers g and g′ with the same structure to project the hidden feature vectors h and h′ obtained in the previous step to a higher dimensional space. The projection layer contains two linear layers. The first layer contains a batch normalization layer and a Relu activation layer. The output length is M1, and the output length of the second layer is M2. Generally, M1 is an integer multiple of M2, for example, M1=512, M2=128.

[0009] Next, in order to calculate the similarity with the current instance, a "first in first out" updated instance queue Q = [a0, a1, a2, ..., a N-1 ], used to save the representation of other instance samples in the previous batch. Here N represents the queue length, a i Represents the feature embedding representation of the instance in the queue. By calculating p(z), z′ and a i The similarity between them is the similarity set P s and And Do normalization and get P t .

[0010]

[0011]

[0012] in Represents the student network prediction value after L2 standardization With the i-th instance a in the queue i The similarity between them, τ is the temperature coefficient of the student network, and (·) represents the inner product operation between two features to measure the similarity. Similarly, represents the L2-normalized teacher network projection z′ and the i-th instance a in the queue i The similarity between them, τ′ is the temperature coefficient of the teacher network. The similarity set between the output feature representation of the student network and the teacher network and the feature representation in the queue can be expressed as:

[0013]

[0014]

[0015] use The similarity set obtained by normalizing the maximum value of can be expressed as:

[0016]

[0017] Push z′ into the queue according to the first-in-first-out principle.

[0018] Then, by minimizing the two similarity sets P obtained in the previous step s and P t The cross entropy optimization network between:

[0019]

[0020] When the network trained according to the above method tends to be stable, that is, the loss value no longer decreases, continue to randomly extract data samples from the data set, randomly enhance them, extract features through the feature encoder, and calculate the similarity with all features in the queue Q to form a similarity set P s and P t , and update the queue Q. t The values ​​of the K sample subsets with the highest values ​​are set to 1. That is:

[0021]

[0022] The similarity is then calculated and the parameters of the student network are updated.

[0023] The resulting feature encoder is used as the initial model for various downstream tasks, such as classification, object detection and segmentation.

[0024] The advantages and beneficial effects of the present invention are as follows:

[0025] 1. The present invention proposes to use the similarity comparison between samples instead of the positive and negative sample instance comparison, thereby avoiding the performance bottleneck caused by inappropriate selection of positive and negative samples.

[0026] 2. The present invention proposes a learning strategy for positive sample expansion, which uses the mined non-self-positive samples to guide the learning of the student network, enriches the diversity of intra-class representations, and improves the representation learning performance.

[0027] 3. The present invention proposes an unsupervised action recognition method based on relational comparison. During the training process, only the unlabeled data itself is used as supervision information, and no manual labeling of action categories is required, which greatly alleviates the consumption of manpower and material resources.

[0028] 4. The unsupervised framework proposed in this invention is evaluated on the commonly used human motion classification tasks based on 3D skeleton datasets, including NTU-60 and NTU-120 datasets, and achieves more accurate motion classification results compared to the current state-of-the-art methods under unsupervised experimental conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The present invention is a flowchart of an unsupervised action recognition method based on skeleton data.

[0030] Figure 2 This is a comparison chart of the results on the NTU-60 dataset.

[0031] Figure 3 This is a comparison chart of the results on the NTU-120 dataset. DETAILED DESCRIPTION

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and through specific embodiments. The following embodiments are only illustrative and not restrictive, and the protection scope of the present invention cannot be limited thereto.

[0033] First, the original bone data is enhanced using spatial and temporal enhancement methods tailored to the characteristics of bone data. Then, two identical networks are used to generate different enhanced representations of the same bone sample. After that, an instance queue is constructed, and non-self positive samples are discovered by comparing the similarity distribution between the instances in the queue and the current instance. Finally, the mined positive sample information is used to regularize the similarity distribution, completing the representation learning of positive sample expansion.

[0034] Figure 1 The structure flow chart of an unsupervised action recognition method based on skeleton data of the present invention.

[0035] The overall architecture process of the present invention is to use two identical encoding networks with projection heads to generate different enhanced representations from the same skeleton sample, and then calculate the similarity distribution between the current instance and the instances in the queue as comparison objects to mine non-self positive samples, and complete the positive sample expansion learning through similarity distribution regularization.

[0036] The data sets used in the present invention are NTU RGB+D and NTU RGB+D 120. The NTU-RGB+D 60 (NTU-60) data set is one of the largest indoor data sets for human action recognition tasks and is also the most widely used public data set. The data set contains a total of 56,880 action clips and 60 action categories, which are performed by 40 performers respectively and are captured simultaneously by three cameras at different horizontal angles and heights in an indoor laboratory environment. The data mode of NTU RGB+D includes 3D skeletons, RGB videos and depth sequences, and 3D skeleton data is used in the experiment of the present invention. One frame of skeleton data contains 25 joint points, and in the camera coordinate system, the three-dimensional joint point position coordinates (X, Y, Z) are given by the provided annotations. The data set has two benchmarks: 1) Cross-subject benchmark: 40,320 fragments are used for training samples and 16,560 fragments are used for verification samples. In this setting, different performers are used to split the training and validation data; 2) Cross view benchmark: 37920 clips are used for training and 18960 clips are used for validation. The camera view numbers of the training data in this setting are 2 and 3, while the camera view numbers of the Figure 1 The data in is used as verification data.

[0037] The NTU-RGB+D 120 (NTU-120) dataset is an extended version of NTU-60. There are a total of 114,480 action clips divided into 120 action categories. Most of the settings of NTU-120 are the same as NTU-60. NTU-120 also contains two benchmarks. 1) Cross subject: contains 63,026 training samples and 50,919 validation samples. The training data and validation data come from different performers. 2) Cross set: 54,468 samples are used for training and 59,477 samples are used for validation. The training data and validation data are divided by different setting IDs.

[0038] First, the input 3D skeleton data format is T frames, V joint points, and C coordinate channels, where T = 300, V = 25, C = 3, then the skeleton sequence s can be expressed as s∈R C×T×V Through the data enhancement module, the given original skeleton data s is randomly transformed into different enhancements. Rotation and cropping are used as enhancement strategies here, where rotation is a linear transformation that moves the skeleton joint points along a fixed direction, which can be regarded as an enhancement of the spatial dimension; cropping is to symmetrically fill multiple frames onto the sequence and then randomly crop them to the original length, which can be regarded as an enhancement of the temporal dimension.

[0039] Then, two teacher networks f with trainable parameters and the same structure are used tand student network s As a feature encoder, the enhanced versions x and x′ are embedded into the latent space respectively. Since the enhanced skeleton sequence is represented in the form of a graph, a graph convolutional neural network is used as an encoder to encode the temporal motion of the human body posture and joints. Finally, a global pooling operation is performed on the enhanced skeleton graph to obtain a 256-dimensional feature vector h and h′. Then two projection layers g and g′ with the same structure are used to project the hidden vectors h and h′ obtained in the previous step to a higher dimensional space. The projection layer contains two linear layers. The first layer contains a batch normalization layer and a Relu activation layer with an output dimension of 512, and the second layer has an output dimension of 128. Through this nonlinear transformation, more information can be generated and retained in h, resulting in a more expressive representation.

[0040] Next, in order to calculate the similarity with the current instance, a "first in first out" updated instance queue Q = [a0, a1, a2, ..., a N-1 ], used to save the representation of other instance samples in the previous batch. Here N represents the queue length, a i Represents the feature embedding representation of the instance in the queue. Then calculate p(z), z′ and a i The similarity between:

[0041]

[0042]

[0043] in Represents the student network prediction value after L2 standardization With the i-th instance a in the queue i The similarity between them, τ is the temperature coefficient of the student network, and (·) represents the inner product operation between two features to measure the similarity. Similarly, represents the L2-normalized teacher network projection z′ and the i-th instance a in the queue i The similarity between them, τ′ is the temperature coefficient of the teacher network. The similarity set between the output features of the student network and the teacher network and the queue can be expressed as:

[0044]

[0045]

[0046] use The similarity set obtained by normalizing the maximum value of can be expressed as:

[0047]

[0048] Then, by minimizing the two similarity sets P obtained in the previous steps and P t The cross entropy optimization network between:

[0049]

[0050] When the network trained according to the above method tends to be stable, that is, the loss value no longer decreases, continue to randomly extract data samples from the data set, randomly enhance them, extract features through the feature encoder, and calculate the similarity with all features in the queue Q to form a similarity set P s and P t , and update the queue Q. t The values ​​of the K sample subsets with the highest values ​​are set to 1. That is:

[0051]

[0052] The similarity is then calculated and the parameters of the student network are updated.

[0053] Finally, the pre-trained model is verified by linear evaluation of the action recognition task. Here, a linear classifier (a fully connected layer followed by a softmax layer) is used for supervised training on the fixed representation of the pre-trained model to obtain the final action classification result.

[0054] The present invention uses the pytorch deep learning framework for experiments, and the specific parameters are shown in the figure below:

[0055]

[0056] After the model is trained to convergence, it is evaluated on the NTU-60 dataset and the NTU-120 dataset. The evaluation results are as follows: Figure 2 and Figure 3As shown, it can be seen that the action recognition of the proposed method on both datasets exceeds the previous mainstream methods. On the NTU-60 dataset, the proposed method can achieve a recognition accuracy of 87.0% in the experiment based on the X-view task, and a recognition accuracy of 83.2% in the experiment based on the X-sub task. The recognition accuracy of the X-sub and X-set tasks on the NTU-120 dataset is 73.0% and 74.0%, respectively. The results show that compared with the unsupervised method, the proposed method has achieved leading performance on all datasets used. Specifically, compared with CrossCLR, the accuracy of the X-view and X-sub tasks on the NTU-60 dataset is increased by 3.6 and 5.4 percentage points, respectively, and the accuracy of the X-sub and X-set tasks on the NTU-120 dataset is increased by 5.1 and 7.3 percentage points, respectively. In addition, the learning effect of the proposed method on the dataset used also surpasses the supervised ST-GCN, and the superior performance proves that the proposed method of the present invention can learn more expressive skeletal action representations.

[0057] The above is only a preferred implementation of the patent of the present invention, but the protection scope of the patent of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and invention concept of the patent within the scope disclosed by the patent of the present invention, which are all within the protection scope of the patent of the present invention.

Claims

1. An unsupervised representation learning method based on relation comparison, characterized in that: The steps include: 1) Extract the original data s from the available image dataset and enhance it to obtain two different enhanced versions x and x′; 2) Construct a pair of feature encoders with exactly the same structure, called the student network f s and teacher network t And use the encoder to extract features from the enhanced instances x and x′ to obtain embedded features h and h′; 3) Construct a pair of nonlinear mapping layers g and g′ with the same structure to map x and x′ to the high-dimensional feature space to obtain high-dimensional representations z and z′. Then, the student network adds a prediction network p with the same structure as the nonlinear mapping layer to generate the prediction representation p(z). The stop gradient operation is used in the teacher network. 4) Construct instance queue Q = [a0, a1, a2, ..., a N-1 ]Save the representation of other instance samples in the previous batch and calculate the similarity with the current instance by calculating p(z), z′ and a i The similarity between them is used to get the similarity set and And After normalization, we get According to the first-in-first-out principle, push z′ into the queue Q. Here N represents the queue length, a i A feature embedding representation representing the instances in the queue; 5) Calculate the loss function Derivative the loss function, use the gradient descent method to update the parameters of the student network, and then randomly extract new data samples from the image dataset and repeat the above operation until the value of the loss function no longer decreases. After every M operations, assign the parameters of the student network to the teacher network; 6) Randomly extract new data samples from the image dataset and repeat to obtain P s and P t , P t The values ​​of the K samples with the highest values ​​are set to 1, and then the loss function is calculated according to step 5), and the parameters of the student network and the teacher network are updated according to step 6) until the value of the loss function no longer decreases, and the obtained feature encoder is output as the result.

2. The unsupervised representation learning method based on relational comparison according to claim 1, characterized in that: The enhancement in step 1) is one or more of time / space enhancement, rotation, cropping, blurring, and color change.

3. The unsupervised representation learning method based on relational comparison according to claim 1, characterized in that: Step 2) The feature encoder is I3D, Resnet or graph convolutional neural network.

4. The unsupervised representation learning method based on relational comparison according to claim 1, characterized in that: Step 3) The projection layer includes two linear layers, the first layer includes a batch normalization layer and a Relu activation layer, the output length is M1, and the output length of the second layer is M2, where M1 is an integer multiple of M2.

5. The unsupervised representation learning method based on relational comparison according to claim 1, characterized in that: Step 4) p(z), z′ and a i The similarity between them is expressed as: in represents the L2-normalized student network prediction value p(z) and the i-th instance a in the queue i The similarity between them, τ is the temperature coefficient of the student network, (·) represents the inner product operation between two features to measure the similarity, represents the L2-normalized teacher network projection z′ and the i-th instance a in the queue i The similarity between them, τ′ is the temperature coefficient of the teacher network.

6. The unsupervised representation learning method based on relational comparison according to claim 1, characterized in that: Step 5) Minimize the two similarity sets P obtained in step 4) s and P t The cross entropy optimization network between .

7. The unsupervised representation learning method based on relational comparison according to claim 1, characterized in that: P t The values ​​of the K sample subsets with the highest values ​​are set to 1, that is:

Images