A behavior recognition method and system based on prototype contrast learning, and a storage medium

Abstract

The application discloses a behavior recognition method and system based on prototype contrast learning, and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: sampling and generating a sample set in a skeleton key point data set; converting the sample set by using at least two random data transformation methods to obtain two groups of enhanced samples; inputting the two groups of enhanced samples into an encoding network to obtain two groups of representation vectors; searching for a prototype vector with the highest similarity to each representation vector in one group of representation vectors in a prototype vector set, and generating a corresponding similar vector set by labeling; constructing a prototype contrast loss function; and performing back propagation by using the prototype contrast loss function, and training the encoder network and all prototype vectors. The behavior recognition method based on prototype contrast learning used by the application can train a model capable of effectively obtaining behavior representation vectors by using a skeleton point sequence sample without behavior category labels and based on a self-encoder.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a behavior recognition method and system based on prototype contrast learning, and a storage medium. Background Art

[0002] In action recognition tasks, limited by data volume and algorithm constraints, action recognition models based on RGB images are often affected by changes in perspective and complex backgrounds, resulting in insufficient generalization performance and poor robustness in practical applications. Action recognition based on skeleton point data can effectively solve this problem.

[0003] In skeletal point data, the human body is represented by the coordinates of several predefined key joints in the camera coordinate system. This can be easily obtained using depth cameras (such as Kinect) and various pose estimation algorithms (such as OpenPose). The key joints of the human body defined by the Kinect depth camera are used. It defines the human body as the three-dimensional coordinates of 25 key joints. Since behaviors often exist in the form of videos, a behavior of length T frames can be represented by a tensor of length T × 25 × 3.

[0004] In skeletal point-based action recognition, the core challenge is to enable the model to learn the action representation vector of the skeletal point sequence. Currently, most methods train the model using a supervised learning paradigm. This means that training the action recognition model requires a large number of skeletal point data samples labeled with action category labels, which are difficult and expensive to obtain. Furthermore, the large number of skeletal point sequence samples that are not labeled with action categories cannot be effectively utilized during the training of the action recognition model. Some methods propose to learn the action representation vector using an unsupervised learning paradigm based on autoencoders. However, the sample reconstruction task in autoencoders is not only complex and inefficient, but the learned representation vectors often contain redundant information, resulting in poor performance in action recognition tasks. Summary of the Invention

[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a behavior recognition method and system based on prototype contrast learning, and a storage medium.

[0006] The present invention adopts the following technical solution: a behavior recognition method based on prototype contrast learning, comprising the following steps:

[0007] Obtain the joint points and corresponding coordinate information in the skeleton to generate a sample set;

[0008] At least two random data transformation methods are used to transform the samples to obtain two groups of enhanced samples;

[0009] Inputting the two sets of enhanced samples into the encoding network respectively to obtain a query vector and a key vector;

[0010] Search the prototype vector with the highest similarity to the key vector in the prototype vector set, and generate the corresponding similar vector set by labeling;

[0011] Construct a prototype contrast loss function L based on the prototype vector, query vector and key vector in the similar vector set pc ;

[0012] Using the prototype contrast loss function L pc Perform backpropagation to train the encoder network and all prototype vectors simultaneously.

[0013] In a further embodiment, two groups of enhanced samples are defined as

[0014] The query vector is obtained as follows: in represents the query vector, f θ is the neural network of the encoder, θ is the parameter of the neural network, that is, the object to be trained;

[0015] The key vector is obtained as follows: in represents the query vector; i represents the sample number.

[0016] 4. In a further embodiment,

[0017] 5. The process of obtaining the similarity vector set is as follows:

[0018] Define a set of prototype vectors, consisting of K prototype vectors:

[0019] Similarity calculation: That is, calculate the cosine similarity between vector a and vector b.

[0020] The prototype vector with the highest similarity is obtained: s i is the number of the prototype vector that has the highest similarity to the key vector of the sample labeled i.

[0021] In a further embodiment, the construction process of the prototype contrast loss function is as follows:

[0022]

[0023]

[0024] Lpc =L p +L c

[0025] Among them, τ is a temperature parameter with a value in the interval (0,1), B is the number of samples in a batch, j represents the label of the sample, and k represents the label of the prototype vector.

[0026] In a further embodiment, the encoder network and all prototype vectors are trained using gradient descent:

[0027]

[0028]

[0029] Where α is the learning rate of the gradient descent method, which is set according to the training situation.

[0030] In a further embodiment, the data transformation method includes: a linear mapping clipping method.

[0031] In a further embodiment, the data transformation method includes: a coordinate axis clipping method.

[0032] A computer system comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.

[0033] A computer-readable storage medium stores a computer program, which implements the steps of the method described above when executed by a processor.

[0034] The beneficial effects of the present invention are as follows: The prototype contrastive learning behavior recognition method employed in the present invention can be trained using skeleton point sequence samples to produce a model that effectively obtains behavior representation vectors, without requiring behavior category labels and without relying on autoencoders. Experiments have shown that this method achieves significantly higher representation learning results while reducing computational effort. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Flowchart of prototype comparative learning behavior identification of the present invention.

[0036] Figure 2 Learning graphs for supervised behavioral representations in the state of the art.

[0037] Figure 3 This is an unsupervised representation learning graph based on autoencoders in the existing technology.

[0038] Figure 4 This is a prototype contrast behavior learning diagram of the present invention.

[0039] Figure 5 The figure is an accurate comparison diagram of the prototype of the present invention and the existing methods of behavioral learning. DETAILED DESCRIPTION

[0040] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0041] Traditional supervised learning methods such as Figure 2 As shown in the figure, after using an encoder network to obtain the representation vector of the sample, the representation vector is input into a linear fully connected layer, and then the probability value of the sample belonging to each category is obtained through the Softmax activation function. The following cross entropy loss function is then used for back propagation to train the encoder network and the linear fully connected layer at the same time:

[0042]

[0043] Among them, B is the batch size, y i is the sample label, p i is the predicted probability output by the Softmax activation function. It can be seen that since the sample labels must be used in the loss function, unlabeled data cannot be used in representation learning in this process. Figure 3 As shown in the figure, it uses an encoder network to obtain the representation vector of the sample, and then inputs the representation vector into a decoder network to generate a skeleton point sequence. The skeleton point sequence and the original sample are back-propagated based on the following squared error loss function to train the encoder network and decoder network at the same time:

[0044]

[0045] Among them, B is the batch size, x i It is the original sample, are samples generated by the decoder. After training the encoder network parameters using unlabeled data, the parameters are fine-tuned using labeled data based on a supervised learning paradigm. This approach allows training on unlabeled skeletal point sequence samples, but requires the addition of a decoder network during training. This results in lower training efficiency and less targeted learned representations for classification tasks like action recognition.

[0046] Therefore, it is proposed Figure 4Compared to autoencoder-based methods, this method directly uses a loss function to backpropagate the encoder. This loss function construction also makes it easier to learn implicit category information, resulting in better performance of the learned representation in action recognition tasks. It also eliminates the sequence reconstruction process in the decoder, resulting in higher training efficiency. Similar to unsupervised action representation learning methods based on autoencoders, after training the encoder network parameters using unlabeled data, these parameters are fine-tuned using a supervised learning paradigm using labeled data.

[0047] Example 1

[0048] This example proposes a prototype contrast behavior learning Figure 1 As shown in the figure, instead of using the encoder + decoder autoencoder structure, the encoder is trained end-to-end by combining contrastive learning at the sample level and contrastive learning at the category level. Specifically, the training process of the method is as follows:

[0049] Step 1: Get the joint points and corresponding coordinate information in the skeleton and generate the sample set X = {x i};x i is the sample numbered i.

[0050] Step 2: Use at least two random data transformation methods to transform the prototype sample x i This is converted to obtain two sets of enhanced samples

[0051] Step 3: Input the two sets of enhanced samples into the encoding network to obtain the query vector and key vector

[0052] Step 4: Search the prototype vector with the highest similarity to each key vector in the prototype vector set, and label it to generate the corresponding similarity vector set W s ;

[0053] Step 5: Prototype vector based on similar vector set Query Vector and key vector Constructing a prototype contrast loss function L pc ;

[0054] Step 6: Use the prototype contrast loss function L pc Perform backpropagation to train the encoder network and all prototype vectors simultaneously.

[0055] In a further embodiment, the query vector is obtained as follows: in represents the query vector, f θis the neural network of the encoder, θ is the parameter of the neural network, that is, the object to be trained;

[0056] The key vector is obtained as follows: in Represents the query vector; i represents the label of the sample.

[0057] In a further embodiment, the process of obtaining the similarity vector set is as follows:

[0058] Define a set of prototype vectors, consisting of K prototype vectors:

[0059] Similarity calculation: That is, calculate the cosine similarity between vector a and vector b.

[0060] The prototype vector with the highest similarity is obtained: s i is the number of the prototype vector that has the highest similarity to the key vector of the sample labeled i.

[0061] In a further embodiment, the process of constructing the prototype contrast loss function is as follows:

[0062]

[0063]

[0064]

[0065] Among them, τ is a temperature parameter with a value in the interval (0,1), B is the number of samples in a batch, j represents the label of the sample, and k represents the label of the prototype vector.

[0066] In a further embodiment, the encoder network and all prototype vectors are trained using gradient descent:

[0067]

[0068]

[0069] Where α is the learning rate of the gradient descent method, which is set according to the training situation.

[0070] In the random data transformation method in step 2, we choose the superposition of the following two strategies:

[0071] 1. Linear mapping clipping method: define each joint point in the entire skeleton as a three-dimensional coordinate in three-dimensional space, perform the following transformation on each joint point, and let the three-dimensional vector corresponding to the three-dimensional coordinate be The new three-dimensional coordinate obtained after the transformation is y, and the transformation is expressed as:

[0072]

[0073]

[0074] Among them, all elements except 1 in the S matrix are cropped elements randomly sampled from [-1,1].

[0075] 2. Coordinate axis clipping: For the three-dimensional coordinates of each joint point in the entire skeleton, randomly select one of the three dimensions and set it to 0.

[0076] Based on the above description, during the entire representation learning process, neither sample label information nor the decoder network is required to construct the sample reconstruction task, and the encoder network can be trained end-to-end. The behavior representation vector learned using this method has the following characteristics:

[0077] (1) The representation vector can learn specific transformation invariance;

[0078] (2) Samples with similar behavioral semantic information will be as close as possible in the representation space;

[0079] (3) All samples will be distributed as evenly as possible in the representation space;

[0080] (4) The vector clusters formed in the representation space based on behavioral semantic information will be distributed as evenly as possible in the representation space.

[0081] Based on the above behavior recognition method, experiments show that the accuracy obtained by using this method is significantly higher than other learning methods in the existing technology, such as Figure 5 As shown. Figure 5 As can be seen, the Top1Accuracy of our method significantly improves over existing unsupervised behavior representation learning methods, even outperforming many supervised representation learning methods on labeled datasets. In the figure, S:Hand-Crafted and S:DNN-Based represent several classic supervised learning methods for skeletal-based behavior recognition, while SS:Depth Image and SS:Skeleton represent several currently top-performing unsupervised behavior representation learning methods based on autoencoders. "Ours" denotes our proposed prototype comparative behavior learning method.

[0082] In another embodiment, a computer system is disclosed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described above are implemented: obtaining samples from bones to generate a sample set;

[0083] At least two random data transformation methods are used to transform the samples to obtain two groups of enhanced samples;

[0084] Inputting the two sets of enhanced samples into the encoding network respectively to obtain a query vector and a key vector;

[0085] Search the prototype vector with the highest similarity to the key vector in the prototype vector set, and generate the corresponding similar vector set by labeling;

[0086] Construct a prototype contrast loss function L based on the prototype vector, query vector and key vector in the similar vector set pc ;

[0087] Using the prototype contrast loss function L pc Perform backpropagation to train the encoder network and all prototype vectors simultaneously.

[0088] A computer-readable storage medium stores a computer program, which implements the steps of the method described above when executed by a processor.

Claims

1. A behavior recognition method based on prototype contrast learning, characterized in that: The following steps are involved: Sampling the skeleton key point dataset to generate a sample set; Use at least two random data transformation methods to transform the samples and obtain two sets of enhanced samples and ; The two sets of enhanced samples are input into the encoding network to obtain the query vector and key vector respectively; the query vector is obtained as follows ： ;in represents the query vector, is the neural network of the encoder, are the parameters of the neural network, that is, the object to be trained; The key vector is obtained as follows: ;in represents the query vector; i Indicates the sample number; Search the prototype vector with the highest similarity to each key vector in the prototype vector set, and generate the corresponding similarity vector set by labeling. Specifically, it includes: defining a set of prototype vector sets, consisting of K prototype vectors: ; Similarity calculation: , that is, calculating the vector With vector The cosine similarity of The prototype vector with the highest similarity is obtained: ; is the number of the prototype vector with the highest similarity to the key vector of the sample labeled i; Construct a prototype contrast loss function based on the prototype vector, query vector and key vector in the similar vector set ; Using the prototype contrast loss function Backpropagation is performed to train the encoder network and all prototype vectors simultaneously.

2. The behavior recognition method based on prototype contrast learning according to claim 1, characterized in that: The construction process of the prototype contrast loss function is as follows: in, is a temperature parameter with a value in the interval (0,1), is the number of samples in a batch, Indicates the sample number, Indicates the label of the prototype vector.

3. The behavior recognition method based on prototype contrast learning according to claim 1, characterized in that: The encoder network and all prototype vectors are trained using gradient descent: Where, is the learning rate of the gradient descent method, which is set according to the training situation.

4. The behavior recognition method based on prototype contrast learning according to claim 1, characterized in that: The data transformation method includes: a linear mapping clipping method.

5. The behavior recognition method based on prototype contrast learning according to claim 1, characterized in that: The data transformation method includes: a coordinate axis clipping method.

6. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images