Skeleton sequence recognition method based on masked image autoencoders and system

Abstract

Disclosed in the present invention are a skeleton sequence recognition method based on masked image autoencoders and a system. The method comprises the following steps: building a skeleton action recognition model, and using the skeleton action recognition model to recognize a skeleton sequence so as to predict the class of an action, the skeleton action recognition model comprising an M-layer spatial-temporal representation learning model and a one-layer classifier, the spatial-temporal representation learning model comprising two masked image autoencoders connected in parallel, and, by means of 1×1 convolution, output ends of the masked image autoencoders being subjected to residual connection with input ends thereof. The present invention uses the M-layer spatial-temporal representation learning model and the one-layer classifier to build the skeleton action recognition model, which uses a fine-grained dependency relationship between different skeleton joints for training and learning, is a highly-efficient skeleton sequence learning model, and is well generalized on different data sets.

Description

A skeleton sequence recognition method and system based on mask image autoencoder Technical Field

[0001] The present invention relates to the technical field of video action representation learning, and more specifically, to a skeleton sequence recognition method and system based on a mask image autoencoder. Background Art

[0002] Human action recognition has attracted increasing attention in video understanding due to its widespread applications in human-computer interaction, intelligent surveillance security, virtual reality, and other fields. In visual perception, humans can identify action categories simply by observing the motion of joints, even without appearance information. Unlike RGB videos, skeleton sequences contain only the coordinates of key human joints, making them high-level, lightweight, and robust to complex backgrounds and various conditions, including viewpoint, scale, and motion speed. Furthermore, with the development of human pose estimation algorithms, methods for localizing human joints (i.e., keypoints) have made significant progress, making it feasible to obtain accurate skeleton sequences. Due to the enormous potential and rapid development of the ability to model the fine-grained and large variations in human motion, skeleton sequences are more suitable for distinguishing similar actions with subtle differences compared to RGB data. To capture discriminative spatiotemporal motion patterns, existing skeleton-based action recognition methods are fully supervised and typically require a large amount of labeled data to train a carefully designed model, which is both time-consuming and labor-intensive. To alleviate the problem of limited labeled training data, self-supervised skeleton-based action recognition methods have recently attracted increasing attention. Some contrastive learning methods employ data augmentation to generate positive and negative pairs, but they rely heavily on the number of contrast pairs. With the popularity of encoder-decoders, some methods, following the graph encoder-decoder paradigm, encourage topological proximity by linking and reconstructing masked skeleton sequences. However, these methods generally perform well on link prediction and node clustering, but struggle with node and graph classification.

[0003] For accurate action recognition, fine-grained dependencies between different skeleton joints (i.e., graph classification) are crucial. However, previous self-supervised learning-based methods often ignore the fine-grained dependencies between different skeleton joints, which limits the generality of self-supervised skeleton representations.

[0004] Summary of the Invention

[0005] In order to solve the problem that the above existing technologies ignore the fine-grained dependencies between different skeleton joints and limit the versatility of self-supervised skeleton representation, the present invention provides a skeleton sequence recognition method and system based on mask image autoencoder.

[0006] In order to achieve the above-mentioned purpose of the present invention, the technical solutions adopted are as follows:

[0007] A skeleton sequence recognition method based on a mask image autoencoder, the method comprising the following steps:

[0008] Establish a skeleton action recognition model, use the skeleton action recognition model to identify skeleton sequences, and predict action categories;

[0009] The skeleton action recognition model includes an M-layer space-time representation learning model and a layer of classifier;

[0010] The space-time representation learning model includes two mask image autoencoders connected in parallel, and the output end of the mask image autoencoder is residually connected to the input end of the mask image autoencoder through a 1×1 convolution.

[0011] Preferably, the mask image autoencoder includes an encoder G E and a decoder G D , where the encoder G E Includes three layers of GIN, decoder G D Includes a layer of GIN.

[0012] Preferably, a graph structure of skeleton joints and the topological structure of skeleton joints is established The skeleton joint topology and skeleton joint features are fused to obtain the skeleton sequence matrix S∈R N×T×2 , N represents the number of skeleton joints, T represents the number of skeleton sequences; the skeleton sequence matrix S is transformed into S∈R with learnable parameters N×T×D , D represents the dimension increase of the original skeleton sequence matrix S;

[0013] For each skeleton joint feature matrix X∈R N×D , graph structure represents a skeleton, where is the node set containing all skeleton joints; A∈{0,1} N×N is an adjacency matrix. If i and j are physically connected, then A i,j =1, otherwise 0; node v i The skeleton joint feature is represented as x i ∈R 1×D , i=1,…,N.

[0014] Furthermore, the masked skeleton joint features are used to train a mask image autoencoder to reconstruct the skeleton sequence. Specifically, the mask image autoencoder is trained to reconstruct the masked skeleton joint features based on the established skeleton joint masking strategy and the reweighted loss function.

[0015] Furthermore, we establish a skeleton joint masking strategy as follows:

[0016] Will Divide by body parts, each part corresponds to a first joint subset, randomly select one or more first joint subsets, and form a second joint subset from one or more first joint subsets Then, a learnable mask token vector [MASK] = x is used to mask. [M] ∈R D Mask each skeleton joint feature of the human skeleton sequence; therefore, the masked skeleton joint features In the masked joint feature matrix Defined as: If but otherwise

[0017] The skeleton joint feature matrix As the input of the mask map autoencoder, the skeleton joint feature matrix Each skeleton joint feature in is defined as

[0018] Therefore, the occluded skeleton is expressed as

[0019] Furthermore, the mask image autoencoder reconstructs the masked skeleton joint features, which is defined as:

[0020] Among them, H represents the intermediate layer feature matrix output by the encoder, and Y represents the skeleton joint feature matrix output by the decoder;

[0021] The goal of the mask autoencoder is to minimize the difference between H and Y.

[0022] Furthermore, the reweighted loss function represents the average value of the similarity gap between the reconstructed skeleton and the input original joint points at all masked nodes, as follows:

[0023] Given the original skeleton joint feature matrix X∈R N×D And the skeleton joint features Y∈R output by the reconstructed decoder N×D , the reweighted loss function is defined as:

[0024] Where x i Represents the original skeleton joint features, contained in X∈R N×D in; z i Represents the reconstructed skeleton joint features, contained in Y∈R N×D In , β represents the scaling factor.

[0025] Furthermore, the skeleton action recognition model recognizes the skeleton sequence and predicts the action category as follows: the input skeleton sequence matrix S∈R N×T×D First, it is added to the learnable time position embedding PE to obtain the skeleton sequence feature matrix

[0026] from Get the individual features of two people and

[0027] The node represents and prior knowledge of nodes Feed it into a mask image autoencoder,

[0028] Among them, G E is the mask graph autoencoder; SP(·) represents sum pooling; Repeat(·; N) represents repeating a single node into N nodes after summing, and then Residual connection to obtain global node representation The mask graph autoencoder obtains global information through a single node representation and constrains some node features through all node representations; similarly,

[0029] Obtained node features Contains the action interaction between person 0 and person 1; According to the update rule of graph convolution, from the multi-layer GCN get The final skeleton sequence feature matrix representation is defined as follows:

[0030] Where W (l) represents the trainable weight matrix of layer l, and σ(·) represents the ReLU activation function.

[0031] Then, a multi-scale spatiotemporal set is used to obtain the final skeleton sequence feature matrix;

[0032] Finally, a classifier predicts the action category based on the final skeleton sequence.

[0033] Preferably, before using the skeleton action recognition model to recognize the skeleton sequence, the skeleton action recognition dataset is input into the skeleton action recognition model, and the skeleton action recognition model is fine-tuned using cross entropy loss.

[0034] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described above are implemented.

[0035] The beneficial effects of the present invention are as follows:

[0036] The present invention constructs a skeleton action recognition model by combining an M-layer space-time representation learning model and a layer of classifier. It utilizes the fine-grained dependencies between different skeleton joints for training and learning. It is an efficient skeleton sequence learning model that can generalize well on different datasets.

[0037] The skeleton action recognition model of the present invention introduces a mask map autoencoder based on skeleton mask, and the mask map autoencoder can perform unsupervised training.

[0038] The masked graph autoencoder constructed in the present invention embeds the skeleton joint sequence into a graph convolutional network and reconstructs the hidden skeleton joints and edges based on the prior topological knowledge of the human body. In order to reliably reconstruct features, the reweighted cosine error (RCE) is introduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] FIG1 is a principle framework diagram of the skeleton action recognition model of the present invention.

[0040] FIG2 is a schematic diagram showing the principle of the mask image autoencoder of the present invention.

[0041] FIG3 is a schematic diagram of training the mask image autoencoder of the present invention.

[0042] FIG4 is a schematic diagram comparing the random selection of nodes for masking in the present invention and the prior art. DETAILED DESCRIPTION

[0043] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] Example 1

[0045] As shown in FIG1 , a skeleton sequence recognition method based on a mask image autoencoder is provided, wherein the method comprises the following steps:

[0046] Establish a skeleton action recognition model, use the skeleton action recognition model to identify skeleton sequences, and predict action categories;

[0047] The skeleton action recognition model (SSL) includes an M-layer spatial-temporal representation learning model (STRL) and a layer of classifier;

[0048] The spatial-temporal representation learning model (STRL) includes two mask map autoencoders (SkeletonMAE, SM) connected in parallel, and the output end of the mask map autoencoder (SkeletonMAE, SM) is residually connected to the input end of the mask map autoencoder (SkeletonMAE) through a 1×1 convolution.

[0049] The present invention constructs a skeleton action recognition model by combining an M-layer spatial-temporal representation learning model (STRL) and a layer of classifier. It utilizes the fine-grained dependencies between different skeleton joints for training and learning. It is an efficient skeleton sequence learning model that can generalize well on different datasets.

[0050] The skeleton action recognition model of the present invention introduces a mask map autoencoder based on skeleton mask, and the mask map autoencoder can perform unsupervised training.

[0051] In a specific embodiment, the mask image autoencoder includes an encoder G E and a decoder G D , where the encoder G E Includes three layers of GIN, decoder G D Includes a layer of GIN.

[0052] In a specific embodiment, N human skeleton joints and T skeleton sequences are preprocessed. A graph structure of skeleton joints and the topological structure of skeleton joints is established. The skeleton joint topology and skeleton joint features are fused to obtain the skeleton sequence matrix S∈R N×T×2 , N represents the number of skeleton joints, T represents the number of skeleton sequences; the skeleton sequence matrix S is transformed into S∈R with learnable parameters N×T×D , D represents the dimension increase of the original skeleton sequence matrix S. In this embodiment, T and D are set to 64 based on experience.

[0053] For each skeleton joint feature matrix X∈R N×D , graph structure represents a skeleton, where is the node set containing all skeleton joints; A∈{0,1} N×N is an adjacency matrix. If i and j are physically connected, then A i,j =1, otherwise 0; node v i The skeleton joint feature is represented as x i ∈R 1×D , i = 1, ..., N. The number of skeleton joints in this embodiment is N = 17.

[0054] In a specific embodiment, the masked skeleton joint features are used to train a mask image autoencoder to reconstruct the skeleton sequence. Specifically, the mask image autoencoder is trained to reconstruct the masked skeleton joint features based on an established skeleton joint masking strategy and a reweighted loss function.

[0055] Furthermore, we establish a skeleton joint masking strategy as follows:

[0056] In order to mask the skeleton joint features, Divide by body parts into six parts: head, limbs, and trunk. Corresponding to the first joint subsets of V0, ..., V5, randomly select one or more first joint subsets, and form a second joint subset from one or more first joint subsets. For masking. For human skeleton sequences, each joint communicates with some of its neighboring joints to represent a specific action category. Therefore, it is infeasible to mask all joint sets for all action categories.

[0057] Then, a learnable mask token vector [MASK] = x [M] ∈R D Mask each skeleton joint feature of the human skeleton sequence; therefore, the masked skeleton joint features In the masked skeleton joint feature matrix Defined as: If but otherwise

[0058] The skeleton joint feature matrix As the input of the mask map autoencoder, the skeleton joint feature matrix Each joint feature in is defined as

[0059] Therefore, the occluded skeleton is expressed as

[0060] The mask map autoencoder, given the masked skeleton joint feature matrix and adjacency matrix A, reconstruct the second joint subset The obscured skeleton joint features in .

[0061] The mask image autoencoder reconstructs the masked skeleton joint features, which is defined as:

[0062] Among them, H represents the intermediate layer feature matrix output by the encoder, and Y represents the skeleton joint feature matrix output by the decoder;

[0063] The goal of the mask autoencoder is to minimize the difference between H and Y.

[0064] In a specific embodiment, a common reconstruction loss for mask map autoencoders in image and video tasks is the mean squared error (MSE). For skeleton sequences, the multidimensional and continuous nature of node features makes it difficult to reliably reconstruct features using MSE, as MSE is sensitive to feature dimensionality and vector normalization. L2 normalization in the cosine error maps vectors onto a unit hypersphere and significantly improves training stability. We use the cosine error as the basis for reconstruction.

[0065] To make the reconstruction criterion lean towards harder examples in the imbalanced easy and hard samples, a reweighted cosine error (RCE) function is introduced for the mask autoencoder. The reweighted cosine error function is based on the idea that the contribution of easy examples in training can be reduced by scaling the cosine error by a power of β ≥ 1. For predictions with high confidence, the corresponding cosine error is usually less than 1, and decays to zero faster when the scaling factor β > 1.

[0066] In this implementation, the reweighted loss function represents the average value of the similarity gap between the reconstructed skeleton joint features and the input original skeleton joint features at all masked nodes, as follows:

[0067] Given the original skeleton joint feature matrix X∈R N×D And the skeleton joint feature matrix Y∈R output by the reconstructed decoder N×D , the reweighted loss function is defined as:

[0068] Where x i Represents the original key point features, contained in X∈R N×D in; z i Represents the reconstructed key point features, contained in Y∈R N×D In , β represents the scaling factor;

[0069] The reweighted loss function reduces the contribution of simple samples in training by scaling the cosine error by a power of β ≥ 1; for predictions with high confidence, the corresponding cosine error is usually less than 1, and when the scaling factor β > 1, it decays to zero faster.

[0070] In this embodiment, β is set to 2. By training the mask image autoencoder to reconstruct the skeleton sequence, the pre-trained mask image autoencoder can fully perceive the human skeleton structure and obtain discerning action representations. After pre-training, the mask image autoencoder can be embedded in the skeleton action recognition model for fine-tuning.

[0071] In a specific embodiment, to evaluate the generalization ability of mask map autoencoders for skeletal action recognition, we built a complete skeletal action recognition model based on the pre-trained mask map autoencoder, namely the skeleton sequence learning framework (SSL). To capture multi-person interactions, we integrated two pre-trained mask map autoencoders to build a spatial-temporal representation learning (STRL) module, as shown in Figure 2(b)-(c). The entire skeletal action recognition model consists of an N-layer STRL model and a classifier. Finally, the skeletal action recognition dataset is input to the skeletal action recognition model, and the cross-entropy loss is used to fine-tune the skeletal action recognition model.

[0072] In this embodiment, the skeleton action recognition model recognizes the skeleton sequence and predicts the action category, as follows: the input skeleton sequence matrix S∈R N×T×D First, it is added to the learnable time position embedding PE to obtain the skeleton sequence feature matrix

[0073] from Get the individual features of two people (P=2) and

[0074] Here, we take the node features of the 0th person as an example, and the operation of the 1st person is implemented similarly. and prior knowledge of nodes Feed into a mask image autoencoder;

[0075] Among them, G E is the mask graph autoencoder; SP(·) represents sum pooling; Repeat(·; N) represents repeating a single node into N nodes after summing, and then Residual connection to obtain global node representation The mask graph autoencoder obtains global information through a single node representation and constrains some node features through all node representations;

[0076] Similarly, Obtained in the same way.

[0077] Obtained node features Contains the action interaction between person 0 and person 1; According to the update rule of graph convolution, from the multi-layer GCN get The final skeleton sequence feature matrix representation is defined as follows:

[0078] Where W (l) represents the trainable weight matrix of layer l, and σ(·) represents the ReLU activation function.

[0079] Then, a multi-scale spatiotemporal set is used to obtain the final skeleton sequence feature matrix;

[0080] Finally, a classifier predicts the action category based on the final skeleton sequence.

[0081] In a specific embodiment, before using the skeleton action recognition model to recognize the skeleton sequence, the skeleton action recognition dataset is input into the mask map autoencoder for unsupervised pre-training, and then the mask map autoencoder is fine-tuned on the skeleton action recognition model and the skeleton action recognition model is used to recognize the action using cross entropy loss.

[0082] As shown in Figure 4, a comparative diagram of the present invention and the prior art in randomly selecting nodes for masking is shown. First, the present invention corresponds to the skeleton MAE, and the prior art is the MAE. Two fine-grained action labels are given as examples in Figure 4. The action in the upper figure is a backflip, and the action in the lower figure is a backflip with body twisting. The present invention masks body parts, because we divide the 17 key points, i.e., joint points, of the human body into 6 parts according to body parts, namely, the head, limbs, and torso. The masking strategy of the present invention is to mask parts. The existing MAE randomly selects some key points from the 17 key points of the human body for masking. The present invention can selectively mask which part of the body according to prior knowledge, thereby improving the performance of the model.

[0083] Example 2

[0084] This embodiment further provides a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described in Example 1 are implemented.

[0085] The memory and processor are connected using a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to the processor.

[0086] Example 3

[0087] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in Example 1.

[0088] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0089] Obviously, the above embodiments of the present invention are merely examples for the purpose of illustrating the present invention, and are not intended to limit the embodiments of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A skeleton sequence recognition method based on mask image autoencoder, characterized in that: The method comprises the following steps: Establish a skeleton action recognition model, use the skeleton action recognition model to recognize skeleton sequences, and predict action categories; The skeleton action recognition model includes an M-layer space-time representation learning model and a layer of classifier; The space-time representation learning model includes two mask image autoencoders connected in parallel, and the output end of the mask image autoencoder is residually connected to the input end of the mask image autoencoder through a 1×1 convolution.

2. The skeleton sequence recognition method based on mask image autoencoder according to claim 1, characterized in that: The mask image autoencoder includes an encoder G E and a decoder G D , where the encoder G E It includes three layers of GIN, decoder G D Includes a layer of GIN.

3. The skeleton sequence recognition method based on mask image autoencoder according to claim 1, characterized in that: Create a graph structure about skeleton joints and the topological structure of skeleton joints The skeleton joint topology and skeleton joint features are fused to obtain the skeleton sequence matrix S∈R N×T×2 , N represents the number of skeleton joints, T represents the number of skeleton sequences; the skeleton sequence matrix S is transformed into S∈R with learnable parameters N×T×D , D represents the dimension increase of the original skeleton sequence matrix S; For each skeleton joint feature matrix X∈R N×D , graph structure represents a skeleton, where is the node set containing all skeleton joints; A∈{0,1} N×N is an adjacency matrix. If i and j are physically connected, then A i,j =1, otherwise 0; node v i The skeleton joint feature is represented as x i ∈R 1×D , i=1,…,N.

4. The skeleton sequence recognition method based on mask image autoencoder according to claim 2, characterized in that: The masked skeleton joint features are used to train a mask image autoencoder to reconstruct a skeleton sequence. Specifically, the mask image autoencoder is trained to reconstruct the masked skeleton joint features based on an established skeleton joint masking strategy and a reweighted loss function.

5. The skeleton sequence recognition method based on mask image autoencoder according to claim 4, characterized in that: Establish the skeleton joint mask strategy as follows: Will Divide by body parts, each part corresponds to a first joint subset, randomly select one or more first joint subsets, and form a first joint subset consisting of one or more first joint subsets. Two-joint subset Then, a learnable mask token vector [MASK] = x is used [M] ∈R D Mask each skeleton joint feature of the human skeleton sequence; Therefore, the masked skeleton joint features In the masked joint feature matrix Defined as: If but otherwise The skeleton joint feature matrix As the input of the mask map autoencoder, the skeleton joint feature matrix Each skeleton joint feature in is defined as Therefore, the masked skeleton is expressed as 6. The skeleton sequence recognition method based on mask image autoencoder according to claim 5, characterized in that: The mask image autoencoder reconstructs the masked skeleton joint features, which is defined as: Among them, H represents the intermediate layer feature matrix output by the encoder, and Y represents the skeleton joint feature matrix output by the decoder; The goal of the mask image autoencoder is to minimize the difference between H and Y.

7. The skeleton sequence recognition method based on mask image autoencoder according to claim 6, characterized in that: The reweighted loss function represents the average value of the similarity gap between the reconstructed skeleton joint features and the input original skeleton joint features on all masked nodes, as follows: Given the original skeleton joint feature matrix X∈R N×D And the skeleton joint feature matrix Y∈R output by the reconstructed decoder N×D , the reweighted loss function is defined as: In the formula, x i Represents the original skeleton joint features, contained in X∈R N×D in; z i Represents the reconstructed skeleton joint features, contained in Y∈R N×D In , β represents the scaling factor.

8. The skeleton sequence recognition method based on mask image autoencoder according to claim 3, characterized in that: The skeleton action recognition model recognizes skeleton sequences and predicts action categories as follows: The input skeleton sequence matrix S∈R N×T×D First, it is added to the learnable temporal position embedding PE to obtain the skeleton sequence feature matrix from Get the individual features of two people and The node represents and prior knowledge of nodes Feed it into a mask image autoencoder, Among them, G E is the mask graph autoencoder; SP(·) represents summation pooling; Repeat(·; N) means repeating a single node into N nodes after summation, and then adding Residual connection to obtain global node representation The mask graph autoencoder obtains global information through a single node representation and constrains some node features through all node representations; similarly, The node features obtained Contains the action interaction between person 0 and person 1; according to the update rule of graph convolution, from the multi-layer GCN get The final skeleton sequence feature matrix representation is defined as follows: Where W (l) represents the trainable weight matrix of the lth layer, and σ(·) represents the ReLU activation function. Then, a multi-scale spatiotemporal set is used to obtain the final skeleton sequence feature matrix; Finally, the classifier predicts the action category based on the final skeleton sequence.

9. The skeleton sequence recognition method based on mask image autoencoder according to claim 1, characterized in that: Before using the skeleton action recognition model to recognize the skeleton sequence, the skeleton action recognition dataset is input into the skeleton action recognition model, and the skeleton action recognition model is fine-tuned using the cross entropy loss.

10. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

Images