A self-supervised action recognition method and device based on hierarchical multi-view

Abstract

The present invention discloses a self-supervised action recognition method and device based on hierarchical multi-perspective, which includes: first a pre-training stage of contrastive learning, in which the original data is enhanced and reorganized to extract features through a key encoder and a query encoder to construct positive and negative samples. During feature extraction, sub-perspectives of different scales are obtained through a perspective generation module, which are encoded, aggregated, and spliced ​​through a Transformer to obtain temporal features, spatial features, and a combination of the two: spatiotemporal features. The spatiotemporal features achieve the goal of bringing positive samples closer and pushing negative samples further away through contrast loss to obtain feature representation capabilities. The feature distance between different perspectives is shortened through the consistency loss between temporal features and spatial features. The present invention is a novel hierarchical perspective modeling method in the field of skeletal action recognition, which can deeply explore the complementary information between different perspectives and can achieve skeletal action recognition with better performance than existing methods.

Description

Technical Field

[0001] The present invention belongs to the technical field of action recognition, and in particular relates to a self-supervised action recognition method and device based on hierarchical multi-perspectives. Background Art

[0002] As one of the subtasks of computer vision, motion recognition appears in many computer vision application scenarios and is widely used in many fields such as video surveillance, human-computer interaction, virtual reality, etc., playing an indispensable role in the development of social intelligence.

[0003] The goal of action recognition is generally to identify the action performed by a person within a segmented video clip. Traditional deep learning action recognition methods extract features by feeding a large amount of video and its labels into a model. The model is then optimized by predicting classifications based on these features and comparing them with the ground-truth labels. Data sources typically include standard RGB video, depth map video, and video in the form of skeleton sequences consisting of human keypoint coordinates. Compared to other data sources, skeleton sequences are more compact and less redundant, making them more robust to changes in viewpoint and appearance.

[0004] Traditionally, deep learning-based skeletal sequence action recognition methods have relied on large amounts of labeled data. However, this labeled data is very expensive and requires significant human and material resources. Self-supervised methods can automatically construct supervisory signals, mining the underlying features in the data for downstream tasks, effectively reducing costs. Current research on skeletal sequence-based self-supervised methods primarily falls into two categories: generative and discriminative. Generative methods primarily reconstruct raw data or specific attributes to mine their inherent features, while discriminative methods primarily use techniques such as contrastive learning to achieve feature clustering.

[0005] Currently, self-supervised methods, especially discriminative methods represented by contrastive learning, are still immature in the field of skeleton sequence-based action recognition research. Existing methods have the following shortcomings:

[0006] First, skeleton sequence data generally includes both temporal and spatial information. Existing methods attempt to reconstruct the data from different perspectives to focus on different information. This approach, which approaches multiple perspectives, is generally called a multi-view approach. However, the complementary information provided by these multiple perspectives is limited, and the different perspectives are generally artificially designed, which prevents the exploration of richer cross-perspective information.

[0007] Second, existing multi-view methods are relatively weak in information interaction under different perspectives and do not fully utilize the high-level semantic similarities under different perspectives. Summary of the Invention

[0008] To address the shortcomings of existing methods, the present invention introduces the Transformer, which excels in modeling sequential data, into self-supervised action recognition under the framework of contrastive learning. The self-attention mechanism is used to characterize the relationship between skeleton key points and sequence frames. Through the proposed learnable perspective generation module, derived perspectives of different scales are generated based on temporal and spatial perspectives. At the same time, a contrastive consistency loss function is proposed as a constraint mechanism during training for high-level semantic consistency across perspectives. The temporal and spatial information in the original data is deeply mined, thus realizing a high-performance skeleton sequence-based action recognition method and device.

[0009] The purpose of the present invention is achieved through the following technical solutions:

[0010] According to a first aspect of the present disclosure, a method for self-supervised action recognition based on hierarchical multi-view is provided, comprising the following steps:

[0011] S1, preprocess the original unlabeled skeleton sequence to obtain the view-normalized skeleton sequence;

[0012] S2, the view normalized skeleton sequence is augmented by two random homologous data to obtain an enhanced skeleton sequence;

[0013] S3: Input the enhanced skeleton sequence into the query encoder and key encoder for encoding, respectively, to obtain the corresponding spatiotemporal features. Each spatiotemporal feature contains the temporal and spatial features of the original unlabeled skeleton sequence.

[0014] S4, performing contrast loss calculation on the spatiotemporal features obtained in step S3, and performing consistency loss calculation on the temporal features and spatial features of the original unlabeled skeleton sequence obtained in step S3, and adding the contrast loss and consistency loss calculation results to obtain a total loss function;

[0015] S5, uses the total loss function to perform self-supervised learning in an end-to-end manner to build a pre-trained model;

[0016] S6, uses the query encoder from the pre-trained model and adds a classifier consisting of a fully connected layer after the query encoder to obtain the downstream action recognition task training model consisting of the encoder and classifier;

[0017] S7, inputting the labeled skeleton sequence into the downstream action recognition task training model, using the encoder in the downstream action recognition task training model to extract features from the labeled skeleton data, classifying the extracted features through a classifier, and performing end-to-end training on the classification results using a cross-entropy loss function to build an action recognition model;

[0018] S8, action recognition using action recognition model.

[0019] Furthermore, the step S1 specifically includes the following steps:

[0020] S11, trim all original unlabeled skeleton sequences to a uniform fixed length to obtain the original uniform fixed-length skeleton sequence;

[0021] S12, first move the original unified fixed-length skeleton sequence in each frame so that the center of the defined original unified fixed-length skeleton sequence is located at the origin of the coordinate system, and then rotate the original unified fixed-length skeleton sequence so that the human spine is parallel to the Z axis of the three-dimensional coordinate system and the shoulders are parallel to the X axis of the three-dimensional coordinate system, thereby obtaining a perspective normalized skeleton sequence.

[0022] Furthermore, the step S2 specifically includes the following steps:

[0023] The view-normalized skeleton sequence is augmented twice using different random numbers, and each data augmentation includes spatial augmentation and temporal augmentation;

[0024] Among them, spatial enhancement includes posture enhancement and node jittering. Pose enhancement randomly changes the viewpoint of the view-normalized skeleton sequence in the coordinate system and the distance between defined observation positions, while node jittering randomly moves the nodes to a point within a certain range near the original position without changing the original topological relationship.

[0025] Temporal augmentation involves randomly selecting a starting frame and then sampling the view-normalized skeleton sequence at random intervals.

[0026] Furthermore, in step S3, the query encoder and the key encoder obtain the temporal features of the original unlabeled skeleton sequence in the spatiotemporal features, and specifically include the following steps:

[0027] S31, reorganizing the enhanced skeleton sequence to obtain skeleton data at different perspectives, and obtaining the frame perspective and node perspective forms of the enhanced skeleton sequence, wherein the frame perspective is to enhance the skeleton sequence. All information of a single frame skeleton is a sequence element, and the node perspective is to enhance the skeleton sequence. All information of a single node is a sequence element;

[0028] S32, first extract the temporal features present in the sequence elements, map the low-dimensional data of the frame view to a high-dimensional embedding space through a fully connected layer, separate the general features, and obtain the frame view embedding layer features;

[0029] S33, recombining the frame perspective embedding layer features through a perspective generation module to obtain a first-level segment perspective, wherein the perspective generation module includes a one-dimensional convolutional neural network, layer normalization, a linear rectification activation function, and a one-dimensional maximum pooling, and recombining the first-level segment perspective using the perspective generation module to obtain a second-level segment perspective, wherein the frame perspective is referred to as a parent perspective, and the different levels of segment perspectives are referred to as child perspectives, and the parent perspective and the different levels of child perspectives constitute a hierarchical temporal perspective;

[0030] S34 first adds position encoding to the enhanced skeleton sequences of different temporal perspectives to obtain different position-encoded temporal perspectives; then, different position-encoded temporal perspectives are encoded by Transformer to obtain different temporal perspective features, where the Transformer's self-attention calculation occurs in the temporal dimension;

[0031] Then, different temporal perspective features are aggregated through one-dimensional maximum pooling to obtain different video-level feature information;

[0032] Finally, different video-level feature information is spliced ​​into the entire hierarchical temporal feature.

[0033] Furthermore, in step S3, the query encoder and the key encoder obtain the spatial features of the original unlabeled skeleton sequence in the spatiotemporal features, which specifically include the following steps:

[0034] First, the node perspective is reorganized through the perspective generation module to obtain component perspectives at different levels. The perspective generation module includes a one-dimensional convolutional neural network, layer normalization, linear rectification activation function and one-dimensional maximum pooling. Then, different component perspectives are aggregated into different spatial perspective features through Transformer encoding. The self-attention calculation of the Transformer occurs in the spatial dimension. Finally, the different spatial perspective features are spliced ​​into the entire hierarchical spatial feature.

[0035] Furthermore, constructing the total loss function in step S4 specifically includes the following steps:

[0036] S41: First, the two spatiotemporal features obtained in step S3 are regarded as positive sample pairs. Then, the spatiotemporal features obtained from the other sequences through the key encoder are regarded as negative samples. The negative samples are saved in a queue. Finally, InfoNCE is used to calculate the contrast loss of the spatiotemporal features. The other sequences refer to the other sequences in the dataset except the currently encoded skeleton sequence during the training process. The dataset is the database of the original unlabeled skeleton sequences in step S1.

[0037] S42: InfoNCE calculation is performed on the temporal features obtained by the query encoder in step S3, the spatial features obtained by the key encoder, and the spatial features obtained by the key encoder of other sequences. InfoNCE calculation is also performed on the spatial features obtained by the key encoder in step S3, the temporal features obtained by the query encoder, and the temporal features obtained by the query encoder of other sequences. The consistency loss is the sum of the two calculation results.

[0038] S43, the final total loss function is the sum of contrast loss and consistency loss.

[0039] Furthermore, the labeled skeleton sequence in step S7 needs to be pre-processed by the following steps before being input into the downstream action recognition task training model:

[0040] S71, trimming all labeled skeleton sequences to a uniform fixed length to obtain labeled unified fixed-length skeleton sequences;

[0041] S72: First, the labeled uniform fixed-length skeleton sequence in each frame is moved so that the center of the defined labeled uniform fixed-length skeleton sequence is located at the origin of the coordinate system. Then, the labeled uniform fixed-length skeleton sequence is rotated so that the human spine is parallel to the Z axis of the three-dimensional coordinate system and the shoulders are parallel to the X axis of the three-dimensional coordinate system, thereby obtaining a labeled view-normalized skeleton sequence.

[0042] S73 , downsampling all labeled view-normalized skeleton sequences so that the number of frames of the labeled view-normalized skeleton sequences is the same as the number of frames of the enhanced skeleton sequences.

[0043] According to the second aspect of this specification, a self-supervised action recognition device based on hierarchical multi-perspective is provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the self-supervised action recognition method based on hierarchical multi-perspective as described in the first aspect.

[0044] The beneficial effects of the present invention are as follows: due to the self-supervised training method, the number of labels required for the model to obtain a good feature expression capability is reduced, that is, only manual labels need to be added in the downstream task training stage, which effectively reduces the labor cost compared with the supervised learning action recognition method. Compared with the self-supervised method of the same type, the present invention adopts a solution for extracting hierarchical spatiotemporal features, and for the first time proposes a unified perspective generation module for automatic generation of learnable sub-perspectives, deeply explores the complementary information between different perspectives, and makes up for the limited complementary information in the multi-perspective method. At the same time, by utilizing the high-level semantic similarity between different perspectives, a consistency loss function between different perspectives is proposed for the first time to improve the feature representation capability. In addition, the present invention adopts Transformer as the basic encoding unit, which is equivalent to the traditional model and has less computational complexity. Ultimately, higher action recognition accuracy and faster reasoning speed can be achieved than similar methods, solving the problem that the complementary information of multiple perspectives in the skeleton-based hierarchical multi-perspective self-supervised action recognition is limited, and different perspectives are generally artificially designed, which cannot explore richer cross-perspective information, and the existing multi-perspective method is relatively weak in information interaction under different perspectives, and does not fully utilize the high-level semantic similarity under different perspectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 A flowchart of a hierarchical multi-view self-supervised action recognition method provided by an exemplary embodiment;

[0047] Figure 2 A schematic diagram of the structure of a mid-level view transformer encoder in an exemplary embodiment;

[0048] Figure 3 FIG1 is a schematic diagram of a contrastive learning network from a hierarchical perspective in an exemplary embodiment;

[0049] Figure 4 A structural diagram of a hierarchical multi-view self-supervised action recognition device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0050] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0051] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0052] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a," "an," and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0053] To solve the problem of action recognition, in one embodiment, a self-supervised action recognition method based on hierarchical multi-perspective is proposed. The self-supervised action recognition method based on hierarchical multi-perspective is based on contrastive learning and Transformer encoding hierarchical perspective feature method, and is mainly divided into two stages: pre-training and task training.

[0054] This hierarchical, multi-view, self-supervised action recognition method aims to input a video consisting of a skeleton sequence of human key points and predict the action category. Specifically, human key points are coordinates in three-dimensional space. Multiple such key points form the human skeleton. For example, a frame of video contains the coordinates of a person's left hand, left foot, and head. The model predicts the action category based on the sequence of these coordinates. The model is first pre-trained, which uses self-supervised contrastive learning to train its feature extraction capabilities. This stage is typically performed on large datasets such as NTU RGB+D 120 to improve generalization performance during downstream task training. This stage eliminates the need for manually annotated labels and focuses on exploring the inherent characteristics of the data. Before entering the model, the data undergoes some pre-processing to facilitate subsequent model processing.

[0055] like Figure 1 As shown in Figure 2, the specific steps of the hierarchical multi-view self-supervised action recognition method are as follows:

[0056] S1, preprocess the original unlabeled skeleton sequence to obtain the view-normalized skeleton sequence;

[0057] In one embodiment, step S1 specifically includes the following steps:

[0058] S11: For a video in the form of a skeleton sequence, its length is often different. To facilitate processing, all videos are fixed to the same number of frames, which is larger than the frame number of the longest video in the dataset. Specifically, all original unlabeled skeleton sequences are pruned to a uniform fixed length to obtain the original uniform fixed-length skeleton sequence. For empty frames caused by frames longer than the video length, the original video is repeated to fill the sequence.

[0059] S12, because the camera position and distance are different when collecting motion data, the skeleton may be located in different areas of the coordinate system. To minimize the impact, the original uniform fixed-length skeleton sequence in each frame will be moved so that the center of the defined original uniform fixed-length skeleton sequence is located at the origin of the coordinate system;

[0060] Similarly, considering that the orientation of the actor is different when collecting data, the original unified fixed-length skeleton sequence will be rotated so that the human spine is parallel to the Z axis of the three-dimensional coordinate system and the shoulders are parallel to the X axis of the three-dimensional coordinate system (that is, the line connecting the left and right shoulders is parallel to the X axis of the coordinate system), obtaining a perspective-normalized skeleton sequence.

[0061] S2, the view-normalized skeleton sequence is augmented by two random data of the same family to obtain an enhanced skeleton sequence; the purpose of data augmentation is to construct positive sample pairs through data transformation without changing the original semantic information of the data, allowing the model to compare and discover its intrinsic characteristics based on semantic consistency, and increase the model's robustness to changes in viewpoint, camera distance, skeleton proportion, etc.

[0062] In one embodiment, step S2 specifically includes the following steps:

[0063] The view-normalized skeleton sequence is augmented twice using different random numbers, and each data augmentation includes spatial augmentation and temporal augmentation;

[0064] Among them, spatial enhancement includes posture enhancement and node jitter.

[0065] Pose enhancement randomly changes the viewpoint of the view-normalized skeleton sequence in the coordinate system and the distance between the defined observation positions;

[0066] Specifically, this posture enhancement is mainly to change the skeleton posture, that is, the distance between the viewpoint and the center of the skeleton and the origin of the coordinate system, so that the model can learn different posture semantic information. This process is implemented by the shear transformation in linear algebra. For each skeleton in the sequence, all 3D nodes x are transformed as follows:

[0067]

[0068] Among them, r 01 , r 02, r 10 , r 12 , r 20 , r 21 Randomly sampled from a uniform distribution with a range of [-1, 1].

[0069] Specifically, general skeleton data is estimated from RGB videos through pose estimation methods and often contains a lot of noise. To enhance the robustness of the model against this situation, we use the method of node jittering to artificially add noise during the training phase. This node jittering randomly moves the nodes to a point within a certain range near the original position without changing the original topological relationship;

[0070] This process randomly selects j out of J nodes of the skeleton as a subset, and for the subset nodes X of each frame in the sequence j the following transformation is performed:

[0071]

[0072] where j < J, and all undetermined elements in the transformation matrix are randomly sampled from a uniform distribution with a range of [-1, 1], but the subset of nodes selected in each frame and the transformation matrix are the same.

[0073] For different actions, there are significant differences in the required time and change speed. Even for the same action, it may vary depending on the performer. To enable the model to adapt to these differences, a resampling method is used to trim the video, that is, a time augmentation method. It first randomly selects a starting frame and then samples the view-normalized skeleton sequence at random intervals. s

[0074] For the sequence X, the specific operation is as follows:

[0075] X = Interpolate(X[L start : L start + TL ratio )

[0076] where Interplolate is an interpolation function that can be used for upsampling and downsampling, T is the sequence length, and L ratio is randomly selected within the range [L mmin , 1.0], and L min is a hyperparameter. After determining L ratio , the starting frame of sampling is randomly selected within the range (0, T - TL ratio ), and then the subsequence S[L start : L start : L start + TL ratio is sampled to a fixed length.

[0077] Two different enhanced skeleton sequences are generated using the above data augmentation method under different random number settings.

[0078] like Figure 2 As shown, S3, the two different enhanced skeleton sequences generated in step S2 are respectively input into the query encoder and the key encoder for encoding, and the corresponding spatiotemporal features are obtained respectively. Each spatiotemporal feature contains the temporal features and spatial features of the original unlabeled skeleton sequence;

[0079] In one embodiment, in step S3, the query encoder obtains the temporal features of the original unlabeled skeleton sequence in the spatiotemporal features, specifically comprising the following steps:

[0080] In step S31, one of the two enhanced skeleton sequences generated in step S2 is input into the query encoder to extract features. The query encoder can be divided into two sub-encoders, namely the temporal level encoder and the spatial level encoder, and its main structure is composed of a Transformer.

[0081] In order to use Transformer to mine temporal and spatial information simultaneously, the enhanced skeleton sequence needs to be reorganized to focus on the original information from different perspectives. Therefore, the enhanced skeleton sequence is reorganized to obtain skeleton data under different perspectives, and the frame perspective and node perspective forms of the enhanced skeleton sequence are obtained respectively. The frame perspective and node perspective in the enhanced skeleton sequence are used as sequence elements. Among them, the frame perspective is all the information of a single frame skeleton in the enhanced skeleton sequence, and the node perspective is all the information of a single node in the enhanced skeleton sequence.

[0082] Specifically, as a time series, video inherently has temporal properties, so its natural arrangement (T, V, C) can be used to mine temporal information, where T is the number of frames, V is the number of key points contained in the skeleton of a frame, and C is the number of channels describing the key point features, with 3 channels for three-dimensional coordinates. This organizational approach is called the frame view and is used to model the relationships between frames. Mining spatial information is primarily achieved by modeling the relationships between nodes and all other nodes, so the (V, T, C) arrangement enables the Transformer to use a self-attention mechanism to establish connections between nodes. This approach is also called the joint view. In actual processing, the V and C dimensions of the frame view and the T and C dimensions of the node view are compressed into a single dimension for ease of processing.

[0083] S32, first extract the temporal features present in the sequence elements, map the low-dimensional data of the frame view to a high-dimensional embedding space through a fully connected layer, separate the general features, and obtain the frame view embedding layer features;

[0084] S33, recombining the frame perspective embedding layer features through a perspective generation module to obtain a first-level segment perspective, wherein the perspective generation module includes a one-dimensional convolutional neural network, layer normalization, a linear rectification activation function, and a one-dimensional maximum pooling, and recombining the first-level segment perspective using the perspective generation module to obtain a second-level segment perspective, wherein the frame perspective is referred to as a parent perspective, and the different levels of segment perspectives are referred to as child perspectives, and the parent perspective and the different levels of child perspectives constitute a hierarchical temporal perspective;

[0085] Specifically, to further explore temporal information, we use a view generation module to obtain derived clip views at different time scales and extract features. The view generation module first uses a one-dimensional convolution in the temporal dimension to absorb contextual information without changing the sequence shape. It then uses layer normalization (LayerNorm) to ensure the stability of the feature distribution. It then uses a linear rectifier activation function (ReLU activation function) to increase the model's nonlinearity. Finally, a one-dimensional max pooling is used to aggregate frames into clips. The process can be expressed as follows:

[0086] X c1 =MaxPool1d 2,2 (ReLU(LayerNorm(Conv1d 5,1,2 (X f ))))

[0087] where X f is the frame view sequence, X c1 For the first level clip view sequence, Conv1d 5,1,2 Represents a one-dimensional convolution with a kernel size of 5, a stride of 1, and a padding of 2, MaxPool1d 2,2 One-dimensional maximum pooling with a convolution kernel of 2 and a stride of 2. The convolution operation and maximum pooling operate on the time dimension, that is, the frame dimension in step 2, while LayerNorm and ReLU are both performed on the feature dimension, that is, the element dimension in step 2.

[0088] The view generation module aggregates two frames into a clip, obtaining the first-level clip view. Similarly, the view generation module can be used to aggregate two clips into a longer clip to obtain the second-level clip view:

[0089] X c2 =MaxPool1d 2，2 (ReLU(LayerNorm(Conv1d 5，1，2 (X c1 ))))

[0090] where Xc2 It is the first-level fragment view sequence. The parent view frame view and the derived sub-views of different levels constitute the hierarchical time view.

[0091] S34, first add position encoding to the enhanced skeleton sequence of different temporal perspectives to obtain different position encoding temporal perspectives, which increases the ability of the subsequent Transformer to distinguish relative positions;

[0092] Specifically, for the sequence X (X is the X in the above f , X f , X f ,......),P n ∈R d Represents the position vector corresponding to the position of the nth frame, d is the vector dimension, and the i-th component is defined as follows:

[0093]

[0094] Among them, the frequency ω k The definition is as follows:

[0095]

[0096] After generating the vector of each corresponding position, the position code P of the entire sequence is obtained. The original sequence and the position code are added and divided by To make the gradient update smoother, it can be expressed as:

[0097]

[0098] Then, different position encoding time perspectives are encoded through Transformer to obtain different time perspective features. Among them, the self-attention calculation of Transformer occurs in the time dimension;

[0099] Specifically, sequences encoding temporal perspectives at different positions are then fed into the encoder for encoding. The Transformer can effectively model the relationships between sequence elements, so we use the Transformer as the basic encoder unit. The Transformer is primarily composed of a multi-head attention module and a feedforward layer.

[0100] The multi-head attention module consists of multiple self-attention modules with the same structure. For the nth element X in the sequence n , we get Q by linear transformation n , K n 、V n Three vectors, other sequence elements are similar. During encoding, the relationship between all frames including themselves is modeled through the self-attention mechanism. The output of the final n-th frame after self-attention calculation is:

[0101]

[0102] Among them, Z is the self-attention output feature, softmax is the activation function, Represents vector K i The transpose of V i From other frames in the sequence.

[0103] The encoded output for other frames is similarly generated. To enable the Transformer to focus on different positions in the sequence, multiple attention modules are used. All attention outputs are then concatenated and dimensionally reduced to form the final multi-head attention output. This output then undergoes a residual connection and LayerNorm to serve as the output of the multi-head attention module. After the multi-head attention layer, it is fed into a feedforward layer, implemented as a fully connected layer that does not alter the data shape.

[0104] Then, different temporal perspective features are aggregated through one-dimensional maximum pooling to obtain different video-level feature information; finally, different video-level feature information is spliced ​​into the entire hierarchical temporal feature.

[0105] Specifically, different time perspectives X t 、X c1 、X c2 ..., after the above encoding process, element-level features are obtained, which are aggregated into video-level feature information through temporal max pooling. Temporal max pooling is achieved by finding the maximum value in the time dimension.

[0106] The overall feature extraction process can be summarized as follows:

[0107] V=TMP(TR(X+P))

[0108] Where X is the input time perspective, which can be X t 、X c1 、X c2 ...any one of them, P is the position code, TR is the Transformer encoder, and TMP is the temporal maximum pooling. Through the above process, the video-level feature information V at different temporal perspectives is obtained. t 、V c1 、V c2 , concatenating them to obtain the temporal feature representation V of the attention time information T .

[0109] In one embodiment, obtaining spatial information of the encoded features by the query encoder in step S3 specifically includes the following steps:

[0110] First, the node perspective is reorganized through the perspective generation module to obtain component perspectives at different levels. The perspective generation module includes a one-dimensional convolutional neural network, layer normalization, linear rectification activation function and one-dimensional maximum pooling. Then, different component perspectives are aggregated into different spatial perspective features through Transformer encoding. The self-attention calculation of the Transformer occurs in the spatial dimension. Finally, the different spatial perspective features are spliced ​​into the entire hierarchical spatial feature.

[0111] Specifically, similarly, the node perspective sequence is operated by steps S321, S322 and S323 to obtain different time perspectives X S 、X p1 、X p2 However, because spatial nodes do not have a natural sequence relationship like between video frames, position encoding is not added to the spatial perspective sequence. Finally, the different spatial perspective features are spliced ​​into the entire hierarchical spatial feature, and the same operation is performed to obtain the spatial feature representation V that focuses on spatial information. S The temporal and spatial features are combined to obtain the spatiotemporal features V ST In order to improve the quality of feature representation in downstream tasks and increase flexibility, a multilayer perceptron is used to project features to a lower dimension and then perform contrastive learning. That is:

[0112] O T =MLP(V T ), O S =MLP(V S ), O ST =MLP(V ST );

[0113] Among them, MLP is a multi-layer perceptron, O T , O S , O ST Corresponding to low-dimensional time features, spatial features and spatiotemporal features.

[0114] In one embodiment, the key encoder has the same structure as the query encoder. The other enhanced skeleton sequence of the two different enhanced skeleton sequences generated in step S2 is input to the key encoder, and the feature is obtained by the same operation as the query encoder in step S3.

[0115] like Figure 3 As shown, S4, the temporal and spatial features obtained in step S3 are subjected to contrast loss calculation, and the temporal and spatial features of the original unlabeled skeleton sequence obtained in step S3 are subjected to consistency loss calculation, and the contrast loss and consistency loss calculation results are added together to obtain the total loss function;

[0116] In one embodiment, constructing the total loss function in step S4 specifically includes the following steps:

[0117] S41: First, the two spatiotemporal features obtained in step S3 are regarded as positive sample pairs. Then, the spatiotemporal features obtained from the other sequences through the key encoder are regarded as negative samples. The negative samples are saved in a queue. Finally, InfoNCE is used to calculate the contrast loss of the spatiotemporal features. The other sequences refer to the other sequences in the dataset except the currently encoded skeleton sequence during the training process. The dataset is the database of the original unlabeled skeleton sequences in step S1.

[0118] S42: InfoNCE calculation is performed on the temporal features obtained by the query encoder in step S3, the spatial features obtained by the key encoder, and the spatial features obtained by the key encoder of other sequences. InfoNCE calculation is also performed on the spatial features obtained by the key encoder in step S3, the temporal features obtained by the query encoder, and the temporal features obtained by the query encoder of other sequences. The consistency loss is the sum of the two calculation results.

[0119] S43, the final total loss function is the sum of contrast loss and consistency loss.

[0120] Specifically, contrastive learning obtains good feature representation by narrowing the feature distance between positive samples and pushing the features of positive and negative samples apart. The two samples of the input query encoder and key encoder enhanced by different random data are regarded as positive sample pairs, and the features of other samples encoded by the key encoder are regarded as negative sample features, and the negative sample features are saved in a queue. InfoNCE is used as the contrast loss function L info , defined as follows:

[0121]

[0122] Among them, M ST is a queue for storing spatiotemporal negative sample features, log is the logarithmic function, exp is the exponential function, and τ is a hyperparameter.

[0123] Whether it is the frame perspective or the node perspective and its derived perspectives, they are all different descriptions of the original data, and their features should all represent the same semantic information. Therefore, the total feature of the time perspective is O T and the overall characteristics of spatial perspective O S More similar in feature space. Therefore, we propose the consistency loss function L CC , defined as follows:

[0124]

[0125] Among them, M S 、M T It is a queue for storing spatial negative sample features and temporal negative sample features, log is the logarithmic function, exp is the exponential function, and τ is a hyperparameter.

[0126] The final total loss function L total It is defined as contrast loss plus consistency loss, that is:

[0127] L total =L info +L CC

[0128] The query encoder is updated by calculating the total loss function and then the gradient. The key encoder uses momentum update as follows:

[0129]

[0130] Where θ is the query encoder parameter, is the key encoder parameter, and α is the momentum coefficient hyperparameter.

[0131] S5 uses the total loss function to perform self-supervised learning in an end-to-end manner to build a pre-trained model. The above is the pre-training stage of the model, the purpose of which is to enable the model to obtain good feature representation capabilities.

[0132] S6. To achieve the goal of action recognition, it is also necessary to fine-tune the downstream task, namely action recognition. That is, use the query encoder in the pre-trained model as the encoder for feature extraction, and add a classifier consisting of a fully connected layer after the query encoder to obtain a downstream action recognition task training model consisting of the encoder and classifier, and fine-tune it on the dataset of the downstream task.

[0133] S7, inputting the labeled skeleton sequence into the downstream action recognition task training model, using the encoder in the downstream action recognition task training model to extract features from the labeled skeleton data, classifying the extracted features through a classifier, and performing end-to-end training on the classified features using a cross entropy loss function to build an action recognition model;

[0134] Among them, the cross entropy loss function is used to update the model:

[0135]

[0136] where q i is the model prediction probability of action category i, p i After calculating the loss, the gradient updates the entire model.

[0137] In one embodiment, the labeled skeleton sequence in step S7 needs to be pre-processed by the following steps before being input into the downstream action recognition task training model:

[0138] S71, trimming all labeled skeleton sequences to a uniform fixed length to obtain labeled uniform fixed-length skeleton sequences; in order to unify the length of all labeled skeleton sequences, the data will be bilinearly resampled to the same sequence length as in the pre-training stage;

[0139] S72: First, the labeled uniform fixed-length skeleton sequence in each frame is moved so that the center of the defined labeled uniform fixed-length skeleton sequence is located at the origin of the coordinate system. Then, the labeled uniform fixed-length skeleton sequence is rotated so that the human spine is parallel to the Z axis of the three-dimensional coordinate system and the shoulders are parallel to the X axis of the three-dimensional coordinate system, thereby obtaining a labeled view-normalized skeleton sequence.

[0140] S73 , downsampling all labeled view-normalized skeleton sequences so that the number of frames of the labeled view-normalized skeleton sequences is the same as the number of frames of the enhanced skeleton sequences.

[0141] S8, uses the action recognition model to perform action recognition, inputs a skeleton sequence, and outputs the predicted probability of belonging to each action category.

[0142] Corresponding to the aforementioned embodiment of the self-supervised action recognition method based on hierarchical multi-views, the present invention also provides an embodiment of a self-supervised action recognition device based on hierarchical multi-views.

[0143] See also Figure 4 An embodiment of the present invention provides a self-supervised action recognition device based on hierarchical multi-perspectives, including a memory and one or more processors. The memory stores executable code. When the processor executes the executable code, it is used to implement the self-supervised action recognition method based on hierarchical multi-perspectives in the above embodiment.

[0144] The embodiment of the self-supervisory action recognition device based on hierarchical multi-perspective of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 4 As shown in the figure, it is a hardware structure diagram of any device with data processing capability where the self-supervised action recognition device based on hierarchical multi-view is located. Figure 4In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0145] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0146] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0147] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for self-supervised action recognition based on hierarchical multi-perspectives in the above embodiment is implemented.

[0148] The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store computer programs and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is about to be output.

[0149] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0150] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0151] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0152] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0153] The above are merely preferred embodiments of one or more embodiments of this specification and are not intended to limit one or more embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of one or more embodiments of this specification should be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A self-supervised action recognition method based on hierarchical multi-view, characterized in that: The following steps are involved: S1, preprocess the original unlabeled skeleton sequence to obtain the view-normalized skeleton sequence; S2, the view normalized skeleton sequence is augmented by two random homologous data to obtain an enhanced skeleton sequence; S3: Input the enhanced skeleton sequence into the query encoder and key encoder for encoding, respectively, to obtain the corresponding spatiotemporal features. Each spatiotemporal feature contains the temporal and spatial features of the original unlabeled skeleton sequence. The query encoder and the key encoder obtain the temporal features of the original unlabeled skeleton sequence in the spatiotemporal features, and specifically include the following steps: S31, reorganizing the enhanced skeleton sequence to obtain skeleton data at different perspectives, and obtaining the frame perspective and node perspective forms of the enhanced skeleton sequence, wherein the frame perspective is to enhance the skeleton sequence. All information of a single frame skeleton is a sequence element, and the node perspective is to enhance the skeleton sequence. All information of a single node is a sequence element; S32, first extract the temporal features present in the sequence elements, map the low-dimensional data of the frame view to a high-dimensional embedding space through a fully connected layer, separate the general features, and obtain the frame view embedding layer features; S33, recombining the frame perspective embedding layer features through a perspective generation module to obtain a first-level segment perspective, wherein the perspective generation module includes a one-dimensional convolutional neural network, layer normalization, a linear rectification activation function, and a one-dimensional maximum pooling, and recombining the first-level segment perspective using the perspective generation module to obtain a second-level segment perspective, wherein the frame perspective is referred to as a parent perspective, and the different levels of segment perspectives are referred to as child perspectives, and the parent perspective and the different levels of child perspectives constitute a hierarchical temporal perspective; S34 first adds position encoding to the enhanced skeleton sequences of different temporal perspectives to obtain different position-encoded temporal perspectives; then, different position-encoded temporal perspectives are encoded by Transformer to obtain different temporal perspective features, where the Transformer's self-attention calculation occurs in the temporal dimension; Then, different temporal perspective features are aggregated through one-dimensional maximum pooling to obtain different video-level feature information; Finally, different video-level feature information is spliced ​​into the entire hierarchical temporal feature; S4, performing contrast loss calculation on the spatiotemporal features obtained in step S3, and performing consistency loss calculation on the temporal features and spatial features of the original unlabeled skeleton sequence obtained in step S3, and adding the contrast loss and consistency loss calculation results to obtain a total loss function; S5, uses the total loss function to perform self-supervised learning in an end-to-end manner to build a pre-trained model; S6, uses the query encoder from the pre-trained model and adds a classifier consisting of a fully connected layer after the query encoder to obtain the downstream action recognition task training model consisting of the encoder and classifier; S7, inputting the labeled skeleton sequence into the downstream action recognition task training model, using the encoder in the downstream action recognition task training model to extract features from the labeled skeleton data, classifying the extracted features through a classifier, and performing end-to-end training on the classification results using a cross-entropy loss function to build an action recognition model; S8, action recognition using action recognition model.

2. The hierarchical multi-view self-supervised action recognition method according to claim 1, characterized in that The step S1 specifically includes the following steps: S11, trim all original unlabeled skeleton sequences to a uniform fixed length to obtain the original uniform fixed-length skeleton sequence; S12, first move the original unified fixed-length skeleton sequence in each frame so that the center of the defined original unified fixed-length skeleton sequence is located at the origin of the coordinate system, and then rotate the original unified fixed-length skeleton sequence so that the human spine is parallel to the Z axis of the three-dimensional coordinate system and the shoulders are parallel to the X axis of the three-dimensional coordinate system, thereby obtaining a perspective normalized skeleton sequence.

3. The hierarchical multi-view self-supervised action recognition method according to claim 1, characterized in that: The step S2 specifically includes the following steps: The view-normalized skeleton sequence is augmented twice using different random numbers, and each data augmentation includes spatial augmentation and temporal augmentation; Among them, spatial enhancement includes posture enhancement and node jittering. Pose enhancement randomly changes the viewpoint of the view-normalized skeleton sequence in the coordinate system and the distance between defined observation positions, while node jittering randomly moves the nodes to a point within a certain range near the original position without changing the original topological relationship. Temporal augmentation involves randomly selecting a starting frame and then sampling the view-normalized skeleton sequence at random intervals.

4. The hierarchical multi-view self-supervised action recognition method according to claim 1, characterized in that In step S3, the query encoder and the key encoder obtain the spatial features of the original unlabeled skeleton sequence in the spatiotemporal features, which specifically include the following steps: First, the node perspective is reorganized through the perspective generation module to obtain component perspectives at different levels. The perspective generation module includes a one-dimensional convolutional neural network, layer normalization, linear rectification activation function and one-dimensional maximum pooling. Then, different component perspectives are aggregated into different spatial perspective features through Transformer encoding. The Transformer's self-attention calculation occurs in the spatial dimension. Finally, the different spatial perspective features are spliced ​​into the entire hierarchical spatial feature.

5. The hierarchical multi-view self-supervised action recognition method according to claim 1, characterized in that The construction of the total loss function in step S4 specifically includes the following steps: S41: First, the two spatiotemporal features obtained in step S3 are regarded as positive sample pairs. Then, the spatiotemporal features obtained from the other sequences through the key encoder are regarded as negative samples. The negative samples are saved in a queue. Finally, InfoNCE is used to calculate the contrast loss of the spatiotemporal features. The other sequences refer to the other sequences in the dataset except the currently encoded skeleton sequence during the training process. The dataset is the database of the original unlabeled skeleton sequences in step S1. S42: InfoNCE calculation is performed on the temporal features obtained by the query encoder in step S3, the spatial features obtained by the key encoder, and the spatial features obtained by the key encoder of other sequences. InfoNCE calculation is also performed on the spatial features obtained by the key encoder in step S3, the temporal features obtained by the query encoder, and the temporal features obtained by the query encoder of other sequences. The consistency loss is the sum of the two calculation results. S43, the final total loss function is the sum of contrast loss and consistency loss.

6. The hierarchical multi-view self-supervised action recognition method according to claim 1, characterized in that: The labeled skeleton sequence in step S7 needs to be preprocessed by the following steps before being input into the downstream action recognition task training model: S71, trimming all labeled skeleton sequences to a uniform fixed length to obtain labeled unified fixed-length skeleton sequences; S72: First, the labeled uniform fixed-length skeleton sequence in each frame is moved so that the center of the defined labeled uniform fixed-length skeleton sequence is located at the origin of the coordinate system. Then, the labeled uniform fixed-length skeleton sequence is rotated so that the human spine is parallel to the Z axis of the three-dimensional coordinate system and the shoulders are parallel to the X axis of the three-dimensional coordinate system, thereby obtaining a labeled view-normalized skeleton sequence. S73 , downsampling all labeled view-normalized skeleton sequences so that the number of frames of the labeled view-normalized skeleton sequences is the same as the number of frames of the enhanced skeleton sequences.

7. A hierarchical multi-view self-supervised action recognition device, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it is used to implement the hierarchical multi-perspective self-supervised action recognition method according to any one of claims 1 to 6.

Images