A skeleton action recognition method based on region-aware motion contrast learning

Abstract

The application relates to a skeleton action recognition method based on region perception motion contrast learning, and relates to the field of computer vision. The method solves the problems that existing contrast learning skeleton action recognition methods focus on global feature contrast learning, ignore the importance of local action patterns for semantic discrimination, and a large amount of redundancy exists in the time and space dimensions of a skeleton sequence, and the method comprises the following steps: skeleton data preprocessing and standardization; multi-stream data conversion, generating three kinds of representations of joint flow, motion flow and skeleton flow; motion perception time sequence enhancement, adaptively retaining key frames based on interframe motion intensity; spatial multi-level data enhancement, implementing progressive three-level data enhancement operation; spatiotemporal feature extraction and region perception mining, extracting features through an encoder and unsupervisedly identifying significant motion regions by using a contrast motion perception region mining method; contrast learning optimization, constructing a dynamic negative sample queue to learn features; and multi-stream model fusion, weightedly integrating recognition results of the data streams.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and specifically to a skeleton action recognition method based on region-aware motion contrast learning. Background Technology

[0002] Human motion recognition is a core task in video understanding, with wide applications in intelligent surveillance, human-computer interaction, and virtual reality. Action recognition methods based on human skeleton data have become a research hotspot due to their insensitivity to background and lighting changes and high computational efficiency. Existing skeleton-based action recognition methods mainly follow a supervised learning paradigm, relying on large-scale labeled data. However, obtaining high-quality skeleton data annotations is costly. Therefore, researchers have turned to self-supervised learning, among which contrastive learning has attracted considerable attention because it can learn effective feature representations. Contrastive learning generates different views of the same sample (positive sample pairs) through data augmentation, and brings positive sample pairs closer together and pushes different samples further apart (negative sample pairs) in the feature space, thus learning feature representations without manual annotation.

[0003] However, existing contrastive learning-based skeleton motion recognition methods mostly focus on the contrastive learning of global features, neglecting the importance of local motion patterns for semantic discrimination. This is because the categories of human actions are usually dominated by local joint or limb movements. For example, the difference between "nodding" and "shaking the head" mainly lies in the direction of head movement. Relying solely on global features may lead to insufficient mining of semantic similarity in the feature space. Furthermore, skeleton data is a high-level semantic modality, and contrastive learning is extremely sensitive to its data augmentation strategies. Existing methods often employ augmentation strategies such as random masking or random frame dropping, which may generate invalid samples that do not conform to the laws of human motion (e.g., random masking destroys the physical connection of joints) or discard time frames containing key semantic information, thus hindering the model from learning discriminative features. Skeleton sequences contain a large amount of redundancy in both time and space dimensions. Static joints or frames with low motion intensity may dominate feature extraction, interfering with the model's capture of core motion patterns. Summary of the Invention

[0004] This invention addresses the shortcomings of existing skeleton action recognition methods based on contrastive learning, which often focus on the contrastive learning of global features, neglecting the importance of local action patterns for semantic discrimination and the significant redundancy in skeleton sequences across time and space. Furthermore, static joints or frames with low motion intensity may dominate feature extraction, interfering with the model's ability to capture core motion patterns. Therefore, this invention proposes a skeleton action recognition method based on region-aware motion contrastive learning.

[0005] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0006] Option 1: This invention proposes a skeleton action recognition method based on region-aware motion contrast learning, the method comprising the following steps:

[0007] Step 1: Obtain the skeleton dataset of the human motion video, parse the skeleton data, extract the 3D coordinates of the body joints, and perform normalization processing to obtain the skeleton sequence.

[0008] Step 2: Adjust all skeleton sequences processed in Step 1 to a skeleton sequence of length T frames using time interpolation. For sequences with fewer than T frames, use bilinear interpolation to complete them; for sequences with more than T frames, use uniform sampling to truncate them.

[0009] Step 3: Divide the data processed in Step 2 into training and testing sets according to the preset evaluation protocol, filter out invalid samples, save the processed data in .npy format, and save the corresponding label information in .pkl format;

[0010] Step 4: Load the data processed in Step 3 using the data loader and convert it into joint flow, motion flow, and skeletal flow; input the three data flows into the network structure for separate training, i.e., each data flow corresponds to an independent model training process;

[0011] Step 5: Perform temporal data augmentation on the skeleton sequence input in Step 4 using the Motion-Aware Temporal Augmentation (MATA) method;

[0012] Step 6: Perform three-level data augmentation on the three data stream skeleton data loaded in Step 4 to generate three skeleton sequences with different augmentation levels.

[0013] Step 7: Process the enhanced sequence obtained after steps 5 and 6 above. , , , ;

[0014] The input to the Query encoder generates a dimension of [B] The spatiotemporal features of [M, hidden_dim, T / 4, V] are then subjected to global average pooling and normalization to obtain the corresponding feature representations of dimension [B, hidden_dim]. , , , ;

[0015] Step 8: Extract region-aware features from the skeleton data obtained in Step 6 using the Comparative Motion-Aware Region Mining Method (CMARM).

[0016] Step 9: Introduce a contrastive learning strategy based on MoCo v2. By constructing a negative sample queue and a positive-negative sample queue for comparison, more discriminative feature representations are learned, thus completing skeleton action recognition based on region-aware motion contrastive learning.

[0017] Furthermore, a preferred embodiment is provided, wherein the method for loading the data processed in step 3 using a data loader in step 4 and converting it into joint flow, motion flow, and skeletal flow is as follows:

[0018] The original skeletal joint coordinate data is used directly as the joint flow dimension [B, C, T, V, M]. The motion flow is generated by calculating the difference between joint coordinates between adjacent frames of the skeleton sequence, with an output dimension of [B, C, T-1, V, M]. The skeleton flow is generated by calculating the vector relationship between connected joints, with an output dimension of [B, C, T, V, M].

[0019] Furthermore, a preferred embodiment is provided, wherein the method for performing temporal data augmentation on the skeleton sequence input in step 4 in step 5 using the Motion-Aware Temporal Augmentation (MATA) method is as follows:

[0020] Step 5.1: Starting from the second frame, calculate its motion intensity. It is defined as the mean square error of the coordinates of all joint points in the previous frame, and is calculated as follows:

[0021] Where C represents the dimension of the human joint coordinates, and V represents the number of joints. Let represent the coordinate vector of the c-th joint in the t-th frame, where t ranges from [2, T].

[0022] Step 5.2: Sort the motion intensity of all frames in each sequence calculated in Step 5.1, set the K value, and set the value range to [0, 1 / 2T]. Retain the high dynamic characteristic frames of TK frames to obtain the subsequence S', whose dimensions are [B, C, TK, V, M].

[0023] Step 5.3: Extend the subsequence S' obtained in Step 5.2 to the length of the original sequence using bilinear interpolation, that is, restore it to the original T-frame length to ensure the consistency of the input dimension;

[0024] That is, enhancing the skeleton sequence The calculation method is as follows:

[0025]

[0026] in, Let represent the bilinear interpolation function, C represent the dimension of the human joint coordinates, V represent the number of joints, and T represent the sequence length, i.e., the number of time steps. This represents the enhanced skeleton sequence after the above steps, with dimensions [B, C, T, V, M].

[0027] Furthermore, a preferred embodiment is provided, wherein the method for generating three skeleton sequences with different enhancement levels in step 6 is as follows:

[0028] Step 6.1: Trim the input skeleton sequence by transforming the angle range [-5°, 5°] into a spatial dimension and performing temporal trimming. Randomly extract M frames from T frames and pad the end with zeros to restore it to T frames, thus obtaining the first-level enhanced skeleton sequence. The dimensions are [B, C, T, V, M];

[0029] Step 6.2: Based on the enhanced input skeleton sequence from Step 1, perform rotation, horizontal flipping, Gaussian noise, and Gaussian blur processing to obtain the secondary enhanced skeleton sequence. The dimensions are [B, C, T, V, M].

[0030] Step 6.3: Perform spatial correlation masking operation based on step 6.2.

[0031] Furthermore, a preferred embodiment is provided, wherein step 6.3 specifically includes the following steps:

[0032] Step 6.3.1: Construct an adjacency matrix P to represent the physical connection relationships between joints. If joint i is adjacent to joint j, set P[i, j] to 1; otherwise, set it to 0.

[0033] Step 6.3.2: Calculate the multi-step adjacency matrix D = P n D[i,j] represents the number of paths from joint i to joint j in n steps, and D[i,j] represents the connection strength of the joints.

[0034] Step 6.3.3: Randomly select X starting joint nodes. For each starting node, select the top Y joint nodes with the highest connection strength in matrix D, set the coordinates of these joints to 0, and then generate the enhancement sequence. The dimensions are [B, C, T, V, M].

[0035] Furthermore, a preferred embodiment is provided, wherein the method for extracting region-aware features based on the Comparative Motion-Aware Region Mining Method (CMARM) in step 8 is as follows:

[0036] Step 8.1: For all skeleton sequences within the batch with dimensions [B, C, T, V, M] The average is calculated to obtain a static anchor point sequence of dimension [1, C, T, V, M]. ;

[0037] Step 8.2: Set the average value of the motion as... skeleton sequence Input Key Encoder In the middle, the corresponding dimension is [B] Dense features of [M, hidden_dim, T / 4, V] And anchor-dense features of dimension [1, hidden_dim, T / 4, V]. Where c represents the channel dimension, t represents the time dimension, and v represents the joint dimension;

[0038] Step 8.3, for and Global average pooling is performed separately to compress the spatiotemporal dimensions, and then the data is passed through a multilayer perceptron. Mapped to global features of dimension [B, hidden_dim]. Global features of anchor points with dimension [1, hidden_dim]. ;

[0039] Step 8.4: Calculate the cosine similarity between the two features from step 8.3. ;

[0040] Step 8.5: Minimize this similarity to drive gradient backpropagation to dense features. The dimension is [B] Gradient plot of M, hidden_dim, T / 4, V] ;

[0041] Step 8.6: Transfer the gradient map Global average pooling is performed along the time T and joint V dimensions, and then activated by ReLU to obtain a dimension [B]. Channel importance weights of [M, hidden_dim] ;

[0042] Step 8.7: For all joints within each body region, replace their saliency values ​​with the average value of all joints within that region to obtain the importance smoothed adjacency matrix. ;

[0043] Step 8.8: Utilize importance weights For original dense features Perform weighted summation and use the bone adjacency matrix Smoothing is performed to obtain dimension [B] Heat map of significant motion regions [M, T / 4, V] ;

[0044] Step 8.9: Calculate the salient motion region feature representation p and the non-salient region feature representation c by using motion region-aware feature pooling (SAFP).

[0045] Step 8.10: Weight the salient motion region features p and the non-salient region features c. and By performing a weighted combination, we obtain dimension [B] The key feature of M, hidden_dim] is represented by kc.

[0046] Furthermore, a preferred implementation is provided, wherein step 9 introduces a MoCo v2-based contrastive learning strategy, and the method for constructing a negative sample queue and a positive-negative sample queue for comparison is as follows:

[0047] Step 9.1: Construct a negative sample queue of size 32768 to store the historical features output by the Key encoder. During each training session, enqueue the features output by the Key encoder for the current batch while removing features from the oldest batch to ensure the diversity of negative samples. Represent the features obtained in steps 7 and 8. , , , and kc are used as the positive sample set;

[0048] Step 9.2: Calculate the similarity between negative samples and positive samples. Use nearest neighbor retrieval to find the negative sample that is most similar to the current positive sample from the negative sample queue. Reclassify the negative sample as a positive sample to expand the positive sample set.

[0049] Furthermore, a preferred embodiment is provided in which step 9 is followed by a step of calculating spatial path loss and temporal path loss.

[0050] Option 2: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in Option 1.

[0051] Option 3: A computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method described in Option 1.

[0052] The advantages of this invention are:

[0053] The skeleton action recognition method based on region-aware motion contrastive learning described in this invention achieves precise focusing on local discriminative features. By contrasting with the motion-aware region mining method CMARM, it can automatically identify and enhance the feature representation of significant motion regions in an unsupervised manner, overcoming the limitation of existing methods that over-rely on global features and ignore key local action patterns.

[0054] The method described in this invention improves the semantic efficiency of temporal modeling. Specifically, the Motion Aware Temporal Enhancement Method (MATA) adaptively selects key frames and removes redundant frames by motion intensity, avoiding the loss of key semantic information that may be caused by randomly discarding frames. This enables the model to capture the core temporal dynamics of actions more efficiently and robustly.

[0055] The method described in this invention enhances the rationality and effectiveness of data augmentation. Specifically, the Spatial Relationship Masking (SCM) method masks based on the physical connections of the human skeleton, generating augmented samples that better conform to the laws of human movement. Compared with random masking strategies, it can more effectively guide the model to learn the structural dependencies between joints and improve the model's generalization ability. Attached Figure Description

[0056] Figure 1 This is a general framework diagram of the skeleton action recognition method based on region-aware motion contrast learning as described in Implementation Method 1.

[0057] Figure 2 This is a flowchart of the skeleton action recognition method based on region-aware motion contrast learning as described in Implementation Method 1. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0059] Implementation Method 1, see [link] Figure 1 and Figure 2 This implementation method proposes a skeleton action recognition method based on region-aware motion contrast learning. Step 1: Obtain the .skeleton files of the NTU RGB+D60 dataset from the NTU RGB+D60 dataset platform and save them in a folder named 'data'. Iterate through all .skeleton files in the 'data' folder, parse the skeleton data, and extract the 3D coordinates (X, Y, Z) of human joints. For sequences with missing joints, interpolation of the same joint coordinates in adjacent frames is used to complete the sequence. If the joint is missing for more than 3 consecutive frames, the statistical mean of the joints corresponding to the same type of action is used to complete the sequence. Map all joint coordinates to the interval [-1, 1] to eliminate scale differences caused by different subjects' heights and body types.

[0060] Step 2: After processing all skeleton sequences in Step 1, adjust them to a length of 64 frames using temporal interpolation. For sequences with fewer than 64 frames, use bilinear interpolation for padding; for sequences with more than 64 frames, use uniform sampling for truncation. Set the training batch size to 64 and construct a tensor with dimensions [64, 3, 64, 25, 2]. The first dimension represents the number of samples, the second represents the coordinate dimensions (x, y, z), the third represents the number of frames in the sequence, the fourth represents the number of key points, and the fifth represents the maximum number of human figures.

[0061] Step 3: Divide the data processed in Step 2 into training and testing sets according to the evaluation protocol (Cross-Subject and Cross-View protocol) recommended by the NTU RGB+D 60 dataset. After filtering out invalid samples, save the data in .npy format and save the label information in .pkl format.

[0062] Step 4: Load the data processed in Step 3 using a data loader and convert it into joint stream, motion stream, and bone stream. Specifically, the original skeletal joint coordinate data is used directly as the joint stream, with dimensions [64, 3, 64, 25, 2]. The motion stream is generated by calculating the difference in joint coordinates between adjacent frames of the skeleton sequence, with an output dimension [64, 3, 63, 25, 2]. The bone stream is generated by calculating the vector relationships between connected joints, with an output dimension [64, 3, 64, 25, 2]. After conversion, the three data streams (joint stream, motion stream, and bone stream) are input into the network structure for separate training; each data stream corresponds to an independent model training process.

[0063] Step 5: Perform temporal data augmentation on the multi-stream skeleton data (joint stream, motion stream, bone stream) loaded in Step 4 using the Motion Aware Temporal Augmentation (MATA) method. Specific steps include:

[0064] Step 5.1: For each frame (starting from frame 2), calculate its motion intensity. Exercise intensity Defined as the mean square error (MSE) between the coordinates of all key points in the previous frame and the coordinates of all key points in the previous frame, its calculation formula is as follows:

[0065]

[0066] Where C represents the dimension of the human joint coordinates, and V represents the number of joints. Let represent the coordinate vector of the c-th joint in the t-th frame, where t ranges from [2, 64].

[0067] Step 5.2: Sort the motion intensity of all frames in each sequence calculated in Step 5.1 above. Since the higher the motion intensity, the more significant the motion features it usually contains, in order to reduce temporal redundancy and ensure that key information is not destroyed, remove the 14 frames with the lowest motion intensity and retain 50 high dynamic feature frames to obtain the subsequence S' with the dimension [64, 3, 50, 25, 2].

[0068] Step 5.3: The subsequence S' obtained in Step 5.2 is extended to the length of the original sequence using bilinear interpolation, i.e., restored to the original 64-frame length, ensuring input dimension consistency. This enhanced skeleton sequence The calculation method is as follows:

[0069]

[0070] in, Let represent the bilinear interpolation function, C represent the dimension of the human joint coordinates, V represent the number of joints, and T represent the sequence length, i.e., the number of time steps. This indicates the enhanced skeleton sequence after the above steps, with dimensions [64, 3, 50, 25, 2].

[0071] Step 6: Perform three-level data augmentation on the multi-stream skeleton data (joint stream, motion stream, bone stream) loaded in Step 4 to generate three skeleton sequences with different augmentation levels. The specific steps are as follows:

[0072] Step 6.1: Perform a shear transform (in the spatial dimension) on the input skeleton sequence with a shearing angle range of [-5°, 5°], then perform temporal pruning, randomly selecting 56 frames from 64 frames, and padding with zeros at the end to restore 64 frames, obtaining the first-level enhanced skeleton sequence. The dimensions are [64, 3, 50, 25, 2].

[0073] Step 6.2: Based on the first-level enhancement described above, further enhance the input skeleton sequence by performing rotation (rotation angle range [-10°, 10°]), horizontal flip (flip probability 0.5), Gaussian noise (noise intensity 0.01), and Gaussian blur (kernel size 3×3) to obtain the second-level enhanced skeleton sequence. The dimensions are [64, 3, 50, 25, 2].

[0074] Step 6.3: Based on the secondary enhancement in Step 6.2 above, perform spatial correlation masking on the input skeleton sequence. The specific steps are as follows:

[0075] Step 6.3.1: Construct an adjacency matrix P to represent the physical connection relationships between joints. If joint i is adjacent to joint j, set P[i, j] to 1; otherwise, set it to 0.

[0076] Step 6.3.2: Calculate the two-step adjacency matrix D = P 2 D[i,j] represents the number of paths from joint i to joint j in 2 steps, and D[i,j] represents the connection strength of the joints.

[0077] Step 6.3.3: Randomly select 4 starting joint nodes. For each starting node, select the top 9 joint nodes with the strongest connection strength in matrix D, set the coordinates of these joints to 0, and then generate the enhancement sequence. The dimensions are [64, 3, 50, 25, 2].

[0078] Step 7: Process the enhanced sequence obtained after steps 5 and 6 above. , , , The input is a Query encoder (composed of a 10-layer ST-GCN spatiotemporal graph convolutional network), which generates spatiotemporal features of dimensions [128, 256, 16, 25]. These spatiotemporal features are then subjected to global average pooling and normalization to obtain feature representations of dimensions [64, 256]. , , , .

[0079] Step 8: Extract region-aware features from the skeleton data obtained in Step 6.1 using the Contrastive Motion-Aware Region Mining (CMARM) method. The specific steps are as follows:

[0080] Step 8.1: For all skeleton sequences with dimensions [64, 3, 64, 25, 2] within the batch. The average is calculated to obtain a static anchor point sequence with dimensions [1, 3, 64, 25, 2]. The calculation formula is shown below:

[0081]

[0082] in, This represents the summation operation. It is the i-th skeleton sequence, and N is the batch size (64 in this example).

[0083] Step 8.2: Set the average value of the motion as... skeleton sequence Input Key Encoder (Same structure as the Query encoder) to obtain dense features with corresponding dimensions [128, 256, 16, 25]. And anchor point dense features of dimension [1, 256, 16, 25]. Where c represents the channel dimension, t represents the time dimension, and v represents the joint dimension.

[0084] Step 8.3: For and Global average pooling (GAP) is performed separately to compress the spatiotemporal dimensions, and then the result is passed through a multilayer perceptron. Mapped to global features of dimension [64, 128]. Global features of anchor points with dimensions [1, 128] .

[0085] Step 8.4: Calculate the cosine similarity between the two features. The calculation formula is shown below:

[0086]

[0087] Where sim(.) represents the similarity function. Represent two vectors and The inner product, and Representing vectors respectively and of Norm.

[0088] Step 8.5: Drive gradient backpropagation to dense features by minimizing this similarity. This yields a gradient map with dimensions [128, 256, 16, 25]. The calculation formula is as follows:

[0089]

[0090] in, This represents the partial derivative function. Gradient plot. This reflects the degree to which different spatiotemporal locations contribute to the differences in characteristics.

[0091] Step 8.6: Then the gradient map Global average pooling is performed along the time T and joint V dimensions, and channel importance weights with dimensions [128, 256] are obtained by applying the ReLU activation function. The calculation formula is as follows:

[0092]

[0093] in, Let T represent the summation function, V be the time dimension, V be the number of key points, and ReLU(). be the activation function.

[0094] Step 8.7: Divide the 25 human body joints in the human skeleton topology map provided by the NTU RGB+D 60 dataset platform into five regions: the torso region contains joints (4,3,21,2,1), the left hand region contains joints (9,10,11,12,24,25), the right hand region contains joints (5,6,7,8,22,23), the left leg region contains joints (17,18,19,20), and the right leg region contains joints (13,14,15,16). For all joints in each body region, replace their significance value with the average of all joints in that region to obtain an importance smooth adjacency matrix. (Size 25×25). This body part-based grouping smoothing mechanism ensures the consistency of adjacent joints in terms of anatomical structure, making the identification of significant motion areas more in line with the natural laws of human movement.

[0095] Step 8.8: Utilize Importance Weights For original dense features Perform weighted summation and use the bone adjacency matrix After smoothing, a heatmap of the significant motion region with dimensions [128, 16, 25] is obtained. The calculation formula is as follows:

[0096]

[0097] Where ReLU(.) is the activation function, and the salient motion region is... Yes, the region where the value exceeds a certain threshold is considered, while the remaining portion represents the region with less significant motion. In this example, the heatmap is processed using the ReLU activation function and numerical clipping. The value is limited to the range of 0 to 1. The closer the value is to 1, the greater the contribution of the spatiotemporal location to action recognition, thus realizing soft attention allocation based on relative importance rather than hard threshold segmentation.

[0098] Step 8.9: Calculate the salient motion region feature representation p using motion-aware feature pooling (SAFP), as shown in equation (8), and the non-salient region feature representation c, as shown in the following formula:

[0099]

[0100] in, Let represent the summation function, T be the time dimension, V be the number of keypoints, and i represent the i-th skeleton sequence in a batch of 64 skeleton data. Heatmaps showing areas of significant motion The salient region feature p focuses on the key motion patterns that contribute the most to action recognition, while the non-salient region feature c retains auxiliary information about the overall motion context.

[0101] Step 8.10: Weight the salient motion region features p and the insignificant region features c. and By performing weighted combination, we obtain the key feature representation kc with dimensions [64, 256].

[0102] Step 9: Introduce a MoCo v2-based contrastive learning strategy. The core idea of ​​this strategy is to learn more discriminative feature representations by constructing a negative sample queue and comparing it with positive and negative samples. The specific steps are as follows:

[0103] Step 9.1: Construct a negative sample queue of size 32768 to store historical features output by the Key encoder. During each training iteration, enqueue the features output by the Key encoder for the current batch while removing features from the oldest batch to ensure the diversity of negative samples. Then, use the feature representations obtained from steps 7 and 8 above. , , , kc is used as the positive sample set.

[0104] Step 9.2: Calculate the similarity between negative and positive samples. Use nearest neighbor retrieval to find the negative sample most similar to the current positive sample from the negative sample queue, and reclassify the negative sample as a positive sample to expand the positive sample set. This process increases the diversity of training data by expanding the positive sample set, aiming to enable the model to learn more discriminative feature representations.

[0105] Step 10: Define the loss function of this network as:

[0106]

[0107] in For spatial path loss, This is the time path loss. Details are as follows:

[0108] The total spatial path loss function is defined as:

[0109]

[0110] in, It is a weighting coefficient, which is set to 0.5 in this example to balance the contributions of the two loss terms. The calculation formula is shown below:

[0111]

[0112] Where k represents the number of query branches, Let represent the feature representation of the i-th branch, and KL be the Kullback-Leibler divergence (KL divergence), which calculates the distance between features. Operations are used to obtain more believable targets for similarity learning, which will strongly enhance view features. Enhanced view features Zoom in The calculation formula is:

[0113]

[0114] Among them, features and Let M be the output features of the Query encoder and Key encoder in the first branch, respectively, and M be the number of negative samples. Let i be the features of the i-th negative sample in the queue that stores negative samples. This refers to temperature hyperparameters.

[0115] Loss function for time path The calculation formula is defined as:

[0116]

[0117] in, Temporal attention-adjusted view features. This loss is achieved by calculating the KL divergence between query features and temporally attention-adjusted view features.

[0118] Step 11: Momentum update encoder parameters, Query encoder parameters The parameters of the key encoder are updated through gradient backpropagation. Updated to: , where m∈[0,1] is the momentum coefficient.

[0119] Step 12: In this embodiment, the NTU RGB+D 60 dataset is used for training. This dataset contains 56,578 samples, 60 action categories, and 25 joints. The training epochs are set to 300. Positive sample augmentation is not performed in the first 100 epochs. After the 100th epoch, the positive sample set is augmented using nearest neighbor retrieval. The training results of the three data streams (joint stream, motion stream, and bone stream) input in Step 4 are weighted and fused using weights [0.6, 0.6, 0.4] to obtain the final training result. Training is considered successful when the loss function converges and performs stably on the training data. The model and corresponding network parameters are then saved.

[0120] In summary, the methods described in this invention include: skeleton data preprocessing and standardization, uniformly processing the original skeleton sequence; multi-stream data conversion, generating three representations: joint stream, motion stream, and skeletal stream; motion-aware temporal enhancement, adaptively retaining keyframes based on inter-frame motion intensity; spatial multi-level data enhancement, implementing progressive three-level data enhancement operations; spatiotemporal feature extraction and region-aware mining, extracting features through an encoder and using a contrastive motion-aware region mining method to unsupervisedly identify salient motion regions; contrastive learning optimization, constructing a dynamic negative sample queue based on the MoCov2 framework for feature learning; and multi-stream model fusion, weightedly integrating the recognition results of each data stream. This invention solves the temporal redundancy problem through the Motion-Aware Temporal Enhancement (MATA) method, ensures the rationality of data enhancement through the Spatial Relevance Masking (SCM) method, and achieves precise focusing of local discriminative features through the Contrastive Motion-Aware Region Mining (CMARM) method. It effectively solves the problems of global feature dominance and spatiotemporal redundancy interference in existing methods, significantly improving the accuracy and generalization ability of action recognition.

[0121] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. This is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0122] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A skeleton action recognition method based on region-aware motion contrast learning, characterized in that, The method includes the following steps: Step 1: Obtain the skeleton dataset of the human motion video, parse the skeleton data, extract the 3D coordinates of the body joints, and perform normalization processing to obtain the skeleton sequence. Step 2: Adjust all skeleton sequences processed in Step 1 to a skeleton sequence of length T frames using time interpolation. For sequences with fewer than T frames, use bilinear interpolation to complete them; for sequences with more than T frames, use uniform sampling to truncate them. Step 3: Divide the data processed in Step 2 into training and testing sets according to the preset evaluation protocol, filter out invalid samples, save the processed data in .npy format, and save the corresponding label information in .pkl format; Step 4: Load the data processed in Step 3 using the data loader and convert it into joint flow, motion flow, and skeletal flow; input the three data flows into the network structure for separate training, i.e., each data flow corresponds to an independent model training process; Step 5: Perform temporal data augmentation on the skeleton sequence input in Step 4 using the Motion-Aware Temporal Augmentation (MATA) method; Step 6: Perform three-level data augmentation on the three data stream skeleton data loaded in Step 4 to generate three skeleton sequences with different augmentation levels. Step 7: Process the enhanced sequence obtained after steps 5 and 6 above. , , , ; The input to the Query encoder generates a dimension of [B] The spatiotemporal features of [M, hidden_dim, T / 4, V] are then subjected to global average pooling and normalization to obtain the corresponding feature representations of dimension [B, hidden_dim]. , , , ; Step 8: Extract region-aware features from the skeleton data obtained in Step 6 using the Comparative Motion-Aware Region Mining Method (CMARM). Step 9: Introduce a contrastive learning strategy based on MoCo v2. By constructing a negative sample queue and a positive-negative sample queue for comparison, more discriminative feature representations are learned to complete skeleton action recognition based on region-aware motion contrastive learning. The method for region-aware feature extraction in step 8 based on the contrastive motion-aware region mining method CMARM is as follows: Step 8.1: For all skeleton sequences within the batch with dimensions [B, C, T, V, M] The average is calculated to obtain a static anchor point sequence of dimension [1, C, T, V, M]. ; Step 8.2: Set the average value of the motion as... skeleton sequence Input Key Encoder In the middle, the corresponding dimension is [B] Dense features of [M, hidden_dim, T / 4, V] And anchor-dense features of dimension [1, hidden_dim, T / 4, V]. Where c represents the channel dimension, t represents the time dimension, and v represents the joint dimension; Step 8.3, for and Global average pooling is performed separately to compress the spatiotemporal dimensions, and then the data is passed through a multilayer perceptron. Mapped to global features of dimension [B, hidden_dim]. Global features of anchor points with dimension [1, hidden_dim]. ; Step 8.4: Calculate the cosine similarity between the two features from step 8.

3. ; Step 8.5: Minimize this similarity to drive gradient backpropagation to dense features. The dimension is [B] Gradient plot of M, hidden_dim, T / 4, V] ; Step 8.6: Transfer the gradient map Global average pooling is performed along the time T and joint V dimensions, and then activated by ReLU to obtain a dimension [B]. Channel importance weights of [M, hidden_dim] ; Step 8.7: For all joints within each body region, replace their saliency values ​​with the average value of all joints within that region to obtain the importance smoothed adjacency matrix. ; Step 8.8: Utilize importance weights For original dense features Perform weighted summation and use the bone adjacency matrix Smoothing is performed to obtain dimension [B] Heat map of significant motion regions [M, T / 4, V] ; Step 8.9: Calculate the salient motion region feature representation p and the non-salient region feature representation c by using motion region-aware feature pooling (SAFP). Step 8.10: Weight the salient motion region features p and the non-salient region features c. and By performing a weighted combination, we obtain dimension [B] The key feature of M, hidden_dim] is represented by kc.

2. The skeleton action recognition method based on region-aware motion contrast learning according to claim 1, characterized in that, The method for loading the data processed in step 3 using a data loader in step 4 and converting it into joint flow, motion flow, and skeletal flow is as follows: The original skeletal joint coordinate data is used directly as the joint flow dimension [B, C, T, V, M]. The motion flow is generated by calculating the difference between joint coordinates between adjacent frames of the skeleton sequence, with an output dimension of [B, C, T-1, V, M]. The skeleton flow is generated by calculating the vector relationship between connected joints, with an output dimension of [B, C, T, V, M].

3. The skeleton action recognition method based on region-aware motion contrast learning according to claim 1, characterized in that, Step 5 involves using the Motion-Aware Temporal Augmentation (MATA) method to augment the skeleton sequence input in Step 4. Step 5.1: Starting from the second frame, calculate its motion intensity. It is defined as the mean square error of the coordinates of all joint points in the previous frame, and is calculated as follows: Where C represents the dimension of the human joint coordinates, and V represents the number of joints. Let represent the coordinate vector of the c-th joint in the t-th frame, where t ranges from [2, T]. Step 5.2: Sort the motion intensity of all frames in each sequence calculated in Step 5.1, set the K value, and set the value range to [0, 1 / 2T]. Retain the high dynamic characteristic frames of TK frames to obtain the subsequence S', whose dimensions are [B, C, TK, V, M]. Step 5.3: Extend the subsequence S' obtained in Step 5.2 to the length of the original sequence using bilinear interpolation, that is, restore it to the original T-frame length to ensure the consistency of the input dimension; That is, enhancing the skeleton sequence The calculation method is as follows: in, Let represent the bilinear interpolation function, C represent the dimension of the human joint coordinates, V represent the number of joints, and T represent the sequence length, i.e., the number of time steps. This represents the enhanced skeleton sequence after the above steps, with dimensions [B, C, T, V, M].

4. The skeleton action recognition method based on region-aware motion contrast learning according to claim 1, characterized in that, The method for generating three skeleton sequences with different enhancement levels in step 6 is as follows: Step 6.1: Trim the input skeleton sequence by transforming the angle range [-5°, 5°] into a spatial dimension and performing temporal trimming. Randomly extract M frames from T frames and pad the end with zeros to restore it to T frames, thus obtaining the first-level enhanced skeleton sequence. The dimensions are [B, C, T, V, M]; Step 6.2: Based on the enhanced input skeleton sequence from Step 1, perform rotation, horizontal flipping, Gaussian noise, and Gaussian blur processing to obtain the secondary enhanced skeleton sequence. The dimensions are [B, C, T, V, M]; Step 6.3: Perform spatial correlation masking operation based on step 6.

2.

5. The skeleton action recognition method based on region-aware motion contrast learning according to claim 4, characterized in that, Step 6.3 specifically includes the following steps: Step 6.3.1: Construct an adjacency matrix P to represent the physical connection relationship between joints. If joint i is adjacent to joint j, set P[i, j] to 1, otherwise set it to 0. Step 6.3.2: Calculate the multi-step adjacency matrix D = P n D[i,j] represents the number of paths from joint i to joint j in n steps, and D[i,j] represents the connection strength of the joints. Step 6.3.3: Randomly select X starting joint nodes. For each starting node, select the top Y joint nodes with the highest connection strength in matrix D, set the coordinates of these joints to 0, and then generate the enhancement sequence. The dimensions are [B, C, T, V, M].

6. The skeleton action recognition method based on region-aware motion contrast learning according to claim 1, characterized in that, Step 9 introduces a contrastive learning strategy based on MoCo v2, which involves constructing a negative sample queue and a positive-negative sample queue for comparison. Step 9.1: Construct a negative sample queue of size 32768 to store the historical features output by the Key encoder. During each training session, enqueue the features output by the Key encoder for the current batch while removing features from the oldest batch to ensure the diversity of negative samples. Represent the features obtained in steps 7 and 8. , , , and kc are used as the positive sample set; Step 9.2: Calculate the similarity between negative samples and positive samples. Use nearest neighbor retrieval to find the negative sample that is most similar to the current positive sample from the negative sample queue. Reclassify the negative sample as a positive sample to expand the positive sample set.

7. The skeleton action recognition method based on region-aware motion contrast learning according to claim 1, characterized in that, Step 9 is followed by steps to calculate spatial path loss and temporal path loss.

8. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

9. A computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method of any one of claims 1-7.

Images