Self-supervised skeleton action recognition method of multi-mode cross-view contrast memory enhancement network

Abstract

The invention provides a self-supervised skeleton action recognition method for a multi-mode cross-view contrast memory enhancement network, and belongs to the technical field of computer vision and artificial intelligence. The problems that rich skeleton information cannot be fully utilized in an existing method, and most researches are insufficient in the aspects of deep feature extraction and diversity contrast learning of skeleton sequences are solved. According to the technical scheme, the method comprises the following steps: S1, carrying out data preprocessing on collected human skeleton data; s2, constructing a graph convolution model encoder for extracting skeleton data information and a nonlinear projection layer required for a subsequent contrast learning task; and S3, processing the processed features by using a cross-view contrast memory enhancement mechanism. The method has the beneficial effects that the graph convolutional network model is adopted as a basic encoder, and a nonlinear projection layer is introduced to map high-dimensional data features, so that the capability of the model in capturing a complex feature relationship is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a self-supervised skeleton action recognition method of a multi-modal cross-view contrast memory enhancement network. Background Art

[0002] Skeleton action recognition is a method for identifying human actions by analyzing skeleton sequences. It has become an important direction in action recognition research because of its protection of data privacy and robustness to environmental changes. With the rapid development of deep learning, deep network structures such as graph convolutional networks (GCN) have been widely used in skeleton action recognition and have achieved good results. At the same time, the introduction of some large-scale action recognition datasets (such as NTU RGB+D and NTU RGB+D120) has provided a solid experimental basis for the research of skeleton action recognition. However, like most visual tasks, training high-performance models usually requires a large amount of high-quality annotated data, which poses a huge challenge to data collection and annotation. Self-supervised learning, as a learning method that does not rely on a large amount of annotated data, provides a possibility to solve the above challenges. Unlike supervised skeleton action recognition, which mainly focuses on developing a better model architecture, self-supervised skeleton action recognition pays more attention to self-training strategies. It avoids dependence on annotated data by using the intrinsic relationship of the data itself for training.

[0003] In recent years, self-supervised representation learning has made significant progress in the field of skeleton action recognition. Self-supervised skeleton action recognition can be roughly divided into two categories: reconstruction-based methods and contrastive learning-based methods. Reconstruction-based methods capture the spatiotemporal correlation of actions by predicting masked skeleton data. Zhang et al. first proposed the use of reconstructed skeletons in long-term global motion dynamics. Contrastive learning-based methods use skeleton transformations to generate positive and negative samples for feature contrast learning. Rao et al. use shearing and clipping to enhance skeleton data. Among these two methods, contrastive learning has become a cutting-edge method for self-supervised skeleton action recognition due to its good interpretability and transferability. Li et al. first proposed a cross-modal knowledge exchange mechanism. Guo et al. proposed using more extreme data enhancement to improve the effect of contrastive learning. Zhou et al. used unique spatiotemporal masking to construct local relationships in partial skeleton sequences. However, most methods still rely mainly on traditional skeleton modal data such as joints, motions, and bones, and fail to make full use of other potentially useful modal information. Since skeleton data lacks background information, the lack of this information limits the model's contrastive expression ability when dealing with complex actions. For contrastive learning methods, many existing models use relatively simple encoder structures in the early feature extraction process, lack effective modeling of high-dimensional complex feature relationships, and have difficulty capturing deep patterns and associations in the data, which affects the robustness and generalization performance of the model. In addition, current self-supervised contrastive learning methods often lack consistent modeling of data from different perspectives, and fail to fully utilize different knowledge information to enhance feature representation capabilities. This results in limited improvement in model performance when processing multi-perspective data. Summary of the invention

[0004] The purpose of the present invention is to provide a self-supervised skeleton action recognition method based on a multi-modal cross-view contrastive memory enhancement network, which solves the technical problems that most existing methods mainly rely on three modal data of skeleton joints, bones and motions, fail to make full use of rich skeleton information, and most studies are insufficient in deep feature extraction and diversity contrastive learning of skeleton sequences.

[0005] The inventive idea of ​​the present invention is: The present invention proposes a Multimodal Cross-View Contrastive Memory-Augmented Network (MCCMN) model. It mainly consists of three parts: First, on the basis of the traditional three modalities, three new data modalities of acceleration, rotation axis and angular velocity are introduced to enrich the representation of skeleton information. Secondly, a graph convolutional network model is used as a feature encoder, and high-dimensional data features are mapped through a nonlinear projection layer to capture deep patterns and associations of the data to improve the robustness of the model. Finally, a cross-view contrast memory enhancement mechanism is proposed. Through a dynamically updated negative sample queue, the diversity of contrast samples is enriched, and the complementary information between different modal views is utilized to enhance the robustness of the features and realize multimodal vision. Figure 1 Contrastive learning.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a self-supervised skeleton action recognition method of a multi-modal cross-view contrast memory enhancement network, comprising the following steps:

[0007] S1. Preprocess the collected human skeleton data to form six-mode view data for multi-mode feature fusion: joint modal view, motion modal view, bone modal view, acceleration modal view, rotation axis modal view, and angular velocity modal view;

[0008] S2. Construct a graph convolutional model encoder for extracting skeleton data information and a nonlinear projection layer required for subsequent contrastive learning tasks. The classic ST-GCN model encoder is used to efficiently extract features from skeleton data to provide richer and higher-quality feature representations for subsequent tasks. The nonlinear projection layer adaptively adjusts the feature dimension. The obtained features will be used as the input of the cross-view contrast memory enhancement mechanism to enhance the contrast between features of different modal views and promote self-supervised training.

[0009] S3, the processed features are processed using a cross-view contrast memory enhancement mechanism to enhance feature representation capabilities, significantly improve generalization performance under different viewing angles, and achieve multimodal vision. Figure 1 The cross-view contrast memory enhancement mechanism is divided into two stages: the first stage provides reliable contrast feature information for each view through single-view contrast learning; the second stage uses the reliable single-view information obtained in the first stage to perform high-confidence cross-view contrast learning.

[0010] Furthermore, the step S1 includes the following steps:

[0011] S11. In the skeleton action recognition task, a skeleton sequence is a collection of action video frames. In each frame, the skeleton is represented as a graph G = (V, E), where V = {v1, v2, ..., v n} represents a vertex set consisting of N joints, E = {e1, e2, ..., e n} represents the edge set consisting of bones between joints. The structure of the skeleton graph can be described using the graph-specific adjacency matrix A, where A ij The value is 0 or 1, indicating that the joint v i and v j Is there an edge connection between them? The entire skeleton sequence can be simply represented by a coordinate set In other words, C is the coordinate dimension, usually C=3, T is the total number of frames in the skeleton sequence, and V is the total number of joints in the current skeleton. In traditional skeletal action recognition, usually only joint, motion and skeletal data are used.

[0012] S12, for the joint modal view, directly obtain the coordinates of the skeleton joint points to indicate the position of each joint point in the three-dimensional space;

[0013] S13, motion mode view is m {：，t+1，：} =x {：，t+1，：} -x {：，t，：} , obtained by calculating the difference in joint coordinates between adjacent frames t, and used to capture the movement information of the joint points;

[0014] S14, the skeleton modal view is It can be obtained by calculating the vectors between adjacent joints V in the same frame, which represents the direction and length of each bone;

[0015] S15. However, these data may have limitations in capturing complex motion features. For example, joint data may have difficulty capturing subtle changes in high-speed motion, while motion and skeletal data may not be able to fully characterize posture changes. To address these issues, this paper introduces three new modal views: acceleration modal view, rotation axis modal view, and angular velocity modal view. In order to ensure temporal consistency, the original skeletal data are adjusted to 50 frames using linear interpolation in the data preprocessing stage, and the time coefficient τ is introduced for normalization. τ is defined as the ratio of 50 frames to the current skeleton sequence frame.

[0016] S16, the acceleration modal view provides a direct measurement of the change in motion speed, which helps to distinguish between fast and slow movements. The acceleration data is calculated by the velocity difference between two frames t:

[0017]

[0018] S17, the rotation axis modal view describes the rotation direction and angle changes of the skeletal joint V, which is particularly useful for capturing complex rotation movements. Its unit direction vector is calculated as:

[0019]

[0020] S18, angular velocity modal view reflects the speed change of joint rotation, which can more accurately describe the dynamic posture change. First, the angle θ between the two bones is obtained from the data of the two bones, and then the angular velocity is obtained by calculating the rotation axis and the angle on the time frame:

[0021]

[0022] w {：，t，：} =r {：，t，：} ×(θ {t+1，：} -θ {t，：} )×τ (4) Furthermore, the step S2 includes the following steps:

[0023] S21. Construct a graph convolution encoder to extract skeleton data features. Graph convolutional networks have significant advantages in extracting spatial features, especially when processing structured data (such as skeleton data). They can automatically learn the implicit structure and relationship in skeleton actions without labeled data or with a small amount of labeled data, thereby improving the effect of feature extraction.

[0024] S22. Construct a nonlinear projection layer to capture more complex feature relationships, effectively process the noise and redundant information in the input data, and lay the foundation for subsequent contrastive learning.

[0025] Furthermore, in the step S21, constructing a graph convolution encoder to extract skeleton data features includes the following steps:

[0026] S211, the adaptability of the classic ST-GCN model to skeleton data and its efficient feature extraction capability can provide richer and higher-quality feature representations for subsequent tasks. The present invention selects it as the basic encoder to extract features of skeleton data;

[0027] S212. In order to reduce the influence of noise and instability in the model training process and construct positive and negative samples for later use, the model designs two encoders during the entire encoder operation stage: q encoder and k encoder (for example, the encoder for joint data is Figure 1denoted as en_q_j and en_k_j in the figure). The q encoder is responsible for encoding the data of the current batch, which is the main feature of the current input data, while the k encoder generally encodes the original data after data enhancement to obtain features. The q encoder's own gradient update is carried out normally, while the k encoder learns to update parameters from the q encoder through the momentum update mechanism to ensure the stability and consistency of its encoding features. This mechanism can effectively maintain the diversity of the negative sample memory sequence mentioned in Section 2.3.1, thereby improving the comparative learning effect of the model. Please note that the preprocessed data input to the encoder has been data enhanced Γ q , Γ k ,This paper adopts the setting in CrossSCLR and adopts cutting and cropping as ,enhancement strategies.

[0028] Furthermore, in the step S22, constructing the nonlinear projection layer includes the following steps:

[0029] S221, after the original data passes through the graph convolution model encoder to extract the feature f, it is passed into a nonlinear projection layer again to obtain the contrast feature z;

[0030] S222. Taking the en_q_j encoder for joint data in the model as an example, the formula is as follows:

[0031]

[0032] in, The joint features extracted by the model en_q_j encoder, The joint contrast features of the encoder are obtained through the nonlinear projection layer. The model can adaptively adjust the feature dimension through the projection layer. The obtained features will be used as the input of the cross-view contrast memory enhancement mechanism to enhance the contrast between the features of different modal views and promote self-supervised training.

[0033] Furthermore, the S3 step includes the following steps:

[0034] S31, the first stage is used to provide reliable contrast feature information for each view through single view contrast learning;

[0035] S32. The second stage uses the reliable individual view information obtained in the first stage to perform high-confidence cross-view contrastive learning.

[0036] Furthermore, in the step S31, the single view contrast learning includes the following steps:

[0037] S311. The construction of positive and negative samples is the core of contrastive learning. In order to ensure the diversity and stability of negative samples, MCCMN adopts a momentum update mechanism to maintain a dynamic negative sample memory sequence. In the first stage, the features of the k encoder are stored in the negative sample memory sequence and compared with the features of the q encoder to obtain positive and negative samples and use the element-by-element product calculation to obtain the contrast information of different modal view samples;

[0038] S312, taking joint features as an example, in the first stage, given the contrast features and the current corresponding key feature The positive sample features are calculated by comparing them, while the negative sample features are calculated by Compared with the feature m stored in the dynamic negative sample memory sequence M, the formula is as follows:

[0039]

[0040] Among them, m i ∈M, γ is a temperature hyperparameter that calculates the similarity between various features through dot products. Please note that the update of the negative sample memory sequence M is performed after the current corresponding batch size is calculated.

[0041] S313. In the first stage, for any modal view i, the final loss function can be obtained by applying formula (6) to the loss aggregation of each modal view:

[0042]

[0043] Furthermore, in the step S32, the cross-view contrastive learning includes the following steps:

[0044] S321, cross-view contrastive learning learns a more robust and generalized feature representation by fusing features from different view data for comparison. The second stage is similar to the first stage. The parameter update of the k encoder itself is frozen, and the momentum is updated according to the q encoder. In the second stage, based on the method of the first stage, the model calculates the negative sample context information between negative samples, and associates the positive and negative samples with the negative sample context to obtain the contrast value between different modal views. Subsequently, the features of the six different k encoders and negative sample memory sequences are concatenated to compare with the features of the q encoder across modal views to obtain the cross-view contrastive learning similarity set, which is finally calculated with the contrast value to obtain the final result;

[0045] S322. Specifically, assuming that the model j modal view guides the model m modal view, the negative sample memory sequences of the model are M j and M m, the index o of the most similar negative sample is learned in view m by comparison, and the formula for guiding the model m modal view to perform cross-view comparison learning from the model j modal view is:

[0046]

[0047] in, and The cross-view contrast learning similarity set S belongs to modal views j and m respectively j and S m , and Represents the part of negative samples that are most similar to the corresponding modality.

[0048] S323. In the second stage, for any modal view v and any modal view u, the final result is obtained by formula (8):

[0049]

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1) Based on the traditional skeleton data processing method, the present invention introduces three new modal data, namely acceleration, rotation axis and angular velocity, and integrates them with joint, motion and bone data, which greatly improves the feature expression ability and provides a richer information source for complex action recognition.

[0052] 2) In order to effectively extract data features in self-supervised learning tasks, the present invention adopts a graph convolutional network model as the basic encoder and introduces a nonlinear projection layer to map high-dimensional data features, thereby enhancing the model's ability to capture complex feature relationships and improving the robustness of the model.

[0053] 3) The present invention uses a cross-view contrast memory enhancement mechanism, and through a dynamically updated negative sample queue and a cross-view contrast learning strategy, further enhances the feature representation capability, significantly improves the generalization performance under different viewing angles, and realizes multimodal vision. Figure 1 Consistent deep feature learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0055] Figure 1 This is the overall network architecture diagram of the present invention.

[0056] Figure 2 This is the ST-GCN model architecture diagram in the present invention.

[0057] Figure 3 This is a structural diagram of the nonlinear projection layer in the present invention.

[0058] Figure 4 Schematic diagram of the cross-view contrast memory enhancement mechanism in the present invention.

[0059] Figure 5 This is a t-SNE visualization diagram of the model performance of the present invention on X-Sub. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0061] Example 1

[0062] See also Figure 1 and Figure 5 The technical solution provided in this embodiment is a skeleton fine motion recognition method based on an action-responsive contrast network, comprising the following steps:

[0063] This embodiment establishes a multi-modal cross-view contrastive memory augmented network (MCCMN) model for self-supervised skeletal action recognition. Figure 1 The framework of the Multimodal Cross-view Contrastive Memory Enhanced Network (MCCMN) is presented. The framework consists of three main parts: multimodal feature fusion, encoding feature extraction, and cross-view contrastive memory enhancement mechanism. The following will introduce the components of MCCMN in detail.

[0064] Multimodal feature fusion: In the skeleton action recognition task, a skeleton sequence is a collection of action video frames. In each frame, the skeleton is represented as a graph G = (V, E), where V = {v1, v2, …, v n} represents a vertex set consisting of N joints, E = {e1, e2, …, e n} represents the edge set consisting of bones between joints. The structure of the skeleton graph can be described using the graph-specific adjacency matrix A, where A ij The value is 0 or 1, indicating that the joint v i and v j Is there an edge connection between them? The entire skeleton sequence can be simply represented by a coordinate set Indicates, where C is the coordinate dimension, generally C=3, T is the total number of frames in the skeleton sequence, and V represents the total number of joints of the current skeleton.

[0065] In traditional skeletal action recognition, only joint, motion and skeletal data are usually used. Joint data is directly obtained from the coordinates of the skeleton joint points, indicating the position of each joint point in three-dimensional space. Motion data is m {:,t+1,:} =x {:,t+1,:} -x {:,t,:} , obtained by calculating the difference in joint coordinates between adjacent frames, is used to capture the movement information of joint points. The skeleton data is It can be obtained by calculating the vectors between adjacent joints in the same frame, indicating the direction and length of each bone. However, these data may have limitations in capturing complex motion features. For example, joint data may have difficulty capturing subtle changes in high-speed motion, while motion and bone data may not be able to fully characterize posture changes. In order to solve these problems, this embodiment introduces three new data: acceleration, rotation axis and angular velocity, i.e., multi-mode features.

[0066] In order to ensure time consistency, the original skeleton data are adjusted to 50 frames using linear interpolation in the data preprocessing stage, and the time coefficient τ is introduced for normalization. τ is defined as the ratio of 50 frames to the current skeleton sequence frame.

[0067] Acceleration data provides a direct measure of the change in speed of an action, helping to distinguish between fast and slow actions. Acceleration data is calculated by taking the velocity difference between two frames t:

[0068]

[0069] The rotation axis data describes the rotation direction and angle change of the bone joint v, which is particularly useful for capturing complex rotation movements. Its unit direction vector is calculated as:

[0070]

[0071] Angular velocity data reflects the change in the speed of joint rotation and can more accurately describe dynamic posture changes. First, the angle θ between the two bones is obtained from the data of the two bones, and then the angular velocity is obtained by calculating the rotation axis and the angle on the time frame:

[0072]

[0073] w {:,t,:} =r {:,t,:} ×(θ {t+1,:} -θ {t,:} )×τ (4)

[0074] By introducing these multimodal features, MCCMN can use more diverse information for action recognition, thus showing stronger adaptability when facing complex and changing actions. For self-supervised tasks, multimodal input can rely on the mutual relationship and information complementarity between multi-dimensional data in the absence of labels to enhance the expressiveness of features and improve the self-supervised learning effect of the model.

[0075] ST-GCN model encoder: Graph convolutional networks (GCN) have significant advantages in extracting spatial features, especially when processing structured data (such as skeleton data). It can automatically learn the implicit structure and relationship in skeleton actions without labeled data or with a small amount of labeled data, thereby improving the effect of feature extraction. The adaptability of the classic ST-GCN model to skeleton data and its efficient feature extraction capabilities can provide richer and higher-quality feature representations for subsequent tasks. This embodiment selects it as the basic encoder to extract the features of skeleton data. The ST-GCN model architecture is as follows: Figure 2 shown.

[0076] In order to reduce the influence of noise and instability in the model training process and construct positive and negative samples for later use, two encoders are designed in the model during the entire encoder operation stage: q encoder and k encoder (for example, the encoder for joint data is Figure 1 denoted as en_q_j and en_k_j in the figure). The q encoder is responsible for encoding the data of the current batch, which is the main feature of the current input data, while the k encoder generally encodes the original data after data enhancement to obtain features. The q encoder's own gradient update is carried out normally, while the k encoder learns to update parameters from the q encoder through the momentum update mechanism to ensure the stability and consistency of its encoding features. This mechanism can effectively maintain the diversity of the negative sample memory sequence mentioned in Section 2.3.1, thereby improving the comparative learning effect of the model. Please note that the preprocessed data input to the encoder has been data enhanced Γ q , Γ k ,This embodiment adopts the settings in CrossSCLR and adopts shearing and cropping as the ,enhancement strategy.

[0077] Nonlinear projection layer: In order to enable the model to capture more complex feature relationships, effectively process noise and redundant information in the input data, and lay the foundation for subsequent contrastive learning, the original data is passed through the graph convolution model encoder to extract the feature f, and then passed to a nonlinear projection layer to obtain the contrast feature z. The nonlinear projection layer is as follows: Figure 3 shown.

[0078] Taking the en_q_j encoder for joint data in the model as an example, the formula is as follows:

[0079]

[0080] in, The joint features extracted by the model en_q_j encoder, The joint contrast features of the encoder are obtained through the nonlinear projection layer. The model can adaptively adjust the feature dimension through the projection layer. The obtained features will be used as the input of the cross-view contrast memory enhancement mechanism to enhance the contrast between the features of different modal views and promote self-supervised training.

[0081] Cross-view contrast memory enhancement mechanism: The cross-view contrast memory enhancement mechanism of this embodiment is divided into two stages: the first stage provides reliable contrast feature information for each view through single-view contrast learning; the second stage uses the reliable individual view information obtained in the first stage to perform high-confidence cross-view contrast learning.

[0082] Single-view contrastive learning: The construction of positive and negative samples is the core of contrastive learning. In order to ensure the diversity and stability of negative samples, MCCMN uses a momentum update mechanism to maintain a dynamic negative sample memory sequence. In the first stage, the features of the k encoder are stored in the negative sample memory sequence and compared with the features of the q encoder to obtain positive and negative samples and use the element-by-element product calculation to obtain the contrast information of different modal view samples, such as Figure 4 shown.

[0083] Taking joint features as an example, in the first stage, given the contrast features and the current corresponding key feature The positive sample features are calculated by comparing them, while the negative sample features are calculated by Compared with the feature m stored in the dynamic negative sample memory sequence M, the formula is as follows:

[0084]

[0085] Among them, m i ∈M, γ is a temperature hyperparameter that calculates the similarity between various features through dot products. Please note that the update of the negative sample memory sequence M is performed after the current corresponding batch size is calculated.

[0086] In the first stage, for any modal view i, the final loss function can be obtained by applying formula (6) to the loss aggregation of each modal view:

[0087]

[0088] Cross-view contrast memory enhancement: Cross-view contrast learning learns more robust and generalized feature representations by fusing features of data from different views, such as Figure 4As shown. The second stage is similar to the first stage. The parameter update of the k encoder itself is frozen, and the momentum is updated according to the q encoder. In the second stage, based on the method of the first stage, the model calculates the negative sample context information between negative samples, and associates the positive and negative samples with the negative sample context to obtain the comparison value between different modal views. Subsequently, the features of the six different k encoders and negative sample memory sequences are concatenated to compare with the features of the q encoder across modal views to obtain the cross-view comparison learning similarity set, and finally the final result is calculated with the comparison value, as shown in Figure 4 Specifically, assuming that the model j modal view guides the model m modal view, their negative sample memory sequences after the model are M j and M m , the index o of the most similar negative sample is learned in view m by comparison, and the formula for guiding the model m modal view to perform cross-view comparison learning from the model j modal view is:

[0089]

[0090] in, and The cross-view contrast learning similarity set S belongs to modal views j and m respectively j and S m , and Represents the part of negative samples that are most similar to the corresponding modality.

[0091] In the second stage, for any modal view v and any modal view u, the final result is obtained by formula (8):

[0092]

[0093] Furthermore, simulation experiments are performed to verify the effectiveness of the method proposed in this embodiment.

[0094] Dataset introduction: NTU RGB+D. The NTU RGB+D dataset contains 56,880 videos from 40 participants, including women, men, young people, and the elderly. These video clips include 60 different action categories, such as drinking water, reading, making phone calls, etc. Each action is recorded from three different vertical camera angles (-45°, 0°, 45°). This dataset can be split in the following two ways: Cross-Subject (X-Sub): The training set includes videos of 20 participants, and the videos of the remaining 20 participants constitute the test set. Cross-View (X-View): The training set uses data recorded with two benchmarks (-45 degrees and 45 degrees), and then the test set is composed of videos of the third benchmark (0 degrees). It should be noted that 302 of the selected erroneous samples need to be ignored.

[0095] Experimental parameter setting: The model proposed in this embodiment is implemented on the PyTorch framework. For the input data, this embodiment uses linear interpolation to resize each skeleton sequence to 50 frames and saves the original number of frames for calculation. In the training of the model, a stochastic gradient descent (SGD) optimizer with a momentum of 0.9 and a weight decay of 0.0001 is used. The batch size is set to 128. In the self-supervised pre-training stage, a learning rate of 0.1 is used to train 300 epochs, and the learning rate is decayed at 250 epochs. The first 150 epochs are set as the first stage, and the last 150 epochs are set as the second stage. The model is then verified using linear evaluation, which uses a learning rate of 0.1 for training 100 epochs and a learning rate decay after 80 epochs.

[0096] Comparative analysis with advanced models: This example compares MCCMN with several other SOTA methods, including: (1) encoder-decoder based methods: LongT GAN, P&C; (2) hybrid learning based methods: MS2L, PCRP; (3) contrastive learning based methods: ASCAL, ISC, CrossSCLR, AimCLR, HYSP, PSTL. In order to evaluate the advantages of the model in this example, a comparison of downstream tasks of action recognition was constructed.

[0097] Table 1 Experimental comparison of different models Unit: %

[0098]

[0099]

[0100] Table 1 shows the performance comparison of the model of this embodiment and other methods in skeletal action recognition. These models all use the linear evaluation method. The results show that the performance of the model of this embodiment is significantly better than the current method. Specifically, the model improves by 1.9% and 1.5% on the X-Sub and X-View of the NTURGB+D dataset respectively compared with the previous best method, indicating the effectiveness of the model of this embodiment in self-supervised action recognition.

[0101] Ablation experiment: In order to verify the effectiveness of each component in the MCCMN model, this embodiment conducted an ablation experiment on X-Sub, and the results are shown in Table 2. Wherein w / o NP means w / o nonlinear projection, which refers to the case where the nonlinear projection layer is removed. w / o CVCMA means w / o cross-view contrastive memory-augmented, which refers to the case where the cross-view contrast memory enhancement is removed. w / o NP&CVCMA means w / onlinearprojection and cross-view contrastive memory-augmented, which refers to the case where the nonlinear projection layer and the cross-view contrast memory enhancement are removed at the same time. The results show that removing the nonlinear projection layer and the cross-view contrast memory enhancement will reduce the performance of the model. Removing the nonlinear projection layer alone will result in a 0.5% loss in model performance. Removing the cross-view contrast memory enhancement alone will result in a 1% loss in model performance. This shows their effectiveness on model performance. Removing both of them will result in the largest loss in model performance, reaching 1.2%, further illustrating that these two components play a vital role in MCCMN.

[0102] Table 2 Performance analysis of different components on the model

[0103] Table 3Different components of the performance analysis of the model

[0104] unit:%

[0105]

[0106] Visualization Analysis: To further analyze the results of the model, this example uses t-SNE to visualize the embedded clusters generated by different types of models, such as Figure 5 As shown in Figure 2, this example randomly selects embeddings of 10 different action categories for sampling and uses different colors for visualization. Figure 5 The results show both the degree to which actions of different categories are separated and the clustering effect between action samples of the same category. Figure 5 From the t-SNE of the four types of models, it can be seen that the MCCMN model of this embodiment significantly improves the clustering of actions, aggregating actions of the same category closer and pushing actions of different categories farther apart, further demonstrating the effectiveness of the model of this embodiment in self-supervised skeleton action recognition.

[0107] Example 2

[0108] See also Figure 1 and Figure 5 The technical solution provided by this embodiment is a skeleton fine motion recognition method based on an action-responsive contrast network, which has the same steps as those in Example 1.

[0109] Furthermore, simulation experiments are performed to verify the effectiveness of the method proposed in this embodiment for semi-supervised learning.

[0110] Table 3 Comparison with mainstream methods of semi-supervised learning Unit: %

[0111]

[0112] For semi-supervised learning, the model of this embodiment is pre-trained on all training data, and then the classifier is fine-tuned using 1% and 10% randomly selected annotation data for linear evaluation. The results in Table 3 show that even with only 1% of labeled data available, the accuracy of the model of this embodiment on X-Sub and X-View is 53.1% and 56.7%, respectively. When 10% of labeled data is available, the performance of the model of this embodiment is further improved to 70.3% and 74.7%, achieving higher performance than other methods.

[0113] Example 3

[0114] See also Figure 1 and Figure 5 The technical solution provided by this embodiment is a skeleton fine motion recognition method based on an action-responsive contrast network, which has the same steps as those in Example 1.

[0115] Furthermore, simulation experiments are performed to verify the effectiveness of multimodal feature fusion and multimodal cross-view information complementarity in the method proposed in this embodiment.

[0116] To explore the effectiveness of multimodal feature fusion on the model, in Table 4, this embodiment designs the performance comparison of the separate input modality view and the joint input modality view of CrossSCLR and MCCMN, both of which use linear evaluation. The results in Table 4 show that: 1) MCCMN has higher performance than the baseline model CrosSCLR under the same input modality view, and the input modality views on X-Sub are improved by 0.5%, 1.1%, 0.9%, 0.4% and 1.8% respectively; 2) The input modality view with multimodal fusion has a significant improvement over the performance of the single input modality view. For example, for MCCMN, the accuracy of the single "joint" on X-Sub is 73.4%, while the accuracy of the 3s-MCCMN with three-modal fusion input reaches 78.2%, an increase of 4.8%. The three new modalities introduced in this embodiment are then fused, with an accuracy of 81.0%, an increase of 7.6% over the single "joint" input modality view; 3) For the 3s-CrosSCLR, which originally had only three modalities, this embodiment adds the three newly introduced modalities to this model. The results show that the performance of 6s-CrosSCLR is 1.4% higher than that of 3s-CrosSCLR, further proving the effectiveness of multimodal fusion. The results of multi-modal MCCMN in different linear evaluation modal views in Table 5 also show that for the same evaluation modal view, the linear evaluation effect of more modal fusion models is better.

[0117] Table 4 Comparison results of the model of this embodiment and the baseline model CrosSCLR in different input modal views Unit: %

[0118]

[0119]

[0120] Table 5 Comparison results of multimodal view models in different linear evaluation modal views Unit: %

[0121]

[0122] Table 5 shows the performance of the trimodal model and hexamodal model of this embodiment under different linear evaluation modal views. The results show that 1) for individual evaluation modal views, the performance of 6s-MCCMN is better than that of 3s-MCCMN. For example, on X-Sub, for the evaluation of only the "joint" modal view, 6s-MCCMN performs 1.7% better than 3s-MCCMN; 2) the effectiveness of multimodal cross-view contrast memory learning is demonstrated. Taking 6s-MCCMN as an example, even for the evaluation of only the "joint" modal view, the model performance can reach a high level of 75.5%. For the evaluation of the traditional trimodal view 3views ("joint+motion+bone"), it can achieve a high level of performance of 80%, and the model performance reaches 81% when evaluating the hexamodal view; 3) it further illustrates the value of using the complementary information of multimodal skeleton data. Taking 6s-MCCMN as an example, in Table 4, the performance of MCCMN for single-input single-evaluation “joint” is 73.4%, while the performance of 6s-MCCMN learned using multimodal information for single-evaluation “joint” reaches 75.5%, which is 2.1% higher than the performance of the single-input model.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A self-supervised skeleton action recognition method based on a multi-modal cross-view contrast memory augmented network, characterized in that: The following steps are involved: S1. Preprocess the collected human skeleton data to form six-mode view data for multi-mode feature fusion: joint modal view, motion modal view, bone modal view, acceleration modal view, rotation axis modal view, and angular velocity modal view; S2. Construct a graph convolutional model encoder for extracting skeleton data information and a nonlinear projection layer required for the subsequent contrastive learning task. The classic ST-GCN model encoder is used to extract features from skeleton data, and the nonlinear projection layer completes the adaptive adjustment of feature dimensions. The obtained features will be used as the input of the cross-view contrast memory enhancement mechanism to enhance the contrast between features of different modal views and promote self-supervised training. S3. The processed features are processed using a cross-view contrast memory enhancement mechanism to enhance feature representation capabilities, significantly improve generalization performance under different viewing angles, and achieve deep feature learning with multimodal view consistency. The cross-view contrast memory enhancement mechanism is divided into two stages: the first stage provides reliable contrast feature information for each view through single-view contrast learning; The second stage uses the reliable individual view information obtained in the first stage to perform high-confidence cross-view contrastive learning.

2. The self-supervised skeleton action recognition method of the multi-modal cross-view contrast memory enhancement network according to claim 1 is characterized in that: The step S1 comprises the following steps: S11. In the skeleton action recognition task, a skeleton sequence is a collection of action video frames. In each frame, the skeleton is represented as a graph G = (V, E), where V = {v1, v2, ..., v n } represents a vertex set consisting of N joints, E = {e1, e2, …, e n } represents the edge set consisting of bones between joints. The structure of the skeleton graph is described using the graph-specific adjacency matrix A, where A ij The value is 0 or 1, indicating that the joint v i and v j Is there an edge connection between them, and the entire skeleton sequence is represented by a coordinate set It is represented by, where C is the coordinate dimension, C=3, T is the total number of frames of the skeleton sequence, and V represents the total number of joints of the current skeleton. In traditional skeleton action recognition, only joint, motion and skeleton data are used; S12, for the joint modal view, directly obtain the coordinates of the skeleton joint points to indicate the position of each joint point in the three-dimensional space; S13, motion mode view is m {:,t+1,:} =x {:,t+1,:} -x {:,t,:} , obtained by calculating the difference in joint coordinates between adjacent frames t, and used to capture the movement information of the joint points; S14, the skeleton modal view is It is obtained by calculating the vector between adjacent joints v in the same frame, which represents the direction and length of each bone; S15. These data have limitations in capturing complex motion features. Joint data are difficult to capture subtle changes in high-speed motion, while motion and bone data cannot fully characterize posture changes. Three new modal views are introduced: acceleration modal view, rotation axis modal view, and angular velocity modal view. In order to ensure time consistency, the original bone data are adjusted to 50 frames using linear interpolation in the data preprocessing stage, and the time coefficient τ is introduced for normalization. τ is defined as the ratio of 50 frames to the current skeleton sequence frame. S16, the acceleration modal view provides a direct measurement of the change in motion speed, which helps to distinguish between fast and slow movements. The acceleration data is calculated by the velocity difference between two frames t: S17, the rotation axis modal view describes the rotation direction and angle changes of the skeletal joint v, which is particularly useful for capturing complex rotation movements. Its unit direction vector is calculated as: S18, angular velocity modal view reflects the speed change of joint rotation and accurately describes the dynamic posture change. First, the angle θ between the two skeletons is obtained from the data, and then the angular velocity is obtained by calculating the rotation axis and the angle on the time frame: w{:,t,:}=r{:,t,:}×(θ{t+1,:}-θ {t,:} )×τ (4)。 3. The self-supervised skeleton action recognition method of the multi-modal cross-view contrast memory enhancement network according to claim 1 or 2 is characterized in that: The step S2 comprises the following steps: S21. Construct a graph convolutional encoder to extract skeleton data features. It performs well in processing structured data, namely skeleton data. In the absence of labeled data or a small amount of labeled data, it automatically learns the implicit structure and relationship in skeleton actions, thereby improving the effect of feature extraction. S22. Construct a nonlinear projection layer to capture complex feature relationships, process noise and redundant information in the input data, and lay the foundation for subsequent contrastive learning.

4. The self-supervised skeleton action recognition method of the multi-modal cross-view contrast memory enhancement network according to claim 3 is characterized in that: In step S21, constructing a graph convolution encoder to extract skeleton data features includes the following steps: S211. The adaptability of the classic ST-GCN model to skeleton data and its efficient feature extraction capability can provide richer and higher-quality feature representations for subsequent tasks. We choose it as the basic encoder to extract features from skeleton data. S212. Construct positive and negative samples for later use. During the entire stage of the encoder operation, the model designs two encoders: q encoder and k encoder. Among them, the q encoder is responsible for encoding the data of the current batch, which is the main feature of the current input data, and the k encoder generally encodes the original data after data enhancement to obtain features. The q encoder's own gradient update is carried out normally, and the k encoder learns to update parameters from the q encoder through the momentum update mechanism to ensure the stability and consistency of its encoding features. This mechanism maintains the diversity of the negative sample memory sequence, thereby improving the comparative learning effect of the model.

5. The self-supervised skeleton action recognition method of the multi-modal cross-view contrast memory enhancement network according to claim 3 is characterized in that: In step S22, constructing the nonlinear projection layer includes the following steps: S221, after the original data passes through the graph convolution model encoder to extract the feature f, it is passed into a nonlinear projection layer again to obtain the contrast feature z; S222. Taking the en_q_j encoder for joint data in the model as an example, the formula is as follows: in, The joint features extracted by the model en_q_j encoder, In order to obtain the joint contrast features of the encoder through the nonlinear projection layer, the model can adaptively adjust the feature dimension through the projection layer. The obtained features will be used as the input of the cross-view contrast memory enhancement mechanism to enhance the contrast between features of different modal views and promote self-supervised training.

6. The self-supervised skeleton action recognition method of the multi-modal cross-view contrast memory enhancement network according to claim 1 is characterized in that: The S3 step includes the following steps: S31, the first stage is used to provide reliable contrast feature information for each view through single view contrast learning; S32. The second stage uses the reliable individual view information obtained in the first stage to perform high-confidence cross-view contrastive learning.

7. The self-supervised skeleton action recognition method of the multi-modal cross-view contrast memory enhancement network according to claim 6 is characterized in that: In the step S31, the single view contrast learning includes the following steps: S311. The construction of positive and negative samples is the core of contrastive learning. In order to ensure the diversity and stability of negative samples, MCCMN adopts a momentum update mechanism to maintain a dynamic negative sample memory sequence. In the first stage, the features of the k encoder are stored in the negative sample memory sequence and compared with the features of the q encoder to obtain positive and negative samples and use the element-by-element product calculation to obtain the contrast information of different modal view samples. S312, in the first stage, given contrast features and the current corresponding key feature The positive sample features are calculated by comparing them, while the negative sample features are calculated by Compared with the feature m stored in the dynamic negative sample memory sequence M, the formula is as follows: Among them, m i ∈M, γ is a temperature hyperparameter. The similarity between various features is calculated by dot product. The update of the negative sample memory sequence M is performed after the current corresponding batch size is calculated. S313. In the first stage, for any modal view i, the final loss function is obtained by applying formula (6) to the loss aggregation of each modal view:

8. The self-supervised skeleton action recognition method of the multi-modal cross-view contrast memory enhancement network according to claim 6 is characterized in that: In step S32, cross-view contrastive learning includes the following steps: S321, cross-view contrastive learning is to fuse the features of different view data for contrast. The second stage is similar to the first stage. The parameter update of the k encoder itself is frozen, and the momentum is updated according to the q encoder. In the second stage, based on the method of the first stage, the model obtains the negative sample context information by calculation between negative samples, and obtains the contrast value between different modal views by associating the positive and negative samples with the negative sample context. Subsequently, the features of the six different k encoders and negative sample memory sequences are concatenated to compare with the features of the q encoder across modal views to obtain the similarity set of cross-view contrastive learning, and finally the final result is calculated with the contrast value. S322, assuming that the model j modal view guides the model m modal view, and their negative sample memory sequences after the model are M j and M m , the index o of the most similar negative sample is learned in view m by comparison, and the formula for guiding the model m modal view to perform cross-view comparison learning from the model j modal view is: in, and The cross-view contrast learning similarity set S belongs to modal views j and m respectively j and S m , and Represents the part of negative samples that are most similar to the corresponding modality; S323. In the second stage, for any modal view v and any modal view u, the final result is obtained by formula (8):

Images