Skeleton fine action recognition method based on action response type comparison network

Abstract

The invention provides a skeleton fine action recognition method based on an action response type comparison network, and belongs to the technical field of computer vision and artificial intelligence. The technical problem that complex and fine skeleton actions are extremely difficult to accurately recognize due to limited appearance information in skeleton data and limitation of a predefined single topological structure is solved. According to the technical scheme, multi-stream preprocessing is carried out on input skeleton data; the method comprises the following steps of: firstly, transmitting data to a motion response type comparison network ARCN, establishing a multi-channel cross-time-domain dynamic skeleton joint attention topology through a motion response type graph convolution network, and comparing and updating the features in space and time dimensions in stages through a fine motion comparator; finally, the cross entropy loss of the classification task and the motion comparison loss based on feature comparison jointly evaluate the motion recognition accuracy. The method has the beneficial effects that the problem of fuzzy boundaries between the fine-grained behaviors is relieved, and the capability of distinguishing the fine-grained skeleton behaviors is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and artificial intelligence, and in particular to a skeleton fine motion recognition method based on an action-responsive contrast network. Background Art

[0002] Human communication is the cornerstone of society. Daily behaviors convey emotions and intentions that affect individuals and society. With the development of intelligent society, action recognition is increasingly used in fields such as human-computer interaction, video surveillance, and abnormal behavior detection, and the requirements for accuracy and detail are also increasing. One of the main challenges of action recognition is to extract rich features in complex backgrounds. Skeleton data has attracted widespread attention due to its robust and stable action features.

[0003] Early methods mainly used recurrent neural networks (RNNs) or convolutional neural networks (CNNs) to analyze joint features, but these methods often ignore the structured representation of the skeleton. Skeleton data naturally forms a topological graph structure, so graph convolutional networks (GCNs) were introduced to model skeleton sequences. GCNs can effectively capture key skeleton action features using the topological structure of the skeleton. This graph-based model has been widely accepted and applied.

[0004] However, everything has two sides. Graph convolution technology can make full use of the advantages of skeleton data, which has rich motion features and strong robustness to complex background information. However, the cost is the loss of background information strongly associated with the action and the limitation of a predefined single fixed topological structure. It only establishes the correlation between naturally connected nodes, while the implicit relationship between non-naturally connected nodes when the action occurs is ignored. This will lead to the loss of key information when performing graph convolution, thereby limiting the expressive power of the graph convolution network and making it more difficult to identify fine movements. Skeleton movements with small differences in movement patterns, such as "reading" and "writing", are difficult to be perceived and utilized by general graph convolution models based on skeleton data alone. Their features are almost mixed together and cannot be distinguished. How can the model learn more discriminative fine movement features from skeleton data? Given that each channel represents different types of dynamic features and the interaction between joints is not constant under different movement characteristics, it is not an ideal choice to uniformly apply a single topology. Some researchers have proposed to establish a topological refinement model to reflect the movement patterns of different movements. Although the limitation of a single topology structure is alleviated to a certain extent by establishing a dynamic topology in the graph convolution layer, it only explores the overall spatial structure of different motion modes. It is very effective for actions with significant differences in motion modes, but the temporal feature expression of learning different motion modes from continuous actions is weak, and the recognition effect of fine movements is insufficient. Secondly, for different motion modes, especially for fine movements, multi-dimensional feature fusion and attention learning of important joints becomes more important. For example, using a spatiotemporal transformer network, the transformer self-attention operator is used through a two-stream network to model the dependency between joints. However, if the joint space and time features are processed separately and then feature fusion is performed, the interactivity of some joint spatiotemporal information may be reduced. Taking "reading" and "writing" as examples, their joint features learned independently in space and time are extremely similar, but the temporal feature expression under similar spatial structures can reflect the subtle differences in their joint features. In addition, since the representation of fine motion features is overly discrete and the boundaries are fuzzy, although there are some research works on fine motion features, research in this area is still limited. In order to fully explore the tiny feature differences between fine movements, it is crucial to explore a compact intra-class and inter-class dispersed motion implicit space for refined motion features. Summary of the invention

[0005] The purpose of the present invention is to provide a skeleton fine motion recognition method based on an action-responsive contrast network. The present invention provides a motion-responsive contrast network for skeleton fine motion recognition tasks. The invention can learn the unique skeletal topology of different fine-grained motions through feature fusion at different spatiotemporal scales, and use feature contrast learning to construct a learnable latent space to capture the subtle feature differences of fine-grained motions, thereby improving the ability to recognize fine-grained human motions and providing strong support for downstream tasks such as human-computer interaction and intelligent monitoring.

[0006] The inventive concept of the present invention is: the present invention proposes an action responsive contrast network (ARCN), which establishes a multi-channel cross-temporal dynamic skeleton joint attention topology and uses feature contrast learning to construct a latent space of fine-grained actions. This method aims to overcome the limitations of a single topological structure and make up for the lack of appearance information in skeleton data from different perspectives. Specifically, the present invention first designs an action responsive topological graph convolutional network, which includes two parts: 1) The action responsive topological module (ART) enhances multi-scale temporal feature learning based on the improved dynamic skeleton topology. It emphasizes the integration of unique action patterns between consecutive frames, focuses on the spatiotemporal topological features of fine-grained actions, and establishes a multi-channel cross-temporal dynamic skeleton topological structure; 2) The action responsive attention module (ARA), which embeds the decoupled spatiotemporal features into the same space for specific joint spatiotemporal attention feature learning. This method better captures the unique joint feature differences of fine-grained operations from both global and local perspectives. Secondly, the present invention applies feature contrastive learning method to explore the latent space of fine-grained actions and proposes a fine-grained action comparator (FAC); by classifying sample encodings into similar and different clusters, updating the global feature representation, and performing multi-level feature contrastive learning at different stages of the model.

[0007] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a skeleton fine action recognition method based on an action-responsive contrast network, comprising the following steps:

[0008] S1. Preprocess the collected human skeleton data to form four stream features: joint data stream, bone data stream, joint velocity data stream, and bone velocity data stream;

[0009] S2. Construct an action-responsive comparison network framework ARCN for skeleton fine action recognition methods. ARCN is mainly composed of two modules: Action-Responsive Graph Convolutional Network (ARGCN) and Fine-Grained Action Comparator (FAC). ARGCN establishes a multi-channel cross-temporal dynamic skeleton joint attention topology to predict the category of action. It consists of an action-responsive topology module (ART) and an action-responsive attention module (ARA). While ARGCN is running, the discriminant features are periodically passed into FAC. FAC performs feature comparison and updates in spatial and temporal dimensions respectively to perform more detailed discrimination of fine actions;

[0010] S3. The preprocessed multi-stream data is input into the network framework ARCN for processing. Each skeleton data will eventually generate two types of loss values: one is the cross entropy loss from the action-responsive graph convolutional network, and the other is the action contrast loss obtained by the fine action comparator based on feature contrast. The two types of loss values ​​corresponding to the four-stream input data are considered together to form the basis for the final estimation of the accuracy of fine action recognition.

[0011] Furthermore, the step S1 includes the following steps:

[0012] S11. Notation definition: In skeleton-based action recognition, a skeleton sequence is a collection of action video frames. In each frame, the skeleton is represented as a graph G = (V, E), where V = {v1, v2, …, v N} represents a vertex set consisting of N joints, E = {e1, e2, …, e M} represents the edge set consisting of bones between joints. The structure of the skeleton graph can be described using the graph-specific adjacency matrix A, where A ij The value is 0 or 1, indicating that the joint v i and v j Whether there is an edge connection between them, the entire skeleton sequence can be simply represented by a coordinate set X = {x∈R C×T×V}, where C is the coordinate dimension, usually C = 3, T is the total number of frames in the skeleton sequence, and V is the total number of joints in the current skeleton;

[0013] S12, for the joint data stream input, the joint dimension is fixed, the defined human spine center node is used as the unified standardized zero node, and the normalized position feature P is obtained by subtracting other joint points from the standard zero node.i |(i=1,2,…,V in )}, the formula is:

[0014] p i =x[:,:,i]-x[:,:,c] (1)

[0015] Among them, p i is the normalized position feature of each joint index of the skeleton sequence X, x is the input feature of the skeleton sequence X, c is the index of the central node of the human spine, and the joint flow input can be directly obtained by connecting X and P in series;

[0016] S13, the skeleton data stream input is obtained by calculating the relative positions between adjacent joints in each frame, that is, the direction vector of each bone;

[0017] S14, the joint velocity data stream input is obtained by calculating the position change of each joint in each frame;

[0018] S15. The bone velocity data stream input is obtained by calculating the change in the direction vector of each bone in each frame.

[0019] Furthermore, the step S2 includes the following steps:

[0020] S21. Build the action-responsive graph convolutional network (ARGCN) of the ARCN framework. ARGCN completes the learning of multi-channel cross-temporal dynamic skeleton joint attention topology, which mainly includes the action-responsive topology module (ART) and the action-responsive attention module (ARA). ART learns the multi-channel cross-temporal dynamic skeleton topology structure of different channels and aggregated action temporal features, and ARA models the importance of skeleton joints in the spatiotemporal dimension.

[0021] S22. Build the fine action comparator (FAC) of the ARCN framework. FAC builds a learnable fine action latent space to explore the hidden information that can distinguish between fine actions.

[0022] Furthermore, in the step S21, constructing the action-responsive graph convolutional network (ARGCN) of the framework ARCN includes the following steps:

[0023] S211, the input skeleton data first passes through the action-responsive topology module (ART) to construct a topology map of a specific motion category;

[0024] S212, the features are continuously transmitted to the action responsive attention module (ARA) to strengthen the feature expression of the main motion joints. After each ARA, the feature map is transmitted to the fine action comparator (FAC) again to strengthen the judgment;

[0025] S213. Obtain the probability distribution of the required motion category through a pooling layer and a fully connected layer with a softmax activation function.

[0026] Furthermore, in the step S211, the input skeleton data is first processed by an action-responsive topology module (ART) including the following steps:

[0027] S2111, normalizing the input data to stabilize the value;

[0028] S2112, through three parallel action-responsive graph convolution blocks to extract some hidden topological connections that are most relevant to the action in space;

[0029] S2113. Perform multi-scale temporal convolution to learn the strength of joint connections when the duration of an action changes.

[0030] Furthermore, in the step S2112, normalizing the input data standard to stabilize the value includes the following steps:

[0031] S21121, input feature map x∈R C×T×V Transformed into a more refined feature representation through a bottleneck layer (1x1 convolution layer);

[0032] S21122, performing pooling along the time dimension to aggregate the time features, thereby obtaining the spatial features after dimensionality reduction processing;

[0033] S21123. Through the designed correlation function, we can deeply explore the relationship between non-adjacent joints and mine the hidden topological correlation matrix A′∈R specific to non-connected joints. C′×V×V :

[0034]

[0035] where a′ ij ∈R C′ is the skeleton vertex (v i , v y ) The corresponding relationship vector in the hidden topological association matrix A′, x i 、x j ∈R C×T is the corresponding skeleton vertex v i 、v j The input features of M1(·) are transformed by performing bottleneck convolution to simulate the correlation between two vertices. φ(·) and φ(·) transform the input features. Bottleneck convolution and time dimension pooling are used here in turn. σ(·) represents the activation function Tanh, and Conv represents the bottleneck layer convolution.

[0036] S21124. Use A′ to improve the original natural connection skeleton topology connection matrix A∈R V×V To obtain the specific action topology matrix H∈R associated with the action C′×V×V :

[0037] H=M2(A,A′)=A+α·A′ (3)

[0038] Among them, M2(·) establishes a specific action topology matrix, and α is a learnable parameter that can adjust the importance of hidden topological associations. Each specific action topology matrix reveals the association pattern of vertex interconnection under specific motion attributes. In order to amplify the differences in different action features and aggregate the global skeleton topology information, the original features are transformed through the bottleneck layer to represent x′ C，：，： ∈R 1×T×V The specific action topology matrix H obtained by each channel c ∈R 1×V×V (c∈{1,…,c′}) and perform channel-by-channel aggregation to obtain the skeleton topology information output G∈R C′×T×V .

[0039] Furthermore, in the step S2113, performing multi-scale time convolution includes the following steps:

[0040] S21131, introduced multi-scale temporal convolution to study the strength of different joint connections across time frames. The module contains four branches, each of which contains a bottleneck convolution to compress the channel. The first three branches also include two temporal convolutions with different dilations and a maximum pooling layer to capture a wider range of action-related information. The results of these four branches are pooled to obtain a multi-channel cross-temporal skeleton topology, which is represented by F∈R C′×T′×V During the training process, residual connections are used to enhance the stability of the network.

[0041] Furthermore, in the step S212, continuing to transfer the features to the action-responsive attention module (ARA) includes the following steps:

[0042] S2121, input multi-channel cross-time domain dynamic skeleton topology representation feature F in ∈R C×T×V After pooling in the time and space dimensions, the specific action features are extracted into corresponding spatial joint features and time frame features. Then, the two feature vectors are connected in series:

[0043] F mid1 =Cat[Pool t (F in ), Pool s (F in )] (4)

[0044] Among them, F mid1 Represents the characteristics of the intermediate transformation, Pool t (·) and Pool s (·) are the average pooling operations on temporal frames and spatial joints, respectively, and Cat represents the concatenation operation.

[0045] S2122, compress information through a bottleneck layer, and mine hidden features through standardization and nonlinear transformation of activation functions:

[0046] F mid2 =ξ(BN(Conv(F mid1 ))) (5)

[0047] Among them, F mid2 represents the features of the intermediate transformation, Conv represents the bottleneck layer convolution, BN represents the normalization operation, and ξ(·) represents the HardSwish activation function.

[0048] S2123. Thereafter, the updated spatial joint features and temporal frame features are separated one by one, and spatial joint and temporal joint feature scores are independently extracted:

[0049] F t , F s =Split(F mid2 , dim=2) (6)

[0050] Among them, Split(·) re-divides the spatiotemporal embedding features into spatial features F s With time feature F t .

[0051] S2124, generate a score feature F representing the joint attention intensity of the entire action sequence through the outer product operation of the spatial joint feature score and the temporal feature score att ∈R C×T×V .

[0052]

[0053] Here, σ(·) represents the Sigmoid activation function.

[0054] S2125. Finally, through the original input feature F in And the obtained score feature F att Perform outer product and standardization processing to complete feature mapping, and compare the obtained features with the original input features F in Perform the residual connection again to obtain the required multi-channel cross-temporal dynamic skeleton joint attention topology F out ∈R C×T×V :

[0055]

[0056] in, and represents the channel outer product and element aggregation, and ψ(·) represents the Swish activation function. It is worth noting that the shape of the output feature remains unchanged from the input feature.

[0057] Furthermore, in the step S22, constructing the fine motion comparator (FAC) of the framework ARCN includes the following steps:

[0058] S221, decouple features in time and space, and explore in depth that fine-grained actions may contain distinguishable hidden features across different dimensions for further research;

[0059] S222, respectively pass the time-space decoupled features into the fine action contrast learning to complete the feature contrast learning of the action latent space in the independent time and space of the fine action.

[0060] Furthermore, in the step S221, decoupling the features in time and space includes the following steps:

[0061] S2211, the input feature sequence is spatially and temporally pooled, followed by bottleneck convolution and feature flattening to capture a deep understanding of temporal and spatial variations respectively without interfering with each other:

[0062] F t ′=Re LU(BN(Conv(Pool s (F in )))) (9)

[0063] F′ s =ReLU(BN(Conv(Pool t (F in )))) (10)

[0064] Among them, F in Represents the incoming features, F t ′ represents the temporal features extracted from the sample, F s ′ represents the spatial features extracted from the sample.

[0065] Furthermore, in the step S222, the steps of respectively transferring the spatiotemporal decoupled features into the fine motion contrast learning include the following steps:

[0066] S2221, for sample i, the temporal feature F extracted after time-space decoupling t ′ and spatial feature F′ s After the same method of contrast learning, the following uses F i replace.

[0067] S2222, given an action m, extract the feature F from a sample i i The predicted label results are encoded using one-hot encoding, and then compared with the true label m encoding to determine whether they are true positives (TP), false negatives (FN), and false positives (FP). TP refers to the positive examples of actions correctly judged by the model. By performing element-wise multiplication of the action label one-hot encoding and the predicted one-hot encoding, we obtain the TP of each category for each sample, which is called the same cluster; FN and FP are the results of incorrect judgment by the model, but FN means that the model mistakenly predicts the current action as other actions, that is, misjudgment. FP means that the model mistakenly predicts other actions as the current action, but they are actually samples of other actions. FN and FP together constitute the overall heterogeneous cluster.

[0068] S2223. Generally speaking, there are more samples in each cluster and their features have better feature consistency. An exponential moving average is used for each cluster to keep the global representation (i.e., prototype feature) updated:

[0069]

[0070] Among them, J m is the global representation of the action m feature, is the same cluster sample of action m, is the number of samples in the same cluster, and α is the momentum term, which is usually set to 0.9 according to experience. By using a global representation, the features of samples belonging to the same cluster can be brought together to reduce intra-class differences.

[0071] S2224. Calculate the feature averages of different clusters so as to optimize these misclassified samples in the subsequent contrastive learning. The formula is as follows:

[0072]

[0073] Among them, F fn 、F fp They represent the average feature representations of the FN sample set and the FP set respectively, and they together constitute the central representation of the heterogeneous clusters. is a heterogeneous FN sample, is a FP sample in a different cluster, and are the number of corresponding samples respectively.

[0074] S2225. In order to quantify the similarity between the sample features and the average feature vector, the cosine similarity between the sample features and various prototype features and the average feature representation of different clusters is calculated to obtain the comparative learning score and use it for subsequent loss calculation:

[0075]

[0076] Among them, Score m It is the comparative learning score between the global representation of action m (class prototype feature) and the current sample i, which is the basis of contrastive learning. fn 、Score fp are the comparative learning scores of FN samples and FP samples with the current sample i respectively.

[0077] S2226. Although FN samples and FP samples belong to different clusters, they are different. For FN samples, we should make them closer to the sample features of the same cluster in the feature space, which may calibrate the model's incorrect prediction of fine motion features. For FP samples, we should make them farther away from the sample features of the same cluster in the feature space, which may more effectively distinguish the subtle differences between fine features. For this reason, we introduce a compensation term φ for FN samples. i , introduce a penalty term Ψ for FP samples i :

[0078]

[0079] Adjust the compensation term φ i and the penalty term Ψ i It helps to correct the misclassification of FN samples in different clusters and prevent the misclassification of FP samples.

[0080] S2227. Finally, the action contrast loss is defined using the various contrast learning scores obtained above.

[0081]

[0082] Among them, p in is the action contrast loss for extracting features from the current action m, is the feature F extracted from the current action m i Action comparison

[0083] loss.

[0084] S2228, the complete FAC obtains the time dimension action contrast loss L tac And the spatial dimension action contrast loss L sac , adding the two-dimensional action contrast loss can form the action contrast loss L AC :

[0085] L AC =L tac +L sac (20)

[0086] Furthermore, S3 comprises the following steps:

[0087] S31. In ARGCN, cross entropy loss is used to train the network:

[0088]

[0089] Where N is the number of samples in the batch. ij is the one-hot encoding representation of action sample i. If and only if j is the target action class of sample i, y ij =1. ij The probability score predicted by the network that sample i belongs to class m.

[0090] S32. In order to extensively learn the latent space of fine movements, each time through ART, the feature map of the dynamic skeleton joint attention topology of the multi-channel cross-time domain is passed to FAC, forming a multi-stage action contrast loss. The multi-stage action contrast loss can be defined as the weighted average sum:

[0091]

[0092] Among them, L MAC It represents the multi-stage action contrast loss finally obtained by FAC of the entire action-responsive contrast network. i represents the number of stages, which are 4 in total, in the second, seventh, eleventh and fourteenth layers of ARGCN respectively. AC is the action contrast loss obtained at each stage of FAC, λ i It is a learnable parameter that adjusts the loss ratio at each stage.

[0093] S33. Combine the cross entropy loss with the multi-stage action contrast loss to form a complete learning objective function:

[0094]

[0095] in, is a learnable hyperparameter that can be used to tune the importance of multi-stage action contrastive loss in ARCN.

[0096] Compared with the prior art, the present invention has the following beneficial effects:

[0097] 1. This paper proposes an action response contrast network (ARCN), which establishes a multi-channel cross-temporal dynamic skeleton joint attention topology and uses feature contrast learning to construct a latent space of fine-grained actions. This method aims to overcome the limitations of a single topological structure and make up for the lack of appearance information in skeleton data from different perspectives.

[0098] 2. The present invention first designs an action-responsive topological graph convolutional network, which consists of two parts: 1) The action-responsive topological module ART enhances multi-scale temporal feature learning based on the improved dynamic skeleton topology. It emphasizes the integration of unique action patterns between consecutive frames, focuses on the spatiotemporal topological features of fine-grained actions, and establishes a multi-channel cross-temporal dynamic skeleton topological structure; 2) The action-responsive attention module ARA, which embeds the decoupled spatiotemporal features into the same space for specific joint spatiotemporal attention feature learning. This method better captures the unique joint feature differences of fine-grained operations from both global and local perspectives.

[0099] 3. The present invention applies feature contrast learning method to explore the latent space of fine-grained actions, and proposes a fine-grained action comparator FAC, which classifies sample encodings into similar and different clusters, updates the global feature representation, and performs multi-level feature contrast learning at different stages of the model; this method alleviates the problem of fuzzy boundaries between fine-grained behaviors and effectively improves the network's ability to distinguish fine-grained skeletal behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0101] Figure 1 It is the overall framework diagram of the action responsive contrast network (ARCN) in the present invention.

[0102] Figure 2 This is a framework diagram of the action-responsive graph convolutional network (ARGCN) in the present invention.

[0103] Figure 3 This is a graph exploring the accuracy of classifying actions by the action-responsive topological graph convolutional network in the present invention on the joint input branch only of the NTU RGB+D dataset X-Sub; wherein, (a) represents the recognition performance of ARGCN (with 80% as a benchmark), blue represents actions with a recognition accuracy greater than 80%, and red represents actions with a recognition accuracy less than 80%; (b) is an error analysis of the low recognition rate of the true label "writing", that is, the error probability that "writing" may be misjudged as other actions.

[0104] Figure 4 It is a specific structural diagram of the fine motion comparator FAC in the present invention.

[0105] Figure 5 It is a dynamic skeleton topology diagram of multi-channel and cross-time domain of random action samples in the deep layer of the model in the present invention.

[0106] Figure 6It is the joint feature attention map of random samples of different actions in the present invention.

[0107] Figure 7 This is a radar chart showing the influence of FAC on model performance on six data sets in the present invention.

[0108] Figure 8 For the related failed actions in the present invention, the confusion matrix diagram of the baseline model with an accuracy lower than 80% under the NTU RGB+D X-Sub benchmark with only joint modality input; wherein (a) and (b) represent the confusion matrices of the failed actions of the baseline model and the model of the present invention, respectively, wherein the coordinate axis represents each action category, and the red rectangle represents similar actions.

[0109] Fig. 9 Schematic diagram of comparison of classification accuracy of six easily confusing actions using different methods in an embodiment of the present invention.

[0110] Fig.10 This is the t-SNE visualization result diagram of different models in the present invention for several random samples of subtle hand movements.

[0111] Fig.11 This is a visualization skeleton diagram of the model in the present invention on 10 frames of some example actions. DETAILED DESCRIPTION

[0112] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0113] Example 1

[0114] See also Figure 1 and Fig.11 This embodiment provides a technical solution, a skeleton fine motion recognition method based on action-responsive contrast network, which is to explore the differential expression of hidden features of fine motion by establishing a multi-channel cross-time dynamic skeleton joint attention topology and performing feature contrast update in low-dimensional space to improve the accuracy of fine motion recognition. The proposed framework is the Action-Responsive Contrastive Network (ARCN) architecture, such as Figure 1As shown in the figure. The whole architecture adopts multi-stream data joint evaluation and is mainly composed of two modules: Action-Responsive Graph Convolutional Network (ARGCN) and Fine-Grained Action Comparator (FAC). Among them, ARGCN establishes a multi-channel cross-temporal dynamic skeleton joint attention topology to predict the category of action. It consists of an action-responsive topology module (ART) and an action-responsive attention module (ARA). ART learns a multi-channel cross-temporal dynamic skeleton topology structure of different channels and aggregated action temporal features, and ARA models the importance of skeleton joints in the spatiotemporal dimension. While ARGCN is running, the discriminant features are passed to FAC in stages. FAC compares and updates features in the spatial and temporal dimensions respectively to perform more detailed discrimination of fine actions. Through the combined effect of ARGCN and FAC, each skeleton data will eventually generate two types of loss values: one is the cross entropy loss from the classification task, and the other is the motion contrast loss based on feature contrast update. The combined consideration of these two types of loss values ​​corresponding to the four-stream input data constitutes the basis for the final estimation of the accuracy of fine motion recognition.

[0115] Action Responsive Graph Convolutional Network (ARGCN): Although natural human skeletal connections are reliable for action recognition, their fixed nature limits the ability to detect hidden information in actions, affecting the recognition of complex and fine-grained actions. By building a dynamic skeleton topology for specific actions and focusing on key parts, this approach can capture global and local action information. Focusing on key action areas can enhance feature representation beyond skeletal connections, thereby improving action understanding and analysis. For these reasons, the present invention designs an action-responsive topology graph convolutional network (ARGCN), such as Figure 2As shown in the figure. Specifically, the ARGCN backbone consists of 14 basic blocks, including 10 action-responsive topological modules (ARTs) and 4 action-responsive attention modules (ARAs). The input skeleton data first passes through ART to construct a topological map of a specific motion category, and then passes through ARA to strengthen the feature expression of the main motion joints. After each ARA, the feature map will be passed back to the fine action comparator (FAC) to strengthen the judgment. Finally, the probability distribution of the required motion category is obtained through the pooling layer and the fully connected layer with a softmax activation function. It is particularly important to note that multi-scale feature matching helps in skeleton action recognition. The 6th and 10th basic blocks are implemented by cross-row multi-scale temporal convolutions with a step size of 2. The corresponding module increases the channel dimension while reducing the time dimension, generating multi-scale features.

[0116] Action-responsive topology module (ART) in ARGCN: In order to fully explore the global and local information of the skeleton topology, the present invention designs ART by taking advantage of the different unnatural topologies of the skeletons of different motion types. It first normalizes the input data to stabilize the values, then uses three parallel action-responsive graph convolution blocks to spatially extract some hidden topological connections that are most relevant to the action, and finally performs multi-scale temporal convolution to learn the strength of joint connections when the duration of the action changes. Figure 2 The upper right shows the specific structure of ART.

[0117] Specifically, in order to effectively infer the hidden skeleton connection representation between non-adjacent joints and reduce the amount of computation, the input feature map x∈R C×T×V It is converted into a more refined feature representation through the bottleneck layer (1x1 convolution layer). Then, pooling is performed along the time dimension to aggregate the time features, thereby obtaining the spatial features after dimensionality reduction. The designed correlation function is then used to deeply explore the relationship between non-adjacent joints and to mine the hidden topological association matrix A′∈R specific to the unconnected joints. C′×V×V :

[0118]

[0119] where a′ ij ∈R C′ is the skeleton vertex (v i , v y ) The corresponding relationship vector in the hidden topological association matrix A′, x i 、x j ∈R C×T is the corresponding skeleton vertex v i 、v iThe input features of M1(·) are calculated by performing bottleneck convolution after calculating the distance between the vertex features after feature transformation to simulate the correlation between the two vertices. φ(·) and φ(·) transform the input features. Bottleneck convolution and time dimension pooling are used here in turn. σ(·) represents the activation function Tanh, and Conv represents the bottleneck layer convolution. Afterwards, A′ is used to improve the original natural connection skeleton topology connection matrix A∈R V×V To obtain the specific action topology matrix H∈R associated with the action C′×V×V :

[0120] H=M2(A,A′)=A+α·A′ (3)

[0121] Among them, M2(·) establishes a specific action topology matrix, and α is a learnable parameter that can adjust the importance of hidden topological associations. Each specific action topology matrix reveals the association pattern of vertex interconnection under specific motion attributes. In order to amplify the differences in different action features and aggregate the global skeleton topology information, the original features are transformed through the bottleneck layer to represent x′ C，：，： ∈R 1×T×V The specific action topology matrix H obtained by each channel c ∈R 1×V×V (c∈{1,…,c′}) and perform channel-by-channel aggregation to obtain the skeleton topology information output G∈R C′×T×V .

[0122] For fine-grained action recognition, since the spatial structures of some actions are extremely similar, it is not enough to only explore the spatial topology of the skeleton. Considering that temporal features can be used as a distinguishing factor for fine-grained actions, this module introduces multi-scale temporal convolution to study the strength of different joint connections across time frames, such as Figure 2 As shown in the upper right. This module contains four branches, each of which contains a bottleneck convolution to compress the channel. The first three branches also include two temporal convolutions with different dilations and a maximum pooling layer to capture more extensive action-related information. The results of these four branches are pooled to obtain a multi-channel cross-temporal skeleton topology, which is represented by F∈R C′×T′×V During the training process, residual connections are used to enhance the stability of the network.

[0123] Action Responsive Attention (ARA) in ARGCN: Multi-dimensional information interaction is very effective in generating better attention maps. Considering the correlation between spatial and temporal information in human action skeleton sequences, it is crucial to identify the joints with the most information in a specific frame of the entire sequence, especially for identifying fine-grained actions with subtle differences. For this reason, the present invention designs ARA, which is summarized as follows: Figure 2 Shown below right.

[0124] First, the input multi-channel cross-temporal dynamic skeleton topology representation feature F in ∈R C×T×V After pooling in the time and space dimensions, the specific action features are extracted into corresponding spatial joint features and time frame features. Then, the two feature vectors are connected in series, information is compressed through a bottleneck layer, and hidden features are mined through standardization and nonlinear transformation of activation functions. After that, the updated spatial joint features and time frame features are separated one by one, and the spatial joint and time joint feature scores are independently extracted. The outer product operation of the spatial joint feature score and the time feature score is performed to generate a score feature F representing the joint attention strength of the entire action sequence. att ∈R C×T×V Finally, through the original input feature F in And the obtained score feature F att Perform outer product and standardization processing to complete feature mapping, and compare the obtained features with the original input features F in Perform the residual connection again to obtain the required multi-channel cross-temporal dynamic skeleton joint attention topology F out ∈R C×T×V It is worth noting that the shape of the output features remains unchanged from the input features. The proposed action-responsive attention module can be expressed as:

[0125] F mid1 =Cat[Pool t (F in ), Pool s (F in )] (4)

[0126] F min2 =ξ(BN(Conv(F mid1 ))) (5)

[0127] F t , F s =Split(F mid2 ,dim=2) (6)

[0128]

[0129] Among them, F mid1 and F mid2 Represents the characteristics of the intermediate transformation, Pool t (·) and Pool s (·) are the average pooling operations on the time frame and the spatial joint, respectively. Cat represents the concatenation operation, Conv represents the bottleneck layer convolution, BN represents the normalization operation, and Split (·) re-divides the spatiotemporal embedding features into the spatial features F s With time feature Ft . and represents channel outer product and element aggregation, ξ(·), σ(·), and ψ(·) represent HardSwish, Sigmoid, and Swish activation functions, respectively.

[0130] Fine Action Comparator (FAC): Even though a multi-channel cross-temporal dynamic skeleton joint attention topology with strong action characteristics is established, there are still problems similar to other models: it is still difficult to identify some fine actions that are only different in small parts, such as Figure 3 . Figure 3 (a) shows the classification accuracy of ARGCN on the joint-only input of the NTU RGB+D dataset X-Sub. Taking 80% as the cutoff point, the red part on the right represents actions with a recognition rate of less than 80%. It can be seen from the figure that some actions are still difficult to recognize, especially "writing". In order to analyze the reason for the low recognition rate of "writing", an error analysis of "writing" is performed, such as Figure 3 As shown in (b). The results show that for samples with the true label "writing", there is a 38.8% probability of being misjudged as "typing on the keyboard", a 30.1% probability of being misjudged as "reading", and a small probability of being misjudged as other actions. The movement patterns between these actions are very similar, and can generally be assisted by apparent information for identification, but due to the lack of apparent information in the skeleton, these actions are very easy to confuse. In order to enhance the recognition of such fine movements, the present invention introduces a feature contrast learning method and designs a fine movement comparator, which constructs a learnable fine movement latent space from another perspective to explore the distinguishable hidden information between fine movements. FAC consists of two parts, such as Figure 4 shown.

[0131] Spatiotemporal decoupling in FAC: Fine-grained actions may contain distinguishable hidden features across different dimensions. To deeply explore such hidden information for further study, the input feature sequence is subjected to spatial and temporal pooling, followed by bottleneck convolution and feature flattening to capture a deep understanding of temporal and spatial variations respectively without interfering with each other:

[0132] F t ′=Re LU(BN(Conv(Pool s (F in )))) (9)

[0133] F′ s =ReLU(BN(Conv(Pool t (F in )))) (10)

[0134] Among them, F in Represents the incoming features, Ft ′ represents the temporal features extracted from the sample, F s ′ represents the spatial features extracted from the sample.

[0135] Fine motion contrast learning in FAC: In skeleton-based action recognition, feature contrast learning can optimize feature representation by comparing sample features and class prototype features, thereby improving the model’s ability to distinguish fine motions. For sample i, the temporal feature F extracted after spatiotemporal decoupling is t ′ and spatial feature F s ′After the same method of contrast learning, the following uses F i replace.

[0136] Given an action m, extract the feature F from a sample i i The predicted label results are encoded with one-hot encoding respectively, and then compared with the true label m encoding to determine whether they are divided into true positive (TP), false negative (FN), and false positive (FP). TP refers to the positive example of the action correctly judged by the model. By performing element-wise multiplication on the one-hot encoding of the action label and the predicted one-hot encoding, we obtain the TP of each category for each sample, which is called the same cluster; FN and FP are the results of incorrect judgment by the model, but FN means that the model mistakenly predicts the current action as other actions, that is, misjudgment. FP means that the model mistakenly predicts other actions as the current action, but they are actually samples of other actions. FN and FP together constitute the overall heterogeneous cluster. Generally speaking, there are more samples in each cluster and their features have better feature consistency. The exponential moving average is used for each cluster to keep the global representation (that is, the prototype feature) updated:

[0137]

[0138] Among them, I m is the global representation of the action m feature, is the same cluster sample of action m, is the number of samples in the same cluster, and α is the momentum term, which is usually set to 0.9 according to experience. By using a global representation, the features of samples belonging to the same cluster can be gathered together to reduce intra-class differences. Then, the feature averages of different clusters are calculated to optimize these misclassified samples in the subsequent contrastive learning. The formula is as follows:

[0139]

[0140] Among them, F fn 、F fp They represent the average feature representations of the FN sample set and the FP set respectively, and they together constitute the central representation of the heterogeneous clusters. is a heterogeneous FN sample, is a FP sample in a different cluster, and are the number of corresponding samples respectively.

[0141] In order to quantify the similarity between sample features and the average feature vector, the cosine similarity between sample features and various prototype features and heterogeneous cluster average feature representations is calculated to obtain the contrastive learning score and use it for subsequent loss calculations:

[0142]

[0143] Among them, Score m It is the comparative learning score between the global representation of action m (class prototype feature) and the current sample i, which is the basis of contrastive learning. fn 、Score fp is the comparative learning score of FN samples and FP samples with the current sample i. Although FN samples and FP samples belong to different clusters, they are different. For FN samples, they should be closer to the sample features of the same cluster in the feature space, which may calibrate the model's incorrect prediction of fine motion features. For FP samples, they should be further away from the sample features of the same cluster in the feature space, which may more effectively distinguish the subtle differences between fine features. For this reason, the compensation term φ is introduced for FN samples. i , introduce a penalty term Ψ for FP samples i :

[0144]

[0145] Adjust the compensation term φ i and the penalty term Ψ i It helps to correct the misclassification of FN samples in different clusters and prevent the misclassification of FP samples.

[0146] Finally, the action contrast loss is defined using the various contrastive learning scores obtained above

[0147]

[0148] Among them, p im is the action contrast loss for extracting features from the current action m, is the feature F extracted from the current action m i The action contrast loss of the complete FAC is the time dimension action contrast loss L tac And the spatial dimension action contrast loss L sac , adding the two-dimensional action contrast loss can form the action contrast loss L AC :

[0149] L AC =Ltac +L sac (20)

[0150] In ARGCN, the cross entropy loss is used to train the network:

[0151]

[0152] Where N is the number of samples in the batch. ij is the one-hot encoding representation of action sample i. If and only if j is the target action class of sample i, y ij =1. ij The probability score predicted by the network that sample i belongs to class m.

[0153] In order to extensively learn the motion latent space of fine movements, each time through ART, the feature map of the dynamic skeleton joint attention topology of multi-channel cross-time domain is passed to FAC, forming a multi-stage motion contrast loss. The multi-stage motion contrast loss can be defined as the weighted average sum:

[0154]

[0155] Among them, L MAC It represents the multi-stage action contrast loss finally obtained by FAC of the entire action-responsive contrast network. i represents the number of stages, which are 4 in total, in the second, seventh, eleventh and fourteenth layers of ARGCN respectively. AC is the action contrast loss obtained at each stage of FAC, λ i The cross entropy loss is combined with the multi-stage action contrast loss to form a complete learning objective function:

[0156]

[0157] in, is a learnable hyperparameter that can be used to tune the importance of multi-stage action contrastive loss in ARCN.

[0158] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0159] Furthermore, simulation experiments are performed to verify the effectiveness of the method proposed in the present invention.

[0160] Table 1. Performance comparison (%) of our model with existing methods on NTU RGB+D, NTU RGB+D 120, NW-UCLA, and UAV-Human datasets. The top part consists of several models without GCN technology, while the other part contains some graph-based models. The best one is marked in bold, and the second one is marked in Underline Mark out.

[0161]

[0162] The model is compared with the current state-of-the-art methods on the NTU RGB+D, NTU RGB+D 120, NW-UCLA, UAV-Human datasets and two fine-grained datasets, FineGYM and Diving48. All reported results are trained and evaluated on a server with Nvidia GeForce RTX 4090 and experiments are conducted using PyTorch. We use standard evaluation metrics to evaluate the performance of the model for skeleton fine action recognition.

[0163] Table 1 shows the comparison with previous methods on NTU RGB+D, NTU RGB+D 120, NW-UCLA, and UAV-Human datasets. In the early stage, some CNN- or RNN-based methods were more accurate than GCN-based models. For example, VA-fusion was 7.9% and 6.7% more accurate than ST-GCN, a classic GCN-based model, on the NTURGB+D dataset. This higher accuracy was due to the fact that VA-fusion combined the characteristics of mature RNN and CNN technologies to fully extract action features. At that time, GCN technology was just starting out and the advantages of skeleton data were not fully explored. With the continuous development of GCN technology, the task of using GCN for skeleton action recognition has been getting better and better. It is worth noting that most of the current state-of-the-art methods use a multi-stream fusion framework. For fair comparison, the invented model uses the fusion results of the four modes of joints, bones, joint motion, and bone motion as the final report results. It can be observed that the invented model outperforms most existing methods on these four datasets. On the NTU RGB+D 120X-Sub, NTU RGB+D 120X-Set, NW-UCLA, and UAV-Human datasets, the invented model achieved accuracies of 89.7%, 91.2%, 97.2%, 44.6%, and 72.0%, respectively. This is because the invented model performs comprehensive interactive extraction of spatiotemporal features and global information aggregation, effectively utilizing the spatiotemporal features of skeleton action sequences to achieve more refined human action recognition. On the NTU RGB+D dataset, the invented model achieved state-of-the-art results, with a reasonable gap of 0.1% from the best model.

[0164] Table 2. Performance comparison (%) of the proposed model with existing methods on the FineGYM and Diving48 fine-grained datasets. The best is marked in bold, the second is marked in Underline Mark out.

[0165]

[0166] In order to further verify the effectiveness of the invented model for fine motion recognition tasks, two fine-grained motion datasets FineGYM and Diving48 were selected for experiments, and the results are shown in Table 2. The invented model achieved the highest result on the FineGYM dataset, which was 8.6%, 3.3%, 3.4%, and 1.2% higher than the ST-GCN, MS-G3D, CTR-GCN, and PoseC3D models, respectively. On the Diving48 dataset, only the current best PoseC3D had a 0.2% gap. The results show that the invented model can effectively learn rich motion representations of fine motions.

[0167] Table 3. Comparison of model performance (%) using different stream data inputs and multi-stream data fusion on six datasets.

[0168]

[0169] Table 4. Comparison of the performance of different model components using only joint data stream input on the NTU RGB+D dataset X-Sub, NTU RGB+D 120 dataset X-Sub, NW-UCLA, UAV-Human dataset CSv1, FineGYM, and Diving48 datasets (%).

[0170]

[0171] ART: Action-responsive topological module ARA: Action-responsive attention module FAC: Fine action contrastor.

[0172] Afterwards, the present invention designs an ablation experiment to study the effectiveness of the invented model. The present invention first verifies the impact of multi-stream data fusion on the final result. Then the effectiveness of each component in the model is verified. Next, the present invention respectively demonstrates the multi-channel cross-time dynamic skeleton topology, spatiotemporal joint attention map, confusion matrix, t-SNE visualization feature representation for some fine movements and model comparison diagram to further illustrate the advantages of the model in fine movement recognition. Finally, the process of visualizing the model working is used to intuitively demonstrate the role of the model of the present invention.

[0173] 1) Impact of multi-stream data fusion on model results: To explore the importance of multi-stream data fusion, the present invention designs Table 3. The results show that compared with the traditional method of using only joint streams for action recognition tasks, as the number of data branches increases, the performance of the model is significantly improved on six datasets including NTU RGB+D, NTU RGB+D 120, NW-UCLA, UAV-Human, FineGYM, and Diving48. The performance of the model is best when four-stream data is fused. Specifically, compared with using a single joint stream, the model's performance on the NTU RGB+D dataset X-Sub and X-View improved by 2.1% and 1.6% respectively, on the NTU RGB+D 120 dataset X-Sub and X-Set improved by 4.1% and 3.7% respectively, on the NW-UCLA dataset improved by 2.6%, on the UAV-Human dataset CSv1 and CSv2 improved by 7.1% and 4.5% respectively, on the FineGYM dataset improved by 3.8%, and on the Diving48 dataset improved by 9.1%. This result further verifies the effectiveness of training models using multi-stream data fusion.

[0174] 2) The impact of each component in the action-responsive contrast network (ARCN): The action-responsive topology module (ART), the action-responsive attention module (ARA), and the fine action contrast (FAC) are the main components of ARCN. In order to evaluate their respective contributions to the effectiveness of the model, CTR-GCN is set as the baseline model, and Table 4 is designed. Only joint data stream input is used for verification on six datasets (NTU RGB+D and NTU RGB+D 120 only X-Sub). It can be seen that all these components help improve the performance of the baseline. The role of each component is explained in detail below:

[0175] a) Effectiveness of the Action Responsive Topology Module (ART): As can be seen from Table 4, compared with the baseline model that only uses multi-channel dynamic skeleton topology, ART that combines action duration information to establish a multi-channel cross-temporal dynamic skeleton topology improves the model effect by 0.3%, 0.3%, 0.8%, 0.3%, 0.3%, 0.5%, and 0.7% on six datasets, respectively. This shows that it is more effective to identify fine movements by combining spatiotemporal information to consider the action pattern of the entire human body to establish a skeleton topology. To verify this statement, in the example Figure 5Visualization of the multi-channel cross-temporal dynamic skeleton topology generated by the 13-layer model, where the input action random sample labels are the easily identifiable "jump" and the easily confused actions "reading" and "writing". It can be observed that: (1) the joint correlation of the generated topological structure tends to be coarse and dense, which can capture the global characteristics of the action during the duration and facilitate the recognition of fine actions; (2) the joint correlation of the topological structure is expressed differently. The joint correlation of the lower part of the human body is stronger during "jumping", and the joint correlation of the hand is stronger during "reading" and "writing", indicating that the model successfully simulates different joint relationships under different movement types; (3) For actions that are easy to distinguish, such as "jumping", the joint correlation of the topological structure is significantly different from that of "reading" and "writing". For the easily confused "reading" and "writing", the model can learn the global and subtle differences. For example, when "reading" and "writing" are performed, the action patterns are different, and the correlation between the spine and other joints is different (joint correlation in the green box). For the hand joints that are important to them, the correlation of the tip of the left hand is stronger during "reading" (joint correlation in the red box). The results show that ART is effective in distinguishing fine actions.

[0176] b) Effectiveness of the Action Responsive Attention Module (ARA): To illustrate the characteristics of ARA, the attention maps of three random samples of actions are depicted at the four layers of the model. The selected actions are “jumping”, “reading”, and “writing”, as shown in Figure 2. Figure 6 As shown. For each subgraph, in the early stage (such as the 2nd layer), the spatiotemporal joint attention feature map obtained by ARA shows stronger selectivity in spatial joint connections, while in the advanced stage (such as the 14th layer), ARA shows additional selectivity in action time. Taking "jump" as an example, in the early stage, ARA learns the joints that are spatially associated with the action in almost all frames, such as the left wrist, right wrist, and right hip when swinging the hand during "jumping". In the advanced stage, ARA mainly concentrates the importance of the most important joints related to "jumping" in a certain range of frames. It should be noted that the model uses cross-row temporal convolution in the 6th and 10th layers, and the frames outside the action duration in the later stage are filled with zeros to keep consistent with the initial input action time frame. In general, ARA pays more attention to information joints in the early stages of the model, and distinguishes information frames in the later stages of the model. It is beneficial to mine fine action recognition that is similar in space but different in time, such as Figure 6The attention maps of the two confusing actions "reading" and "writing" are shown. In the primary stage, the joint feature attention maps of "reading" and "writing" are very similar, while in the advanced stage, the joint importance related to "reading" tends to be expressed in the early time frame, while the joint importance related to "writing" tends to be expressed in the later time frame. In Table 4, using ARA can improve the model performance by approximately 0.1%-0.5% on the six datasets. Therefore, it can be concluded that ARA is effective for fine action recognition.

[0177] c) Effectiveness of Fine Action Comparator (FAC): From the results in Table 4, we can see the great effect of FAC on the model. On four conventional action recognition datasets (NTU RGB+D, NTU RGB+D 120, NW-UCLA, UAV-Human), the model with FAC has improved by 0.6%, 0.5%, 1.6%, and 0.8% respectively compared with the model without FAC. On two fine-grained action recognition datasets (FineGYM, Diving48), the effect has improved by 1% and 1.3%. In order to more intuitively demonstrate the role of FAC, we designed Figure 7 . Figure 7 The figure shows the effect of the presence or absence of FAC on the classification performance of the model on six datasets. The red range represents the classification performance of the baseline model, and the blue range represents the classification performance of the baseline model after adding FAC. The blue completely surrounds the red, indicating that for each dataset, the classification performance of the baseline model after adding FAC is better than before.

[0178] 3) Effectiveness of the final model (ARCN):

[0179] The results in Table 4 show that on the six data sets, the performance of the model of the present invention is improved to varying degrees compared with the baseline model, and the best performance is improved by 1.9% on the NW-UCLA data set.

[0180] To further analyze the performance of the model, we compare our model with the baseline model in recognizing some fine-grained actions. Figure 8 (a) and Figure 8 (b) highlights the actions where the accuracy of the baseline model using joint modality input on NTU RGB+D X-Sub falls below 80%. Figure 8From (a) and (b), we can see three groups of similar actions, marked with red rectangles, namely, "reading" and "writing", "playing with mobile phone" and "typing on keyboard", "taking off shoes" and "putting on shoes". The first four actions are mainly completed by slight shaking of the two hands. They are extremely similar in spatial configuration and temporal dynamics, and the probability of misjudgment is relatively high. The latter two actions have similar spatial structures but different temporal dynamics. From the results of (a) and (b), compared with the baseline model, the recognition accuracy of the model proposed in this paper has been improved to varying degrees in these fine actions. The recognition accuracy of actions such as "reading", "writing", "playing with mobile phone", and "typing on keyboard" has increased by 7%, 3%, 9%, and 7% respectively, and the recognition degree of "putting on shoes" and "taking off shoes" has reached more than 80%. In addition, the probability of misjudgment of fine actions has decreased to varying degrees. For example, the probability of misjudging "writing" as "typing on the keyboard" has dropped by 5%, the probability of misjudging "reading" as "writing" has dropped by 6%, and the probability of misjudging "wearing shoes" as "taking off shoes" has dropped by 8%. This shows that the model can mine the subtle differences between fine movements from multiple perspectives of time and space.

[0181] Fig. 9 The above six fine movements are compared between the method of the present invention and other methods. The results show that our method has achieved excellent performance, and the accuracy of identifying the six movements has reached the highest level. In particular, the performance of "reading", "playing with mobile phones", and "typing on keyboards" is improved by at least 4.3%, 4.0%, and 4.7% respectively compared with other methods, indicating that our method can classify fine movements more accurately than other methods.

[0182] In addition, in order to further explore the degree of distinction of fine movements by the model in this paper, the present invention uses t-SNE on the more complex NTURGB+D120 dataset X-Sub to visualize the distribution of the features finally learned by the four models for random samples of five random fine hand movements in space, such as Fig.10 As shown. These actions mainly involve subtle interactions between fingers, which are very similar in spatial structure. In order to more intuitively feel the effect of the model on these difficult-to-distinguish actions, we calculated the normalized mutual information (NMI) to quantitatively support the discrimination of the learned representations, where the closer the NMI is to 1, the better the clustering effect. From the results, the model of the present invention can make the clustering of easily confused action features more compact and more distinguishable.

[0183] 4) Model visualization: To show how the model works, Fig.11The skeleton sequence diagrams of five actions after being processed by the model are shown. It can be seen from the figure that different actions have different dynamic skeleton topologies. For fine actions that are easy to confuse, such as "reading" and "writing", the number and strength of their unnatural joint topological connections (blue lines) are different. For actions that are easy to distinguish, such as "reading" and "kicking something", they can be distinguished from the strength of their natural joint topological connections (red lines). For example, the hand joint connection strength of "reading" is higher, and the leg joint connection strength of "kicking something" is higher. In addition, the model successfully focuses on the most informative joints when different movements occur, that is, the arm joints are the most important when "reading" and "writing", the leg joints are the most important when "kicking something", and the arm and leg joints are more important when "throwing something" and "jumping". For fine actions that are easy to confuse, such as "reading" and "writing", the model has learned that the importance of their joint points is different in different frames based on their different dynamic skeleton topologies. This means that the proposed model works well and can effectively perform fine action recognition tasks.

[0184] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A skeleton fine motion recognition method based on action-responsive contrast network, It is characterized in that The following steps are involved: S1. Preprocess the collected human skeleton data to form four stream features: joint data stream, bone data stream, joint velocity data stream and bone velocity data stream; S2. Construct an action-responsive contrast network framework ARCN for skeleton fine motion recognition method, where ARCN is mainly composed of two modules: an action-responsive graph convolutional network and a fine motion comparator; ARGCN establishes a multi-channel cross-temporal dynamic skeleton joint attention topology to predict the category of action. It consists of an action-responsive topology module ART and an action-responsive attention module ARA. While ARGCN is running, the discriminant features are periodically passed into FAC, which compares and updates the features in the spatial and temporal dimensions respectively. S3. The preprocessed multi-stream data is input into the network framework ARCN for processing. Each skeleton data will eventually produce two types of loss values: one is the cross entropy loss from the action-responsive graph convolutional network, and the other is the motion contrast loss obtained by the fine motion comparator based on feature contrast. The two types of loss values ​​corresponding to the four-stream input data are jointly considered to form the basis for the final estimation of the accuracy of fine motion recognition.

2. The skeleton fine motion recognition method based on the action-responsive contrast network according to claim 1, It is characterized in that The S1 step includes the following steps: S11. In skeleton-based action recognition, a skeleton sequence is a collection of action video frames. In each frame, the skeleton is represented as a graph G = (V, E), where V = {v 1 ,v 2 ,…,v N } represents a vertex set consisting of N joints, E = {e 1 ,e 2 ,…,e M } represents the edge set consisting of bones between joints. The structure of the skeleton graph is described using the graph-specific adjacency matrix A, where A ij The value is 0 or 1, indicating that the joint v i and v j Is there an edge connection between them? The entire skeleton sequence is represented by a coordinate set X = {x∈R C×T×V } represents, where C is the coordinate dimension, C=3, T is the total number of frames in the skeleton sequence, and V represents the total number of joints in the current skeleton; S12, for the joint data stream input, the joint dimension is fixed, the defined human spine center node is used as the unified standardized zero node, and the normalized position feature P is obtained by subtracting other joint points from the standard zero node. i |(i=1,2,…,V in )}, the formula is: p i =x[:,:,i]-x[:,:,c] (1) Among them, p i is the normalized position feature of each joint index of the skeleton sequence X, x is the input feature of the skeleton sequence X, c is the index of the central node of the human spine, and the joint stream input is directly obtained by connecting X and P in series; S13, the skeleton data stream input is obtained by calculating the relative positions between adjacent joints in each frame, that is, the direction vector of each bone; S14, the joint velocity data stream input is obtained by calculating the position change of each joint in each frame; S15. The bone velocity data stream input is obtained by calculating the change in the direction vector of each bone in each frame.

3. The skeleton fine motion recognition method based on the action-responsive contrast network according to claim 1, It is characterized in that The S2 step includes the following steps: S21. Build the action-responsive graph convolutional network ARGCN of the ARCN framework. ARGCN completes the learning of multi-channel cross-temporal dynamic skeleton joint attention topology, which includes the action-responsive topology module ART and the action-responsive attention module ARA. ART learns the multi-channel cross-temporal dynamic skeleton topology structure of different channels and aggregated action temporal features, and ARA models the importance of skeleton joints in the spatiotemporal dimension. In the step S21, constructing the action-responsive graph convolutional network ARGCN of the framework ARCN includes the following steps: S211, the input skeleton data first passes through the action-responsive topological module ART to construct a topological graph of a specific motion category; In the step S211, the input skeleton data is first processed by the action-responsive topology module ART, which includes the following steps: S2111, normalizing the input data to stabilize the value; S2112, through three parallel action-responsive graph convolution blocks to extract some hidden topological connections that are most relevant to the action in space; S2113, perform multi-scale temporal convolution to learn the strength of joint connections when the duration of the action changes; S212, the features are continuously transmitted to the action responsive attention module ARA to strengthen the feature expression of the motion joints. After each ARA, the feature map is transmitted to the fine action comparator FAC again to strengthen the judgment; S213, obtaining the probability distribution of the required motion category through a pooling layer and a fully connected layer with a softmax activation function; S22. Build the fine motion comparator FAC of the ARCN framework. FAC builds a learned fine motion latent space to explore the hidden information that can distinguish between fine motions.

4. The skeleton fine motion recognition method based on the action-responsive contrast network according to claim 3, It is characterized in that In the step S2112, the input data is normalized to stabilize the value, including the following steps: S21121, input feature map x∈R C×T×V Transformed into refined feature representation through bottleneck 1x1 convolution layer; S21122, performing pooling along the time dimension to aggregate the time features, thereby obtaining the spatial features after dimensionality reduction processing; S21123. Through the designed correlation function, we can deeply explore the relationship between non-adjacent joints and mine the hidden topological correlation matrix A′∈R specific to non-connected joints. C′×V×V : where a′ ij ∈R C′ is the skeleton vertex (v i , v y ) The corresponding relationship vector in the hidden topological association matrix A′, x i 、x j ∈R C×T is the corresponding skeleton vertex v i 、v j The input features, M 1 (·) simulates the correlation between two vertices by calculating the distance between the vertex features after feature transformation and then performing bottleneck convolution. φ(·) and φ(·) transform the input features. Bottleneck convolution and time dimension pooling are used here in turn. σ(·) represents the activation function Tanh, and Conv represents the bottleneck layer convolution. S21124. Use A′ to improve the original natural connection skeleton topology connection matrix A∈R V×V To obtain the specific action topology matrix H∈R associated with the action C′×V×V : H=M 2 (A,A')=A+α·A' (3) Among them, M 2 (·) Establish a specific action topology matrix, α is a learnable parameter that adjusts the importance of hidden topological associations. Each specific action topology matrix reveals the association pattern of vertex interconnection under specific motion attributes. In order to amplify the differences in different action features and aggregate the global skeleton topology information, the original features are transformed through the bottleneck layer to represent x′ C,:,: ∈R 1×T×V The specific action topology matrix H obtained by each channel c ∈R 1×V×V (c∈{1,…,c′}) and perform channel-by-channel aggregation to obtain the skeleton topology information output G∈R C′×T×V .

5. The skeleton fine motion recognition method based on the action-responsive contrast network according to claim 3, It is characterized in that In the step S2113, performing multi-scale time convolution includes the following steps: S21131, introduced multi-scale temporal convolution to study the strength of different joint connections across time frames. The module contains four branches, each of which contains a bottleneck convolution to compress the channel. The first three branches also include two temporal convolutions with different dilations and a maximum pooling layer to capture a wider range of action-related information. The results of these four branches are pooled to obtain a multi-channel cross-temporal skeleton topology, which is represented by F∈R C′×T′×V , the stability of the network is enhanced through residual connections during training.

6. The skeleton fine motion recognition method based on the action-responsive contrast network according to claim 4, It is characterized in that In the step S212, the feature is continuously transferred to the action-responsive attention module ARA, including the following steps: S2121, input multi-channel cross-time domain dynamic skeleton topology representation feature F in ∈R C×T×V After pooling in the time and space dimensions, the specific action features are extracted into corresponding spatial joint features and time frame features; the two feature vectors are connected in series: F mid1 =Cat[Pool t (F in ),Pool s (F in )] (4) Among them, F mid1 Represents the characteristics of the intermediate transformation, Pool t (·) and Pool s (·) are the average pooling operations on the time frame and the spatial joint, respectively, and Cat represents the concatenation operation; S2122, compress information through a bottleneck layer, and mine hidden features through standardization and nonlinear transformation of activation functions: F mid2 =ξ(BN(Conv(F mid1 ))) (5) Among them, F mid2 represents the features of the intermediate transformation, Conv represents the bottleneck layer convolution, BN represents the normalization operation, and ξ(·) represents the HardSwish activation function; S2123. Separate the updated spatial joint features and temporal frame features one by one, and independently extract the spatial joint and temporal joint feature scores: F t ,F s =Split(F mid2 ,dim=2) (6) Among them, Split(·) re-divides the spatiotemporal embedding features into spatial features F s With time feature F t ; S2124, generate a score feature F representing the joint attention intensity of the entire action sequence through the outer product operation of the spatial joint feature score and the temporal feature score att ∈R C×T×V ; Where σ(·) represents the Sigmoid activation function; S2125, through the original input feature F in And the obtained score feature F att Perform outer product and standardization processing to complete feature mapping, and compare the obtained features with the original input features F in Perform residual connection again to obtain the required multi-channel cross-temporal dynamic skeleton joint attention topology F out ∈R C×T×V : in, and represents the channel outer product and element aggregation, ψ(·) represents the Swish activation function, and the shape of the output feature remains unchanged from the input feature.

7. The skeleton fine motion recognition method based on the motion-responsive contrast network according to claim 3, It is characterized in that In the step S22, constructing the fine motion comparator FAC of the framework ARCN includes the following steps: S221, decouple features in time and space, and deeply explore the distinguishable hidden features of fine-grained actions across different dimensions; In the step S221, decoupling the features in time and space includes the following steps: S2211, the input feature sequence is spatially and temporally pooled, followed by bottleneck convolution and feature flattening to capture a deep understanding of temporal and spatial variations respectively without interfering with each other: F t ′=ReLU(BN(Conv(Pool s (F in )))) (9) F s ′=ReLU(BN(Conv(Pool t (F in )))) (10) Among them, F in Represents the incoming features, F t ′ represents the temporal features extracted from the sample, F s ′ represents the spatial features extracted from the sample; S222, respectively passing the spatiotemporal decoupled features into the fine action contrast learning to complete the feature contrast learning of the action latent space in the independent spatiotemporal fine action; In the step S222, the steps of respectively transferring the spatiotemporal decoupled features into the fine motion contrast learning include the following steps: S2221, for sample i, the temporal feature F extracted after time-space decoupling t ′ and spatial feature F s ′After the same method of contrast learning, the following uses F i replace; S2222, given an action m, extract the feature F from a sample i i The predicted label results are encoded with one-hot encoding, and then compared with the true label m encoding to determine whether they are true positive TP, false negative FN, and false positive FP. TP refers to the positive example of the action correctly judged by the model. The action label one-hot encoding and the predicted one-hot encoding are element-wise multiplied to obtain the TP of each category of each sample, which is called the same cluster; FN and FP are the results of the model's incorrect judgment. FN means that the model mistakenly predicts the current action as other actions, that is, misjudgment; and FP means that the model mistakenly predicts other actions as the current action. FN and FP together constitute the overall different clusters; S2223. Use exponential moving average to maintain global representation for each cluster, i.e., prototype feature update: Among them, J m is the global representation of the action m feature, is the same cluster sample of action m, is the number of samples in the same cluster, α is the momentum term, and by using the global representation, the features of samples in the same cluster are grouped together to reduce intra-class differences; S2224. Calculate the feature averages of different clusters and optimize these misclassified samples in the subsequent contrastive learning. The formula is as follows: Among them, F fn 、F fp Represent the average feature representations of the FN sample set and the FP set, respectively, which together constitute the central representation of the heterogeneous clusters. is a heterogeneous FN sample, is a FP sample in a different cluster, and are the number of corresponding samples, respectively; S2225. In order to quantify the similarity between the sample features and the average feature vector, the cosine similarity between the sample features and various prototype features and the average feature representation of different clusters is calculated to obtain the comparative learning score and use it for subsequent loss calculation: Among them, Score m It is the comparative learning score between the prototype feature of the global representation class of action m and the current sample i, which is the basis of contrastive learning. fn 、Score fp is the comparative learning score of FN sample and FP sample with the current sample i respectively; S2226. Although FN samples and FP samples belong to different clusters, for FN samples, they should be made closer to the sample features of the same cluster in the feature space, and the calibration model will make incorrect predictions of fine motion features. For FP samples, they should be made farther away from the sample features of the same cluster in the feature space, and a compensation term φ is introduced for FN samples. i , introduce a penalty term Ψ for FP samples i : Adjust the compensation term φ i and the penalty term Ψ i It helps to correct the misclassification of FN samples in different clusters and prevent the misclassification of FP samples; S2227, using the various contrastive learning scores obtained above to define the action contrast loss Among them, p im is the action contrast loss for extracting features from the current action m, is the feature F extracted from the current action m i Action contrast loss; S2228, the complete FAC obtains the time dimension action contrast loss L tac And the spatial dimension action contrast loss L sac , add the two-dimensional action contrast loss to form the action contrast loss L AC : L AC =L tac +L sac (20) 8. The skeleton fine motion recognition method based on the motion-responsive contrast network according to claim 1, It is characterized in that The step S3 comprises the following steps: S31. In ARGCN, cross entropy loss is used to train the network: Where N is the number of samples in the batch, y ij is the one-hot encoding representation of action sample i, if and only if j is the target action class of sample i, y ij =1, p ij The probability score that the network predicts that sample i belongs to class m; S32. In order to extensively learn the latent space of fine movements, each time through ART, the feature map of the dynamic skeleton joint attention topology of the multi-channel cross-time domain is passed to FAC, forming a multi-stage action contrast loss. The multi-stage action contrast loss is defined as the weighted average sum: Among them, L MAC represents the multi-stage action contrast loss finally obtained by FAC of the entire action-responsive contrast network, i represents the number of stages, a total of 4 stages, respectively in the second layer, the seventh layer, the eleventh layer and the fourteenth layer of ARGCN, L AC is the action contrast loss obtained at each stage of FAC, λ i For learnable parameters, adjust the loss ratio at each stage; S33. Combine the cross entropy loss with the multi-stage action contrast loss to form a complete learning objective function: in, is a learnable hyperparameter used to adjust the importance of multi-stage action contrast loss in ARCN.

Images