Action recognition method and system based on human bone structure

Abstract

The application provides a motion recognition method and system based on human skeleton structure, and the method comprises the following steps: acquiring position data of all joints of human skeleton, and constructing a first data set and a second data set according to the position data of all joints of human skeleton; constructing an adaptive adjacency matrix based on human structure symmetry coding, and respectively performing normalized spatial graph convolution on the first data set and the second data set according to the adaptive adjacency matrix; respectively performing feature extraction on third spatio-temporal features and fourth spatio-temporal features to obtain first features and second features, and fusing the first features and the second features; and inputting the fused features into a two-stage recognition model for recognition to obtain human motion category information. The application can realize high-precision recognition of multi-granularity motion while ensuring real-time performance.

Description

Technical Field

[0001] This invention relates to the field of human motion recognition technology, and in particular to a motion recognition method and system based on human skeletal structure. Background Technology

[0002] Human action recognition (HAR), a core research area in computer vision and artificial intelligence, has demonstrated crucial application value in numerous real-world scenarios. With the rapid development of sensor technology and deep learning algorithms, HAR methods based on skeletal data have gradually become a research focus in this field. This is due to the excellent robustness of skeletal data to changes in lighting and background interference, and its inherent close correlation with human movements.

[0003] In the early stages of deep learning-based human action recognition research, recurrent neural networks (RNNs) and convolutional neural networks (CNNs) were the core methods. RNNs capture temporal dependencies by transforming skeletal data into frame-by-frame vector sequences, but suffer from the vanishing gradient problem and ignore spatial topological relationships between joints. Long Short-Term Memory (LSTMs) networks, while solving the long-distance dependency problem, still cannot effectively model spatial correlations. CNNs were introduced into the field due to their powerful spatial feature extraction capabilities, but they destroy the inherent structural information of the skeleton when extracting spatial features. None of these methods can effectively integrate the spatiotemporal features of skeletal data, prompting researchers to explore modeling methods suitable for non-Euclidean topologies.

[0004] Subsequently, Graph Convolutional Neural Networks (GCNs) were applied to the field of human action recognition, capable of directly processing graph-structured data and capturing spatial dependencies. The pioneering work ST-GCN was the first to model skeletal sequences as spatiotemporal graphs, using a fixed adjacency matrix to capture local spatiotemporal features. However, the fixed topology cannot capture the non-physical connections that dynamically arise during the action, limiting its ability to model complex actions. 2S-AGCN introduced adaptive graph convolutional layers to improve the dynamic adaptability of the topology, but it relied excessively on data-driven learning, easily neglecting the physical connections of the human skeleton.

[0005] As research progresses, attention mechanisms are widely used in GCNs. Some studies design spatiotemporal attention modules to highlight key nodes or propose channel attention mechanisms to solve the feature redundancy problem. However, these attention methods are mostly single-branch or multi-dimensionally separated architectures, which cannot adaptively adjust the feature capture strategy according to the complexity of the action.

[0006] Based on current research progress, the following limitations still exist in current human action recognition methods based on GCN: (1) Existing topology modeling methods do not consider deep fusion of symmetry priors, making it difficult to accurately capture the core collaborative patterns of symmetrical actions; (2) When traditional attention mechanisms are combined with GCN, they lack the ability to pay attention to heterogeneous action features and cannot adaptively adjust feature extraction strategies according to action complexity; (3) Existing networks mostly adopt a "single-stage unified modeling" architecture, which cannot capture multi-level features and makes it difficult to achieve a balance between inference efficiency and recognition accuracy. These problems severely limit the accuracy and generalization ability of human action recognition, restricting its practical application and promotion. Camera sensor methods often ignore subtle differences in actions, resulting in deviations between skeleton data and actual actions. Summary of the Invention

[0007] The purpose of this invention is to provide a motion recognition method and system based on human skeletal structure, aiming to solve at least one of the problems in the background art.

[0008] In a first aspect, the present invention provides a motion recognition method based on human skeletal structure, the method comprising:

[0009] Obtain the positional data of all joints in the human skeleton, and construct a first dataset about all joints and a second dataset about human structure based on the positional data of all joints in the human skeleton.

[0010] An adaptive adjacency matrix based on the symmetry encoding of human body structure is constructed, and normalized spatial graph convolution and temporal convolution are performed on the first dataset and the second dataset respectively according to the adaptive adjacency matrix to obtain the third spatiotemporal feature and the fourth spatiotemporal feature corresponding to the first dataset and the second dataset respectively.

[0011] Feature extraction is performed on the third and fourth spatiotemporal features respectively to obtain a first feature and a second feature corresponding to each dataset. The first feature and the second feature are then fused to obtain a first fused feature and a second fused feature corresponding to the third and fourth spatiotemporal features respectively.

[0012] The first fusion feature and the second fusion feature are respectively input into the two-stage recognition model for recognition to obtain human action category information.

[0013] In some embodiments, the step of acquiring positional data of all joints in the human skeleton and constructing a first dataset about all joints and a second dataset about human structure based on the positional data of all joints in the human skeleton includes:

[0014] The human body structure is divided into five parts: trunk, left arm, right arm, left leg, and right leg. Based on the positional data of the joints included in each part, the key node features corresponding to each part are calculated.

[0015] ;

[0016] in, Key node features N represents the total number of joints contained in the corresponding part of the human body structure. This refers to the position data of the i-th joint contained in the corresponding part of the human body structure;

[0017] The features of the five key nodes are summarized to obtain the second dataset.

[0018] In some embodiments, the step of constructing an adaptive adjacency matrix based on human structural symmetry encoding includes:

[0019] Symmetrical coding of human joints:

[0020] ;

[0021] in, Let i be a set of positive integers, where i is the reference number of the right limb joint or midline joint. The encoding for the k-th node. For the k-th node;

[0022] Skeletal encoding is performed based on joint encoding results, and the initial adjacency matrix elements are defined through algebraic operations on adjacent joint encoding values. :

[0023] ;

[0024] in, For symbolic functions, and These are the absolute values ​​of the encoded values ​​of section i and node j, respectively. Let be the symbol of the encoded value of node i. The symbol for the encoded value of the node;

[0025] By setting the initial adjacency matrix as a learnable parameter and dynamically adjusting the edge connection weights through backpropagation, an adaptive adjacency matrix based on symmetric structure encoding is obtained.

[0026] In some embodiments, the step of performing normalized spatial graph convolution and temporal convolution on the first dataset and the second dataset respectively based on the adaptive adjacency matrix to obtain the third spatiotemporal feature and the fourth spatiotemporal feature corresponding to the first dataset and the second dataset respectively includes:

[0027] Perform normalized spatial graph convolution according to the following formula:

[0028] ;

[0029] in, To output the feature matrix, For the input feature matrix, Let A be the degree matrix, I be the SAAM matrix, I be the identity matrix, and W be the weight matrix;

[0030] Perform temporal convolution according to the following formula:

[0031] ;

[0032] in, It is a third or fourth spatiotemporal feature. This is a 1D time-varying convolution operation.

[0033] In some embodiments, the step of extracting features from the third spatiotemporal feature and the fourth spatiotemporal feature respectively to obtain a first feature and a second feature corresponding to each dataset, and fusing the first feature and the second feature to obtain a first fused feature and a second fused feature corresponding to the third spatiotemporal feature and the fourth spatiotemporal feature respectively includes:

[0034] The first feature is extracted using the following formula:

[0035] ;

[0036] in, For simple attention weights, For the Sigmoid activation function, This is a 1×1 convolution operation to restore the number of channels to the original dimension. for Activation function For batch normalization operations, For 1×1 convolution operations with the number of channels reduced to C / reduction ratio, For adaptive max pooling operation, X represents simple local action features, and X represents the input features.

[0037] Extract the second feature using the following formula:

[0038] ;

[0039] in, For complex attention weights, For adaptive average pooling operation, Features of complex global actions;

[0040] The fusion is performed according to the following formula:

[0041] ;

[0042] in, For splicing features, For adaptive fusion weights of complex global action features, For simple local action features, adaptive fusion weights, for Activation function This is a 1×1 convolution operation with a kernel size of 1×1. It can be either the first fusion feature or the second fusion feature.

[0043] In some embodiments, the step of inputting the first fusion feature and the second fusion feature into a two-stage recognition model for recognition to obtain human action category information includes:

[0044] The first predicted probability distribution is generated according to the following formula:

[0045] ;

[0046] in, This is the first predicted probability distribution. Basic model, This is the second fusion feature, where K is the number of action categories;

[0047] pass The function will Convert to confidence score :

[0048] ;

[0049] For each sample, extract the maximum confidence score. Compared with the initial predicted category ;

[0050] like If the initial prediction result is not obtained, it will be retained directly as the final human action category information.

[0051] like Then, according to the following formula, the second prediction probability distribution is:

[0052] ;

[0053] in, This represents the second predicted probability distribution corresponding to node i. The first fusion feature corresponding to node i, To enhance the model;

[0054] Update the prediction category to ;

[0055] in, The corrected predicted human action category for sample i.

[0056] Secondly, the present invention provides a motion recognition system based on human skeletal structure, the system comprising:

[0057] The dataset construction module is used to obtain the position data of all joints of the human skeleton, and construct a first dataset about all joints and a second dataset about human structure based on the position data of all joints of the human skeleton.

[0058] The adjacency matrix construction module is used to construct an adaptive adjacency matrix based on the symmetry encoding of human body structure, and to perform normalized spatial graph convolution and temporal convolution on the first dataset and the second dataset respectively according to the adaptive adjacency matrix to obtain the third spatiotemporal feature and the fourth spatiotemporal feature corresponding to the first dataset and the second dataset respectively.

[0059] The feature extraction module is used to extract features from the third spatiotemporal feature and the fourth spatiotemporal feature respectively, to obtain a first feature and a second feature corresponding to each dataset, and to fuse the first feature and the second feature to obtain a first fused feature and a second fused feature corresponding to the third spatiotemporal feature and the fourth spatiotemporal feature respectively;

[0060] The feature recognition module is used to input the first fused feature and the second fused feature into the two-stage recognition model for recognition, and obtain human action category information.

[0061] Thirdly, the present invention provides a storage medium that stores one or more programs that, when executed by a processor, implement the above-described motion recognition method based on human skeletal structure.

[0062] Fourthly, the present invention provides an electronic device, the electronic device comprising a memory and a processor, wherein:

[0063] The memory is used to store computer programs;

[0064] When the processor executes the computer program stored in the memory, it implements the above-mentioned motion recognition method based on human skeletal structure.

[0065] Compared with the prior art, the present invention has the following advantages:

[0066] This invention constructs an adaptive adjacency matrix based on the symmetry encoding of human body structure, combining symmetry prior information with data-driven learning to optimize the graph topology and enhance the model's ability to model symmetric cooperative actions. By focusing on local salient features and global cooperative features through two branches, and employing an adaptive fusion mechanism to dynamically balance their contributions, it improves the feature representation ability for actions of varying complexity. Finally, a two-stage dynamic recognition architecture is introduced, cascading key node features (coarse-grained) and full-joint features (fine-grained), combined with a confidence-based dynamic sample allocation strategy, achieving high-precision recognition of multi-granularity actions while ensuring real-time performance. This method effectively solves the problems of neglecting symmetry in topology modeling, lack of specificity in feature extraction, and rigid network architecture in existing technologies, achieving excellent recognition performance on multiple datasets. Attached Figure Description

[0067] Figure 1 This is a flowchart of an action recognition method based on human skeletal structure proposed in an embodiment of the present invention;

[0068] Figure 2 A schematic diagram of the human skeleton and joints;

[0069] Figure 3 This is a schematic diagram of the motion recognition system based on human skeletal structure proposed in an embodiment of the present invention.

[0070] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art. The terms "comprising" and similar expressions used in this embodiment mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, but does not exclude other elements or objects.

[0072] like Figure 1 As shown, an embodiment of the present invention proposes an action recognition method based on human skeletal structure, the method comprising steps S101 to S104, wherein:

[0073] Step S101: Obtain the position data of all joints of the human skeleton, and construct a first dataset about all joints and a second dataset about human structure based on the position data of all joints of the human skeleton;

[0074] It should be noted that the acquisition of spatiotemporal coordinate data of human skeletal joints includes the position data of all joints in the human skeleton. This first dataset contains the position data of each joint, i.e., the full joint data.

[0075] Furthermore, the full-joint data was divided into five parts according to human structure: torso, left arm, right arm, left leg, and right leg. The average of the node features within each group was used to obtain five key node features for basic model training; simultaneously, full-joint features were retained for enhanced model training. Data normalization was performed to remove outliers and noise, ensuring data quality.

[0076] In some embodiments, key node features corresponding to each part of the human body structure are calculated based on the positional data of the joints included in each part:

[0077] ;

[0078] in, Key node features N represents the total number of joints contained in the corresponding part of the human body structure. The first dataset contains the positional data of the i-th joint in the corresponding human body structure; the second dataset is obtained by summarizing the features of the five key nodes.

[0079] For example, such as Figure 2 As shown, the human body is divided into five regions, and the joints contained in each region are as follows:

[0080] I. The head and trunk include: 1. head, 2. neck, 3. shoulders, 4. mid-spine, 5. base of the spine;

[0081] II. The right upper limb includes: 16. Right shoulder, 17. Right elbow, 18. Right wrist, 19. Right hand, 20. Right thumb, 21. Right fingertips;

[0082] III. The left upper limb includes: 6. left shoulder, 7. left elbow, 8. left wrist, 9. left hand, 10. left thumb, 11. left fingertips;

[0083] IV. The right lower limb includes: 22. right hip, 23. right knee, 24. right ankle, 25. right foot;

[0084] V. The left lower limb includes: 12. left hip, 13. left knee, 14. left ankle, 15. left foot.

[0085] Step S102: Construct an adaptive adjacency matrix based on the symmetry encoding of human body structure, and perform normalized spatial graph convolution on the first dataset and the second dataset according to the adaptive adjacency matrix to obtain the third spatiotemporal features and the fourth spatiotemporal features corresponding to the first dataset and the second dataset respectively.

[0086] It should be noted that in this step, human joints are symmetrically encoded. Bilateral symmetrical joints are numbered using a rule of opposite numbers, while midline asymmetrical joints are numbered with positive integers. Based on the joint encoding results, the elements of the initial adjacency matrix are defined through algebraic operations on the encoded values ​​of adjacent joints. This initial adjacency matrix is ​​set as a learnable parameter, and the edge connection weights are dynamically adjusted through backpropagation during model training, ultimately resulting in an adaptive adjacency matrix that combines symmetry and data adaptability.

[0087] Specifically, the human body structure itself possesses strict symmetry semantics, which are fully reflected in human movement. For example, when running, if the left hand swings forward due to inertia, the right leg will simultaneously step forward; similarly, when the right hand swings forward, the left leg will step forward. In this movement, human joints exhibit strict symmetry semantics. Therefore, when modeling the human body topology, this symmetry semantics needs to be incorporated into the adjacency matrix. The human body structure can be divided into the head, torso, left and right hands, and left and right legs. For bilaterally symmetrical joints, a numbering rule of opposite numbers is used; for midline asymmetrical joints, positive integer numbers are used. This design is inspired by mathematical coordinate axes (opposite numbers are symmetric about the x-axis or y-axis). This embodiment extends this concept to the human body structure to represent symmetry semantics, firstly by symmetrically encoding the human joints according to the following formula:

[0088] ;

[0089] in, Let i be a set of positive integers, where i is the reference number of the right limb joint or midline joint. The encoding for the k-th node. This is the kth node; this encoding method enables the node number to reflect the inherent characteristics and symmetry of each node, laying a semantic foundation for subsequent weight calculation.

[0090] Because the coding of human joints inherently contains semantic information about the physical characteristics of the human body, and bones represent the connection relationship between two adjacent joints, there is a strong dependency between bones and their adjacent joints. We can use the coding information of two adjacent joints to encode bones, more fully expressing this dependency. Furthermore, the calculation method of multiplying adjacent joint codes not only transforms this dependency into quantifiable coding features—for example, the absolute values ​​of the codes for joints of the same limb are similar, so the multiplication results are close and highly correlated, while the multiplication results of codes for joints of different limbs differ significantly—this coding difference naturally enables the distinction between local limb movements and global cross-limb coordinated movements.

[0091] Then, based on the joint encoding results, bone encoding is performed, and the initial adjacency matrix elements are defined through algebraic operations on adjacent joint encoding values. :

[0092] ;

[0093] in, For symbolic functions, and These are the absolute values ​​of the encoded values ​​of section i and node j, respectively. Let be the symbol of the encoded value of node i. The symbol for the encoded value of the node;

[0094] By setting the initial adjacency matrix as a learnable parameter and dynamically adjusting the edge connection weights through backpropagation, an adaptive adjacency matrix based on symmetric structure encoding is obtained.

[0095] In summary, considering the significant differences in human movement characteristics—joints that are not physically connected during human movement sometimes exhibit the same movement patterns. For example, when dancing or running, the hands and feet need to move in coordination, making the relationship between them particularly noteworthy. Based on these factors, the adjacency matrix should be able to learn dynamically from the data, adding connecting edges and dynamically adjusting edge weights. To enable the adjacency matrix to have dynamic learning capabilities, the encoded adjacency matrix is ​​set as a learnable parameter, allowing it to dynamically adjust its initial weights during backpropagation, ultimately resulting in an adaptive adjacency matrix based on symmetric structure encoding. In this way, the adaptive adjacency matrix based on symmetric structure encoding integrates symmetric encoding semantics and adaptively learned dynamic weights into the feature aggregation process, enabling the model to simultaneously capture the symmetry of human structure and the dynamic specificity of movement.

[0096] Furthermore, in some embodiments, normalized spatial graph convolution is performed according to the following formula:

[0097] ;

[0098] in, The output feature matrix is ​​a node feature matrix obtained after graph convolution operation, which completes feature aggregation and dimensional transformation, providing effective features for subsequent tasks (such as human behavior recognition). The input feature matrix has each row corresponding to the input feature of a node in the graph, and each column corresponds to a feature channel. For example, in a human skeleton graph, it can represent the C-dimensional features of N joint nodes, such as coordinates. The degree matrix is ​​the adjacency matrix. The corresponding diagonal matrix serves to assist in subsequent normalization operations, preventing imbalances in feature value scaling due to differences in the number of neighbors among different nodes. A is the SAAM matrix, a core matrix describing the connection relationships in the graph structure, used to locate the neighbor range of a node. I is the identity matrix, with the same shape as the adjacency matrix A, except that the elements on the diagonal are 1s and the rest are 0s. Its function is to add "self-connections" to each node, allowing it to retain its original features when aggregating features, avoiding information loss caused by relying solely on neighbor features. W is the learnable weight matrix, with the shape "input feature dimension (C) × output feature dimension (C')". It is a core learnable parameter in graph convolution, used to perform linear transformations on the aggregated features, adjusting the feature dimension and filtering feature information. Its parameters are continuously optimized through model training to adapt to specific tasks. This is the normalized matrix. It is an optimized form of the original adjacency matrix, normalized by the inverse square root of the degree matrix. This effectively alleviates the problem of unstable feature values ​​caused by differences in the number of neighbors of different nodes, making convolution operations more efficient.

[0099] Next, perform temporal convolution according to the following formula:

[0100] ;

[0101] in, It is a third or fourth spatiotemporal feature. This is a 1D time-varying convolution operation.

[0102] Step S103: Extract features from the third and fourth spatiotemporal features respectively to obtain a first feature and a second feature corresponding to each dataset, and fuse the first and second features to obtain a first fused feature and a second fused feature corresponding to the third and fourth spatiotemporal features respectively;

[0103] In this step, the action information in the human action recognition task includes both simple movement patterns of local joints and complex collaborative relationships between global joints. Graph Convolutional Networks (GCNs) often employ a uniform processing strategy for local details and global structure during feature learning, making it difficult to balance the differentiated needs of these two types of action information. To address this issue, parallel simple attention branches and complex attention branches are used to enhance key local features and global features respectively, and an adaptive fusion mechanism is introduced to dynamically balance their weights, thereby achieving refined capture of action features. Where C is the number of channels, T is the temporal length, and V is the number of joints. For discriminative features of local joint motion, the simple attention branch learns attention weights in the channel dimension by aggregating local information: First, adaptive max pooling compresses the input features to a 1×1 channel dimension, aggregating local spatial information of the feature map to generate channel-level information; then, 1×1 convolution reduces the number of channels to C / reduction (where reduction is the dimensionality reduction coefficient), which reduces parameter redundancy, alleviates overfitting, thereby reducing computational complexity and improving generalization ability; after introducing nonlinearity through batch normalization and ReLU activation function, the model can learn complex dependencies between channels; then, 1×1 convolution restores the original number of channels; finally, sigmoid activation generates simple attention weights. The simple local action features are obtained by element-wise multiplying the input features with the simple action local attention weights. The calculation process is as follows:

[0104] ;

[0105] in, For simple attention weights, For the Sigmoid activation function, This is a 1×1 convolution operation to restore the number of channels to the original dimension. for Activation function For batch normalization operations, For a 1×1 convolution operation with the number of channels reduced to C / reduction ratio, For adaptive max pooling operation, For simple local action features, This process uses input features. By enhancing the local spatial motion of key joints and suppressing interference from irrelevant regions, it achieves precise focusing on local features.

[0106] To capture complex collaborative patterns of global features, the complex attention branch learns attention weights in the channel dimension by aggregating global information. First, adaptive average pooling compresses the input features to a 1×1 spatial dimension. Unlike simple attention, the compression operation in complex attention is achieved through global average pooling. This is because global average pooling condenses the spatial information of each channel into a scalar, allowing the model to capture the global average information of the entire feature map. It is by utilizing this property that channels are better represented as a unified whole. Subsequently, feature transformation is performed using the same 1×1 convolution, batch normalization, and ReLU structure as in the simple attention branch; finally, complex attention weights are generated through 1×1 convolution and sigmoid activation. The calculation process for complex action features is as follows:

[0107] ;

[0108] in, For complex attention weights, For adaptive average pooling operation, It represents complex global action features.

[0109] To dynamically balance the contributions of local and global features, an adaptive fusion method is used to dynamically allocate weight ratios for the two types of features. The adaptive fusion concatenates simple local action features and complex global action features along the channel dimension:

[0110] ;

[0111] The concatenated features are aggregated using adaptive average pooling to gather global information, then mapped to a 2D weight vector via a 1×1 convolution, and finally activated by Softmax to generate feature fusion weights.

[0112] ;

[0113] Finally, the mixture is fused according to the following formula:

[0114] ;

[0115] in, For splicing features, For adaptive fusion weights of complex global action features, For simple local action features, adaptive fusion weights, for Activation function This is a 1×1 convolution operation with a kernel size of 1×1. It can be either the first fusion feature or the second fusion feature.

[0116] Step S104: Input the first fusion feature and the second fusion feature into the two-stage recognition model for recognition to obtain human action category information.

[0117] It should be noted that in skeleton-based action recognition tasks, different actions exhibit features of varying granularity, with differing requirements for the number of joints and feature representation. Traditional single-model architectures struggle to handle these multi-granularity features simultaneously. To address this issue, this paper proposes a Dynamic Concatenation (DC) method. This method constructs a cascaded structure of a base model and an enhancement model, and designs an adaptive selection mechanism based on prediction confidence to achieve differentiated processing of actions of varying difficulty. The Dynamic Concatenation method utilizes the base model... b The system quickly generates initial predictions and assesses their confidence level. Based on this confidence level, samples are then categorized into simple coarse-grained action samples and complex fine-grained action samples. Only the difficult-to-identify fine-grained action samples trigger the augmentation model. A second, refined analysis is performed, forming a cascaded strategy of "first general classification, then precise optimization." This approach allows the base model to handle coarse-grained samples, while the enhancement model improves the recognition accuracy of fine-grained samples. The dynamic cascaded method includes a base model... Enhanced model With dynamic threshold controller Three components.

[0118] The process of the dynamic cascading method is as follows:

[0119] The first predicted probability distribution is generated according to the following formula:

[0120] ;

[0121] in, This is the first predicted probability distribution. Basic model, This is the second fusion feature, where K is the number of action categories;

[0122] The Softmax function will Convert to confidence score :

[0123] ;

[0124] For each sample, extract the maximum confidence score. Compared with the initial predicted category ;

[0125] like If the initial prediction result is not obtained, it will be retained directly as the final human action category information.

[0126] like Then, according to the following formula, the second prediction probability distribution is:

[0127] ;

[0128] in, This represents the second predicted probability distribution corresponding to node i. The first fusion feature corresponding to node i, To enhance the model;

[0129] Update the prediction category to ;

[0130] in, Predict the human action category after correcting sample i.

[0131] For traditional static thresholds To address the issue of inability to adapt to changes in data distribution, this embodiment proposes a dynamic threshold mechanism: given a batch The base model is for each sample Generate confidence Dynamic threshold The calculation process is as follows:

[0132] ;

[0133] in, Indicates confidence level the median of This is the scaling factor. This is a minimum threshold constraint. This mechanism allows the threshold to be automatically adjusted based on the overall difficulty of the current batch of samples: when the overall sample size is relatively simple, As the difficulty increases, more samples are processed by the base model; when the overall difficulty of the samples is high... This reduces the risk of damage and ensures that complex samples can be accurately identified through enhanced models.

[0134] In summary, based on the aforementioned action recognition method based on human skeletal structure, this method constructs an adaptive adjacency matrix based on the symmetry encoding of human structure, combining symmetry prior information with data-driven learning to optimize the graph topology and enhance the model's ability to model symmetric collaborative actions. By focusing on local salient features and global collaborative features through two branches, and employing an adaptive fusion mechanism to dynamically balance their contributions, the method improves the feature representation ability for actions of varying complexity. Finally, a two-stage dynamic recognition architecture is introduced, cascading key node features (coarse-grained) and full-joint features (fine-grained), combined with a confidence-based dynamic sample allocation strategy, achieving high-precision recognition of multi-granularity actions while ensuring real-time performance. This method effectively solves the problems of neglecting symmetry in topology modeling, lack of specificity in feature extraction, and rigid network architecture in existing technologies, achieving excellent recognition performance on multiple datasets.

[0135] like Figure 3 As shown, one embodiment of the present invention proposes a motion recognition system based on human skeletal structure, the system comprising:

[0136] The dataset construction module 100 is used to obtain the position data of all joints of the human skeleton, and construct a first dataset about all joints and a second dataset about human structure based on the position data of all joints of the human skeleton.

[0137] The adjacency matrix construction module 200 is used to construct an adaptive adjacency matrix based on the symmetry encoding of human body structure, and to perform normalized spatial graph convolution and temporal convolution on the first dataset and the second dataset respectively according to the adaptive adjacency matrix to obtain the third spatiotemporal feature and the fourth spatiotemporal feature corresponding to the first dataset and the second dataset respectively.

[0138] The feature extraction module 300 is used to extract features from the third spatiotemporal feature and the fourth spatiotemporal feature respectively, to obtain a first feature and a second feature corresponding to each dataset, and to fuse the first feature and the second feature to obtain a first fused feature and a second fused feature corresponding to the third spatiotemporal feature and the fourth spatiotemporal feature respectively;

[0139] The feature recognition module 400 is used to input the first fused feature and the second fused feature into the two-stage recognition model for recognition to obtain human action category information.

[0140] In another aspect, the present invention also proposes a storage medium on which one or more programs are stored, which, when executed by a processor, implement the above-described motion recognition method based on human skeletal structure.

[0141] In another aspect, the present invention also proposes an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to realize the above-mentioned motion recognition method based on human skeletal structure.

[0142] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain stored, communicated, propagated, or transmitted programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0143] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0144] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0145] While embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations fall within the scope and spirit of the invention as set forth in the claims. Furthermore, the invention described herein may have other embodiments and can be implemented or carried out in various ways.

Claims

1. A motion recognition method based on human skeletal structure, characterized in that, The method includes: Obtain the positional data of all joints in the human skeleton, and construct a first dataset about all joints and a second dataset about human structure based on the positional data of all joints in the human skeleton. An adaptive adjacency matrix based on the symmetry encoding of human body structure is constructed, and normalized spatial graph convolution and temporal convolution are performed on the first dataset and the second dataset respectively according to the adaptive adjacency matrix to obtain the third spatiotemporal feature and the fourth spatiotemporal feature corresponding to the first dataset and the second dataset respectively. Symmetrical coding of human joints: ； in, Let i be a set of positive integers, where i is the reference number of the right limb joint or midline joint. The encoding for the k-th node. For the k-th node; Skeletal encoding is performed based on joint encoding results, and the initial adjacency matrix elements are defined through algebraic operations on adjacent joint encoding values. : ； in, For symbolic functions, and These are the absolute values ​​of the encoded values ​​of node i and node j, respectively. Let be the symbol of the encoded value of node i. The symbol for the encoded value of node j; The initial adjacency matrix is ​​set as a learnable parameter, and the edge connection weights are dynamically adjusted through backpropagation to obtain an adaptive adjacency matrix based on symmetric structure encoding. Feature extraction is performed on the third and fourth spatiotemporal features respectively to obtain a first feature and a second feature corresponding to each dataset. The first feature and the second feature are then fused to obtain a first fused feature and a second fused feature corresponding to the third and fourth spatiotemporal features respectively. The first fusion feature and the second fusion feature are respectively input into the two-stage recognition model for recognition to obtain human action category information; The first predicted probability distribution is generated according to the following formula: ； in, This is the first predicted probability distribution. Basic model, This is the second fusion feature, where K is the number of action categories; pass The function will Convert to confidence score : ； For each sample, extract the maximum confidence score. Compared with the initial predicted category ; like If the initial prediction result is not obtained, it will be retained directly as the final human action category information. like Then, according to the following formula, the second prediction probability distribution is: ； in, This represents the second predicted probability distribution corresponding to node i. The first fusion feature corresponding to node i, To enhance the model; Update the prediction category to ; in, The corrected predicted human action category for sample i.

2. The action recognition method based on human skeletal structure according to claim 1, characterized in that, The steps of obtaining the positional data of all joints in the human skeleton and constructing a first dataset about all joints and a second dataset about human structure based on the positional data of all joints in the human skeleton include: The human body structure is divided into five parts: trunk, left arm, right arm, left leg, and right leg. Based on the positional data of the joints included in each part, the key node features corresponding to each part are calculated. ； in, Key node features N represents the total number of joints contained in the corresponding part of the human body structure. This refers to the position data of the i-th joint contained in the corresponding part of the human body structure; The features of the five key nodes are summarized to obtain the second dataset.

3. The action recognition method based on human skeletal structure according to claim 2, characterized in that, The step of performing normalized spatial graph convolution and temporal convolution on the first dataset and the second dataset respectively based on the adaptive adjacency matrix to obtain the third spatiotemporal feature and the fourth spatiotemporal feature corresponding to the first dataset and the second dataset respectively includes: Perform normalized spatial graph convolution according to the following formula: ； in, To output the feature matrix, For the input feature matrix, Let A be the degree matrix, I be the SAAM matrix, I be the identity matrix, and W be the weight matrix; Perform temporal convolution according to the following formula: ； in, It is a third or fourth spatiotemporal feature. This is a 1D time convolution operation.

4. The action recognition method based on human skeletal structure according to claim 3, characterized in that, The steps of extracting features from the third and fourth spatiotemporal features respectively to obtain a first feature and a second feature corresponding to each dataset, and fusing the first and second features to obtain a first fused feature and a second fused feature corresponding to the third and fourth spatiotemporal features respectively, include: The first feature is extracted using the following formula: ； in, For simple attention weights, The Sigmoid activation function is used. This is a 1×1 convolution operation to restore the number of channels to the original dimension. for Activation function For batch normalization operations, For 1×1 convolution operations with the number of channels reduced to C / reduction ratio, For adaptive max pooling operation, X represents simple local action features, and X represents the input features. The second feature is extracted using the following formula: ； in, For complex attention weights, For adaptive average pooling operation, Features of complex global actions; The fusion is performed according to the following formula: ； in, For splicing features, For adaptive fusion weights of complex global action features, For simple local action features, adaptive fusion weights, for Activation function This is a 1×1 convolution operation with a kernel size of 1×1. It can be either the first fusion feature or the second fusion feature.

5. A motion recognition system based on human skeletal structure, used to implement the motion recognition method based on human skeletal structure as described in any one of claims 1-4, characterized in that, The system includes: The dataset construction module is used to obtain the position data of all joints of the human skeleton, and construct a first dataset about all joints and a second dataset about human structure based on the position data of all joints of the human skeleton. The adjacency matrix construction module is used to construct an adaptive adjacency matrix based on the symmetry encoding of human body structure, and to perform normalized spatial graph convolution and temporal convolution on the first dataset and the second dataset respectively according to the adaptive adjacency matrix to obtain the third spatiotemporal feature and the fourth spatiotemporal feature corresponding to the first dataset and the second dataset respectively. The feature extraction module is used to extract features from the third spatiotemporal feature and the fourth spatiotemporal feature respectively, to obtain a first feature and a second feature corresponding to each dataset, and to fuse the first feature and the second feature to obtain a first fused feature and a second fused feature corresponding to the third spatiotemporal feature and the fourth spatiotemporal feature respectively; The feature recognition module is used to input the first fused feature and the second fused feature into the two-stage recognition model for recognition, so as to obtain human action category information.

6. A storage medium, characterized in that, The storage medium stores one or more programs that, when executed by a processor, implement the motion recognition method based on human skeletal structure as described in any one of claims 1-4.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein: The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the action recognition method based on human skeletal structure as described in any one of claims 1-4.

Images