Human motion recognition method and system based on cascaded graph convolution network

Abstract

The application provides a human motion recognition method and system based on a cascaded graph convolution network, which comprises the following steps: firstly, obtaining human skeleton sequence data, calculating acceleration amplitude based on joint coordinate information, constructing a dynamic adjacency matrix based on short-term mutual information and long-term mutual information between joints, and weighting and fusing the dynamic adjacency matrix with a physically connected adjacency matrix to obtain a motion-aware layered adjacency matrix; then extracting multi-type high-order features based on the joint coordinate information and the motion-aware layered adjacency matrix; finally, constructing a three-level cascaded graph convolution network from coarse to fine, taking the motion-aware layered adjacency matrix as the adjacency matrix of a graph convolution layer, taking the multi-type high-order features as input features, controlling the cascaded reasoning process through a confidence threshold, and realizing adaptive recognition of action complexity. The application can fully depict the dynamic correlation between joints, improve the fine-grained action discrimination ability, and significantly reduce the model calculation cost.

Description

Technical Field

[0001] This invention relates to the field of action recognition technology, and in particular to a method and system for human action recognition based on cascaded graph convolutional networks. Background Technology

[0002] Action recognition is a crucial research area in computer vision, with wide applications in intelligent security, human-computer interaction, motion analysis, and risk warning. Based on the different types of input data, existing methods are mainly divided into three categories: optical flow-based video, RGB image-based video, and skeleton-based video. Among these, optical flow and RGB-based methods are susceptible to environmental factors such as lighting changes and cluttered backgrounds, exhibiting poor robustness. In contrast, skeleton-based methods use the three-dimensional coordinates of human joints as input, offering advantages such as low background dependence and strong resistance to external interference. Furthermore, with the increasing maturity of depth sensors and human pose estimation technologies, the cost of acquiring skeleton data is continuously decreasing, making this method the mainstream approach in action recognition. In skeleton-based action recognition, graph convolutional networks, due to their ability to naturally model the non-Euclidean graph structure of the human skeleton, have significant advantages in capturing spatial dependencies between joints and have become the current mainstream technical framework.

[0003] However, existing skeleton action recognition methods based on graph convolutional networks still have the following shortcomings: First, the expressive power of the adjacency matrix is ​​limited. Most methods only construct a fixed adjacency matrix based on the physical connection of human joints, or learn uniform weights in a globally shared manner. This fails to fully characterize the dynamic changes in the relationship between joints under different actions, and also fails to explore the strong correlation between non-physically connected joints such as hands and feet in specific actions, resulting in insufficient recognition ability for long-distance collaborative actions. Second, the feature representation is singular. Most methods only rely on joint coordinate information and fail to make full use of higher-order features such as velocity, acceleration, and energy changes, making it difficult to effectively distinguish fine-grained actions with highly similar trajectories, such as waving and jumping. Third, the model has high computational overhead. Existing methods use a uniform complete network for inference on all samples without dynamically allocating computational resources according to the complexity of the action, resulting in redundant computation for a large number of simple actions and low inference efficiency. To address the aforementioned issues, this invention proposes a human motion recognition method based on cascaded graph convolutional networks. By constructing a motion-aware hierarchical adjacency matrix, it fully models the dynamic relationships between joints, enriches feature representation by fusing multiple types of high-order features, and achieves adaptive inference of motion complexity through a multi-scale cascaded network, significantly reducing computational overhead while ensuring recognition accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide a human action recognition method and system based on cascaded graph convolutional networks, aiming to solve the problems of existing skeleton action recognition methods based on graph convolutional networks, such as the inability of the adjacency matrix to fully characterize the dynamic relationship between joints, the single feature representation making it difficult to distinguish fine-grained actions, and the high computational cost of the model leading to low inference efficiency.

[0005] In a first aspect, the present invention provides a human action recognition method based on a cascaded graph convolutional network, the method comprising:

[0006] Obtain human skeleton sequence data, wherein the skeleton sequence data includes the three-dimensional coordinate information of several human joints in each frame;

[0007] The acceleration amplitude of each joint is calculated based on the three-dimensional coordinate information. Short-term mutual information and long-term mutual information between joint pairs are calculated using both sliding window and complete sequence methods based on the acceleration amplitude. Short-term dynamic adjacency matrices and long-term dynamic adjacency matrices are generated based on the short-term and long-term mutual information, respectively. The short-term and long-term dynamic adjacency matrices are then fused to obtain a dynamic adjacency matrix. The dynamic adjacency matrix is ​​then weighted and fused with the human body physical connection adjacency matrix to obtain a motion perception hierarchical adjacency matrix.

[0008] Based on the motion-aware hierarchical adjacency matrix, spatial tetrahedral volume features, skeletal potential-kinetic coupling features, and skeletal map Laplacian spectrum features are extracted. The motion-aware hierarchical adjacency matrix is ​​used as the adjacency matrix of the graph convolutional layer, and various features are used as the input features of the graph convolutional layer to construct a multi-scale cascaded graph convolutional network composed of three-level sub-networks.

[0009] The skeleton sequence data to be identified is input into the multi-scale cascaded graph convolutional network to obtain the final identification result.

[0010] Secondly, the present invention provides a human action recognition system based on a cascaded graph convolutional network, the system comprising:

[0011] The skeleton data acquisition module is used to acquire human skeleton sequence data, which includes the three-dimensional coordinate information of several human joints in each frame.

[0012] The adjacency matrix construction module is used to calculate the acceleration amplitude of each joint based on the three-dimensional coordinate information, calculate the short-term mutual information and long-term mutual information between joint pairs using both the sliding window method and the complete sequence method based on the acceleration amplitude, generate short-term dynamic adjacency matrices and long-term dynamic adjacency matrices based on the short-term mutual information and long-term mutual information respectively, and fuse the short-term dynamic adjacency matrix and long-term dynamic adjacency matrix to obtain a dynamic adjacency matrix; and perform weighted fusion of the dynamic adjacency matrix with the human body physical connection adjacency matrix to obtain a motion perception hierarchical adjacency matrix;

[0013] The cascaded network construction module is used to extract spatial tetrahedral volume features, skeletal potential-kinetic coupling features, and skeletal map Laplacian spectrum features based on the motion-aware hierarchical adjacency matrix. The motion-aware hierarchical adjacency matrix is ​​used as the adjacency matrix of the graph convolutional layer, and various features are used as the input features of the graph convolutional layer to construct a multi-scale cascaded graph convolutional network composed of three-level sub-networks.

[0014] The action recognition module is used to input the skeleton sequence data to be recognized into the multi-scale cascaded graph convolutional network to obtain the final recognition result.

[0015] Thirdly, the present invention provides a storage medium that stores one or more programs, which, when executed by a processor, implement the above-described human action recognition method based on cascaded graph convolutional networks.

[0016] Fourthly, the present invention provides an electronic device, the electronic device comprising a memory and a processor, wherein:

[0017] The memory is used to store computer programs;

[0018] When the processor executes the computer program stored in the memory, it implements the above-described human action recognition method based on cascaded graph convolutional networks.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] This invention overcomes the limitations of traditional fixed physical adjacency matrices by constructing a motion-aware hierarchical adjacency matrix. It can adaptively characterize the dynamic relationships between joints, including non-physically connected joints, under different actions, effectively improving the recognition ability of long-distance collaborative actions. By integrating multiple types of high-order features, such as spatial tetrahedral volume features, skeletal potential-kinetic coupling features, and skeletal map Laplacian spectrum features, the model's input information is enriched, significantly enhancing the ability to discriminate fine-grained actions with highly similar trajectories, such as waving and jumping. At the same time, by designing a multi-scale cascaded graph convolutional network from coarse to fine, simple actions can be recognized at the first level with only a few key joints, while complex actions are progressively entered into more refined joint subnets. This achieves adaptive allocation of computational resources according to the complexity of the action, significantly reducing the model's inference overhead while ensuring overall recognition accuracy. Attached Figure Description

[0021] Figure 1 This is a flowchart of a human action recognition method based on a cascaded graph convolutional network proposed in an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of the structure of a human motion recognition system based on a cascaded graph convolutional network proposed in an embodiment of the present invention.

[0023] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art. The terms "comprising" and similar expressions used herein mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, but does not exclude other elements or objects.

[0025] like Figure 1 As shown, an embodiment of the present invention proposes a human action recognition method based on a cascaded graph convolutional network. The method includes steps S101 to S104, wherein:

[0026] Step S101: Obtain human skeleton sequence data, wherein the skeleton sequence data includes the three-dimensional coordinate information of several human joints in each frame;

[0027] Step S102: Calculate the acceleration amplitude of each joint based on the three-dimensional coordinate information. Calculate the short-term and long-term mutual information between joint pairs using both the sliding window method and the complete sequence method based on the acceleration amplitude. Generate a short-term dynamic adjacency matrix and a long-term dynamic adjacency matrix based on the short-term and long-term mutual information, respectively. Merge the short-term and long-term dynamic adjacency matrices to obtain a dynamic adjacency matrix. Weightedly fuse the dynamic adjacency matrix with the human body physical connection adjacency matrix to obtain a motion-sensing hierarchical adjacency matrix.

[0028] It should be noted that traditional adjacency matrices are typically constructed based solely on the physical connections of human joints, failing to adequately capture relationships between joints that are not physically connected. To address this limitation, this paper proposes the Motion-Aware Hierarchical Adjacency Matrix (MAHAM), which can be self-learned using sample data and achieves superior performance compared to traditional adjacency matrices when the sample size is sufficient.

[0029] This human body physical adjacency matrix is ​​the natural physical adjacency matrix of the human body. It is a mathematical expression of the anatomical topological relationships between human skeletal joints and belongs to the category of symmetric sparse matrices. This matrix only describes whether there is a direct physical connection between bone segments between joint nodes. If a direct anatomical connection exists, it is marked as connected; if there is no direct connection and the association is only transmitted through intermediate joints, it is marked as disconnected.

[0030] This embodiment constructs a dynamic association layer by calculating the mutual information between joints to obtain a non-physical adjacency matrix, specifically including the following process:

[0031] First, the acceleration is calculated using the second-order central difference method, and then its norm is determined; the calculation process is shown in the following formula.

[0032] ;

[0033] in, Let be the coordinates of joint i at times t+1, t, and t-1, respectively. It is the acceleration of joint i at time t. They represent Acceleration components along the x, y, and z axes.

[0034] Then, short-term mutual information is obtained according to the following formula:

[0035] ;

[0036] in, For the short-term mutual information between joint i and joint j, S represents the number of samples, and W represents the number of windows per sample; , They are samples s at time t. The acceleration amplitudes of joints i and j, It is the starting time point of the window. The average acceleration amplitude of joint i over five time windows; Let represent the correlation between the acceleration amplitudes of joint i and joint j in the s-th sample and the w-th time window.

[0037] Long-term mutual information can be obtained using the following formula:

[0038] ;

[0039] in, For long-term mutual information between joint i and joint j. They are samples s at time t. The acceleration amplitudes of joints i and j, To represent the total duration of the entire time series, The average acceleration over the entire time series, This refers to the overall synergy or correlation between the changes in acceleration amplitude of joints i and j during the entire motion of sample s.

[0040] Then, construct the short-term dynamic adjacency matrix and the long-term dynamic adjacency matrix according to the following formulas:

[0041] ;

[0042] ;

[0043] in, , Let be the short-term dynamic adjacency matrix and the long-term dynamic adjacency matrix of joint i and joint j, respectively. These are short-term and long-term adaptive thresholds, respectively.

[0044] The short-term adaptive threshold is calculated using the following formula:

[0045]

[0046] The long-term adaptive threshold is calculated using the following formula:

[0047]

[0048] in, These are the mean values ​​of short-term mutual information and long-term mutual information, respectively. , These are the standard deviations of short-term mutual information and long-term mutual information, respectively. For mathematical expectation, For variance, This is the adjustment coefficient for the short-term adaptive threshold. This is the adjustment coefficient for the long-term adaptive threshold. For short-term mutual information, For long-term mutual information.

[0049] Construct a dynamic adjacency matrix using the following formula:

[0050]

[0051] in, It is a dynamic adjacency matrix.

[0052] Finally, then Natural physical adjacency matrix of the human body The weighted fusion yields the final motion-aware hierarchical adjacency matrix.

[0053] Step S103: Extract spatial tetrahedral volume features, skeletal potential-kinetic coupling features, and skeletal map Laplacian spectrum features based on the motion-aware hierarchical adjacency matrix, and use the motion-aware hierarchical adjacency matrix as the adjacency matrix of the graph convolutional layer, and use various features as the input features of the graph convolutional layer to construct a multi-scale cascaded graph convolutional network composed of three-level sub-networks.

[0054] It should be noted that existing skeleton-based action recognition methods mainly rely on joint coordinate features, resulting in limited feature diversity and a high error rate for fine-grained actions with subtle inter-class differences. To efficiently recognize actions, this step designs three types of high-order features: spatial tetrahedral volume features, skeletal potential-kinetic coupling features, and skeletal atlas Laplacian spectrum features. These features include the following 11 sub-features: instantaneous tetrahedral volume, temporal rate of volume change and volume fluctuation entropy, instantaneous gravitational potential energy, instantaneous kinetic energy, potential-kinetic energy conversion phase difference, potential-kinetic energy ratio, topological spectrum features, full-frame spectral entropy, and temporal mean of spectral features.

[0055] Specifically, in some embodiments, four non-coplanar joints (shoulder, elbow, hip, and knee, denoted as...) are selected. The system constructs a spatial tetrahedron. By calculating the instantaneous volume and temporal rate of change of the volume of the tetrahedron frame by frame, the dynamic changes of the three-dimensional opening and closing state of the limbs are quantified. On this basis, the volume fluctuation entropy is introduced to quantify the degree of disorder of the temporal changes of the volume, realizing the progressive process from "single frame state → inter-frame change → change pattern", effectively capturing the subtle fluctuation features of the movement, while suppressing redundant noise.

[0056] (1) Instantaneous tetrahedral volume

[0057] ;

[0058] in: (Joint vector) It is a cross product. For dot product, For absolute values, Let be the instantaneous volume of the skeleton image in frame t. It can characterize the size of the three-dimensional space formed by the four non-coplanar joints at frame t, and directly reflect the three-dimensional opening and closing posture of the limb.

[0059] (2) Volume change rate over time

[0060] ;

[0061] The temporal rate of volume change reflects the magnitude of the change in tetrahedral volume between two adjacent frames, i.e., the inter-frame rate of change in limb opening and closing posture: It indicates that the limbs are in an open position. This indicates that the limb is in a contracted state. This indicates that the limb posture remains unchanged. The rate of change of volume over time can initially capture the dynamic trend of the movement, but it cannot quantify the degree of disorder in this change. Therefore, volume fluctuation entropy is further introduced.

[0062] (3) Volume fluctuation entropy

[0063] ;

[0064] Where: K is the volume sequence The number of non-overlapping intervals (that is, discretizing the continuous volume change rate value into K non-overlapping intervals). The probability (normalized frequency) of the k-th interval is the number of volume change rate values ​​contained in that interval. The ratio to the total number of frames T.

[0065] Volume fluctuation entropy is used to quantify the disorder of the rate of change of volume over time: when the opening and closing rhythm of limbs is stable during movement (such as waving hands at a constant speed), the dispersion of the rate of change of volume is small, corresponding to... The value is relatively small; when the opening and closing rhythm of the limbs is variable (such as random swinging), the dispersion of the rate of volume change is large, corresponding to... The value is relatively large. This indicator can be used to further distinguish fine-grained movements with similar instantaneous volumes but different dynamic change patterns, while filtering out meaningless volume fluctuations caused by joint noise, thus suppressing redundant information.

[0066] Furthermore, in some embodiments, the potential-kinetic energy coupling characteristics of the skeleton can reveal the intrinsic motion laws of movement from the perspective of energy conversion, making up for the deficiency that a single kinematic feature cannot characterize the energy changes of movement. We calculate instantaneous potential energy and kinetic energy based on the three-dimensional coordinates of the joints, and further extract the coupling characteristics of the two to achieve the extension of the feature dimension from "motion state" to "energy change".

[0067] (1) Instantaneous gravitational potential energy

[0068] ;

[0069] Where N is the total number of joints in the human body, and m is the mass of a joint. Let i be the mass weight of joint i. It is the acceleration due to gravity. Let be the y-axis coordinate of joint i in the skeleton image of frame t.

[0070] Instantaneous gravitational potential energy can characterize the vertical positional energy of a human limb in frame t, when the limb is raised during movement. Enlargement, limb reduction Reduce. For example, in the action of "jumping," at the moment of takeoff... Gradually increase in size, upon landing Rapidly decrease.

[0071] (2) Instantaneous kinetic energy

[0072] ;

[0073] in, for Instantaneous kinetic energy of the skeleton image in frame t , Let be the velocity vector of joint i at time t.

[0074] Instantaneous kinetic energy can characterize the total kinetic energy of human joint motion in frame t. The faster the joint motion, the higher the kinetic energy. The larger, The higher the value, the better. For example, in a "waving" motion, when the arm joint speed is at its maximum, It has reached its peak.

[0075] (3) Phase difference in potential energy-kinetic energy conversion

[0076] The phase difference characteristic of potential energy to kinetic energy conversion can effectively distinguish different types of actions by the frequency of phase difference fluctuations.

[0077] ;

[0078] in, Let be the potential-kinetic energy conversion phase difference of the skeleton image in frame t. and They are respectively and The instantaneous phase obtained by Hilbert transform , These are the imaginary and real parts of the Hilbert transform result, respectively. Let be the total instantaneous potential energy of the skeleton image in frame t. For Hilbert transform, It is a bivariate arctangent function.

[0079] The range of values ​​is ,when When potential energy and kinetic energy change synchronously, such as when walking at a constant speed, the raising of limbs (increase in potential energy) and the increase in speed (increase in kinetic energy) are synchronized; when When the potential energy and kinetic energy change in opposite directions, such as when jumping and landing, the potential energy decreases while the kinetic energy increases (speed increases). For "running" and "walking", the former has a high frequency of phase difference fluctuations, while the latter has gentler fluctuations.

[0080] (4) Potential energy to kinetic energy ratio

[0081] ;

[0082] in, For a small value (taken as 10⁻) 6 This prevents the denominator from being zero. The potential energy-kinetic energy ratio is used to characterize the energy dominance type of an action. At that time, potential energy dominates; At that time, kinetic energy dominates, and different actions are determined based on the dominant type of energy.

[0083] Furthermore, in some embodiments, the motion-aware hierarchical adjacency matrix map obtained above is decomposed using the Laplacian matrix to extract the inherent features of the skeletal topology, characterizing the essential changes in the skeletal structure during movement, and providing a topological basis for action recognition. The calculation process is as follows:

[0084] Based on the motion-aware hierarchical adjacency matrix obtained above Constructing the Laplacian matrix of the skeletal map This transforms skeletal topological information into computable matrix features, specifically the Laplace matrix. It can preserve the global structural information of the skeletal topology. Its diagonal elements reflect the core nature of the joints, while the off-diagonal elements reflect the linkage inhibition relationships between joints, laying the foundation for subsequent spectral feature extraction. The calculation formula is as follows:

[0085] ;

[0086] in, It is an N×N dimensional Laplace matrix. Given an N×N diagonal matrix, only the diagonal elements are non-zero, and the rest are 0. The adjacency matrix is ​​a hierarchical adjacency matrix for motion sensing.

[0087] The diagonal elements are defined as:

[0088] ;

[0089] in, Degree matrix The element in the i-th row and i-th column, Let the connection weight between node i and node k be . This represents the sum of the linkage strengths of the i-th joint with all other joints in frame t. The larger its value, the more central the i-th joint is in the skeletal topology.

[0090] Laplacian matrix for each frame Eigenvalue decomposition is performed to extract eigenvalues ​​as the core spectral features of the skeletal topology. The specific steps are as follows:

[0091] ① Eigenvalue decomposition: for Eigenvalue decomposition yields N eigenvalues, which are then sorted in ascending order. . Laplace matrix The i-th eigenvalue represents the inherent characteristics of the skeletal topology graph. Different eigenvalues ​​correspond to different topological patterns. The larger the eigenvalue, the stronger the stability of the corresponding topological structure.

[0092] ② Constructing the spectral coefficient vector: Select all eigenvalues ​​to construct a single-frame Laplacian spectral coefficient vector:

[0093] ;

[0094] ③ Full-frame spectral coefficient matrix: Concatenate the spectral coefficient vectors of all frames (t=1 to T) in frame order to obtain the full-frame Laplace spectral coefficient matrix, and integrate the topological spectral features of all frames:

[0095] ;

[0096] It is a T×N matrix (T is the total number of frames, N is the total number of joints), which fully preserves the spectral feature changes of the skeletal topology throughout the entire action process, without any frame-level or feature-level redundancy removal, and can be directly used as input for action recognition models.

[0097] To characterize the temporal disorder of topological patterns, we introduce the full-frame spectral entropy feature, calculated as follows:

[0098] ;

[0099] The sum of all feature values ​​in a single frame is used for normalization, so that the sum of the proportions of each feature value is 1. The larger the spectral entropy value, the more drastic the change in the skeleton topology pattern of the whole frame, which can effectively distinguish the complexity of the action.

[0100] To characterize the overall trend of topological features, we introduce the temporal mean of spectral features, calculated as follows:

[0101]

[0102] The temporal mean of the i-th feature value across the entire frame reflects the average state of the topological pattern corresponding to the feature value throughout the entire action process, and can be used to distinguish the differences in topological features between different actions.

[0103] The Laplacian spectral feature of the skeletal atlas transforms the connection relationship of skeletal joints into spectral domain features, fully preserving the topological information and linkage rules of all joints in the entire frame. It does not require filtering any redundant content and can accurately capture the inherent changes in skeletal topology during the movement.

[0104] Furthermore, it should be noted that most existing GCN-based methods utilize all joint information for action recognition, leading to increased computational complexity. This paper proposes a cascaded network with a multi-scale graph structure, constructing progressive cascaded networks at scales of 5, 10, and 15 joints respectively. Most samples can be recognized in the first step of the cascaded network. Specifically, based on the natural connections of the human skeleton, coarse-grained graphs containing 5 key joints, medium-grained graphs containing 10 key joints, and fine-grained graphs containing 15 key joints are constructed as the graph topology structures of the three-level sub-networks.

[0105] Each sub-network level contains a batch normalization layer, seven spatiotemporal graph convolutional modules, a global average pooling layer, and a fully connected classification layer connected in sequence. The time stride of the fourth and seventh spatiotemporal graph convolutional modules is set to 2.

[0106] Each sub-network's graph convolutional layer uses the motion-aware hierarchical adjacency matrix as its adjacency matrix and the spatial tetrahedral volume feature, skeletal potential-kinetic coupling feature, and skeletal graph Laplacian spectrum feature as its input features to extract spatiotemporal features of the skeleton graph structure at its respective granularity.

[0107] A first confidence threshold is set between the first-level subnetwork and the second-level subnetwork, and a second confidence threshold is set between the second-level subnetwork and the third-level subnetwork.

[0108] For example, the five joints corresponding to the coarse-grained map are the head, left elbow, right elbow, left knee, and right knee; the ten joints corresponding to the medium-grained map are the head, middle of the spine, left elbow, left wrist, right elbow, right wrist, left knee, left ankle, right knee, and right ankle; and the fifteen joints corresponding to the fine-grained map are the head, neck, middle of the spine, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, left hip, left knee, left ankle, right hip, right knee, and right ankle.

[0109] Step S104: Input the skeleton sequence data to be identified into the multi-scale cascaded graph convolutional network to obtain the final identification result.

[0110] It should be noted that in this step, the skeleton sequence data to be identified needs to be input into the first-level sub-network of the multi-scale cascaded graph convolutional network. This sub-network extracts and classifies the spatiotemporal features of the five key joints based on the coarse-grained graph, and outputs the first identification result and its corresponding first confidence score.

[0111] If the first confidence level is not lower than the first confidence threshold, then the first recognition result will be output as the final action recognition result.

[0112] If the first confidence level is lower than the first confidence threshold, the skeleton sequence data is input into the second-level sub-network. The sub-network performs spatiotemporal feature extraction and classification on 10 key joints based on the medium-granularity map, and outputs the second recognition result and its corresponding second confidence level.

[0113] If the second confidence level is not lower than the second confidence threshold, then the second recognition result will be output as the final action recognition result.

[0114] If the second confidence level is lower than the second confidence threshold, the skeleton sequence data is input into the third-level sub-network. This sub-network extracts and classifies the spatiotemporal features of 15 key joints based on the fine-grained map, and outputs the third recognition result as the final action recognition result.

[0115] For example, suppose the human motion recognition method of this application is deployed in a video surveillance system. Input is a skeleton sequence data (30 frames, each containing the 3D coordinates of 25 human joints) showing a person waving. When this skeleton sequence data is fed into a multi-scale cascaded graph convolutional network, it first enters the first-level sub-network. This sub-network performs spatiotemporal feature extraction and classification based only on a coarse-grained map (containing only 5 key joints: head, left elbow, right elbow, left knee, and right knee), outputting the first recognition result "waving" and its corresponding first confidence score of 0.92. Since the system's preset first confidence threshold is 0.85, and the first confidence score of 0.92 is not lower than this threshold (0.92 ≥ 0.85), the inference process terminates at the first level, directly outputting "waving" as the final motion recognition result. There is no need to call the second and third-level sub-networks. The entire recognition process uses only 5 joints (20% of the total number of joints), significantly reducing computational overhead and achieving the design goal of "fast recognition of simple actions." If the input is another skeleton sequence containing the "dancing" action, the first-level sub-network extracts features based on 5 key joints and outputs the first recognition result "walking" and its corresponding first confidence score of 0.62. Since this confidence score is lower than the first confidence threshold of 0.85 (0.62 < 0.85), the system determines that the credibility of the first-level recognition result is insufficient and automatically inputs the skeleton sequence data into the second-level sub-network. This sub-network performs more refined spatiotemporal feature extraction and classification based on a medium-granularity map (including 10 key joints: head, middle of the spine, left elbow, left wrist, right elbow, right wrist, left knee, left ankle, right knee, and right ankle), and outputs the second recognition result "dancing" and its corresponding second confidence score of 0.78. Since the system's preset second confidence threshold is 0.80, and the second confidence score of 0.78 is still lower than this threshold (0.78 < 0.80), the system determines that the credibility of the second-level recognition result is still insufficient. Therefore, the skeleton sequence data is further input into the third-level sub-network. This sub-network performs the most refined spatiotemporal feature extraction and classification based on a fine-grained map (including 15 key joints: head, neck, mid-spine, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, left hip, left knee, left ankle, right hip, right knee, and right ankle), outputting the third recognition result "dancing" and its corresponding third confidence score of 0.95. At this point, since the final level of the cascaded network has been reached, the system no longer performs confidence judgment and directly outputs "dancing" as the final action recognition result.The above process embodies the core design idea of ​​the multi-scale cascaded reasoning mechanism of this application: most simple actions (such as walking, standing, waving) can be recognized with high confidence and the reasoning can be terminated early at the first level based on coarse-grained skeleton information (5 key joints). Only a few complex actions (such as dancing, gymnastics) need to go deeper into the second or third level to call more fine-grained joint information. This progressive reasoning strategy enables most samples to be recognized at the first level while ensuring recognition accuracy, which significantly reduces the overall computational complexity and achieves the optimal balance between recognition accuracy and computational efficiency.

[0116] In summary, this invention overcomes the limitations of traditional fixed physical adjacency matrices by constructing a motion-aware hierarchical adjacency matrix. This allows for adaptive characterization of dynamic relationships between joints, including those not physically connected, under different actions, effectively improving the recognition capability of long-distance collaborative actions. By fusing multiple types of high-order features, such as spatial tetrahedral volume features, skeletal potential-kinetic coupling features, and skeletal map Laplacian spectrum features, the model's input information is enriched, significantly enhancing its ability to discriminate fine-grained actions with highly similar trajectories, such as waving and jumping. Furthermore, by designing a multi-scale cascaded graph convolutional network from coarse to fine, simple actions can be recognized at the first level with only a few key joints, while complex actions gradually enter more refined joint subnets. This achieves adaptive allocation of computational resources according to action complexity, significantly reducing model inference overhead while ensuring overall recognition accuracy.

[0117] like Figure 2 As shown, one embodiment of the present invention proposes a human action recognition system based on a cascaded graph convolutional network, the system comprising:

[0118] The skeleton data acquisition module 10 is used to acquire human skeleton sequence data, which includes the three-dimensional coordinate information of several human joints in each frame.

[0119] The adjacency matrix construction module 20 is used to calculate the acceleration amplitude of each joint based on the three-dimensional coordinate information, calculate the short-term mutual information and long-term mutual information between joint pairs using the sliding window method and the complete sequence method respectively based on the acceleration amplitude, generate a short-term dynamic adjacency matrix and a long-term dynamic adjacency matrix based on the short-term mutual information and the long-term mutual information respectively, and fuse the short-term dynamic adjacency matrix and the long-term dynamic adjacency matrix to obtain a dynamic adjacency matrix; and perform weighted fusion of the dynamic adjacency matrix and the human body physical connection adjacency matrix to obtain a motion perception hierarchical adjacency matrix.

[0120] The cascaded network construction module 30 is used to extract spatial tetrahedral volume features, skeletal potential-kinetic coupling features and skeletal map Laplacian spectrum features based on the motion-aware hierarchical adjacency matrix, and uses the motion-aware hierarchical adjacency matrix as the adjacency matrix of the graph convolutional layer, and uses various features as the input features of the graph convolutional layer to construct a multi-scale cascaded graph convolutional network composed of three-level sub-networks.

[0121] The action recognition module 40 is used to input the skeleton sequence data to be recognized into the multi-scale cascaded graph convolutional network to obtain the final recognition result.

[0122] In another aspect, the present invention also proposes a storage medium on which one or more programs are stored, which, when executed by a processor, implement the above-described human motion recognition method based on cascaded graph convolutional networks.

[0123] In another aspect, the present invention also proposes an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to realize the above-mentioned human action recognition method based on cascaded graph convolutional networks.

[0124] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain stored, communicated, propagated, or transmitted programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0125] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0126] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0127] While embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations fall within the scope and spirit of the invention as set forth in the claims. Furthermore, the invention described herein may have other embodiments and can be implemented or carried out in various ways.

Claims

1. A human action recognition method based on cascaded graph convolutional networks, characterized in that, The method includes: Obtain human skeleton sequence data, wherein the skeleton sequence data includes the three-dimensional coordinate information of several human joints in each frame; The acceleration amplitude of each joint is calculated based on the three-dimensional coordinate information. Short-term mutual information and long-term mutual information between joint pairs are calculated using both sliding window and complete sequence methods based on the acceleration amplitude. Short-term dynamic adjacency matrices and long-term dynamic adjacency matrices are generated based on the short-term and long-term mutual information, respectively. The short-term and long-term dynamic adjacency matrices are then fused to obtain a dynamic adjacency matrix. The dynamic adjacency matrix is ​​then weighted and fused with the human body physical connection adjacency matrix to obtain a motion perception hierarchical adjacency matrix. Based on the motion-aware hierarchical adjacency matrix, spatial tetrahedral volume features, skeletal potential-kinetic coupling features, and skeletal map Laplacian spectrum features are extracted. The motion-aware hierarchical adjacency matrix is ​​used as the adjacency matrix of the graph convolutional layer, and various features are used as the input features of the graph convolutional layer to construct a multi-scale cascaded graph convolutional network composed of three-level sub-networks. The skeleton sequence data to be identified is input into the multi-scale cascaded graph convolutional network to obtain the final identification result.

2. The human action recognition method based on cascaded graph convolutional networks according to claim 1, characterized in that, The steps of calculating the acceleration amplitude of each joint based on the three-dimensional coordinate information, and calculating the short-term mutual information and long-term mutual information between joint pairs using the sliding window method and the complete sequence method respectively based on the acceleration amplitude include: Calculate the acceleration amplitude using the following formula: ； in, Let be the coordinates of joint i at times t+1, t, and t-1, respectively. It is the acceleration of joint i at time t. They represent Acceleration components along the x, y, and z axes; Short-term mutual information can be obtained using the following formula: ； in, For the short-term mutual information between joint i and joint j, S represents the number of samples, and W represents the number of windows per sample; , They are samples s at time t. The acceleration amplitudes of joints i and j, It is the starting time point of the window. The average acceleration amplitude of joint i over five time windows; The correlation between the acceleration amplitudes of joint i and joint j in the s-th sample and the w-th time window; Long-term mutual information can be obtained using the following formula: ； in, For long-term mutual information between joint i and joint j. Let be the acceleration amplitudes of joints i and j of sample s at time t, respectively. To represent the total duration of the entire time series, The average acceleration over the entire time series, This refers to the overall synergy or correlation between the changes in acceleration amplitude of joints i and j during the entire motion of sample s.

3. The human action recognition method based on cascaded graph convolutional networks according to claim 2, characterized in that, The method generates a short-term dynamic adjacency matrix and a long-term dynamic adjacency matrix based on the short-term mutual information and the long-term mutual information, respectively, and then merges the short-term dynamic adjacency matrix and the long-term dynamic adjacency matrix to obtain a dynamic adjacency matrix. The step of weightedly fusing the dynamic adjacency matrix with the human physical connection adjacency matrix to obtain the motion-aware hierarchical adjacency matrix includes: Construct the short-term dynamic adjacency matrix and the long-term dynamic adjacency matrix according to the following formulas: ； ； in, , Let be the short-term dynamic adjacency matrix and the long-term dynamic adjacency matrix of joint i and joint j, respectively. These are short-term and long-term adaptive thresholds, respectively. The short-term adaptive threshold is calculated using the following formula: ； The long-term adaptive threshold is calculated using the following formula: ； in, These are the mean values ​​of short-term mutual information and long-term mutual information, respectively. , These are the standard deviations of short-term mutual information and long-term mutual information, respectively. For mathematical expectation, For variance, This is the adjustment coefficient for the short-term adaptive threshold. This is the adjustment coefficient for the long-term adaptive threshold. For short-term mutual information, For long-term mutual information; Construct a dynamic adjacency matrix using the following formula: ； in, It is a dynamic adjacency matrix.

4. The human action recognition method based on cascaded graph convolutional networks according to claim 3, characterized in that, The steps of extracting spatial tetrahedral volume features, skeletal potential-kinetic coupling features, and skeletal atlas Laplacian spectrum features based on the motion-aware hierarchical adjacency matrix include: The volume characteristics of the spatial tetrahedron include the instantaneous volume of the tetrahedron, the rate of change of volume over time, and the volume fluctuation entropy. Four non-coplanar joints were selected, and the shoulder, elbow, hip, and knee were denoted as follows: Construct a spatial tetrahedron and calculate its instantaneous volume, rate of change of volume over time, and entropy of volume fluctuation: ； ； ； in, It is a cross product. For dot product, For absolute values, For joints Pointing joint The vector, For joints Pointing joint The vector, For joints Pointing joint The vector, Let be the instantaneous volume of the skeleton image in frame t. Let be the temporal rate of change of the volume of the skeleton image in frame t. Let K be the volume fluctuation entropy, and K be the volume sequence. The number of non-overlapping intervals. The probability of falling into the k-th interval is the number of volume change rate values ​​contained within that interval. The ratio to the total number of frames T.

5. The human action recognition method based on cascaded graph convolutional networks according to claim 4, characterized in that, The steps of extracting spatial tetrahedral volume features, skeletal potential-kinetic coupling features, and skeletal atlas Laplacian spectrum features based on the motion-aware hierarchical adjacency matrix include: The skeletal potential-kinetic energy coupling characteristics include instantaneous gravitational potential energy, instantaneous kinetic energy, potential-kinetic energy conversion phase difference, and potential-kinetic energy ratio. The instantaneous gravitational potential energy can be obtained using the following formula: ； in, Let be the instantaneous gravitational potential energy of the skeleton image in frame t. N Let m be the total number of joints in the human body, and m be the mass of each joint. Let i be the mass weight of joint i. It is the acceleration due to gravity. For joint i in the skeleton image of frame t y Axis coordinates; The instantaneous kinetic energy can be obtained using the following formula: ； in, Let be the instantaneous kinetic energy of the skeleton image in frame t. Let i be the velocity vector of joint i at time t; The potential energy to kinetic energy conversion phase difference can be obtained using the following formula: ； in, Let be the potential-kinetic energy conversion phase difference of the skeleton image in frame t. and They are respectively and The instantaneous phase obtained by Hilbert transform , These are the imaginary and real parts of the Hilbert transform result, respectively. for Skeleton image of frame t The total instantaneous potential energy, For Hilbert transform, It is a bivariate arctangent function; The potential energy to kinetic energy ratio can be obtained using the following formula: ； in, Let be the potential energy to kinetic energy ratio of the skeleton image in frame t. Let be the potential energy and kinetic energy of the skeleton image in frame t, respectively. It is a non-zero constant.

6. The human action recognition method based on cascaded graph convolutional networks according to claim 5, characterized in that, The steps of extracting spatial tetrahedral volume features, skeletal potential-kinetic coupling features, and skeletal atlas Laplacian spectrum features based on the motion-aware hierarchical adjacency matrix include: The Laplacian spectral features of the skeletal atlas include topological spectral features, full-frame spectral entropy, and temporal mean of spectral features; The Laplacian matrix of the skeletal atlas can be obtained using the following formula: ； in, It is an N×N dimensional Laplace matrix. Given an N×N diagonal matrix, only the diagonal elements are non-zero, and the rest are 0. For motion-aware hierarchical adjacency matrices, Let be the degree of the i-th node. Let be the connection weight between node i and node k; right Eigenvalue decomposition yields N eigenvalues, which are then sorted in ascending order. ; Select all eigenvalues ​​to construct the Laplacian spectral coefficient vector of the skeleton image in frame t. : ； The spectral coefficient vectors of all frames are concatenated in frame order to obtain the full-frame Laplace spectral coefficient matrix, and the topological spectral features of all frames are then integrated. : ； in, These are the Laplacian spectral coefficient vectors of the skeleton images in frames 1, 2, and T, respectively. The full-frame spectral entropy can be obtained using the following formula: ； in, For the full frame spectral entropy, , These are the Laplace matrices. The i-th and j-th eigenvalues; The temporal mean of the spectral features can be obtained using the following formula: ； in, Laplace matrix The temporal mean of the i-th feature value across the entire frame.

7. The human action recognition method based on cascaded graph convolutional networks according to claim 6, characterized in that, The steps of using the motion-aware hierarchical adjacency matrix as the adjacency matrix of the graph convolutional layer, and using various features as the input features of the graph convolutional layer to construct a multi-scale cascaded graph convolutional network composed of three-level sub-networks include: Based on the natural connection relationships of the human skeleton, coarse-grained graphs containing 5 key joints, medium-grained graphs containing 10 key joints, and fine-grained graphs containing 15 key joints are constructed as the graph topology structures of the three-level sub-networks. Each sub-network level contains a batch normalization layer, seven spatiotemporal graph convolutional modules, a global average pooling layer, and a fully connected classification layer connected in sequence. The time stride of the fourth and seventh spatiotemporal graph convolutional modules is set to 2. Each sub-network's graph convolutional layer uses the motion-aware hierarchical adjacency matrix as its adjacency matrix and the spatial tetrahedral volume feature, skeletal potential-kinetic coupling feature, and skeletal graph Laplacian spectrum feature as its input features to extract spatiotemporal features of the skeleton graph structure at its respective granularity. A first confidence threshold is set between the first-level subnetwork and the second-level subnetwork, and a second confidence threshold is set between the second-level subnetwork and the third-level subnetwork.

8. The human action recognition method based on cascaded graph convolutional networks according to claim 7, characterized in that, The step of inputting the skeleton sequence data to be identified into the multi-scale cascaded graph convolutional network to obtain the final recognition result includes: The skeleton sequence data to be identified is input into the first sub-network of the multi-scale cascaded graph convolutional network. The sub-network performs spatiotemporal feature extraction and classification on 5 key joints based on the coarse-grained graph, and outputs the first identification result and its corresponding first confidence score. If the first confidence level is not lower than the first confidence threshold, then the first recognition result will be output as the final action recognition result. If the first confidence level is lower than the first confidence threshold, the skeleton sequence data is input into the second-level sub-network. The sub-network performs spatiotemporal feature extraction and classification on 10 key joints based on the medium-granularity map, and outputs the second recognition result and its corresponding second confidence level. If the second confidence level is not lower than the second confidence threshold, then the second recognition result will be output as the final action recognition result. If the second confidence level is lower than the second confidence threshold, the skeleton sequence data is input into the third-level sub-network. This sub-network extracts and classifies the spatiotemporal features of 15 key joints based on the fine-grained map, and outputs the third recognition result as the final action recognition result.

9. A human motion recognition system based on cascaded graph convolutional networks, characterized in that, The system includes: The skeleton data acquisition module is used to acquire human skeleton sequence data, which includes the three-dimensional coordinate information of several human joints in each frame. The adjacency matrix construction module is used to calculate the acceleration amplitude of each joint based on the three-dimensional coordinate information, calculate the short-term mutual information and long-term mutual information between joint pairs using both the sliding window method and the complete sequence method based on the acceleration amplitude, generate short-term dynamic adjacency matrices and long-term dynamic adjacency matrices based on the short-term mutual information and long-term mutual information respectively, and fuse the short-term dynamic adjacency matrix and long-term dynamic adjacency matrix to obtain a dynamic adjacency matrix; and perform weighted fusion of the dynamic adjacency matrix with the human body physical connection adjacency matrix to obtain a motion perception hierarchical adjacency matrix; The cascaded network construction module is used to extract spatial tetrahedral volume features, skeletal potential-kinetic coupling features, and skeletal map Laplacian spectrum features based on the motion-aware hierarchical adjacency matrix. The motion-aware hierarchical adjacency matrix is ​​used as the adjacency matrix of the graph convolutional layer, and various features are used as the input features of the graph convolutional layer to construct a multi-scale cascaded graph convolutional network composed of three-level sub-networks. The action recognition module is used to input the skeleton sequence data to be recognized into the multi-scale cascaded graph convolutional network to obtain the final recognition result.

10. A storage medium, characterized in that, The storage medium stores one or more programs that, when executed by a processor, implement the human motion recognition method based on cascaded graph convolutional networks as described in any one of claims 1-8.

Images