Skeleton action recognition method based on motion center space-time diagram convolution
By acquiring the temporal features and motion centers of human skeleton image sequences and combining them with spatiotemporal graph convolutional networks, the problem of insufficient human skeleton information mining in existing technologies is solved, and the accuracy of action recognition is improved.
Patent Information
- Application Number
- CN202511702431.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-19
- Publication Date
- 2026-02-06
AI Technical Summary
In existing technologies, there is relatively little work done on mining human skeletal information, which increases the difficulty of classification in recognition models and reduces the accuracy of action recognition.
By acquiring human skeleton image sequences, temporal features are extracted to extract the motion center of the moving object. Skeletal features are generated by combining three-dimensional spatial position information. Spatiotemporal dynamic features are extracted using a spatiotemporal graph convolutional network to establish a skeleton action recognition model for training and recognition.
It improves the accuracy of action recognition by making full use of spatial structure and temporal change features, reducing the difficulty of model recognition and improving the performance of action recognition.
Smart Images
Figure CN121482866A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of human motion recognition technology, and in particular to a skeleton motion recognition method based on spatiotemporal graph convolution of the center of motion. Background Technology
[0002] As a crucial step in the process of achieving intelligentization, human motion recognition technology is a highly anticipated research field, involving the interdisciplinary integration of theories and technologies from computer vision, video processing, pattern recognition, and data mining. Currently, the application of human motion recognition technology is widespread. For example, in intelligent transportation systems, Internet of Things (IoT) technology achieves comprehensive monitoring of traffic conditions through the interconnection of various sensors and vehicle communication devices. Human motion recognition technology can assist in analyzing the behavioral characteristics of pedestrians on the road, thereby making corresponding decisions to ensure the safety and smoothness of the driving process. In intelligent monitoring systems, this technology can assist managers in responding to various dangerous situations in a timely manner, improving management efficiency.
[0003] Current mainstream action recognition technologies primarily process image sequences, extracting features from these sequences and inputting them into classifiers to complete the recognition task. Common feature extraction methods are mainly divided into two categories: manual feature extraction and deep learning model feature extraction. However, due to interference factors such as background, lighting, and personal objects in video images, and because directly using video images for action recognition increases the computational complexity of the model with increasing resolution, action recognition methods based on human skeletons have become a key research focus compared to directly processing video images. Currently, among the many action recognition algorithms based on human skeletons, graph convolutional methods based on human skeletons are less affected by the surrounding environment and require less data for the recognition model. Therefore, graph convolutional networks have become one of the mainstream methods for human action recognition. Their extraction of spatial features of human skeletons is becoming increasingly mature. However, current research focuses on the design of graph convolutional network architectures, while relatively little work is done on feature mining of human skeleton information. Human skeleton sequences not only contain rich motion-related information, but incomplete information mining will limit the performance of the model. At the same time, human skeleton information also contains a lot of interference information, which can easily increase the classification difficulty of the recognition model and reduce the accuracy of action recognition. Summary of the Invention
[0004] This invention provides a skeleton action recognition method based on spatiotemporal graph convolution of the center of motion, in order to overcome the technical problem that the existing technology has relatively little work on mining human skeleton information, which easily leads to increased classification difficulty of the recognition model and reduced action recognition accuracy.
[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A skeleton action recognition method based on spatiotemporal graph convolution of the motion center, the specific steps of which include: S1. Obtain a sequence of human skeleton diagrams of a moving object, wherein the sequence of human skeleton diagrams contains three-dimensional spatial location information of the moving object; Temporal features are extracted from the human skeleton image sequence to obtain a temporal feature map representing the temporal features of each moving object; S2. Extract the temporal feature map to obtain the motion center of the moving object; S3. Obtain skeleton features containing spatiotemporal information based on the motion center and three-dimensional spatial position information; S4. Establish a skeleton motion recognition model, and train the skeleton motion recognition model based on the skeleton features to obtain the trained skeleton motion recognition model; The skeleton motion recognition model includes a feature extraction module and a classification module; The feature extraction module is used to extract the spatiotemporal dynamic features from the skeleton features and transmit them to the classification module; The classification module is used to output skeleton action classification results based on the spatiotemporal dynamic features; S5. Identify skeleton actions based on the trained skeleton action recognition model to obtain the action classification results of the moving object. Further, in S1, the specific steps for extracting temporal features from the human skeleton image sequence to obtain a temporal feature map representing the temporal features of each moving object include: The three-dimensional spatial position information of each joint point in the human skeleton diagram sequence is mapped by a mapping function to obtain the time feature value of each joint point. The temporal feature map is obtained based on the temporal feature values of all key points.
[0006] Furthermore, the mean function is used as the mapping function, and the mapping calculation is performed on each joint point in three coordinate dimensions based on the mapping function to obtain the time feature value of each joint point. The calculation formulas for the time feature value of each joint point are shown in equations (1), (2) and (3): (1) (2) (3) In the formula, n Indicates the frame rate of the motion video. , and They represent the first j Frame number iEach joint point along X , Y , Z The coordinate values of the axis. , and The first element in the time series feature graph obtained after mapping is represented as the first element. i Each joint point along X , Y , Z Time characteristic values in the axial direction.
[0007] Further, in S2, the specific steps for extracting the temporal feature map to obtain the motion center of the moving object include: S21. Make all joints in the timing feature map move along... X , Y , Z The coordinate values of the axes constitute three sets of coordinates; S22. Calculate the coordinates of the motion center independently in three coordinate dimensions, including: obtaining the maximum and minimum coordinate values in each coordinate set, calculating the mean using the maximum and minimum coordinate values in each coordinate set, and using the mean obtained from the three coordinate sets as the initial coordinates of the motion center of the moving object. The calculation formula is as follows: (4) In the formula, Represents the time series feature graph along X, Y or Z Axial direction M A set of coordinates of the positions of each joint. express X, Y or Z The average of the maximum and minimum values in the set of coordinates of joint points along the axis. M This represents the number of key points in the time-series feature map. S23. Calculate the mean of the temporal feature values of all relevant nodes in the temporal feature map across the three coordinate dimensions. The calculation formula is: (5) In the formula, Key points in the time series feature graph i Along X, Y or Z Eigenvalues along the axial direction; S24. Calculate the initial coordinates of the center of motion of the moving object and... mean The calculation formula is: (6) S25. Obtain the three-dimensional position coordinates of the motion center of the moving object. .
[0008] Furthermore, the calculation formula for obtaining skeleton features containing spatiotemporal information based on the motion center and three-dimensional spatial location information is as follows: (7) In the formula, , and These represent the first and second lines in the sequence of human skeleton diagrams. i Frame number j Spatial coordinates of each joint point This represents the feature representation vector generated for the joints of a moving object.
[0009] Furthermore, the feature extraction module includes several ST-GCN modules, and extracts spatiotemporal dynamic features from the skeleton features through the several ST-GCN modules; Each of the ST-GCN modules includes a GCN module and a TCN module; The GCN module is used to extract spatial features from the input data and transmit them to the TCN module; The TCN module is used to perform convolution operations on the input data and add the convolution result to the data input to the GCN module to obtain the output features.
[0010] Beneficial Effects: This invention acquires a sequence of human skeleton images of a moving object, extracts temporal features from the sequence, and uses temporal feature maps to represent the temporal characteristics of each moving object. It then extracts the motion center specific to each moving object using these temporal feature maps. The motion center is combined with the three-dimensional spatial position information of the action to generate skeleton features. Using these skeleton features as input, a skeleton-action recognition model extracts the spatiotemporal dynamic features from the skeleton features, ultimately completing action recognition. This invention fully utilizes the spatial structural information and temporal variation characteristics of each moving object, and by introducing a motion center to express the characteristics of each moving object in different actions, it improves the accuracy of action recognition. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a flowchart of a skeleton motion recognition method based on spatiotemporal graph convolution of the motion center in this invention; Figure 2 The diagram shows the human skeleton used in this embodiment of the invention, where numbers 1-25 represent 25 joints of the human skeleton. Figure 3 This is a flowchart illustrating the acquisition of the time-series feature map in an embodiment of the present invention; Figure 4 This is a flowchart of the action recognition process based on the skeleton action recognition model in an embodiment of the present invention; Figure 5 This is a network architecture diagram of each ST-GCN module in an embodiment of the present invention; Figure 6 This is a schematic diagram of the feature representation vector of a single person's action "falling down" and a two-person action "slapping" during a certain frame in an embodiment of the present invention. Detailed Implementation
[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0014] Addressing the shortcomings and limitations of current mainstream feature extraction methods, this invention aims to break through existing frameworks and extract skeleton features in a novel way. Since all computer operations are data-driven, feature selection significantly impacts the final recognition performance. For example, when different individuals perform the same action, their range of motion, center of gravity height, and action duration differ, potentially generating entirely different feature matrices. When the feature matrices corresponding to the same type of action differ significantly, these matrices increase the difficulty for the model to recognize the action, leading to a decrease in the model's recognition rate. In the field of action recognition, the problem of large intra-class differences and small inter-class differences in similar actions persists; reducing this difference can effectively improve recognition performance.
[0015] Based on the above ideas, this embodiment provides a skeleton action recognition method based on spatiotemporal graph convolution of the center of motion, such as... Figure 1 As shown, the specific steps include: S1. Obtain a sequence of human skeleton diagrams of a moving object, wherein the sequence of human skeleton diagrams contains three-dimensional spatial location information of the moving object; Temporal features are extracted from the human skeleton image sequence to obtain a temporal feature map representing the temporal features of each moving object; Specifically, this embodiment is based on motion video and uses Openpose to extract the human skeleton image sequence of the moving object from the motion video, that is, the human skeletal joints.
[0016] Specifically, human skeleton sequences not only contain motion-related information, but also a large amount of interference information. Effective features can filter out redundant interference information while preserving action features, thereby reducing the difficulty of model classification and improving action recognition rate.
[0017] S2. Extract the temporal feature map to obtain the motion center of the moving object; S3. Based on the motion center and three-dimensional spatial location information, obtain the skeleton feature (Combined Spatial-Temporal Information skeleton feature, CSTI feature) containing spatiotemporal information. Specifically, in this embodiment, the skeleton features fully consider the spatiotemporal information of the action.
[0018] S4. Establish a skeleton motion recognition model, and train the skeleton motion recognition model based on the skeleton features to obtain the trained skeleton motion recognition model; The skeleton motion recognition model includes a feature extraction module and a classification module; The feature extraction module is used to extract the spatiotemporal dynamic features from the skeleton features and transmit them to the classification module; The classification module is used to output skeleton action classification results based on the spatiotemporal dynamic features; S5. Identify skeleton actions based on the trained skeleton action recognition model to obtain the action classification results of the moving object. Specifically, this embodiment proposes a skeleton feature representation based on the center of motion. The design idea of this feature is to dynamically generate a unique center of motion for each moving object according to its characteristics. The center of motion reflects the motion characteristics of the object under the action. Using it as a reference point, the entire motion can be regarded as unfolding around the reference point. Then, feature expression vectors of each moving object are generated based on the center of motion.
[0019] In a specific embodiment, S1, the specific steps of extracting temporal features from the human skeleton image sequence to obtain a temporal feature map representing the temporal features of each moving object include: The three-dimensional spatial position information of each joint in the human skeleton image sequence is mapped by a mapping function to obtain the temporal feature value of each joint. The temporal feature map is obtained based on the temporal feature values of all joints.
[0020] Specifically, in this embodiment, such as Figure 2As shown, the time series feature map contains time feature values of 25 key points.
[0021] Specifically, in acquiring temporal feature maps, the two most important features describing an action are temporal and spatial features. Spatial features reflect information such as the position and amplitude of the action, while temporal features reflect information such as the speed and rhythm of the action. Compared to spatial features, modeling temporal features is often more difficult because spatial features can be directly obtained from the position coordinates of the human skeleton's joints, while the modeling method for temporal features is more flexible. Since the human skeleton map is composed of skeletal joints and their connections, the positional changes of the same skeletal joint during movement are the basis for acquiring temporal features in this embodiment. The temporal information of an action is composed of the temporal information of all joints that make up the human skeleton map. The extraction of temporal features cannot only consider the correlation between a few frames; since an action is a complete process, changes in a single frame during this process will also lead to changes in the overall temporal characteristics. Therefore, this embodiment will comprehensively consider the information of all skeletal frames when extracting the temporal features of an action. Specifically, such as Figure 3 As shown, the main process of temporal feature extraction is illustrated, with key points as the starting point. a For example, in the case of 1, n In a frame-by-frame motion video, as the action progresses, the joints... a 1 will be presented n These are different states. n After being processed by a mapping function, each different state is mapped to a temporal feature value representing the joint point under this action. If a skeleton graph contains h There are 10 skeletal joints, and the temporal feature value of each joint can be obtained through a mapping function, ultimately resulting in... h Each time feature value is used to form a new temporal feature map, in which each key point can reflect its temporal characteristics in the overall motion process.
[0022] Specifically, in this embodiment, the mean function is used as the mapping function. Based on the mapping function, the mapping calculation is performed on each joint point in three coordinate dimensions to obtain the time feature value of each joint point. The calculation formulas for the time feature value of each joint point are shown in Equations (1), (2) and (3): (1) (2) (3) In the formula, n Indicates the frame rate of the motion video. , and They represent the first j Frame number i Along each skeletal joint pointX , Y , Z The coordinate values of the axis. , and The first element in the time series feature graph obtained after mapping is represented as the first element. i Along each skeletal joint point X , Y , Z Time characteristic values in the axial direction.
[0023] Specifically, in order to avoid destroying the spatial information of actions during the mapping process, this embodiment adopts a mapping function that follows the same principle as the temporal feature process: maintaining correlation without destroying spatial information, and compressing data at the same time.
[0024] Specifically, in this embodiment, for each moving object in a motion video, a temporal feature map is used to represent the temporal characteristics of all moving objects participating in the action. Simultaneously, the temporal feature map also retains the spatial position information of each moving object. Then, the motion center is obtained using the temporal feature map. In a specific embodiment, S2, the specific steps of extracting the temporal feature map to obtain the motion center of the moving object include: S21. Make all joints in the timing feature map move along... X , Y , Z The coordinate values of the axes constitute three sets of coordinates; S22. Calculate the coordinates of the motion center independently in three coordinate dimensions, including: obtaining the maximum and minimum coordinate values in each coordinate set, calculating the mean using the maximum and minimum coordinate values in each coordinate set, and using the mean obtained from the three coordinate sets as the initial coordinates of the motion center of the moving object. The calculation formula is as follows: (4) In the formula, Represents the time series feature graph along X, Y or Z Axial direction M A set of coordinates of the positions of each joint. express X, Y or Z The average of the maximum and minimum values in the set of coordinates of joint points along the axis. M This represents the number of key points in the time-series feature map. Specifically, the average of the maximum and minimum values reflects the initial position of the motion center in the current coordinate dimension. This position is only a preliminary calculation result and needs further correction.
[0025] S23. Calculate the mean of the temporal feature values of all relevant nodes in the temporal feature map across the three coordinate dimensions. The calculation formula is: (5) In the formula, Key points in the time series feature graph i Along X, Y or Z Eigenvalues along the axial direction; S24. Calculate the initial coordinates of the center of motion of the moving object and... mean The calculation formula is: (6) S25. Obtain the three-dimensional position coordinates of the motion center of the moving object. .
[0026] Specifically, in extracting the motion center, spatial modeling of the temporal feature map is required. The ultimate goal is to calculate the motion center belonging to the moving object from the temporal feature map and obtain the feature matrix of the moving object based on the motion center. In this embodiment, a mapping relationship is used to map the temporal feature map to the motion center. This motion center will become the reference point of the moving object under this action, and the final skeleton features will be obtained based on this reference point.
[0027] Specifically, for each motion video, the position coordinates of the joints reflect their spatial position changes during the motion. This embodiment generates a unique motion center for each moving object, and the number of motion centers is consistent with the number of moving objects in the video. The generated motion centers are used throughout all skeletal frames of the video, with each frame using this motion center as a reference point. For videos containing multiple moving objects, this embodiment does not limit the number of moving objects.
[0028] In a specific embodiment, the calculation formula for obtaining skeleton features containing spatiotemporal information based on the motion center and three-dimensional spatial location information is as follows: (7) In the formula, , and These represent the first and second lines in the sequence of human skeleton diagrams. i Frame number j Spatial coordinates of each joint point This represents the feature representation vector generated for the joints of a moving object. In this embodiment, to better express spatial features, the feature representation vector is represented by the direction vector from the motion center of each moving object to its joints.
[0029] In this embodiment, the feature vector input into the skeleton motion recognition model is designed as a feature expression vector pointing from the center of motion to each joint in the human skeleton diagram during the motion process.
[0030] Specifically, such as Figure 6 As shown, the feature vector representation of a frame during the two actions of "falling down" (single person) and "slapping" (two people) is displayed. The green feature points are the motion centers of each moving object, and the red line segments are the feature vector representations pointing from the motion centers to each joint point in the frame.
[0031] In a specific embodiment, the feature extraction module includes several Spatial-Temporal Graph Convolutional Networks (ST-GCN) modules, and extracts the spatiotemporal dynamic features from the skeleton features through the several ST-GCN modules; Each of the ST-GCN modules includes a GCN module and a TCN module; The GCN module is used to extract spatial features from the input data and transmit them to the TCN module; The TCN module is used to perform convolution operations on the input data and add the convolution result to the data input to the GCN module to obtain the output features.
[0032] Specifically, this embodiment utilizes 10 ST-GCN modules to extract features serially, such as... Figure 4 As shown, the first ST-GCN module has 3 input channels and 64 output channels. After passing through three ST-GCN modules with 64 input and 64 output channels, the output is used as the input of the fifth ST-GCN module. After passing through the fifth ST-GCN module, the number of channels increases from 64 to 128. Then, after passing through two ST-GCN modules with 128 input and 128 output channels, the output is fed into the eighth ST-GCN module. After passing through the eighth ST-GCN module, the number of channels increases from 128 to 256. After passing through two ST-GCN modules with 256 input and 256 output channels, feature extraction is completed.
[0033] In this embodiment, the classification module is an FCN (Fully Convolutional Network) classifier, and the output of the tenth ST-GCN module is input into the FCN classifier to complete the final classification task.
[0034] Specifically, in each ST-GCN module, such as Figure 5 As shown, the GCN module is used to extract spatial features from the human skeleton image, and then a convolution kernel size of [missing value] is applied. The TCN module, composed of convolutional layers, performs convolution operations on the output feature map of the GCN module, and finally obtains the output feature through residual connections.
[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A skeleton motion recognition method based on motion center spatio-temporal graph convolution, characterized in that, The specific steps include: S1. Obtain a human skeleton graph sequence of a moving object, the human skeleton graph sequence containing three-dimensional spatial position information of the moving object; S1. Obtain a human skeleton graph sequence of a moving object, the human skeleton graph sequence containing three-dimensional spatial position information of the moving object; S1. Obtain a human skeleton graph sequence of a moving object, the human skeleton graph sequence containing three-dimensional spatial position information of the moving object; S2. Extract the time sequence feature map to obtain a motion center of the moving object; S3. Obtain skeleton features containing spatiotemporal information based on the motion center and the three-dimensional spatial position information; S4. Establish a skeleton action recognition model, train the skeleton action recognition model based on the skeleton features, and obtain a trained skeleton action recognition model; The skeleton action recognition model includes a feature extraction module and a classification module; The feature extraction module is configured to extract spatiotemporal dynamic features in the skeleton features and transmit the spatiotemporal dynamic features to the classification module; The classification module is configured to output a skeleton action classification result based on the spatiotemporal dynamic features; 2. The motion center-based spatio-temporal graph convolutional network method for skeleton action recognition according to claim 1, characterized in that, S5. Identify a skeleton action based on the trained skeleton action recognition model to obtain an action classification result of the moving object. In S1, the specific steps of performing time sequence feature extraction on the human skeleton graph sequence to obtain a time sequence feature map for representing time features of each moving object include: Map the three-dimensional spatial position information of each joint node in the human skeleton graph sequence through a mapping function to obtain a time feature value of each joint node; 3. The motion center-based spatio-temporal graph convolutional network method of skeleton action recognition according to claim 2, characterized in that, Obtain the time sequence feature map based on the time feature values of all joint nodes. (1) (2) (3) In the formula, n frame rate of the action video, , and respectively represent the coordinate values of the first j frame the first i joint along the X , Y , Z axis, , and represent the time feature values of the first i joint in the time sequence feature map obtained after the mapping processing along the X , Y , Z axis direction.
4. The motion center-based spatio-temporal graph convolutional network method of skeleton action recognition according to claim 3, characterized in that, Use a mean function as the mapping function, perform mapping calculation on each joint node in three coordinate dimensions based on the mapping function, and obtain a time feature value of each joint node. The time feature value calculation formula of each joint node is shown in formula (1), formula (2) and formula (3): S21. make all the joint nodes in the timing feature map along the coordinate values of the three coordinate sets of the axis; X , Y , Z axis. In S2, the specific steps of extracting the time sequence feature map to obtain a motion center of the moving object include: (4) In the formula, represents a set of node position coordinates along the S22. Independently calculate the coordinate values of the motion center in three coordinate dimensions, including: obtaining the maximum and minimum values of the coordinates in each coordinate set respectively, and calculating the mean value using the maximum and minimum values of the coordinates in each coordinate set, and taking the mean values obtained based on the three coordinate sets as the initial coordinates of the motion center of the moving object, and the calculation formula is: or Z axis direction in the time sequence feature map, M represents X, Y or Z the average of the maximum value and the minimum value in the node position coordinate set along the axis direction, M is the number of nodes in the time sequence feature map; S23. Calculate the mean value of the time sequence feature values of all the junctions in the three coordinate dimensions in the time sequence feature map The calculation formula is: (5) In the formula, representing a node in a time-series feature map i along X, Y or Z characteristic value in the axial direction; S24. calculate the initial coordinates of the motion center of the moving object and the mean value of the initial coordinates of the motion center of the moving object , the calculation formula is: (6) S25. Obtain the three-dimensional position coordinates of the motion center of the moving object as .
5. The motion center-based spatio-temporal graph convolutional network method of skeleton action recognition according to claim 4, characterized in that, X, Y (7) In the formula, , and respectively represent the spatial position coordinates of the i frame of the j th joint in the human skeleton graph sequence, represent the feature expression vector generated for the joint of the moving object.
6. The motion center-based spatio-temporal graph convolutional network based skeleton action recognition method according to claim 5, characterized in that, The calculation formula for obtaining skeleton features containing spatiotemporal information based on the motion center and the three-dimensional spatial position information is: The feature extraction module includes a plurality of ST-GCN modules, and the spatiotemporal dynamic features in the skeleton features are extracted through the plurality of ST-GCN modules; Each ST-GCN module includes a GCN module and a TCN module; The GCN module is configured to extract spatial features in input data and transmit the spatial features to the TCN module; The TCN module is configured to perform convolution operation on the input data, and add the convolution operation result to the data input into the GCN module to obtain output features.