Three-dimensional human motion recognition method based on trajectory manifold

Abstract

The application relates to a three-dimensional human body motion recognition method based on a trajectory manifold, and is characterized in that the method comprises the following steps: step 1, pre-processing a human body motion data set; step 2, for a certain action of all samples, performing three times of B-spline curve fitting on the motion trajectory of each joint, splicing all joint curves in the same sample to form a B-spline curve cluster, and step 3, calculating curve discretization information, and obtaining shape information on Kendall space through removal of translation, scaling and rotation operations and the like; the method has the superior technical effect that a manifold construction method based on a motion trajectory curve cluster is proposed, a geodesic complete motion trajectory manifold is constructed, the motion time sequence is regarded as a three-dimensional curve by the motion curve cluster, time-space conversion is completed, the B-spline motion trajectory description can realize continuous representation of motion by using fewer parameters in a continuous domain, and matching and alignment between different motions can be realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of human motion recognition of video skeleton sequence, and particularly relates to a three-dimensional human motion recognition method based on trajectory manifold. BACKGROUND

[0002] With the continuous development of computer technology, it has become increasingly mature to obtain human motion information from motion videos. However, in the face of massive motion file data, how to represent, analyze, process and use them is still a great challenge.

[0003] Most of the time, the human body is in a state of motion. However, due to the complexity and non-rigidity of the human body, it is very difficult for computers to recognize the motion that the human eye can easily recognize. In view of the fact that real human motion needs to consider multiple complex factors such as skeleton, skin and muscle, various simplified human model methods have appeared, among which the 3D skeleton represents the joints and bones that mark the key parts of the body, and can express the joint position as a set of three-dimensional coordinates. This representation can simplify the motion process, remove the influence of motion background information, and has the advantages of computational flexibility and view point insensitivity.

[0004] The research focus of motion recognition using human skeleton data is feature representation. In computer graphics, many methods focus on representing human skeleton motion with geometric information. This kind of method regards the joint displacement change in the time (frame) sequence as an input point in the Euclidean space, and then fits the motion trajectory of each joint. Compared with other methods, this type of method has low computational cost and can consider the smooth continuity of motion, but it does not fully exhibit the view invariance of motion.

[0005] Another method is the manifold-based motion representation. This kind of method maps the position information of the skeleton sequence to a nonlinear shape space, and hopes to find a more descriptive representation in the new space, and on this basis, obtain a metric for comparing these trajectories. The manifold-based method fuses space-time to represent skeleton motion, which has scale-invariant robustness compared with traditional space representation, but most of the manifold-based methods do not consider the continuity and smoothness of motion.

[0006] In addition, features based on the topology of the body are also used to construct the motion representation of the skeleton, which takes into account the physical structure of the human body, and generally improves the motion recognition capability. Such methods can be roughly divided into three categories: the first method divides the joints into multiple parts, extracts features from only one part at a time, and combines the features of each part at the top of the model. By analyzing the influence of different part features on the whole body motion, the relationship between body parts can be captured, but it often accompanies a complex classification model, and the integration of the results also increases the computational load. The second method is based on the motion representation of the physical link joints. This method takes the relationship between the skeleton and the joint dependency as a feature to describe human motion, and improves the flexible handling of joints by considering the physical structure of the human body. However, it is not robust to local occlusion caused by motion or shooting, and processing the features of each joint will result in a large amount of computational cost and high-dimensional features. The third method captures more information between joints through deep learning methods, and constructs new edges between joints that are not physically adjacent but related to motion. Such methods can directly use the original skeleton information to automatically discover and create features, and find more valuable structural information, but deep learning methods are usually computationally expensive and are not suitable for online real-time applications.

[0007] In view of the shortcomings of the above algorithms, combined with the application of human skeleton motion recognition in various fields and related work, the current difficulty of the problem lies in the processing of time and space features: the processing in time needs to ensure the consistency of the time and speed of human motion, and the processing in space needs to consider not only the robustness to geometric scale changes (such as translation, rotation and global scaling of the scene), but also the structure (topology) information of the human skeleton. Considering that the representation based on manifold can remove the influence of view point, human body size and motion speed changes, the representation based on the topology of the skeleton can increase the motion constraint information, and the method combining topology and shape information can fully integrate the spatiotemporal features of the skeleton, thereby overcoming the above difficulties. However, an innovative problem is how to integrate the two, and if the human motion features can be constructed on this basis and a suitable classification method is proposed, the recognition accuracy will be greatly improved.

[0008] In the prior art, Chinese invention patent application No. CN202110521607.4 discloses a three-dimensional real-time human posture recognition method, which is based on a hierarchical regressor to capture three-dimensional human motion in real time from depth images. It solves the problem that traditional methods based on random decision forests cannot obtain reasonable and accurate three-dimensional human posture estimation results when facing body occlusion during human motion. It also improves the accuracy and efficiency of three-dimensional human posture estimation. The method includes: S1, using a depth camera to obtain a depth image I (I = 1,..., i); the pixel points of the depth image I are represented as x, and the depth values and three-dimensional points corresponding to the pixel points of the depth image I are represented as d(x) and p, respectively; S2, defining the joint freedom of three-dimensional human posture as a vector q; using a hierarchical regressor for three-dimensional human posture regression, which ensures posture reasonableness and higher accuracy compared to traditional recognition methods; and defining a new three-dimensional human regression target.

[0009] For another example, Chinese invention patent application No. CN202111431658.4 discloses a human motion posture migration method and device, control equipment, and readable storage medium, relating to the field of robot control technology. The application performs three-dimensional human posture recognition on the video content of a target motion video of a target person, obtains three-dimensional human motion posture information of the target motion video, and redirects the three-dimensional human motion posture information of the target motion video to a target humanoid robot according to the joint distribution of the robot body of the target humanoid robot, to obtain robot motion posture information that matches the target humanoid robot, and complete the human motion posture migration job for the target person. This way, the content analysis operation on a regular motion video can achieve a low-cost and scene-limited human motion posture acquisition effect, effectively reducing the implementation cost of the entire human motion posture migration scheme, and expanding the application scope of the human motion posture migration scheme.

[0010] For another example, Chinese invention patent application No. CN202211017362.2 relates to a three-dimensional human motion feature expression method for intelligent behavior recognition, which solves the problems of dimension redundancy or under-expression of existing human motion features based on three-dimensional Cartesian coordinate system data description in machine learning models in the field of computer vision. The proposed three-dimensional human motion feature has a smaller data dimension. Starting from the concepts of joint constraint and freedom in the mechanical field, the human body is regarded as a mechanical motion mechanism with complex degrees of freedom, and the construction of intelligent human behavior recognition features is carried out based on the mechanism motion freedom, to optimize the description of human motion posture. When used in human behavior recognition engineering in computer vision, the proposed three-dimensional human motion feature can accurately and comprehensively express human posture, and can also guarantee the minimum data dimension, to greatly improve the training speed of deep learning models, and support high-speed and accurate personnel behavior recognition and digital twin modeling.

[0011] The above invention patent applications cannot solve the method of combining topological and shape information in order to fully integrate the skeleton space-time characteristics. SUMMARY

[0012] The application provides a three-dimensional human motion recognition method based on trajectory manifold, which can provide a flexible framework for parameterized representation of motion, and also ensures the continuity and scale invariance of motion.

[0013] The application adopts the following technical solutions:

[0014] A three-dimensional human motion recognition method based on trajectory manifold, comprising:

[0015] Step 1, pre-processing the human motion data set, including converting the sample into coordinate data of each joint, and removing noise points;

[0016] Step 2, for a certain action of all samples, performing three B-spline curve fitting on the motion trajectory of each joint, splicing all joint curves in the same sample to form a B-spline curve cluster;

[0017] Step 3, calculating the curve discretization information, obtaining the shape information on Kendall space by removing translation, scaling and rotation operations, and calculating the geodesic distance d K (·,·) between shapes;

[0018] Step 4, calculating the variance of the change of the joint angle with time and the correlation coefficient according to the human skeleton tree structure, normalizing the sum of the variance and the correlation coefficient of each joint to obtain the joint probability w j ;

[0019] Step 5, combining the shape information and topological information of the motion, calculating the distance between each two samples in the data set to generate a distance matrix P;

[0020] Step 6, dividing the data set into a training set and a test set according to the distance, calculating the distance matrix of the training set and generating a kernel, training a ppfSVM classifier according to the distance kernel of the training set and the class label;

[0021] Step 7, calculating the distance from the test sample to each sample in the training set, predicting the label of the test sample in ppfSVM, and outputting the class of the sample.

[0022] Further, in step 1, input three-dimensional skeleton motion data x containing J joint skeleton models, the three-dimensional skeleton motion data x is a set of three-dimensional coordinate discrete points of each joint at different time, the three-dimensional coordinates of each joint are stored in a file separately, and the noise is removed according to the point cloud distribution characteristics of a large number of continuous data points.

[0023] Further, in step 2, the motion trajectory of each joint is fitted with a cubic B-spline curve, and all joint curves are spliced to form a B-spline curve cluster:

[0024] According to the B-spline curve fitting method, the data points are fitted with a cubic B-spline curve where d i (i = 0, 1,..., n) are control vertices, the number of control points is n, N i,3 (u) is a cubic normal B-spline basis function, and the B-spline curve fitting steps are: (a) selecting a uniform parameterization method for data point parameterization; (b) determining the node vector through the re-node endpoint condition and the UAVG technique; (c) solving the control vertex, and using the least square method to iteratively optimize the objective function about the control vertex;

[0025] The J joints that constitute the entire motion of the human skeleton have J B-spline curves, and according to the formula, the curve set can be expressed as a motion trajectory curve cluster C = {c 1 (u),..., c J (u)}, which describes all joint curves contained in the human skeleton.

[0026] Further, in step 3, the joint curve is discretized, scale normalization is used to eliminate the influence of scaling on motion analysis, and Procrustes analysis is used to remove rotation to obtain shapes in Kendall space, and the geodesic distance between shapes is calculated:

[0027] Assume that Z j (j = 1,..., J) represents the information of the discretized B-spline curve of each joint, the mean of the point set is removed to eliminate the influence of the absolute position of the motion in the space from the representation space, scale normalization is used to eliminate the influence of scaling on motion analysis, and Procrustes analysis is used to remove rotation, that is, all shapes in the pre-shape space are removed from rotation, and at this time, the point s j in Kendall space representing the j = 1,..., J human joint motion curve characteristics can be obtained.

[0028] For any two motions x m1 ,x m2 , assume that the shape in Kendall space corresponding to the jth joint trajectory is The geodesic distance of the two motions in the joint trajectory shape is:

[0029]

[0030] The metric of two motions in Kendall space is the sum of geodesic distances of all joints:

[0031]

[0032] Further, in step 4, first calculate the change of each joint angle over time and calculate the variance, giving E as the adjacency matrix representation of the skeleton graph:

[0033]

[0034] For the entire motion dataset, assuming the number of samples is N, for a certain time, the sample average angle set of joint v i (i = 1,...,J) is:

[0035]

[0036] Where: represents the angle of joint v i in the n-th sample.

[0037] For all joints V = {v i |i = 1,2,...,J}, for each same position angle, according to its change over time, calculate the variance (where k takes values v i The number of angles contained). Since some joints have more than one related angle, the overall angle change is represented by the sum of all element variances , and finally the sample average overall angle variance set

[0038] To reflect the hierarchical relationship of related joints, by introducing two joints v p ,v c (p,c = 1,...,J) with parent-child relationship, the correlation coefficient ρ p is used to represent the degree of influence of the parent joint on the child joint, and the hierarchical structure joint probability is represented as:

[0039] p(v i ) = w i ,

[0040] s.t.w i ≥ 0, i = 1,...,J

[0041] To construct the mapping relationship p(v i ), first find all pairs of parent-child joints in the tree structure, then find the angle sequence of the corresponding parent-child joints from the sample average angle time sequence set and calculate the correlation coefficient ρp ;

[0042] For all father-son joints (Where: k is the number of paired parent-child joints), calculate the correlation coefficient according to the formula, and add it to the corresponding parent joint to update the topology information of each parent joint:

[0043]

[0044] Where: c i (i = 1, ..., n) p ) represents all child joints contained in a parent joint, and the parent joint has n p For each child node, and for a leaf node in a tree structure, its joint information is represented as σ'. e =σ e Calculate σ' for all joints i (i = 1, ..., J), then normalize to obtain the probability w of a single joint. i In the topology of human skeletal motion, the joint probabilities after fusing hierarchical information are W = {w1, w2, ..., w...} J The larger the value, the greater the effect of joint movement on other adjacent joints.

[0045] Furthermore, in step 5, the shape information and topological information of the motion are combined. That is, the geodesic distance between corresponding joints of the sample is calculated, and the joint probability is used as a coefficient to multiply the geodesic distance of the joint. For the entire human body, the metric in the motion trajectory manifold combined with the topological information is expressed as:

[0046]

[0047] in: Indicates two actions x m1 ,x m2 Geodesic distance of the j-th joint;

[0048] For all motion data in the dataset, calculate the pairwise distance between each sample, and after normalization, obtain the distance matrix P, where the elements of P are...

[0049] P ij =d NE (x i ,x j ), i,j=1,...,N.

[0050] Furthermore, in step 6, the dataset is divided into training and test sets, the distance matrix of the training set is calculated, a linear kernel function is applied to it to generate a kernel, and ppfSVM (human motion classifier) ​​is trained based on the training set distance matrix kernel and class labels.

[0051] ppfSVM assumes that the training set is given by an N×N dimensional pairwise distance matrix P and a target class set Y for each sample data, where P is a subset of the target class set Y. ij =d(x i ,x j Let i,j=1,...,N be the pairwise distances between samples, and assume that the distance matrix satisfies reflexivity P. ii =0 and symmetry P ij =P ji Then, a linear SVM model can be established, represented by P and the mapping φ(x), to classify samples x, where φ(x)=[F(x,x1),...,F(x,x2)]. m )] T This represents the distance between x and all data in the training set.

[0052] Furthermore, in step 7, the distance matrix of the test set (to each sample in the training set) is input, ppfSVM is used to predict the label of the test sample, and the category of the sample is output.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] 1. The 3D human motion recognition method based on trajectory manifolds described in this invention proposes a manifold construction method based on motion trajectory curve clusters. This constructs a geodesically complete motion trajectory manifold, where the motion curve clusters treat the motion time series as 3D curves, achieving spatiotemporal interchange. B-spline motion trajectory description can achieve continuous representation of motion in the continuous domain using fewer parameters, enabling matching and alignment between different motions. The geodesic distance defined on the motion trajectory manifold based on motion trajectory curve clusters exhibits similarity transformation invariance, providing better descriptiveness for different motion categories.

[0055] 2. The three-dimensional human motion recognition method based on trajectory manifold described in this invention defines a probability-based three-dimensional skeleton motion topology expression and proposes a skeleton motion measurement model that integrates topological information for motion trajectory manifold. Through the topology expression based on hierarchical probability, the influence of each skeleton part and joint on the overall skeleton during the motion process can be analyzed independently, and the constraint influence between adjacent joints is fully considered, thereby more accurately describing the human motion correlation and obtaining a more accurate expression of human motion.

[0056] 3. The 3D human motion recognition method based on trajectory manifold described in this invention proposes a kernel classification method based on motion trajectory manifold to achieve 3D skeleton motion recognition. This trajectory manifold-based 3D human motion recognition method transforms the distance matrix between samples into a proximity function, effectively solving nonlinear classification problems and ensuring that the classification process is unaffected by translation, rotation, and scaling in the skeleton motion data. The algorithm has been experimentally verified using publicly available motion databases, and the proposed method demonstrates higher classification accuracy compared to existing classical methods. Attached Figure Description

[0057] Figure 1 This is an implementation framework diagram of the three-dimensional human motion recognition method based on trajectory manifold described in this invention. Detailed Implementation

[0058] like Figure 1 As shown, the input data of this invention comes from a human motion dataset with skeleton information. The representation of human motion is divided into a manifold representation part and a topological representation part. The generation of B-spline curve clusters and the motion representation part in Kendall space constitute the manifold representation part. The topological extraction part mainly includes generating joint probability information by statistically analyzing joint angles. Finally, the training and classification parts are completed by converting the pairwise distance between samples into a kernel function and applying a ppsSVM classifier.

[0059] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments:

[0060] Detailed description of the technical features involved in the trajectory manifold-based three-dimensional human motion recognition method of this invention:

[0061] 1. Extraction and processing of skeletal motion sequence data:

[0062] For the obtained 3D skeleton information human motion dataset, the coordinate information of each joint is extracted from the TXT file (sample) with motion coordinate information to obtain the joint motion sequence file (TXT file). Noise is removed from all files, that is, data whose coordinate values ​​are located at the origin for a long time are removed.

[0063] 2. Motion trajectory curve family based on B-spline:

[0064] During movement, certain parts of the body may be obscured by objects or other parts, which can affect the coordinate estimation of certain joints and thus affect the accuracy of motion representation. When describing skeletal motion information as geometry, considering that the acquired skeletal motion coordinate data is often dense and noisy, many studies use curve fitting to process joint sequences.

[0065] Definition of motion trajectory curve:

[0066] For a human action, given the number of frames N T The three-dimensional skeleton data, assuming the human body is in t∈{1,2,...,N T The three-dimensional coordinates of the j-th, j=1,...,J joints in frame j are represented as follows: The input sequence is the coordinates of the joint from the first frame to the last frame. To further represent the trajectory of the joint over time, curve c is introduced. j (t), satisfying:

[0067]

[0068] Then the curve C(t) is called the trajectory curve, where Ω is the set of all approximating curves.

[0069] Definition of a family of motion trajectory curves:

[0070] The motion trajectory curve represents the position trajectory of a joint. For a human skeleton model with J joints, the motion curve of the j-th joint (j∈{1,...,J}) can be represented by c. j (t), the motion trajectory curve family is all the curves that make up the motion trajectory curves of all the joints of the human skeleton:

[0071] C = {c 1 (t),...,c J (t)},

[0072] 1) B-spline curve of motion trajectory

[0073] To ensure the continuity and smoothness of motion, it is necessary to select appropriate parametric curves for analysis and processing. B-spline curves possess characteristics such as continuity, differentiability, and local and affine invariance. Furthermore, the local motion pattern can be modified by altering control points. Using B-spline curves to represent motion not only ensures the continuity and smoothness of the movement but also guarantees the consistency of motion speed through parameterization. In this invention, for each joint motion sequence file, according to the B-spline curve fitting method, the data points p... i (i = 0, ..., m) Fit a cubic B-spline curve

[0074]

[0075] Where, d i (i = 0, 1, ..., n) represents the control vertices (number of which is n), u is a parameter, and N i,3(u) is a cubic normalized B-spline basis function. In order to fit the data points into a B-spline curve, the most important thing is to back-calculate the control points. The more control points there are, the better the curve fitting effect will be. B-spline curve fitting will be transformed into solving the linear least squares of the optimal control points. Its essence is the determination of the node vector. The steps of B-spline curve fitting are: (a) parameterization of data points; (b) determination of node vector; (c) finding control vertices.

[0076] Considering that the sequence is sampled at a fixed number of frames, in step (a), a uniform parameterization method is selected during parameterization to obtain the parameter value sequence t. i (i = 0, ..., m); In step (b), based on endpoint interpolation and the curve domain requirements, four-node endpoints are first used, so u0 = u1 = ... = u3 = 0, u n+1 =u n+2 =...=u n+3+1 =1, and then the remaining node vector values ​​are determined using UAVG technology:

[0077]

[0078] For a cubic B-spline fitted curve, when 3 < j ≤ n, the node u is called a node u. 3+j Let j be the j-th internal node;

[0079] For step (c), Piegl provides an algorithm to try to find a B-spline curve:

[0080]

[0081] First, let p0 = c(0), p m = c(1), then the objective function is:

[0082]

[0083] It concerns n-1 control vertices d i The minimum value of (i = 1, ..., n-1) is obtained. The least squares method is applied to calculate the final fitted B-spline curve.

[0084] 2) Motion trajectory curve family based on B-spline

[0085] By following the steps above, a B-spline curve for a single joint can be constructed. The J joints that constitute the entire movement of the human skeleton have J B-spline curves. According to the formula, this set of curves can be represented as a family of motion trajectory curves C = {c 1 (u),...,c J (u)} describes all the joint curves contained in the entire human skeleton.

[0086] 3. Manifold representation of motion trajectory:

[0087] In this step, the present invention will propose a manifold representation method based on motion trajectory based on shape space theory, which represents human motion characteristics in a scale-invariant manner, ensuring the robustness of motion representation to geometric scale changes (such as scene translation, rotation and global scaling). At the same time, combined with shape metric theory, a human motion metric method based on geodesic distance is implemented on the basis of motion representation, which compares the similarity of motion by the distance between trajectory shapes, thereby reducing computational costs.

[0088] 1) Representation of motion trajectory curve families based on Kendall space

[0089] Based on shape space theory, Kendall space's advantage lies in eliminating the influence of translation, scaling, and rotation on curve analysis. Building upon B-spline curves of joint trajectories, this invention proposes a representation model of skeletal motion trajectories in Kendall space. First, the B-spline curves of the moving joints are discretized, ensuring all curves have the same number of shape points (m), and the relevant data is stored. Assuming Z... j This represents a set of three-dimensional coordinates formed by discretizing the B-spline curves of each joint. Here, the motion curve manifold M is given:

[0090] M=R (m×3) {0},Z j =(z j 1,z j 2,...,z j m ),Z j ∈M,

[0091] The influence of the absolute position of motion in space is eliminated from the representation space by removing the mean of the point set:

[0092]

[0093]

[0094] in, This represents the point set after removing translation operations, using scale normalization to eliminate the impact of scaling on motion analysis:

[0095]

[0096] This represents the point set after scaling is removed, resulting in a pre-shape space represented by a high-dimensional sphere. During rotation removal, since the original point set is three-dimensional, Protodyakonov analysis is used to process it, removing rotation from all shapes pairwise in the pre-shape space. This yields the shape (points) s of the human body's j-th joint motion curve feature in Kendall space. j The representation of skeletal motion in Kendall space is given by a set of shapes S = (s 1 ,...,s J )composition;

[0097] 2) Motion trajectory measurement based on geodesic distance

[0098] Since shapes in Kendall space are not limited by translation, scaling, and rotation operations, and there are corresponding measurement methods, this invention combines geodesic distance to solve the problem of measuring and evaluating motion trajectories.

[0099] After a joint motion curve is mapped onto Kendall space, it becomes a point in that space. The human motion curve family is a set of points in Kendall space. For two movements x... m1 ,x m2 First, ensure that the joints represented by the joint indexes are consistent. Assume that the human joint motion curve feature corresponding to joint j∈J is mapped in Kendall space as follows: According to the principles of Riemannian metric, the geodesic distance between the shapes of two motion trajectories is defined as:

[0100]

[0101] The above analysis shows that Riemannian metrics can represent the differences in shape characteristics of different joint curves. By mapping the joint curves corresponding to two actions onto Kendall space and calculating the geodesic distance, a measurement method for the Kendall shape model of human joint motion is further obtained. The geodesic distance is calculated for each joint, and the measurement of the two actions in Kendall space is the sum of the geodesic distances of all joints, as shown in the following formula:

[0102]

[0103] 4. Probability-based topological representation of skeleton motion:

[0104] There are complex interactions and coordination relationships between human joints, which are crucial for accurately analyzing human motion. Therefore, it is necessary to consider the characteristics of the entire human joint more comprehensively in order to analyze human motion more accurately. To solve this problem, topology-based features are used to represent skeletal motion. These features take into account the physical structure of the human body, increase motion constraints, and improve the accuracy of motion representation. However, most of them are currently based on deep learning. Although deep learning methods are powerful, they require a large amount of data to achieve the expected performance. The goal of this invention is to manually construct topology-based feature representations so as to fuse them with distance methods for shape feature representation.

[0105] 1) Kinematic topology representation based on joint angles:

[0106] The human skeleton diagram implicitly contains information about angles. Since joint angles are composed of several joints that are adjacent to each other in the diagram, the spatial relationship between joints is taken into account. Moreover, the degree of joint movement can also be described by the change of angle over time. Based on this, we will start with the angles formed by the connection relationship of adjacent joints, statistically analyze the time change information, and use this as a probability feature to finally construct a topological structure based on the joint probability diagram. This description implicitly connects the bones together, independent of the body's position and orientation, and is unrelated to the size of the skeleton or the angle of the shot.

[0107] For a human skeleton diagram G = (V, E), where V = {v i |i=1,2,...,J} represents the set of all joints, E represents the set of bones between joints, and for any joint point v i First use each v i The angle change statistics are used to characterize the topological information of the joints, and the adjacency matrix representation of the skeleton graph is given:

[0108]

[0109] For any frame of an action, joint v i The angle is expressed as:

[0110]

[0111] In the above formula, Indicates joint v i ,v j The unit vector formed;

[0112] Consider all frames N of the action T , for v i The β-th angle (which takes the value of the angle associated with the joint) and its change over time constitute a set of sequence information. This vector contains both temporal information and spatial joint relationship information. The variance of this set of numbers is calculated to obtain v. i The magnitude of the change in the included angle over time:

[0113]

[0114] Where: β t This represents the angle sought at time t. This represents the average of the set of numbers, since v i There may be more than one relevant perspective regarding v. i To calculate the overall angular change, we need to sum the variances of all elements:

[0115]

[0116] Where: k takes the value of v i The number of angles included.

[0117] For the entire motion dataset, assuming the number of samples is N, the joint features of the entire set are characterized by calculating the sample mean of the joint angle sequence. Specifically, for a given moment, the joint v is calculated. i The set of sample average angles (i = 1, ..., J):

[0118]

[0119] in: v represents the value of the nth sample. i The set consisting of all angles.

[0120] For all joints V = {v i |i=1,2,...,J},will Substituting into the formula, we obtain the set of average population angle changes for the sample. It includes the degree of influence of each joint on the motion in all samples;

[0121] 2) Motion topology construction based on hierarchical probability:

[0122] For certain movements, there exists a certain joint v i The movement of a joint affects the movement of other joints, especially when joint v... i and joint v jThis effect is particularly significant when there are physical connections between them. For example, in a waving motion, the change in hand position is caused by the elbow joint, and the movement trajectory of the hand is caused by the elbow. This relationship is the key to recognizing the action. This hierarchical structure reflects the strong influence between adjacent joints and can be well used to distinguish human movements. In view of this, a spatiotemporal graph topology based on the hierarchical structure is constructed, and all parent-child joint related information in the human skeleton tree structure is used to add supplementary topological information to each joint.

[0123] The human skeleton can also be described using a tree structure. According to human kinematics, the movement of parent joints is often more important than the movement of child joints: the movement of a parent joint may cause the movement of its child joints, but the movement of a child joint cannot affect its parent joint. The tree structure of joints is a hierarchical description of movement in an intuitive way, with each level representing the parent-child relationship between adjacent joints.

[0124] 3) Definition of joint probability in hierarchical structure:

[0125] To fully reflect the hierarchical relationship between the relevant joints, two joints v with a parent-child relationship are introduced. p ,v c The correlation coefficient υ of the angle variation (p,c=1,...,J) p To represent the degree of influence of the parent joint on the child joint, the probability p(v) of the hierarchical joint is expressed as follows. i ) is represented as:

[0126] p(v i ) = w i ,

[0127] stw i ≥0, i=1,...,J

[0128] To construct the mapping relationship p(v) i First, find all pairs of parent-child joints in the tree structure. Let a certain pair of parent-child joints be represented as (v...). p ,v c Then, find the corresponding parent-child joint angle sequence from the sample average angle time series set. And calculate the correlation coefficient for this set of data:

[0129]

[0130] Where: σ p ,σ c Let Cov(θ) represent the variance of the change in the angle between the parent and child joints, respectively. p ,θ c () represents the covariance:

[0131] Cov(θp ,θ c )=E[(θ p -μ p )(θ c -μ c )],

[0132] Where: E[·] is the expectation operator, μ p ,μ c ρ is the population average. p This indicates the importance of the parent joint to the child joint. The larger the value, the greater the influence of the parent joint's movement on the child joint's movement, and it can accurately describe the interaction between the parent and child joints.

[0133] For all father-son joints (Where: k is the number of paired parent-child joints), calculate the correlation coefficient according to the formula, and add the correlation coefficient to the corresponding parent joint. This allows us to update the joint information of each parent joint based on the previous step:

[0134]

[0135] Where: c i (i = 1, ..., n) p ) represents all child joints contained in a parent joint, and the parent joint has n p For each child node, the joint information of a leaf node in the tree structure is represented as σ'. e =σ e Calculate all σ' using the formula i (i = 1, ..., J), and then normalization yields the representation of the joint probability:

[0136]

[0137] The joint probability features after fusing hierarchical information are W = {w1, w2, ..., w J The larger the value, the greater the effect of joint movement on other adjacent joints;

[0138] 5. Motion trajectory shape distance based on fusion joint probability:

[0139] First, the geodesic distance between corresponding joints of the samples is calculated, and the joint probability is multiplied by the geodesic distance of the joint as a coefficient. For the entire human body, after fusing the hierarchical joint probabilities, the distance in the motion trajectory manifold is expressed as:

[0140]

[0141] in: Indicates two actions x m1 ,xm2 For the distance at the i-th joint, if we combine the hierarchical joint probability with the shape representation of Kendall space, and substitute the geodesic distance into the above formula, then the distance between samples can be expressed as:

[0142]

[0143] At this point, each element in the pairwise distance matrix P between samples is P. ij =d NE (x i ,x j ), i,j=1,...,N;

[0144] 6. Kernel classification method based on trajectory manifold:

[0145] This invention proposes a manifold-based classification method that relies on the Riemannian metric (geodesic distance in this invention) between sample data:

[0146] 1) Motion classification based on motion trajectory manifold:

[0147] Based on the applications and related work of human skeleton motion recognition in various fields, the current challenges lie in addressing the following issues: the ability to integrate spatiotemporal information; invariance to changes in viewpoint, human scale, and motion speed; the ability to describe human topological constraints; and robustness to noise such as background or occlusion.

[0148] Despite numerous efforts to address these challenges, a comprehensive solution remains elusive. This invention formulates motion recognition as a problem of calculating the distance matrix between shapes on a manifold that incorporates topological features. Manifold shape representations can remove the influence of viewpoint, human scale, and motion speed variations, while skeleton topological representations can add motion constraint information. The method combining topological and shape information can fully describe the spatiotemporal characteristics of motion. Based on this, a suitable classification algorithm is proposed to achieve motion recognition.

[0149] The manifold-based motion recognition problem is defined as follows: Let the sample set X = {x} i}, i = 1, ..., N, corresponding label Y = {y i}, i=1,...,N,y i For x i The corresponding category label and its value is Y = {1, ..., N} C}, N C If the number of classes is the primary factor, then the classification problem based on Riemannian manifolds can be formulated as follows: for a sample set X in a Riemannian manifold, the goal is to find a function... This enables the labeled training set to classify test samples containing different elements in Riemann space.

[0150] Distance-based classifiers use the distances between a set of labeled training samples and the distances between the test sample and the training samples to estimate the class label of the test sample. Combining this with distances for human motion trajectories on the Kendall manifold, the motion recognition problem on the Riemannian manifold is further refined into motion classification based on the distance matrix between shapes in the motion trajectory manifold:

[0151] Suppose X is the set of shapes in the motion trajectory manifold, Y is the set of action class labels, and d(·) is the distance between shapes on the manifold. Let the distance matrix P represent the pairwise distance between training samples, i.e., P ij =d(x i ,x j ), where: x i ∈X,i=1,2,...,N represents the i-th training sample, y i Let ∈Y, i=1,2,...,N represent the class label corresponding to the i-th sample. Then, the classification problem based on the distance matrix of the motion trajectory can be summarized as: based on the distance d(x,x) between the test sample x and all training samples... i We use i = 1, 2, ..., N to estimate the class label y of the test sample.

[0152] 2) Kernel function based on the distance matrix of the motion trajectory:

[0153] Support Vector Machine (SVM) is a well-known representative of kernel methods. Its kernel is defined as any data type. The kernel is introduced to solve the problem of mapping low-dimensional training data to high-dimensional space to handle linearly inseparable "non-linear" problems. The kernel in SVM needs to guarantee symmetry and positive semi-definiteness (PSD). The pairwise distance matrix is ​​a symmetric matrix, but it is usually not PSD, which means that SVM cannot be used directly. In order to solve the problem that the kernel is not PSD, the Pairwise Proximity Function SVM (ppfSVM) algorithm was developed. It does not have any restrictions on the kernel function such as positive semi-definiteness, differentiability, or continuity.

[0154] The ppfSVM method constructs a set of input data, represented by the distance between each sample and all other samples in the training set, and then applies the SVM to the transformed data. Corresponding to this invention, assuming X = {x i |i=1,...,N} represents N data sets of the motion trajectory curve family, and the nearest neighbor function Representing the distance between two shapes, with no restrictions on the function F (neither symmetry nor continuity), the data mapping φ(x) is defined as:

[0155] φ(x)=[F(x,x1),...,F(x,x N )] T ,

[0156] This mapping will sample Represented as an N-dimensional vector, it contains the sample x. i The distance to all other samples is thus represented by an N×N matrix P, where P is the distance to all other samples. ij =F(x) i ,x j ), i,j=1,...,N, and a linear kernel is used on this data representation:

[0157]

[0158] Then the kernel matrix would become:

[0159] K = PP T ,

[0160] Since the kernel matrix is ​​the square of the distance matrix, it is always positive semi-definite.

[0161] 3) ppfSVM Human Motion Classifier:

[0162] Suppose the training set is given by an N×N dimensional pairwise distance matrix P and the target class set Y for each sample data, where P is a set of pairs ... ij =d(x i ,x j Let i,j=1,...,N be the pairwise distances between samples, and assume that the distance matrix satisfies reflexivity P. ii =0 and symmetry P ij =P ji Then, a linear SVM model represented by P and the mapping φ(x) can be established to classify the sample x.

[0163] SVM learns the nonlinear decision boundary by using kernels. After introducing pairwise proximity functions, the decision rule becomes:

[0164] g(x) = sign(α) T YPφ(x)+ω0),

[0165] Observing the above equation, when the square of the distance matrix is ​​used as the linear kernel, since K is positive semi-definite, solving the convex quadratic optimization problem also becomes a dual problem:

[0166]

[0167]

[0168] Where: I is the penalty coefficient. In SVM, once the optimal coefficient is found, the test sample x' can be evaluated by calculating the penalty coefficient against the elements x in the training set. i The distance d(x',x) i We will use this to categorize them.

[0169] Therefore, the proposed motion classification algorithm follows this process:

[0170] 1. Feature Extraction: Given a human skeleton motion sequence, cubic B-spline curve fitting is performed based on the three-dimensional coordinate time information of each joint. The curve is then removed from translation, scaling, and rotation and mapped to Kendall space to obtain shape information. Based on the skeleton topology, joint angles and related information are statistically analyzed, and joint probabilities are calculated.

[0171] 2. Calculate the distance matrix kernel: Based on the shape information of each pair of actions, calculate the geodesic distance of the corresponding joint shape, multiply it by the corresponding joint probability and accumulate it to obtain the distance between each pair of actions, and generate the distance matrix.

[0172] 3. Training and classification: Calculate the distance matrix kernel of the training set, train ppfSVM based on the kernel and class labels, and predict the label of the test sample.

[0173] 7. Experimental Example

[0174] To verify the effectiveness of this invention, experiments were conducted on three datasets: UTKinect, Florence3D, and MSRAction3D. On the UTKinect and Florence3D datasets, the proposed method achieved 100% recognition accuracy, surpassing all current state-of-the-art motion recognition methods. On the MSRAction3D dataset, it achieved 93.82% recognition accuracy, demonstrating its competitive performance among methods based on manifold and other features. Furthermore, to demonstrate the performance of the proposed classifier, experiments were conducted on the datasets using different classifiers, calculating the classification accuracy of ppfSVM, SVM, KNN, decision tree, XGBoost, and MLP classifiers (all parameters using default values). The experimental results show that the ppfSVM classifier generally outperformed other traditional classifiers, indicating that the classifier constructed in this paper can be effectively used for the proposed motion similarity features on this dataset. Compared with SVM, ppfSVM shows a significant performance improvement in classification results. This is because ppfSVM is a classifier specifically constructed for pairwise distances based on SVM, and is therefore more suitable for the similarity description proposed in this paper.

[0175] Comparative experimental results show that the motion representation and metric proposed in this invention achieve competitive results in motion recognition based on skeleton sequences. The ablation experimental results constitute the contribution of each module in the method: 1) Trajectory representation helps to better encode temporal dynamic information and smooth motion. The Kendall space representation can remove the robustness of rotation and scaling and plays a key role in motion recognition; 2) The topological structure based on joint and skeleton fusion captures useful dependencies between joints, adding more spatial structural information to describe motion, which is also important for the accuracy of motion recognition; 3) The proposed ppfSVM classifier is feasible and effective and can be applied to distance-based motion recognition.

[0176] This invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope. All such changes and modifications fall within the scope of the invention as defined by the appended claims.

Claims

1. A three-dimensional human motion recognition method based on trajectory manifold, characterized in that... Includes the following steps: Step 1: Preprocess the human motion dataset, including converting the samples into coordinate data for each joint and removing noise. Step 2: For a certain action of all samples, perform cubic B-spline curve fitting on the motion trajectory of each joint, and combine all joint curves in the same sample to form a B-spline curve cluster. Step 3: Calculate the discretized information of the curves. By removing translation, scaling, and rotation operations, the shape information in Kendall space is obtained. At the same time, the geodesic distance d between the shapes is calculated. K (·,·); Discretize the joint curves, use scale normalization to eliminate the effect of scaling on motion analysis, remove rotation using Protodyakonov analysis, obtain the shapes in Kendall space, and calculate the geodesic distances between shapes: Assume Z j The information of each joint's B-spline curve is discretized, j = 1, ..., J. The influence of the absolute position of the motion in space is eliminated by removing the mean of the point set. Scale normalization is used to eliminate the effect of scaling on motion analysis. When eliminating rotation, Protodyakonov analysis is used, that is, all shapes in the pre-shape space are pairwise rotated. At this point, the feature of the human joint motion curve of the j = 1, ..., Jth individual can be obtained at a point s in Kendall space. j ; For any two actions x m1 ,x m2 Suppose the shape of the trajectory of the j-th joint in Kendall space is represented as... The geodesic distance between the two motions at this joint trajectory shape is: The measurement of two actions in Kendall space is the sum of the geodesic distances composed of all joint distances: Step 4: Statistically analyze the changes in joint angles over time and calculate the variance. Calculate the correlation coefficient based on the human skeletal tree structure. Sum the variance and correlation coefficient for each joint and normalize the result to obtain the joint probability w. j ; Step 5: Combine the shape information and topological information of the motion to calculate the distance between each pair of samples in the dataset. Generate distance matrix P; Step 6: Divide the dataset into training and test sets according to the distance, calculate the distance matrix of the training set and generate the kernel, and train the ppfSVM classifier based on the distance kernel of the training set and the class labels; Step 7: Calculate the distance from the test sample to each sample in the training set, predict the label of the test sample in ppfSVM, and output the category of the sample.

2. The three-dimensional human motion recognition method based on trajectory manifold according to claim 1, characterized in that, In step 1, the input is a 3D skeleton motion data x containing J joint skeleton models. The 3D skeleton motion data x is a set of discrete 3D coordinate points of each joint at different times. The 3D coordinates of each joint are stored in a separate file. Noise is removed based on the point cloud distribution characteristics of a large amount of data with the origin as the origin.

3. The three-dimensional human motion recognition method based on trajectory manifold according to claim 1, characterized in that, In step 2, the motion trajectory of each joint is fitted with a cubic B-spline curve, and all joint curves are combined to form a B-spline curve family: Based on the B-spline curve fitting method, a cubic B-spline curve is fitted to the data points. Where d i To control vertices, i = 0, 1, ..., n, the number of control points is n, N i,3 (u) is a cubic normalized B-spline basis function. The steps for fitting the B-spline curve are as follows: (a) Select a uniform parameterization method to parameterize the data points; (b) Determine the node vector through the heavy node endpoint conditions and UAVG technology; (c) Find the control vertices and perform iterative optimization of the objective function with respect to the control vertices using the least squares method. The J joints that constitute the entire movement of the human skeleton have J B-spline curves. According to the formula, this set of curves can be represented as a family of motion trajectory curves C = {c 1 (u),...,c J (u)} describes all the joint curves contained in the entire human skeleton.

4. The three-dimensional human motion recognition method based on trajectory manifold according to claim 1, characterized in that, In step 4, the variation of the angle between each joint over time is first calculated and the variance is calculated, giving E as the adjacency matrix representation of the skeleton graph: For the entire motion dataset, assuming the number of samples is N, for a certain moment, joint v i The set of sample average angles for i = 1, ..., J is: in: In the nth sample, joint v i The set of all angles; For all joints V = {v i For the angle |i=1,2,...,J} at each corresponding position, calculate its variance based on its variation over time. Where k takes the value of v i The number of angles included, given that some joints have more than one associated angle, is calculated by summing the variances of all elements. This is used to represent the sample mean population angular variance set. To illustrate the hierarchical relationship between related joints, two joints v with a parent-child relationship are introduced. p v c The correlation coefficient ρ of the angle change between p, c = 1, ..., J p To represent the degree of influence of the parent joint on the child joint, the probability of the hierarchical joint is expressed as: p(v i )=w i , s.t.w i ≥0,i=1,...,J To construct the mapping relationship p(v) i First, find all paired parent-child joints in the tree structure, and then find the angle sequence of the corresponding parent-child joint from the set of average angle time series of the samples. And calculate the correlation coefficient ρ for this set of data. p ; For all father-son joints Where: k is the number of paired parent-child joints. The correlation coefficient is calculated according to the formula and added to the corresponding parent joint, which allows the topological information of each parent joint to be updated. Where: c i This represents all child joints contained in the parent joint, i = 1, ..., n p And the parent joint has n p For each child node, and for a leaf node in the tree structure, its joint is consistent with the original information, denoted as σ′. e =σ e Calculate σ for all joints i ', i = 1, ..., J, and then normalize to obtain the probability w of a single joint. i In the topology of human skeletal motion, the joint probabilities after fusing hierarchical information are W = {w1, w2, ..., w...} J The larger the value, the greater the effect of joint movement on other adjacent joints.

5. The three-dimensional human motion recognition method based on trajectory manifold according to claim 1, characterized in that, In step 5, the shape information and topological information of the motion are combined. That is, the geodesic distance between corresponding joints of the sample is calculated, and the joint probability is used as a coefficient to multiply the geodesic distance of the joint. For the entire human body, the metric in the motion trajectory manifold combined with the topological information is expressed as: in: Indicates two actions x m1 ,x m2 Geodesic distance of the j-th joint; For all motion data in the dataset, calculate the pairwise distance between each sample, and after normalization, obtain the distance matrix P, where the elements of P are... P ij =d NE (x i ,x j ),i,j=1,...,N。 6. The three-dimensional human motion recognition method based on trajectory manifold according to claim 1, characterized in that, In step 6, the dataset is divided into training and test sets, the distance matrix of the training set is calculated, a linear kernel function is applied to it to generate a kernel, and ppfSVM is trained based on the training set distance matrix kernel and class labels. ppfSVM assumes that the training set is given by an N×N dimensional pairwise distance matrix P and a target class set Y for each sample data, where P is a subset of the target class set Y. ij =d(x i ,x j Let i,j=1,...,N be the pairwise distances between samples, and assume that the distance matrix satisfies reflexivity P. ii =0 and symmetry P ij =P ji Then, a linear SVM model can be established, represented by P and the mapping φ(x), to classify samples x, where φ(x)=[F(x,x1),...,F(x,x2)]. m )] Τ This represents the distance between x and all data in the training set.

7. The three-dimensional human motion recognition method based on trajectory manifold according to claim 1, characterized in that, In step 7, the distance matrix from the test set to each sample in the training set is input, ppfSVM is used to predict the label of the test sample, and the category of the sample is output.

Images