Monocular human motion trajectory estimation method and system based on graph neural network

Abstract

The application provides a monocular human motion trajectory estimation method and system based on a graph neural network, and relates to the technical field of human motion trajectory estimation. The application designs a human motion trajectory estimation network based on a graph neural network, uses multiple stacked space-time modules to mine time and space features in human motion, wherein the space model uses a graph neural network to explore joint features under human topological structure priori and uses limb grouping to explore spatial correlation of joints in the limb group, the time model uses a Transformer structure to model each joint motion trajectory and uses limb grouping to explore time trajectory features of the whole limb group, so that more accurate trajectory estimation is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human motion trajectory estimation, in particular to a monocular human motion trajectory estimation method and system based on a graph neural network. BACKGROUND

[0002] In the field of action standard degree discrimination during rehabilitation training of patients or ordinary people's movement, human motion trajectory needs to be estimated.

[0003] At present, human motion trajectory estimation has the following defects:

[0004] 1. Most of the methods based on trajectory tracking are based on a target detection framework, which identifies the candidate region coordinates of each object to be detected and the human body in the video image, and does not give the joint trajectory of the human body, so it cannot perform downstream tasks such as action trajectory estimation of specific body parts such as limbs.

[0005] 2. The existing 3D human pose estimation method based on video can obtain the joint coordinates of each frame in the video image, but basically does not explore the time trajectory generated by each joint.

[0006] 3. The existing good 3D human pose estimation method based on video is mostly based on traditional architectures such as CNN and Transformer, and rarely involves constructing a graph neural network through a human body graph topology. SUMMARY

[0007] (I) Technical problems solved

[0008] In view of the deficiencies of the prior art, the present application provides a monocular human motion trajectory estimation method and system based on a graph neural network, which solves the problem of poor effect of existing human motion trajectory estimation technology.

[0009] (II) Technical solutions

[0010] To achieve the above purpose, the present application is realized by the following technical solutions:

[0011] A monocular human motion trajectory estimation method based on a graph neural network, the method comprising:

[0012] Obtaining a data set containing monocular video and ground truth labels, the ground truth labels including: ground truth two-dimensional coordinates of joints and ground truth three-dimensional coordinates of joints;

[0013] Training a human motion trajectory estimation model based on the data set;

[0014] Obtaining the human motion trajectory of the monocular video to be estimated based on the trained human motion trajectory estimation model;

[0015] The human motion trajectory estimation model comprises:

[0016] The two-dimensional posture sequence acquisition module is configured to acquire a two-dimensional posture sequence in the monocular video, wherein the two-dimensional posture sequence comprises two-dimensional coordinates of N joints sorted in time sequence of the monocular video.

[0017] The first dimension lifting module is configured to lift a dimension of a feature of the two-dimensional coordinates of the joints of the two-dimensional posture sequence to a preset dimension C.

[0018] The L sequentially connected spatio-temporal blocks each comprise a spatial block and a temporal block sequentially connected, wherein the spatial block is configured to learn spatial features, and the temporal block is configured to learn temporal features.

[0019] The low-pass prediction module is configured to acquire the three-dimensional coordinates of the joints based on a feature output by the last temporal block.

[0020] Further, each spatial block sequentially comprises a multi-hop graph attention module and a spatial limb attention module.

[0021] The multi-hop graph attention module comprises:

[0022]

[0023]

[0024]

[0025] wherein,

[0026] represents a feature latent representation of the kth hop;

[0027] A k represents an adjacency matrix of the kth hop;

[0028] X g represents an original input feature of each multi-hop graph attention module;

[0029] W k represents a learnable weight matrix;

[0030] α represents a multi-feature latent representation weight of each node;

[0031] W α represents a mapping function;

[0032] represents that the feature latent representations of each hop are concatenated with the original input feature C X g in the feature dimension;

[0033] Xgo represents the updated node feature of the multi-hop graph attention module;

[0034] β represents an attenuation factor;

[0035] ⊙ represents element multiplication;

[0036] represents the updated node feature of the multi-hop graph attention module; and the original input feature x g are stacked;

[0037] The spatial limb attention module is:

[0038] X p = X go [group p ]

[0039]

[0040] X ps = X ps + MLP (X ps )

[0041] wherein,

[0042] X p represents the feature vector of the pth group obtained by dividing the updated node feature according to the limb grouping graph;

[0043] X ps represents the feature after the intragroup self-attention learning of the pth group;

[0044] represents different mapping functions corresponding to the pth group;

[0045] T represents a transpose function;

[0046] softmax represents a normalized exponential function;

[0047] MLP represents a multi-layer perceptron;

[0048] represents the X p after the feature distribution normalization of X p by layer normalization.

[0049] Further, each of the time blocks comprises a time limb attention module, a second dimension increasing module, at least one trajectory attention module, and a first dimension decreasing module connected in sequence;

[0050] The time limb attention module is:

[0051]

[0052] X pt = X pt + MLP(X pt )

[0053] wherein,

[0054] X pt represents the output of the time-limb attention module;

[0055] represents X ps after feature distribution normalization on X ps by layer normalization;

[0056] The second dimension increasing module is configured to increase the dimension of the feature X T to M times of the preset dimension, and the feature X T represents a feature obtained by rearranging X pt according to the corresponding joint index and performing dimension transformation on the time dimension and the joint dimension;

[0057] The trajectory attention module is:

[0058]

[0059] X T = X T + dctFC(X T )

[0060] wherein,

[0061] represents X T after feature distribution normalization on X T by layer normalization;

[0062] The dctFC represents a full connection layer:

[0063] dctFC = IDCT(FC(DCT(X T )))

[0064] DCT represents discrete cosine transform, IDCT represents inverse discrete cosine transform, FC represents a full connection layer, and the output dimension and the input dimension of the full connection layer are both M*C;

[0065] The first dimension decreasing module is configured to map the dimension of X T output by the trajectory attention module back to the preset dimension C.

[0066] Further, the low-pass prediction module comprises a full connection layer and a linear layer; the full connection layer is dctFC; and the linear layer is configured to map the dimension from the preset dimension C to a target dimension 3.

[0067] Further, in training the model, the loss function comprises a joint position loss error L pos , a trajectory consistency loss error L c , and a trajectory similarity loss error L s .

[0068] The joint position loss error L pos comprises an Euclidean distance constraint L pos2 and an L1 norm constraint L pos1 .

[0069] The Euclidean distance constraint is:

[0070]

[0071] The L1 norm constraint is:

[0072]

[0073] The trajectory consistency loss error L c is:

[0074]

[0075] The trajectory similarity loss error L s is:

[0076]

[0077] wherein,

[0078] T represents the number of video frames;

[0079] N represents a predefined number of joints;

[0080] y 3d is the true value three-dimensional coordinates of the joints;

[0081] y

[0082] represents the three-dimensional coordinates of the nth indexed joint in the t-th frame output by the model;

[0083] DCT represents the discrete cosine transform.

[0084] Further, the first dimension increasing module is an embedding layer; and the second dimension increasing module and the first dimension decreasing module are both linear layers.

[0085] A monocular human motion trajectory estimation system based on a graph neural network, comprising:

[0086] a data acquisition module configured to acquire a dataset containing monocular videos and ground truth labels, the ground truth labels including ground truth two-dimensional coordinates of joints and ground truth three-dimensional coordinates of joints;

[0087] a model training module configured to train a human motion trajectory estimation model based on the dataset;

[0088] a human motion trajectory estimation module configured to acquire a human motion trajectory of a monocular video to be estimated based on the trained human motion trajectory estimation model;

[0089] wherein the human motion trajectory estimation model comprises:

[0090] a two-dimensional pose sequence acquisition module configured to acquire a two-dimensional pose sequence in the monocular video, the two-dimensional pose sequence including two-dimensional coordinates of N joints sorted in a time sequence of the monocular video;

[0091] a first dimension lifting module configured to lift a dimension of a feature of the two-dimensional coordinates of the joints of the two-dimensional pose sequence to a preset dimension C;

[0092] L sequentially connected spatio-temporal blocks, each of the spatio-temporal blocks including a spatial block and a temporal block sequentially connected, wherein the spatial block is configured to learn spatial features, and the temporal block is configured to learn temporal features;

[0093] a low-pass prediction module configured to acquire three-dimensional coordinates of the joints based on a feature output by a last temporal block.

[0094] Further, each of the spatial blocks sequentially includes a multi-hop graph attention module and a spatial limb attention module;

[0095] the multi-hop graph attention module is:

[0096]

[0097]

[0098]

[0099] wherein,

[0100] represents a feature latent representation of the kth hop;

[0101] A k represents an adjacency matrix of the kth hop;

[0102] X g represents an original input feature of each multi-hop graph attention module;

[0103] Wk represents a learnable weight matrix;

[0104] α represents a multi-feature latent representation weight of each node;

[0105] W α represents a mapping function;

[0106] represents the feature latent representation of each hop and the original input feature X g is spliced in the feature dimension;

[0107] X go represents the updated node feature of the multi-hop graph attention module;

[0108] β represents a decay factor;

[0109] ⊙ represents element product;

[0110] represents the feature latent representation of each hop and the original input feature X g is stacked;

[0111] The spatial limb attention module is:

[0112] X p = X go [group p ]

[0113]

[0114] X ps = X ps +MLP(X ps )

[0115] wherein,

[0116] X p represents the feature vector of the pth group according to the limb grouping graph division;

[0117] X ps represents the feature after the intra-group self-attention learning of the pth group;

[0118] represents a different mapping function corresponding to the pth group;

[0119] T represents a transpose function;

[0120] softmax represents a normalized exponential function;

[0121] MLP represents a multi-layer perceptron;

[0122] This indicates that X is normalized by layer. p X after feature distribution normalization p ;

[0123] Each time block includes a time-limb attention module, a second dimension-upgrading module, at least one trajectory attention module, and a first dimension-reducing module connected in sequence.

[0124] The time-body attention module is:

[0125]

[0126] X pt =X pt +MLP(X pt )

[0127] in,

[0128] X pt This represents the output of the Time-Body Attention module;

[0129] This indicates that X is normalized by layer. ps X after feature distribution normalization ps ;

[0130] The second dimension-up module is used to transform feature X T The dimension is increased to M times the preset dimension, and the feature X T Indicates X pt The features obtained by rearranging according to the corresponding joint index and transforming the time dimension and joint dimension;

[0131] The trajectory attention module is:

[0132]

[0133] X T =X T +dctFC(X T )

[0134] in,

[0135] X Tnorm This indicates that X is normalized by layer. T X after feature distribution normalization T ;

[0136] dctFC indicates that the fully connected layer is:

[0137] dctFC = IDCT(FC(DCT(X) T )))

[0138] DCT stands for Discrete Cosine Transform, IDCT stands for Inverse Discrete Cosine Transform, and FC stands for Fully Connected Layer. The output dimension and input dimension of a fully connected layer are both M*C.

[0139] The first dimensionality reduction module is used to convert the X output by the trajectory attention module T The dimension is mapped back to the preset dimension C.

[0140] Furthermore, the low-pass prediction module includes a fully connected layer and a linear layer; the fully connected layer is a dctFC; the linear layer is used to map the dimension from the preset dimension C to the target dimension 3.

[0141] Furthermore, during model training, the loss function includes the joint position loss error L. pos Trajectory consistency loss error L c And trajectory similarity loss error L s ;

[0142] Wherein, the joint position loss error L pos Including Euclidean distance constraint L pos2 and L1 norm constraint L pos1 ;

[0143] The Euclidean distance constraint is as follows:

[0144]

[0145] The L1 norm constraint is:

[0146]

[0147] The trajectory consistency loss error L c for:

[0148]

[0149] The trajectory similarity loss error L s for:

[0150]

[0151] Where T represents the number of video frames;

[0152] N represents the predefined number of joints;

[0153] y 3d These are the true three-dimensional coordinates of the joint;

[0154] Represents the true three-dimensional coordinates of the nth index joint in frame t, where n = 1, 2, ..., N;

[0155] represents the third frame of the t-th index joint three-dimensional coordinates output by the model;

[0156] DCT represents a discrete cosine transform.

[0157] (III) Advantages

[0158] The application provides a monocular human motion trajectory estimation method and system based on a graph neural network.

[0159] 1. The application designs a human motion trajectory estimation network based on a graph neural network, which uses multiple stacked space-time modules to mine the time and space features in human motion. The spatial model uses a graph neural network to explore the joint features under the prior of human topological structure and uses limb grouping to explore the spatial correlation of joints within the limb group. The time model uses a Transformer structure to model the motion trajectory of each joint and uses limb grouping to explore the time trajectory features of the entire limb group. This spatial graph topology and time sequence Transformer modeling can better utilize the spatial and temporal characteristics of human motion, while supplementing the human structure prior grouping, more completely utilizing all information of human motion, and achieving more accurate trajectory estimation.

[0160] 2. Existing graph neural network-based spatial feature learning only aggregates first-order neighbor information or directly aggregates multi-order neighbor information. The multi-hop graph attention module designed in the application aggregates each order of neighbors of a node into a latent representation, and calculates the weighted aggregation weight of each order of latent representation based on the latent representation, so that the spatial feature representation in human motion is more accurate, and the time trajectory based on space is more easily explored. Not only does it better develop the idea of graph neural network node aggregation, but it also has high computational efficiency.

[0161] 3. Existing research methods do not study the spatial correlation of joints within the limb group after grouping, while the application proposes that the nodes within the limb group have strong spatial correlation, which enhances the learning of human joint spatial structure feature representation of the model. In addition, the application designs a center-based human skeleton topology graph based on the grouping structure, which enhances the center stability of the human skeleton topology graph to obtain a more stable motion trajectory.

[0162] 4、The existing trajectory estimation research method is mostly based on the spatial position constraint of each node in the trajectory in the loss constraint of the trajectory, the application designs a frequency domain coefficient-based trajectory similarity error loss function based on the frequency domain-based trajectory representation of the trajectory estimation task and the frequency domain constraint to estimate the trajectory to make it more similar to the original trajectory, and adds frequency domain feature learning in the time trajectory attention module and the low-pass prediction module to further improve the trajectory feature learning, improves the similarity of the predicted trajectory of the model, and realizes high smoothness and high precision of the trajectory estimation task. BRIEF DESCRIPTION OF DRAWINGS

[0163] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0164] Figure 1 The flowchart of the present application;

[0165] Figure 2 The structure diagram of the human motion trajectory estimation model of the present application;

[0166] Figure 3 The structure diagram of the spatiotemporal block of the present application;

[0167] Figure 4 The center-based human skeleton topology of the present application;

[0168] Figure 5 The limb grouping diagram of the present application. DETAILED DESCRIPTION

[0169] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application is described clearly and completely. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0170] The embodiments of the present application provide a monocular human motion trajectory estimation method and system based on graph neural network, which solves the problem of poor effect of existing human motion trajectory estimation technology.

[0171] In order to better understand the above technical solutions, the above technical solutions will be described in detail in combination with the drawings in the specification and specific embodiments.

[0172] Embodiment 1:

[0173] As Figure 1 shown, the application provides a monocular human motion trajectory estimation method based on a graph neural network, which comprises the following steps:

[0174] obtaining a data set containing monocular video and ground truth labels, wherein the ground truth labels include ground truth two-dimensional coordinates of joints and ground truth three-dimensional coordinates of joints;

[0175] training a human motion trajectory estimation model based on the data set;

[0176] obtaining a human motion trajectory of a monocular video to be estimated based on the trained human motion trajectory estimation model;

[0177] wherein the human motion trajectory estimation model comprises:

[0178] a two-dimensional pose sequence acquisition module, configured to acquire a two-dimensional pose sequence in the monocular video; the two-dimensional pose sequence comprises two-dimensional coordinates of N joints sorted in time sequence of the monocular video;

[0179] a first dimension lifting module, configured to lift the dimension of the features of the two-dimensional coordinates of the joints of the two-dimensional pose sequence to a preset dimension C;

[0180] L sequentially connected spatio-temporal blocks, and each spatio-temporal block comprises a spatial block and a temporal block connected in sequence; wherein the spatial block is configured to learn spatial features; and the temporal block is configured to learn temporal features;

[0181] a low-pass prediction module, configured to acquire three-dimensional coordinates of joints based on features output by the last temporal block.

[0182] The embodiment has the following beneficial effects:

[0183] The application designs a human motion trajectory estimation network based on a graph neural network, uses a plurality of stacked spatio-temporal blocks to mine time and space features in human motion, wherein the spatial model uses a graph neural network to explore joint features under human topological structure priori and uses limb grouping to explore spatial correlation of joints in the limb group, the temporal model uses a Transformer structure to model motion trajectories of the joints and uses limb grouping to explore time trajectory features of the limb group as a whole. This spatial graph topological structure and time sequence Transformer modeling can better utilize the characteristics of human motion in space and time, while supplementing the human structure prior grouping, more completely utilizing all information of human motion, and realizing more accurate trajectory estimation.

[0184] The implementation process of the embodiment of the application will be described in detail as follows:

[0185] An applicable scenario of the embodiment is that when a patient is performing rehabilitation training or an ordinary person is exercising, the motion of the patient or the ordinary person needs to be judged, and at this time, the motion trajectory of the human body in the video needs to be recognized.

[0186] In the embodiment, the monocular video refers to a video containing a human body motion picture collected by a monocular camera module.

[0187] The human body motion trajectory refers to the motion trajectory of N joints (N = 17 in the embodiment) of the human body in a real three-dimensional coordinate system.

[0188] The embodiment is implemented through the following steps:

[0189] S1, obtaining a data set containing a monocular video and a true value label.

[0190] In specific implementation, the data set can be an open source data set containing a true value label or a self-established data set, each data set containing a plurality of monocular videos, and each monocular video corresponding to a group of true value labels, the true value label including a true value two-dimensional coordinate of a joint and a true value three-dimensional coordinate of the joint. The two-dimensional coordinate of the joint is the position of the joint in the picture of the monocular video, and the three-dimensional coordinate of the joint represents the position of the joint in the real three-dimensional coordinate system.

[0191] S2, training a human body motion trajectory estimation model based on the data set.

[0192] In specific implementation, in order to better meet the use scenario of the embodiment, the inventors have designed a human body motion trajectory estimation model for identifying the human body motion trajectory in the monocular video.

[0193] Specifically, the structure of the human body motion trajectory estimation model is as shown in Figure 2 and sequentially includes:

[0194] 1) a two-dimensional posture sequence acquisition module, used for extracting a two-dimensional posture sequence from the monocular video; the two-dimensional posture sequence is a two-dimensional coordinate x 2d ∈(T,N,2) of N joints sorted in time sequence of the monocular video, serving as an input for subsequent estimation of three-dimensional coordinates of the human body, wherein T represents the total number of frames of a monocular video, N represents a predefined number of human body joints, and 2 represents the number of dimensions.

[0195] The embodiment does not make specific limitation on the specific structure and implementation mode of the two-dimensional posture sequence acquisition module, for example, the module can use an existing Cascaded Pyramid Network (CPN), HRNet, mediapipe, etc. to realize the function.

[0196] 2) a first dimension lifting module, configured to lift the dimension of the joint two-dimensional coordinates x 2d from 2 to a preset dimension C of the features of ∈(T, N, 2); the preset dimension C can be set according to the trade-off of model efficiency, parameter quantity, inference speed, etc. to meet the subsequent model exploration in a high-dimensional space.

[0197] Specifically, the first dimension lifting module is an embedding layer, and the high-dimensional features after dimension lifting can be expressed as x emb ∈(T, N, C).

[0198] 3) L sequentially connected spatio-temporal blocks, configured to alternately learn spatial and temporal features.

[0199] As shown in Figure 3 , each spatio-temporal block further includes a spatial block and a temporal block connected in sequence.

[0200] The spatial block is configured to learn spatial features; each spatial block includes a multi-hop graph attention module based on a graph neural network and a spatial limb attention module based on a limb part; the multi-hop graph attention module is configured to learn spatial human topological structure features, and the spatial limb attention module is configured to learn motion correlation within different limbs in space.

[0201] The multi-hop graph attention module can be expressed as:

[0202]

[0203]

[0204]

[0205] wherein,

[0206] represents the k-th hop feature latent representation;

[0207] A k represents the k-th hop adjacency matrix;

[0208] X g represents the original input features of each multi-hop graph attention module; the first spatio-temporal block in the human motion trajectory estimation model takes the high-dimensional features x emb ∈(T, N, C) as the original input features X g ;

[0209] W k ∈R C×C, denotes a learnable weight matrix;

[0210] denotes the multi-hop feature latent representation weight of each node;

[0211] W α denotes a mapping function;

[0212] denotes the multi-hop feature latent representation of each hop; and the original input feature X g are spliced in the feature dimension;

[0213] X go denotes the updated node feature of the multi-hop graph attention module;

[0214] denotes a decay factor;

[0215] denotes element product;

[0216] denotes the multi-hop feature latent representation of each hop; and the original input feature X g are stacked.

[0217] denotes that the multi-hop feature latent representation is obtained according to the center-based human body skeleton topology graph as shown in Figure 4 ; the center-based human body skeleton topology graph is a human body skeleton topology graph adding two virtual bones of limb starting nodes (i.e. left and right shoulder and left and right hip nodes) and spine joints and hip nodes and chest nodes, and the joint index and joint are: 0-hip, 1-left hip, 2-left knee, 3-left foot, 4-right hip, 5-right knee, 6-right foot, 7-spine, 8-chest, 9-nose, 10-head, 11-right shoulder, 12-right elbow, 13-right wrist, 14-left shoulder, 15-left elbow, 16-left wrist.

[0218] denotes that the multi-hop feature latent representation weight of each node is calculated;

[0219] Due to the characteristics of the graph representation in the human body topology structure, i.e. the correlation between the k-hop neighbor of a node and the node decays as k increases, in the embodiment, a decay factor β is designed to constrain the correlation, and the decay coefficient is 0.8, for example, when k=3, β=[1, 0.8, 0.8 2 , 0.8 3 ].

[0220] denotes that the element product of the decay factor and the multi-hop feature latent representation weight is multiplied by the multi-hop feature latent representation in the corresponding dimension to obtain the updated node feature X go as the output of the multi-hop graph attention module.

[0221] The spatial limb attention module can be represented as:

[0222] X p =X go [group p ]

[0223]

[0224] X ps =X ps +MLP(X ps )

[0225] in,

[0226] X p =X go [group p ] indicates as Figure 5 The updated node features X shown are based on the limb grouping map. go Divide into P groups to allow for attentional learning within each group to study the spatial movement correlations between different limbs. Therefore, X p Represents the eigenvector of the p-th group;

[0227] X ps This represents the characteristics of group p after it has undergone in-group self-attention learning.

[0228] Indicate the different mapping functions corresponding to the p-th group;

[0229] T represents the transpose function;

[0230] softmax represents the normalization exponential function; its function is to normalize the output to meet the feature weighting requirements.

[0231] MLP stands for Multilayer Perceptron; MLP is a two-layer multilayer perceptron, where the feature dimension of the middle layer is consistent with the original dimension, and it uses layer normalization functions and rectified linear units.

[0232] This indicates that X is normalized by layer. p X after feature distribution normalization p .

[0233] After the operation on the space block is completed, do not put all X... ps Instead of returning to the original index, the input is given to the time block for feature learning in the time limb attention module.

[0234] The time block is used for time feature learning, and the time block comprises: a time limb attention module based on an attention mechanism, a second dimension increasing module, at least one trajectory attention module based on a Transformer, and a first dimension decreasing module.

[0235] The time limb attention module can be represented as:

[0236]

[0237] X pt =X pt +MLP(X pt )

[0238] wherein,

[0239] X pt represents an output of the time limb attention module;

[0240] represents X ps after feature distribution normalization of X ps by layer normalization;

[0241] Unlike the spatial limb attention module which focuses on the correlation between nodes within each group, the time limb attention module regards each group as a whole and explores the motion features of the entire group over time.

[0242] The second dimension increasing module is configured to increase the dimension of the feature X T to M times of a preset dimension, for example, M = 2, to meet the feature learning of long sequence trajectory data; specifically, the second dimension increasing module is a linear layer, and the X T ∈(N,T,C) represents a feature obtained by rearranging X pt according to the corresponding joint index and performing dimension transformation on the time dimension and the joint dimension, which is completed before dimension increasing.

[0243] The trajectory attention module can be represented as:

[0244]

[0245] X T =X T +dctFC(X T )

[0246] wherein,

[0247] X Tnorm represents X T after feature distribution normalization of X T by layer normalization;

[0248] dctFC represents a fully connected layer designed for time trajectory, which can be expressed as:

[0249] dctFC = IDCT (FC (DCT (X T ))

[0250] DCT represents a discrete cosine transform, IDCT represents an inverse discrete cosine transform, FC represents a fully connected layer, and the output dimension and the input dimension of the fully connected layer are both M*C;

[0251] The first dimension reduction module after the trajectory attention module is used to map the dimension of X T output by the trajectory attention module back to the preset dimension C to meet the long sequence trajectory data feature learning; specifically, the first dimension reduction module is a linear layer.

[0252] The L spatio-temporal blocks alternately perform in this way, so that the spatial and temporal features are alternately fused.

[0253] 4) a low-pass prediction module for obtaining three-dimensional coordinates x 3d ∈(T, N, 3) of each joint based on the features output by the last time block.

[0254] Specifically, the low-pass prediction module includes a fully connected layer and a linear layer; wherein the fully connected layer is a fully connected layer designed for time trajectory dctFC; dctFC still maintains the data dimension as the preset dimension C, and the linear layer maps the feature dimension from the preset dimension C to the target dimension 3.

[0255] And when training the model, the loss function includes joint position loss error L pos , trajectory consistency loss error L c and trajectory similarity loss error L s .

[0256] The joint position loss error L pos includes the Euclidean distance constraint L pos2 and the L1 norm constraint L pos1 , so that the model gets better prediction results for the position of each joint at each time.

[0257] The Euclidean distance distance constraint is:

[0258]

[0259] The L1 norm constraint is:

[0260]

[0261] The trajectory consistency loss error L c, which is used to constrain the Euclidean distance of the trajectory of each joint predicted by the model to avoid abnormal outliers and obtain smoother trajectories, and can be expressed as:

[0262]

[0263] The trajectory similarity loss error L s , which is used to compare the similarity of the trajectory of each joint predicted by the model and the true value trajectory, and constrain the predicted trajectory to be similar to the true value trajectory, since the predicted trajectory and the true value trajectory are the predicted value and the true value of the same trajectory, and the design of the point pair distance loss error L pos and the key point position loss error L s based on the difference of the frequency domain coefficient of the discrete cosine transform is coincided, the trajectory similarity loss error L s of the embodiment based on the difference of the frequency domain coefficient of the discrete cosine transform can be expressed as:

[0264]

[0265] Wherein, T represents the number of video frames; N represents the number of predefined joints; y 3d is the true value three-dimensional coordinates of the joint; represents the true value three-dimensional coordinates of the nth index joint of the tth frame, n = 1, 2,..., N;

[0266] represents the three-dimensional coordinates of the nth index joint of the tth frame output by the model;

[0267] DCT represents the discrete cosine transform, which calculates the discrete cosine transform coefficient for each coordinate system under each joint respectively, and the calculation formula is:

[0268]

[0269] Wherein,

[0270] C a,b represents the discrete cosine transform coefficient;

[0271] δ is a constant, and when b is 1, δ is Otherwise, δ is

[0272] x a,t represents the value of the three-dimensional coordinate system component of the a th joint at the t th time;

[0273] b represents the b th discrete cosine transform coefficient;

[0274] a represents the a th three-dimensional coordinate system component, and there are 3N in total;

[0275] x represents the input of DCT, that is, x 3d or y 3d .

[0276] S3, obtaining the human motion trajectory of the monocular video to be estimated based on the trained human motion trajectory estimation model.

[0277] Embodiment 2:

[0278] A monocular human motion trajectory estimation system based on a graph neural network, comprising:

[0279] A data acquisition module is configured to acquire a data set containing a monocular video and a true value label, wherein the true value label comprises true value two-dimensional coordinates of joints and true value three-dimensional coordinates of joints.

[0280] A model training module is configured to train a human motion trajectory estimation model based on the data set.

[0281] A human motion trajectory estimation module is configured to obtain the human motion trajectory of the monocular video to be estimated based on the trained human motion trajectory estimation model.

[0282] The human motion trajectory estimation model comprises:

[0283] A two-dimensional pose sequence acquisition module is configured to acquire a two-dimensional pose sequence in the monocular video, wherein the two-dimensional pose sequence comprises two-dimensional coordinates of N joints sorted in time sequence of the monocular video.

[0284] A first dimension lifting module is configured to lift the dimension of the features of the two-dimensional coordinates of the joints of the two-dimensional pose sequence to a preset dimension C.

[0285] L sequentially connected spatio-temporal blocks, and each spatio-temporal block comprises a spatial block and a time block connected in sequence, wherein the spatial block is configured to learn spatial features, and the time block is configured to learn temporal features.

[0286] A low-pass prediction module is configured to acquire the three-dimensional coordinates of the joints based on the features output by the last time block.

[0287] It can be understood that the monocular human motion trajectory estimation system based on the graph neural network provided by the embodiment of the present application corresponds to the monocular human motion trajectory estimation method based on the graph neural network described above, and the explanation, examples, beneficial effects and other parts of the related content can refer to the corresponding content in the monocular human motion trajectory estimation method based on the graph neural network, which will not be repeated here.

[0288] In summary, compared with the prior art, the present application has the following beneficial effects:

[0289] 1. The application designs a human motion trajectory estimation network based on a graph neural network, which uses multiple stacked space-time modules to mine the time and space characteristics in human motion. The spatial model uses a graph neural network to explore the joint characteristics under the prior of human topological structure and uses limb grouping to explore the spatial correlation of joints within the limb group. The time model uses a Transformer structure to model the motion trajectory of each joint and uses limb grouping to explore the time trajectory features of the whole limb group. This spatial graph topology and time sequence Transformer modeling can better utilize the spatial and temporal characteristics of human motion, while supplementing the human structure prior grouping, more completely utilizing all information of human action, and achieving more accurate trajectory estimation.

[0290] 2. Existing graph neural network-based spatial feature learning only aggregates first-order neighbor information or directly aggregates multi-order neighbor information. The multi-hop graph attention module designed in the application aggregates each order of neighbor of the node into a latent representation, and calculates the weighted aggregation weight of each order of latent representation based on the latent representation. This not only better develops the idea of graph neural network node aggregation, but also has high computational efficiency.

[0291] 3. The existing research method does not study the spatial correlation of joints within the limb group after grouping, while the application proposes that the nodes within the limb group have strong spatial correlation, which enhances the model's learning of human joint spatial structure features. In addition, the application designs a human skeleton topology graph based on the center based on the grouping structure, which enhances the center stability of the human skeleton topology graph to obtain a smaller jitter motion trajectory.

[0292] 4. The existing trajectory estimation research method is mostly based on the spatial position constraint of each node in the trajectory in the loss constraint of the trajectory. The embodiment of the application constructs a trajectory similarity loss function, uses discrete cosine transform to calculate the frequency coefficient of the trajectory, and constrains the trajectory in the frequency domain, which improves the similarity of the predicted trajectory of the model.

[0293] It should be noted that, through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus necessary universal hardware platforms. Based on such an understanding, the above technical solutions can be embodied in the form of a software product in essence or in the form of a part of the prior art that makes a contribution. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments. In this article, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0294] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for estimating monocular human motion trajectory based on graph neural networks, characterized in that, The method includes: Obtain a dataset containing monocular video and ground truth labels, wherein the ground truth labels include: ground truth two-dimensional coordinates of joints and ground truth three-dimensional coordinates of joints; Training a human motion trajectory estimation model based on a dataset; The human motion trajectory of the monocular video to be estimated is obtained based on the trained human motion trajectory estimation model; The human motion trajectory estimation model includes: A two-dimensional pose sequence acquisition module is used to acquire a two-dimensional pose sequence from a monocular video; the two-dimensional pose sequence includes the two-dimensional coordinates of N joints ordered in the time sequence of the monocular video; The first dimension-up module is used to increase the dimension of the joint two-dimensional coordinate features of the two-dimensional pose sequence to a preset dimension C; L sequentially connected spatiotemporal blocks, each spatiotemporal block comprising a spatial block and a temporal block; wherein the spatial block is used for spatial feature learning; and the temporal block is used for temporal feature learning. The low-pass prediction module is used to obtain the three-dimensional coordinates of the joint based on the features output by the last time block. Each of the spatial blocks includes a multi-hop graph attention module and a spatial limb attention module in sequence. The multi-hop graph attention module is: in, The latent characteristic representation of the k-th jump; A k Let represent the adjacency matrix of the k-th hop; X g This represents the original input features of each multi-hop graph attention module; W k Represents the learnable weight matrix; α represents the weight of the multi-feature latent representation for each node; W α Represents a mapping function; This indicates that each jump feature is latently represented. With the original input features X g Concatenate along the feature dimension; X go This represents the node features updated by the attention module in a multi-hop graph. β represents the attenuation factor; ⊙ represents the element-wise product; This indicates that each jump feature is latently represented. With the original input features X g Perform a stacking operation; The spatial limb attention module is: X p =X go [group p ] X ps =X ps +MLP(X ps ) in, X p This represents the feature vector of the p-th group obtained by dividing the updated node features according to the limb grouping diagram; X ps This represents the characteristics of group p after it has undergone in-group self-attention learning. Indicate the different mapping functions corresponding to the p-th group; T represents the transpose function; softmax represents the normalized exponential function; MLP stands for Multilayer Perceptron; This indicates that X is normalized by layer. p X after feature distribution normalization p ; Each time block includes a time-limb attention module, a second dimension-upgrading module, at least one trajectory attention module, and a first dimension-reducing module connected in sequence. The time-body attention module is: X pt =X pt +MLP(X pt ) in, X pt This represents the output of the Time-Body Attention module; This indicates that X is normalized by layer. ps X after feature distribution normalization ps ; The second dimension-up module is used to transform feature X T The dimension is increased to M times the preset dimension, and the feature X T Indicates that X pt The features obtained by rearranging according to the corresponding joint index and transforming the time dimension and joint dimension; The trajectory attention module is: X T =X T +dctFC(X T ) in, This indicates that X is normalized by layer. T X after feature distribution normalization T ; dctFC indicates that the fully connected layer is: dctFC=IDCT(FC(DCT(X T ))) DCT stands for Discrete Cosine Transform, IDCT stands for Inverse Discrete Cosine Transform, and FC stands for Fully Connected Layer. The output dimension and input dimension of a fully connected layer are both M*C. The first dimensionality reduction module is used to convert the X output by the trajectory attention module T The dimension is mapped back to the preset dimension C.

2. The method for estimating monocular human motion trajectory based on graph neural networks as described in claim 1, characterized in that, The low-pass prediction module includes a fully connected layer and a linear layer; the fully connected layer is a dctFC; the linear layer is used to map the dimension from the preset dimension C to the target dimension 3.

3. The method for estimating monocular human motion trajectory based on graph neural networks as described in claim 1, characterized in that, When training the model, the loss function includes the joint position loss error L. pos Trajectory consistency loss error L c And trajectory similarity loss error L s ; Wherein, the joint position loss error L pos Including Euclidean distance constraint L pos2 and L1 norm constraint L pos1 ; The Euclidean distance constraint is as follows: The L1 norm constraint is: The trajectory consistency loss error L c for: The trajectory similarity loss error L s for: Where T represents the number of video frames; N represents the predefined number of joints; y 3d These are the true three-dimensional coordinates of the joint; Represents the true three-dimensional coordinates of the nth index joint in frame t, where n = 1, 2, ..., N; This represents the 3D coordinates of the nth index joint in the t-th frame of the model output; DCT stands for Discrete Cosine Transform.

4. The method for estimating monocular human motion trajectory based on graph neural networks as described in claim 1, characterized in that, The first dimension-up module is an embedded layer; the second dimension-up module and the first dimension-down module are both linear layers.

5. A monocular human motion trajectory estimation system based on graph neural networks, characterized in that, The system includes: The data acquisition module is used to acquire a dataset containing monocular video and ground truth labels, wherein the ground truth labels include: ground truth two-dimensional coordinates of joints and ground truth three-dimensional coordinates of joints; The model training module is used to train a human motion trajectory estimation model based on a dataset. The human motion trajectory estimation module is used to obtain the human motion trajectory of the monocular video to be estimated based on the trained human motion trajectory estimation model. The human motion trajectory estimation model includes: A two-dimensional pose sequence acquisition module is used to acquire a two-dimensional pose sequence from a monocular video; the two-dimensional pose sequence includes the two-dimensional coordinates of N joints ordered in the time sequence of the monocular video; The first dimension-up module is used to increase the dimension of the joint two-dimensional coordinate features of the two-dimensional pose sequence to a preset dimension C; L sequentially connected spatiotemporal blocks, each spatiotemporal block comprising a spatial block and a temporal block; wherein the spatial block is used for spatial feature learning; and the temporal block is used for temporal feature learning. The low-pass prediction module is used to obtain the three-dimensional coordinates of the joint based on the features output by the last time block. Each of the spatial blocks includes a multi-hop graph attention module and a spatial limb attention module in sequence. The multi-hop graph attention module is: in, The latent characteristic representation of the k-th jump; A k Let represent the adjacency matrix of the k-th hop; X g This represents the original input features of each multi-hop graph attention module; W k Represents the learnable weight matrix; α represents the weight of the multi-feature latent representation for each node; W α Represents a mapping function; This indicates that each jump feature is latently represented. With the original input features X g Concatenate along the feature dimension; X go This represents the node features updated by the attention module in a multi-hop graph. β represents the attenuation factor; ⊙ represents the element-wise product; This indicates that each jump feature is latently represented. With the original input features X g Perform a stacking operation; The spatial limb attention module is: X p =X go [group p ] X ps =X ps +MLP(X pS ) in, X p This represents the feature vector of the p-th group based on the limb grouping diagram; X ps This represents the characteristics of group p after it has undergone in-group self-attention learning. Indicate the different mapping functions corresponding to the p-th group; T represents the transpose function; softmax represents the normalized exponential function; MLP stands for Multilayer Perceptron; This indicates that X is normalized by layer. p X after feature distribution normalization p ; Each time block includes a time-limb attention module, a second dimension-upgrading module, at least one trajectory attention module, and a first dimension-reducing module connected in sequence. The time-body attention module is: X pt =X pt +MLP(X pt ) in, X pt This represents the output of the Time-Body Attention module; This indicates that X is normalized by layer. ps X after feature distribution normalization ps ; The second dimension-up module is used to transform feature X T The dimension is increased to M times the preset dimension, and the feature X T Indicates that X pt The features obtained by rearranging according to the corresponding joint index and transforming the time dimension and joint dimension; The trajectory attention module is: X T =X T +dctFC(X T ) in, This indicates that X is normalized by layer. T X after feature distribution normalization T ; dctFC indicates that the fully connected layer is: dctFC=IDCT(FC(DCT(X T ))) DCT stands for Discrete Cosine Transform, IDCT stands for Inverse Discrete Cosine Transform, and FC stands for Fully Connected Layer. The output dimension and input dimension of a fully connected layer are both M*C. The first dimensionality reduction module is used to convert the X output by the trajectory attention module T The dimension is mapped back to the preset dimension C.

6. The monocular human motion trajectory estimation system based on graph neural networks as described in claim 5, characterized in that, The low-pass prediction module includes a fully connected layer and a linear layer; the fully connected layer is a dctFC; the linear layer is used to map the dimension from the preset dimension C to the target dimension 3.

7. The monocular human motion trajectory estimation system based on graph neural networks as described in claim 5, characterized in that, When training the model, the loss function includes the joint position loss error L. pos Trajectory consistency loss error L c And trajectory similarity loss error L s ; Wherein, the joint position loss error L pos Including Euclidean distance constraint L pos2 and L1 norm constraint L pos1 ; The Euclidean distance constraint is as follows: The L1 norm constraint is: The trajectory consistency loss error L c for: The trajectory similarity loss error L s for: Where T represents the number of video frames; N represents the predefined number of joints; y 3d These are the true three-dimensional coordinates of the joint; Represents the true three-dimensional coordinates of the nth index joint in frame t, where n = 1, 2, ..., N; This represents the 3D coordinates of the nth index joint in the t-th frame of the model output; DCT stands for Discrete Cosine Transform.

Images