A human motion sequence synthesis method and system based on motion graphs

Abstract

The present application relates to a kind of human motion sequence synthesis method and system based on motion graph.The present application first extracts key frame from human motion sequence to construct motion graph, transition edge is added in motion graph, and the transition sub-sequence corresponding to transition edge is generated;Based on dynamic graph, human motion sequence completion task is designed, and a kind of transfer number related graph path search algorithm is used to find multiple feasible paths between start node and end node, so as to generate a set of diversified motion sequence completion results;For human motion sequence prediction task, the method of forward walking on motion graph starting from start node is used, and the method can return multiple possible end nodes, so that human motion prediction task is attributed to human motion completion task with some different end nodes.The present application can generate real and diverse human motion sequence, and has excellent generalization ability between different data.

Description

Technical Field

[0001] This invention designs a method for synthesizing human motion sequences based on motion graphs, and particularly relates to a method and system for constructing motion graphs based on human motion data for human motion sequence prediction and motion sequence completion, belonging to the field of computer vision. Technical Background

[0002] With the development of social intelligence, virtual reality and augmented reality technologies have been increasingly widely applied in fields such as film, animation, game production, and education. The ultimate goal of virtual reality technology is to enable humans in real life to interact naturally and realistically with the virtual world. Human motion sequence synthesis, as one of the key technologies of virtual reality, can drive human skeletal models, giving virtual intelligent agents realistic human motion effects. It can be used to assist in human-centered video generation. Human motion sequence generation refers to generating unknown human motion sequences based on the movement of human joints in three-dimensional space within a given time interval. This invention mainly involves two sub-tasks: completion and prediction of human motion sequences.

[0003] Current research in human motion synthesis primarily focuses on human action prediction, a task aimed at predicting human motion sequences for the next few frames given a set of visible frames. Mainstream human motion sequence prediction methods can be categorized into data-driven models and generative models. Traditional data-driven methods for human motion prediction, such as Hidden Markov Models (HMMs), Gaussian dependent variable models, and graph-based methods, have proven effective for simple periodic movements (e.g., walking and swinging a golf club). With the development of deep learning, deep neural network models have been used to handle complex motion sequence generation tasks. Due to the temporal nature of human motion sequences, Recurrent Neural Networks (RNNs) are naturally used for this task. However, some existing works often use the estimate of the current RNN step as the input for the next action prediction. This approach tends to accumulate errors throughout the sequence generation task, resulting in unrealistic predicted human motion sequences. In recent years, some works have proposed using feedforward networks to generate human motion sequences. Compared to RNN-based methods, these can generate more accurate human sequences, but they can only handle short-term sequence generation. This method leads to a rapid decline in sequence fidelity over a long period of time, and also increases the complexity of the model. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a method and system for synthesizing human motion sequences based on motion graphs, which is a unified method for solving the problems of human motion sequence completion and prediction.

[0005] The technical solution adopted in this invention is as follows:

[0006] A method for synthesizing human motion sequences based on motion graphs includes the following steps:

[0007] A motion graph is constructed based on a human motion sequence dataset. Nodes in the motion graph correspond to human poses, and edges in the motion graph correspond to human motion sequences.

[0008] Transition nodes are selected from the motion graph to establish virtual paths between two motion sequences or between jump connections within a motion sequence; transition edges are added between human motion sequences through the transition nodes, and transition subsequences corresponding to the transition edges are generated;

[0009] For the human motion sequence completion task, a graph path search algorithm related to the number of transitions is used to find multiple feasible paths between the start node and the end node in the motion graph, and the human motion sequence completion result is recovered from multiple feasible paths;

[0010] For the human motion sequence prediction task, a method of moving forward on the motion graph starting from the starting node is used to obtain multiple possible termination nodes, thus reducing the human motion sequence prediction task to a human motion sequence completion task with some different termination nodes.

[0011] Further, the transition node is selected using the following steps:

[0012] 1) Set the subset A of the node set V of the motion graph G to an empty set. First, record the maximum value of the minimum cost from each node to all nodes in the graph, and use it as the maximum penalty value U. max For subset A, compute the minimum cost from all nodes in the graph to subset A, denoted as U. A Then the penalty reduction function corresponding to subset A is defined as F(A) = U max –U A ;

[0013] 2) For any node r∈V / A that belongs to the set of all nodes V in the graph but not to the subset A, add it to the subset A as a candidate node, and calculate the relative reduction δ of the penalty function after introducing r. r ←F(A∪{r})-F(A), select the node r* with the largest reduction in the corresponding penalty function;

[0014] 3) Expand r* into subset A, A←A∪{r*}, and recalculate the value corresponding to F(A);

[0015] 4) Repeat steps 2) and 3) until the number of nodes in subset A meets the requirements. The resulting subset A serves as a transition node.

[0016] Furthermore, the transition subsequence is generated through discrete cosine transform and a learnable graph convolutional network.

[0017] Furthermore, the human motion sequence completion task includes the following steps:

[0018] Human pose search is performed by finding the nodes corresponding to the starting pose and the ending pose in the motion graph G. The nodes on the motion graph are clustered using a spectral clustering algorithm to obtain multiple human pose clusters. Then, a hash algorithm is used to quantize and encode different human pose clusters, thereby accelerating the speed of human pose search from the motion graph.

[0019] Given the initial pose x of the human body s and termination posture x t The initial pose x is supplemented using a graph path search algorithm related to the number of transitions. s and termination posture x t The intermediate motion sequence.

[0020] Furthermore, the human pose search includes:

[0021] First, for a given human pose p to be queried, calculate the distance between human pose p and each cluster center, and locate the human pose cluster that is closest to p;

[0022] Then, the hash code of human pose p in the human pose cluster is calculated, and the Hamming distance is used to quickly find the human pose most similar to human pose p in the human pose cluster.

[0023] Furthermore, the graph path search algorithm related to the number of transitions includes the following steps:

[0024] 1) Calculate the minimum jump matrix between all nodes in the graph;

[0025] 2) Suppose we need to find a path from node s to node t with a transition number of j*, and assume we have already obtained the path P = {p1, p2, ..., p...} o Meanwhile, the number of transitions for the current path P is j. cur Try to determine if p can be used o The successor node p is added to the current search space; from p o The minimum number of hops to p is j p The minimum number of hops from p to t has been pre-stored in the minimum hop matrix D. p,t In the middle, if j cur +j p +D p,tIf the number of jumps in the current path increases by 1, only M paths need to be retained in the current search space to reduce the search space from exponential to polynomial level, where M is a hyperparameter.

[0026] 3) After obtaining the path from node s to node t, the corresponding motion sequence is recovered based on the path; for the edges in the original human motion sequence, the corresponding motion subsequence is directly extracted from the original sequence; for the transition edges between sequences, the transition subsequence generation method is used to generate the corresponding motion sequence, thereby completing the motion sequence completion from the starting posture to the ending posture.

[0027] Furthermore, the human motion sequence prediction task involves performing a pose search within the motion graph to find the initial pose x. s The most similar node set S' is used, and then motion sequence prediction is performed in two steps: First, a forward propagation related to the number of jumps is performed on each node s in S' in the motion graph; the forward propagation related to the number of jumps means that the path is moved forward from s to obtain candidate paths with a specified number of jumps J. By controlling J, paths of different lengths are generated, and they are sorted according to the scores on the paths. The graph paths with the highest scores will be retained as the final results; then, based on the obtained paths, the predicted human motion sequence is recovered.

[0028] A motion graph-based human motion sequence synthesis system employing the above method includes:

[0029] The motion graph construction module is used to construct a motion graph based on a human motion sequence dataset. Nodes in the motion graph correspond to human poses, and edges in the motion graph correspond to human motion sequences.

[0030] The transition sequence generation module is used to select transition nodes from the motion graph and to establish virtual paths between two motion sequences or between jump connections within a motion sequence; it adds transition edges between human motion sequences through the transition nodes and generates transition sub-sequences corresponding to the transition edges;

[0031] The human motion sequence completion module is used to find multiple feasible paths between the start node and the end node in the motion graph for the human motion sequence completion task using a graph path search algorithm related to the number of transitions, and to recover the human motion sequence completion result from multiple feasible paths.

[0032] The human motion sequence prediction module is used to predict human motion sequences by walking forward on the motion graph from the starting node to obtain multiple possible termination nodes, thereby reducing the human motion sequence prediction task to a human motion sequence completion task with some different termination nodes.

[0033] The key technical contributions and beneficial effects of this invention can be summarized in two aspects:

[0034] 1. The technology described in this invention introduces a new data representation called motion graph, and an accompanying graph search method for generating realistic and diverse human motion sequences from unidirectional inputs (e.g., human motion sequence prediction tasks) or bidirectional inputs (e.g., human motion sequence completion tasks).

[0035] 2. The method proposed in this invention differs fundamentally from existing generative model-based motion sequence synthesis methods. In the rarely studied human motion sequence completion task, the method of this invention can generate realistic and diverse human motion sequences. In the human motion sequence prediction task, the motion graph-based method of this invention exhibits excellent generalization ability across different datasets. The data-driven method proposed in this invention provides some inspiration for more robust human motion sequence synthesis tasks. Attached Figure Description

[0036] Figure 1 This is a schematic diagram summarizing the overall process of the present invention.

[0037] Figure 2 This is a schematic diagram of the specific method flow of the present invention.

[0038] Figure 3 This is the convergence graph of the submodular optimization algorithm.

[0039] Figure 4 A schematic diagram of the completion results of multi-scale human motion sequences. Detailed Implementation

[0040] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0041] The core of the method proposed in this invention is a novel directed graph-based representation that encodes a wealth of knowledge about human kinematics. This invention refers to it as a motion graph. Figure 1The flowchart illustrates the method. Given a complete human motion sequence dataset (such as training data from Human3.6M or HumanEva-I), keyframes are first extracted from the sequence at regular intervals, serving as nodes in the MotionGraph. The temporal order of these frames naturally defines the directed structure of the MotionGraph. To enable traversal between different sequences, transition edges are further added between sequences, generating virtual transition subsequences by connecting the starting two transition nodes of each transition edge. Based on the constructed MotionGraph, for the human motion sequence completion task, this invention designs a graph path search algorithm related to the number of transitions to find multiple feasible paths from the starting node to the ending node, thereby generating a diverse set of motion sequence completion results. For the human motion sequence prediction task, which only specifies the starting node in the graph corresponding to the initial human pose, this invention uses a method of walking forward on the motion graph starting from the starting node. This method can return multiple possible ending nodes. Thus, this invention can reduce the human motion prediction task to a human motion completion task with some different ending nodes.

[0042] This invention proposes a method for human motion sequence completion and prediction based on motion graphs, the process of which is as follows: Figure 2 As shown, it includes the following main contents:

[0043] 1) Construct a motion graph from the training data. This graph is used to encode the motion information of various human bodies within the same sequence and the smooth transitions between different sequences. In this step, the present invention designs an algorithm to find suitable transition nodes, i.e., transition nodes, to connect different human motion sequences.

[0044] 2) For the transition edge between two transition nodes across sequences, use the model to generate the virtual transition subsequence corresponding to the transition edge.

[0045] 3) Use spectral clustering algorithm to cluster nodes on the motion graph to obtain multiple human pose clusters, and then use hash algorithm to quantize and encode different clusters, thereby accelerating the speed of human pose retrieval from the motion graph.

[0046] 4) Design a graph path search algorithm related to the number of transitions to find many feasible paths from the starting node to the ending node of the graph, and recover the motion sequence completion result of the human body from the path.

[0047] 5) By using the method of walking forward on the motion graph starting from the starting node, multiple possible ending nodes are obtained, thus reducing the human motion sequence prediction task to a human motion sequence completion task with some different ending nodes, and then step 4) can be used to solve it.

[0048] 1. Construction of motion graphs

[0049] (1) Topological Structure of Motion Graphs: Human motion sequences can be defined as samples of human body parameters, such as the angles of each joint or the three-dimensional coordinates of the joints. Human motion sequences can be part of certain daily activities, such as walking, making a phone call, greeting, etc. Given a human motion sequence database... The database contains N human motion sequences, each sequence S i ={x i，1 x i，2 x i，3 , ..., x i，T The dataset includes T consecutive human poses. A motion graph G is constructed based on the human pose database D. G is a directed graph where all edges correspond to a sequence of human motions. For each sequence S... i One frame (i.e., a keyframe) is extracted from each I-frame as a node in the motion graph, and each node corresponds to a human pose. Considering the scale of the motion graph and the requirement of the generation model for the length of the known sequence, I is set to 20 in this embodiment.

[0050] (2) Calculation of graph node similarity: For any pair of nodes in the graph i，t S j，t’ When determining the similarity between the two nodes, we consider two factors: the distance between the human poses of the nodes and the motion similarity of the subsequences of the two nodes.

[0051] Human pose distance: for node pairs i，t S j，t’ >, let x i，t and x j，t’ This provides a vectorized representation of their corresponding poses. Before calculating the distance between two poses, we first align them using Procrustes Analysis. Then, we calculate the Euclidean distance between the aligned poses, i.e.:

[0052]

[0053]

[0054] Motion similarity: Visually similar human poses require further differentiation using dynamic information about human motion. Therefore, we take the next 10 frames as the subsequence corresponding to each node in the graph. For node S... i，t Its corresponding motion subsequence is {x} i，t-9 x i，t-8 , ..., x i，t ​​We use a simple strategy to calculate the motion vector corresponding to this node:

[0055] m i，t ={x i，t-8 -x i，t-9 x i，t-7 -x i，t-8 , ..., x i，t -x i，t-1}

[0056] Similarly, node S can also be calculated. i，t’ The corresponding motion vector m i，t’ We use cosine distance to measure the distance between two motion vectors:

[0057]

[0058] We will consider the two distances d mentioned above pose and d mot Merge them together to obtain the two nodes S in the graph. i，t and S j，t’ Final distance:

[0059]

[0060] Where D pose and D mot These represent the maximum values ​​of the attitude distance and motion distance used for normalization purposes, respectively. α is the harmonic parameter, which is set to 0.7 in this embodiment.

[0061] (3) Selection of Transition Nodes: Transition nodes in the motion graph G are designed to establish virtual paths between two motion sequences or between jump connections within a motion sequence. The selection of transition nodes significantly affects the quality of the final generated motion graph. Insufficient sampling of transition nodes leads to low connectivity of the graph, resulting in a fragmented motion graph. Furthermore, to ensure the smoothest possible transitions between motion sequences, the cost of connecting the selected transition nodes to other nodes in the graph should be sufficiently low. Formally, given a fully connected graph G = (V, E), where V is the set of nodes and E is the set of edges, our goal is to find a subset A in V where the path cost from other nodes to this subset A is minimized, and the resulting subset A is used as a transition node. This problem can be effectively formulated as a submodular optimization problem. The greedy algorithm for solving subset A includes the following steps:

[0062] 1) Set subset A to an empty set. First, record the maximum value of the minimum cost (cost refers to the sum of the distances d along the edges of the path from one node to another) from each node in the graph to all nodes in the graph, and use it as the maximum penalty value U. maxFor subset A, compute the minimum cost from all nodes in the graph to subset A, denoted as U. A Then the penalty reduction function corresponding to subset A is defined as F(A) = U max -U A .

[0063] 2) For any node r∈V / A that belongs to the set of all nodes V in the graph but not to the subset A, add it to the subset A as a candidate node, and calculate the relative reduction δ of the penalty function after introducing r. r ←F(A∪{r})-F(A). Select the node r* with the largest reduction in the corresponding penalty function.

[0064] 3) Expand r* into subset A, A←A∪{r*}, and recalculate the value corresponding to F(A).

[0065] 4) Repeat steps 2) and 3) until the number of nodes in subset A meets the requirements. The resulting subset A serves as a transition node.

[0066] The convergence curve of the penalty reduction F(A) is as follows: Figure 3 As shown, the penalty reduction gradually stabilizes as the number of selected nodes increases. In this embodiment, 4000 nodes are ultimately selected to form set A as transition nodes. Then, we use the k-nearest neighbor method to select the k nodes with the smallest distance for each transition node to establish an edge. In the experiment, we set k to 5.

[0067] 2. Transition sequence generation

[0068] The human poses connecting two transition nodes may differ, and this difference is perceptible to humans. Therefore, this example generates transition subsequences for the two transition nodes to fill this gap. This example can be viewed as a miniature human motion completion problem: the subsequences of the two transition nodes are known, and conceptually, they can serve as the starting and ending human poses for the completion problem. Transition sequence generation takes the two known subsequences and inserts the "missing" subsequence between them. For example, given a motion sequence X = {x1, x2, x3, ..., x...} of length 30 frames... 30}, where x 1～10 and x 21～30 These are the representations of the two known subsequences, x. 11～20 It is the transition subsequence of insertion and deletion. The method proposed in this section first gives the transition subsequence x 11～20An initial representation is generated, and then the motion sequence X is passed through Discrete Cosine Transform (DCT) and a learnable Graph Convolutional Network (GCN) to obtain a new representation of the transition subsequence, thus generating the transition subsequence. The transformation process of the initial motion sequence X by Discrete Cosine Transform and Learnable Graph Convolutional Network is described below.

[0069] Discrete cosine transform and learnable graph convolutional neural networks can jointly encode the temporal information and spatial structure of human motion nodes in a motion sequence, obtaining a transition motion sequence containing spatiotemporal context information. This example uses... Let this represent the trajectory of the k-th node over T time frames. Given a trajectory... Its first DCT coefficient is calculated as follows:

[0070]

[0071] Where δ ij This represents the Kronecker delta function, where δ = j when i equals j. ij Equal to 1, otherwise δ ij Equals 0. DCT encoding allows us to model the temporal information of each joint using DCT coefficients. Given such coefficients, the original human pose representation can be obtained through Inverse Discrete Cosine Transform (IDCT):

[0072]

[0073] Where t∈{1,2,3,...,T}.

[0074] After obtaining the DCT encoding of the sequence, a GCN is used to encode the spatial structure of the human pose. Let's assume we model the human body as a fully connected graph with K nodes. Then, we can encode the spatial structure of the human pose using a weighted adjacency matrix. This represents the strength of the connecting edges in the graph. The graph shows the convolutional layer p... As the input matrix, F is the size of the output features from the previous layer. Given this information and a set of trainable weight parameters... The output format of the convolutional layer is H. p+1 =σ(A (p) H (p) W (p) ), where σ(·) is the activation function.

[0075] Through the above transformation, a new human motion sequence representation X' = {x'1, x'2, x'3, ..., x'} is obtained. 30}, where x' 11～20This is the generated transition sequence.

[0076] 3. Human motion sequence completion

[0077] (1) Human posture search:

[0078] For the problem of completing or predicting human motion sequences, this example first needs to find the nodes corresponding to the starting and ending poses in the motion graph G. This process is called pose search. Pose search is accomplished through two operations: pose alignment and calculation of human pose distance. As mentioned earlier, before calculating the similarity between two human poses, they need to be precisely aligned. Due to the dense sampling of human motion sequences in the dataset, the motion graph G usually contains a large number of vertices. Performing an alignment operation for each pose query will inevitably lead to significant time consumption. To accelerate the pose search process, this invention adopts a divide-and-conquer strategy to pre-index all information in the motion graph G. A key insight for this operation is that human poses tend to cluster locally in certain feature spaces. Therefore, this example only needs to find the cluster of the human pose to be queried, which can approximately find the human pose most similar to the query pose. Given a motion graph G with a total of N nodes, all nodes in the graph are aligned pairwise, and their Euclidean distance is calculated to obtain the distance matrix D corresponding to the motion graph. In D, 0 represents two identical human poses, and the higher the value, the greater the difference between the two poses. The distance matrix D can be converted into a similarity matrix using a Gaussian kernel:

[0079] S = exp(-D 2 / (2*δ 2 ),

[0080] Here, δ represents the width of the Gaussian kernel, which can be estimated empirically based on the training data. Next, this example runs a spectral clustering algorithm on the similarity matrix S, resulting in C classes (i.e., C human pose clusters). For each class, the center pose xi of class i is calculated as follows:

[0081]

[0082] Where C i d is the set of all human poses in the i-th human pose cluster, where x is the representation of each human pose. pose As defined in Section 1, after obtaining the cluster center xi, all poses in cluster i are aligned with the center pose xi. To further accelerate the human pose search process, vector-based hashing is performed in each cluster. Binary quantization is then performed using the Iterative Quantization (ITQ) method. For a specific human pose cluster, it is assumed that there are n human poses. Represented in matrix form Our goal is to learn a binary encoding matrix B∈{0,1} n×h Where h represents the length of the binary code. The first step is to project the human pose data X onto [the graph] using Principal Component Analysis (PCA). The binary hash code matrix B is estimated by optimizing the quantization loss:

[0083]

[0084] Where Q(B, R) represents the quantization loss function, ||·|| F Let R denote the Frobenius norm, and R be a c×c orthogonal matrix. After obtaining the optimal R, the binary encoding matrix B = sgn(VR) can be obtained. * ), where sgn(.) is a binary encoding function; sgn(v) = 1 when v >= 0, otherwise sgn(v) = 0. For C classes, C binary encoding matrices can be obtained {B1, B2, ..., B...}. C}

[0085] Now, each node in the motion graph G has a unique cluster index and a corresponding compressed hash code. For a given human pose p to be queried, firstly, the distance between human pose p and each cluster center is calculated to locate the human pose cluster closest to p. Then, the hash code of human pose p in that human pose cluster is calculated, and the Hamming distance is used to quickly find the human pose most similar to human pose p from the human pose cluster.

[0086] (2) Path search on the graph:

[0087] Given the initial pose x of the human body s and termination (or target) posture x t Human motion sequence completion tasks require completing the intermediate motion sequences between two poses. Using the pose search technique mentioned earlier, this can be achieved by finding the intermediate motion sequences in the motion graph. s Find the most similar node set S', which is x t Find the most similar set of nodes T'. The motion sequence completion task can be divided into two parts: First, find all possible paths in the graph for all start-end pairs (s, t) ∈ S'×T', and then reconstruct the path from x... s To x t Missing intermediate motion sequences.

[0088] Although this invention enhances the connectivity of the graph, the transition subsequences between two transition nodes may still not be smooth enough. This invention uses jumps to represent the process of transitioning from one human motion sequence to another along a path. Fewer jumps on a graph path tend to generate more realistic human motion sequences, but the diversity of such generated motion sequences is limited. It is readily known that for a directed acyclic graph, the time complexity of finding all paths from the source node to the target node is O(2^3). N ), where N is the number of nodes in the graph. For a path search with a specified number of hops, the time complexity is O(N). J ), where J is the number of hops. Searching for all paths to the target in the graph is an NP-hard problem that cannot be solved in polynomial time.

[0089] Since it is not necessary to find all paths from the starting node to the target node in practical applications, this invention proposes a heuristic approximate search algorithm. This algorithm has polynomial time complexity while ensuring a certain degree of search performance. One of the most widely used algorithms for path search problems is the breadth-first search (BFS) algorithm. However, the search space of this algorithm grows exponentially with the number of jumps. Here, this invention proposes a heuristic guidance strategy to filter out unreasonable candidate nodes in the search space. The heuristic approximate search algorithm of this invention, namely, a graph path search algorithm related to the number of transitions (i.e., the specified number of jumps J), has the following steps:

[0090] 1) First, use the Floyd-Warshall algorithm to calculate the minimum jump matrix D between all nodes in the graph.

[0091] 2) Suppose we need to find a path from node s to node t with j* transitions. Assume we already have the path P = {p1, p2, ..., p...} o Meanwhile, the number of transitions for the current path P is j. cur Let's try to determine if we can put p o The successor node p is added to the current search space. We can know that from p... o The minimum number of hops to p is j p The minimum number of hops from p to t has been pre-stored in the minimum hop matrix D. p,t In the middle. If j cur +j p +D p,tIf the number of jumps in the current path is greater than j*, then p is an irrational node and will not be added to the search space. Since not all possible paths from node s to node t are needed, for every 1 increase in the number of jumps in the current path, only M paths need to be retained in the current search space, so as to reduce the search space from exponential to polynomial level. M is a hyperparameter, and in this example, M is 1000.

[0092] The time complexity of calculating the jump matrix D using the Floyd-Warshall algorithm is O(N). 3 However, this time cost is a one-time event and does not need to be calculated for each query. Furthermore, the time complexity of each query in the heuristic graph search algorithm of this invention is O(JMN). Therefore, the overall time complexity of this algorithm is polynomial.

[0093] 3) After obtaining the path from node s to node t, the corresponding motion sequence is reconstructed based on the path. For edges in the original human motion sequence in the database, their corresponding motion subsequences are directly extracted from the original sequence. For transition edges between sequences, the transition sequence generation method introduced in Section 2 is used to generate the corresponding motion sequence. In this way, the motion sequence completion from the starting posture to the ending posture is completed.

[0094] 4. Human motion sequence prediction

[0095] For the task of predicting human motion sequences, we only know the initial pose x. s The goal is to predict future human motion sequences. Using a pose search algorithm, x can be found in the motion graph. s The most similar node set S' is then used. Motion sequence prediction is performed in two steps. First, a jump-count-related forward propagation is conducted for each node s∈S' in the motion graph. Jump-count-related forward propagation means moving forward from s to obtain candidate paths with a specified number of jumps J. Paths of different lengths can be generated by controlling J, and they are sorted according to their scores. The top-ranked graph paths are retained as the final result. Then, the predicted human motion sequence can be recovered based on the paths obtained above.

[0096] 5. Visualization Results

[0097] Figure 4 The image shows the results of human motion sequence completion at three different time scales (1s, 10s, and 30s). To better illustrate the components of the generated sequence, the human poses are not sampled at equal time intervals.

[0098] Other embodiments of the present invention:

[0099] 1) This invention mainly involves the completion and prediction of human motion sequences based on the Human3.6M and HumanEva-I datasets. This invention is also applicable to other datasets. During implementation, the number of nodes in the motion graph and other hyperparameters mentioned above may change accordingly with the dataset.

[0100] 2) The method based on discrete cosine coding and graph convolutional neural network used in the transition sequence generation part of this invention can also be replaced with other stronger generation models to obtain better generation results.

[0101] Based on the same inventive concept, another embodiment of the present invention provides a motion graph-based human motion sequence synthesis system employing the method of the present invention, comprising:

[0102] The motion graph construction module is used to construct a motion graph based on a human motion sequence dataset. Nodes in the motion graph correspond to human poses, and edges in the motion graph correspond to human motion sequences.

[0103] The transition sequence generation module is used to select transition nodes from the motion graph and to establish virtual paths between two motion sequences or between jump connections within a motion sequence; it adds transition edges between human motion sequences through the transition nodes and generates transition sub-sequences corresponding to the transition edges;

[0104] The human motion sequence completion module is used to find multiple feasible paths between the start node and the end node in the motion graph for the human motion sequence completion task using a graph path search algorithm related to the number of transitions, and to recover the human motion sequence completion result from multiple feasible paths.

[0105] The human motion sequence prediction module is used to predict human motion sequences by walking forward on the motion graph from the starting node to obtain multiple possible termination nodes, thereby reducing the human motion sequence prediction task to a human motion sequence completion task with some different termination nodes.

[0106] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0107] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0108] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0109] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A method for synthesizing a human motion sequence based on a motion graph, characterized in that, The method comprises the following steps: According to the human motion sequence dataset, a motion graph is constructed, the nodes in the motion graph correspond to human poses, and the edges in the motion graph correspond to human motion sequences; Transition nodes are selected from the motion graph to establish virtual paths between two motion sequences or jump connections within a motion sequence; transition edges are added between human motion sequences through the transition nodes, and transition sub-sequences corresponding to the transition edges are generated; For the human motion sequence completion task, a transition transfer number related graph path search algorithm is used to find multiple feasible paths between the start node and the end node from the motion graph, and the human motion sequence completion result is recovered from the multiple feasible paths; For the human motion sequence prediction task, a forward walking method starting from the start node on the motion graph is used to obtain multiple end nodes, so that the human motion sequence prediction task is reduced to a human motion sequence completion task with different end nodes; The transition transfer number related graph path search algorithm comprises the following steps: 1) Calculate the minimum jump matrix between all nodes in the graph; 2) Suppose that the transition number of the path from node s to node t needs to be found j*, and suppose that the path P = {p1, p2, …, p o} has been obtained, and the transition number of the current path P is j cur , try to determine whether the successor node p of p o can be added to the current search space; the minimum hop number from p o to p is j p , and the minimum hop number from p to t has been pre-stored in the minimum hop matrix D p,t , if j cur +j p +D p,t >j*, then p is an unreasonable node and is not added to the search space; the number of jumps in the current path is increased by 1, and only M paths need to be retained in the current search space, so as to reduce the search space from exponential level to polynomial level, where M is a hyperparameter; 3) After obtaining the path from node s to node t, the corresponding motion sequence is recovered according to the path; for the edges in the original human motion sequence, the corresponding motion sub-sequence is directly taken from the original sequence; for the transition edges between sequences, the corresponding motion sequence is generated by using a transition sub-sequence generation method, so as to complete the motion sequence completion from the start pose to the end pose.

2. The method of claim 1, wherein, The transition nodes are selected by the following steps: 1) Set the subset A of the node set V of the motion graph G to be empty, first record the maximum value of the minimum cost from each node in the graph to all nodes in the graph as the maximum penalty value U max ; for the subset A, calculate the minimum cost from all nodes in the graph to the subset A, denoted as U A , then the penalty reduction function corresponding to the subset A is defined as F(A) = U max - U A ; 2) For any node r∈V / A, which belongs to the set of all nodes V in the graph but not to the subset A, add it as a candidate node to the subset A, and calculate the relative reduction δ of the penalty function after introducing r r ←F(A∪{r})-F(A), select the node r* corresponding to the maximum reduction of the penalty function 3) Expand r* into the subset A, A←A∪{r*}, and recalculate the value corresponding to F(A); 4) Repeat steps 2) and 3) until the number of nodes in the subset A meets the requirement, and the formed subset A is used as the transition node.

3. The method of claim 1, wherein, The transition sub-sequence is generated by discrete cosine transform and a learnable graph convolution network.

4. The method of claim 1, wherein, The human motion sequence completion task comprises the following steps: Human pose search is performed, that is, the nodes corresponding to the start pose and the end pose in the motion graph G are found; a spectral clustering algorithm is used to cluster the nodes on the motion graph to obtain multiple human pose clusters, and then a hash algorithm is used to quantitatively encode different human pose clusters, so as to speed up the human pose search speed on the motion graph; a starting pose x of a human body s and a terminal pose x t , a middle motion sequence of the starting pose x s and the terminal pose x t is supplemented by using a transition transfer number related graph path search algorithm.

5. The method of claim 4, wherein, The human pose search comprises: First, for a given human pose p to be queried, the distance between the human pose p and each cluster center is calculated to locate the human pose cluster closest to the human pose p; Then, the hash code of the human pose p in the human pose cluster is calculated to quickly find the human pose most similar to the human pose p from the human pose cluster by Hamming distance.

6. The method of claim 1, wherein, The human motion sequence prediction task is to search for a starting pose x in the motion graph s The corresponding most similar node set S', and then the motion sequence prediction is performed in two steps: first, the jump number related forward propagation is performed on each node s in S' in the motion graph; the jump number related forward propagation refers to forward walking from s to obtain a candidate path with a specified jump number J, different length paths are generated by controlling J, and the paths are sorted according to the scores on the paths, and the paths at the front of the sorting are reserved as the final result; then, the predicted human motion sequence is recovered based on the obtained path.

7. A human motion sequence synthesizing system based on a motion graph using the method according to any one of claims 1 to 6, characterized by, It comprises: A motion graph construction module is configured to construct a motion graph according to a human motion sequence dataset, the nodes in the motion graph correspond to human poses, and the edges in the motion graph correspond to human motion sequences; A transition transfer sequence generation module is configured to select transition nodes from the motion graph to establish virtual paths between two motion sequences or jump connections within a motion sequence; transition edges are added between human motion sequences through the transition nodes, and transition sub-sequences corresponding to the transition edges are generated; The human motion sequence completion module is used for a human motion sequence completion task, adopts a transition transfer number related graph path search algorithm to find a plurality of feasible paths between a start node and a terminal node from a motion graph, and recovers a human motion sequence completion result from the plurality of feasible paths. The human motion sequence prediction module is used for a human motion sequence prediction task, uses a method of forward walking on the motion graph starting from the start node to obtain a plurality of terminal nodes, and thus reduces the human motion sequence prediction task to a plurality of human motion sequence completion tasks with different terminal nodes.

8. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by a computer to implement the method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a computer to implement the method in any one of claims 1-6.

Images