Digital human motion generation method based on sample synthesis

Abstract

The invention discloses a digital human action generation method based on sample synthesis. According to the method, as many different actions as possible can be mined from a single sample sequence. Compared with an existing data driving method, the existing data driving method usually needs very long offline training time. The core of the generation method is to generate an action matching algorithm, utilize bi-directional visual similarity as a generation cost function of action matching, and operate in multiple stages to gradually refine initially input random noise using sample action matching. The method provided by the invention is an innovative action generation method. Under the guidance of the sample action sequence, the action sequence which is high in quality and diversified and does not deviate from the sample is generated.

Description

Technical Field

[0001] The present invention relates to a method for generating digital human actions based on sample synthesis, and belongs to the technical field of the combination of computer graphics and artificial intelligence. Background Art

[0002] With the in-depth application of machine learning technology in the fields of computer vision and graphics, it has provided an important technical innovation solution for digital human motion synthesis. In the field of action sequences generated based on artificial intelligence, how to synthesize diverse and high-quality action sequences from limited action samples is a challenging problem. Traditional data-driven methods rely on a large amount of motion capture data, which is expensive to obtain and is limited to specific bone structures and movement styles. In addition, the authenticity of the effect based on the generative model (GAN) needs to be improved when processing complex bone structures, and it is very easy to produce visual artifacts or over-smoothing when generating new actions.

[0003] In order to overcome these defects, the present invention proposes a method for generating digital human motion based on sample synthesis. The method does not rely on a large amount of training data, but only "mines" as many action sequences as possible from a single or a few example sequences to achieve diversified digital human motion synthesis. Compared with traditional methods, the method of the present invention has the characteristics of short training time, high generation quality, and strong adaptability to complex bone structures.

[0004] The contribution of the present invention is to apply a multi-stage action generation algorithm, and to control the generated action content in a hierarchical manner by gradually refining the generated sequence actions in each stage.

[0005] In terms of technical implementation, the present invention adopts skeleton-aware action sequence extraction, bidirectional similarity measurement, multi-stage synthesis strategy, and noise-based generation diversity. Skeleton-aware action sequence extraction decomposes the skeleton into multiple overlapping sub-parts, and considers the bone structure when extracting the action sequence, making the synthesized action more natural and diverse. The bidirectional similarity measurement ensures that the synthesized action contains the key sequence of the sample action and avoids generating irrelevant or distorted sequences. The multi-stage synthesis strategy makes the generated result consistent with the style of the sample action by gradually refining the action sequence from coarse to fine. Adding noise input in the coarsest generation stage can ensure that the generated action sequence result is not limited to a single mode. Summary of the invention

[0006] The present invention provides a method for generating digital human motion based on sample synthesis, and the technical solution adopted is as follows:

[0007] Step 1: Use foot displacement O, joint rotation R, and ground contact label L to represent the complete action sequence;

[0008] Step 2: Extract the action skeleton sequence of the sample and connect the sample action sequence and the ground contact label;

[0009] Step 3: Calculate the distance matrix, introduce bidirectional similarity as a protection mechanism, and match and fuse the initially generated action sequence and sample action sequence that conform to the statistical distribution through the normalized distance matrix;

[0010] Step 4: Use progressive motion synthesis to iteratively refine the action sequence stage by stage from step 1.

[0011] 1. The method for generating digital human motion based on sample synthesis according to claim 1, characterized in that the specific steps of step 1 are as follows:

[0012] Step 1.1: Since the local foot displacement has the characteristic of time invariance, the local foot displacement is used Instead of foot displacement O, T is the time interval of the sample, and represents joint rotation, where J is the number of joints and Q is the number of rotation features;

[0013] Step 1.2: By calculating L tj =1[||FK([R,O]) tj || 2 <ε] Set the contact label L∈{0,1} T×C , where T represents the action sequence, C represents the number of foot joints, and L tj The meaning is: if the calculated velocity norm of the forward motion joint j∈C in frame t∈{1,...,T} is less than the threshold ε, then L tj The label is 1 otherwise, 0. The introduction of contact labels can alleviate the common foot sliding artifacts.

[0014] 2. The method for generating digital human motion based on sample synthesis according to claim 1, characterized in that the specific steps of step 2 are as follows:

[0015] Step 2.1: Input sample sequence to action generation matching algorithm;

[0016] Step 2.2: The motion generation matching algorithm obtains data information representing the foot displacement O and joint rotation R. The difference in global coordinates between consecutive actions is calculated to obtain the local foot displacement V;

[0017] Step 2.3: Split the skeleton B into multiple overlapping parts according to function, flatten each part, obtain the local feature vector of the body movement, and connect the foot contact label L to the local feature vector.

[0018] 3. The method for generating digital human motion based on sample synthesis according to claim 1, characterized in that the specific steps of step 3 are as follows:

[0019] Step 3.1: The action generation matching algorithm generates a new action sequence through the initial action sequence X and the sample action sequence Y, ensuring that the generated action postures are diverse and do not deviate from the sample statistical distribution. The action generation matching algorithm calculates the distance norm of the action sequences X and Y and saves it to the distance matrix D;

[0020] Step 3.2: To ensure that the generated actions maintain visual integrity, the action generation matching algorithm assigns a higher score to the key action sequence similarity, increasing the probability that the key action sequence is selected and appears in the output sequence; the score is normalized using the distance matrix of each sequence When the key action sequence is missing, the normalized value becomes larger and the score becomes smaller. When the key action sequence appears, the normalization factor is closer to α. The parameter α is used to adjust the completeness of the generated action sequence.

[0021] 4. The method for generating digital human motion based on sample synthesis according to claim 1, characterized in that the specific steps of step 4 are as follows:

[0022] Step 4.1: Set the upsampling factor r>1, upsample the action sequence, obtain action sequences with different resolutions, and form an action sample pyramid;

[0023] Step 4.2: In the first stage, direct generation is adopted. Random noise N is input into the action matching algorithm, action sequences are extracted, and the normalized distance matrix M is calculated. For each action sequence in the initial input, the best matching action sequence is found in the sample action sequence. The initial input and the matching action sequence are mixed to form a new action. The values ​​of the overlapping joints of the skeleton are averaged to form the action sequence of the digital human.

[0024] Step 4.3: Upsample the generated action sequence and iterate the above steps to make the difference between the generated action sequence and the sequence distribution of the sample small enough and no longer have obvious improvement.

[0025] 5. The method for generating digital human motion based on sample synthesis according to claim 4, characterized in that in step 3.1, the reduction process is as follows:

[0026] (3.11) Define X as the set of action subsequences extracted from the initial action sequence N, and Y as the set of action subsequences extracted from the sample sequence T;

[0027] (3.12) Calculate the MSE distance between each query sequence X in N and each action sample sequence Y in T.

[0028] 6. The method for generating digital human motion based on sample synthesis according to claim 4, characterized in that in step 3.2, the reduction process is as follows:

[0029] (3.21) For each element of the distance matrix, calculate the normalized score S, and use the score to replace the corresponding element of the distance matrix to obtain the normalized distance matrix adjusted by the control coefficient. For each query action sequence X, find the action sequence with the minimum normalized score. When there are multiple identical element values, use average voting to merge to obtain the matching action sequence;

[0030] (3.22) For each partial body motion sequence obtained, the sequence average of the overlapping parts of the skeleton is calculated to form the final result.

[0031] The beneficial effects of the present invention are: introducing noise input and inheriting the traditional action matching method, it can generate high-fidelity, smooth and diverse action sequences. It does not require long-term offline training and can synthesize high-quality actions in a very short time. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is an example diagram of the extraction of digital human skeleton motion perception sequence according to the present invention.

[0033] Figure 2 It is the present invention that generates the matching and mixing graphs.

[0034] Figure 3 The present invention generates an action matching graph in multiple stages. DETAILED DESCRIPTION

[0035] The present invention provides a method for generating digital human motion based on sample synthesis, which will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0036] 1. Generate action matches

[0037] Figure 1 The present invention provides a digital human skeleton motion perception sequence extraction process. In the synthesis stage, the action sequence is processed by upsampling, and the action matching algorithm extracts the sequence.

[0038] The complex digital human skeleton B is decomposed into multiple sub-skeletons according to the movement function {B 1 , ... B b …, B B}, where B b ∈B is a sub-skeleton of the digital human skeleton, and has j bThe whole body action T is represented by joint rotation R, local foot displacement V and ground contact label L. The spatial features of joint rotation R are flattened, and the local foot displacement vector and the local action at the center of gravity are concatenated. Then the ground contact label is concatenated to the feature vectors of all body part actions. The whole body action T is represented by multiple body part actions {T 1 , ...T b , ...T B}. From the action T of each partial skeleton b The motion action sequence of length p frames is extracted from the , the extraction step length is 1 frame, and the set of action sequences is expressed as {X b} (extracted from the initial action input N) and {Y b}(extracted from sample action T).

[0039] Figure 2 The sample graph of the generated action and mixture is given. The action sequence set {X} and {Y} are extracted from the input action N and the sample action T, and the formula is calculated. Compute the pairwise L between each identical part b action block in X and Y 2 Distance Matrix D b . represents the i-th sequence extracted from the initial input, is the jth sequence extracted from the sample action. Using the normalized distance matrix Compute similarity bidirectionally. Ensure that the generated action sequences not only contain the sequences in the examples, but also that the sequences in the synthesis do not deviate too much from the style of the examples.

[0040] For each initial input sequence Find the most similar sequence among the sample actions using the distance matrix The average voting method is used to mix the values ​​of the collected action sequences to form a synthetic partial body action F b Finally, the values ​​of the overlapping joints between the skeletons are averaged and all the synthesized partial body motions are assembled into a digital human motion sequence.

[0041] 2. Multi-stage action matching synthesis

[0042] Figure 3 The process flow chart of the present invention is given. The entire action generation process S is divided into multiple stages, each of which aims to generate a more refined action sequence. Each stage takes the action sequence generated at the previous level as the initial input and gradually refines it by matching and fusion.

[0043] In the initial stage, random Gaussian noise F is input 0 ~N(μ,σ 2). At each stage s (e.g. s=2, 3, ..., S), the input is the output sequence F of the previous stage s-1 The result after upsampling is denoted as F s =F s-1 ↑ r The matching algorithm matches the initial input sequence F at the current stage s and the action sequence set T extracted from the sample actions s After matching, generating and mixing, the synthetic action sequence F of the current stage is generated. s , and output it to the next stage F s+1 The model iterates this process until it generates a complete action sequence F S .

[0044] In summary, the present invention discloses a method for generating digital human motion based on sample synthesis, which generates diversified motion sequences using sample motion sequences. The beneficial effects of the present invention are: by introducing noise input and inheriting the traditional motion matching method, high-fidelity, smooth and diversified motion sequences can be generated. Long-term offline training is not required, and high-quality motion can be synthesized in a very short time.

[0045] The above describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the protection scope of the present invention in any form, and all technical solutions obtained by equivalent replacement and the like fall within the protection scope of the present invention.

[0046] The parts not involved in the present invention are the same as the prior art or can be implemented by using the prior art.

Claims

1. A method for generating digital human actions based on sample synthesis, characterized in that The following steps are involved: Step 1: Use foot displacement O, joint rotation R, and ground contact label L to represent the complete action sequence; Step 2: Extract the action skeleton sequence of the sample and connect the sample action sequence and the ground contact label; Step 3: Calculate the distance matrix, introduce bidirectional similarity as a protection mechanism, and match and fuse the initially generated action sequence and sample action sequence that conform to the statistical distribution through the normalized distance matrix; Step 4: Use progressive motion synthesis to iteratively refine the action sequence stage by stage from step 1.

2. The method for generating digital human motion based on sample synthesis according to claim 1, characterized in that: The specific steps of step 1 are as follows: Step 1.1: Since the local foot displacement has the characteristic of time invariance, the local foot displacement is used Instead of foot displacement O, T is the time interval of the sample, and represents joint rotation, where J is the number of joints and Q is the number of rotation features; Step 1.2: By calculating L tj =1[||FK([R,O]) tj ||2<ε] Set the contact label L∈{0,1} T×C , where T represents the action sequence, C represents the number of foot joints, and L tj The meaning is: if the calculated velocity norm of the forward motion joint j∈C in frame t∈{1,...,T} is less than the threshold ε, then L tj The label is 1 otherwise, 0. The introduction of contact labels can alleviate the common foot sliding artifacts.

3. The method for generating digital human motion based on sample synthesis according to claim 1, characterized in that: The specific steps of step 2 are as follows: Step 2.1: Input sample sequence to action generation matching algorithm; Step 2.2: The motion generation matching algorithm obtains data information representing the foot displacement O and joint rotation R. The difference in global coordinates between consecutive actions is calculated to obtain the local foot displacement V; Step 2.3: Split the skeleton B into multiple overlapping parts according to function, flatten each part, obtain the local feature vector of the body movement, and connect the foot contact label L to the local feature vector.

4. The method for generating digital human motion based on sample synthesis according to claim 1, characterized in that: The specific steps of step 3 are as follows: Step 3.1: The action generation matching algorithm generates a new action sequence through the initial action sequence X and the sample action sequence Y, ensuring that the generated action postures are diverse and do not deviate from the sample statistical distribution. The action generation matching algorithm calculates the distance norm of the action sequences X and Y and saves it to the distance matrix D; Step 3.2: To ensure that the generated actions maintain visual integrity, the action generation matching algorithm assigns a higher score to the key action sequence similarity, increasing the probability that the key action sequence is selected and appears in the output sequence; the score is normalized using the distance matrix of each sequence When the key action sequence is missing, the normalized value becomes larger and the score becomes smaller. When the key action sequence appears, the normalization factor is closer to α. The parameter α is used to adjust the completeness of the generated action sequence.

5. The method for generating digital human motion based on sample synthesis according to claim 1, characterized in that: The specific steps of step 4 are as follows: Step 4.1: Set the upsampling factor r>1, upsample the action sequence, obtain action sequences with different resolutions, and form an action sample pyramid; Step 4.2: In the first stage, direct generation is adopted. Random noise N is input into the action matching algorithm, action sequences are extracted, and the normalized distance matrix M is calculated. For each action sequence in the initial input, the best matching action sequence is found in the sample action sequence. The initial input and the matching action sequence are mixed to form a new action. The values ​​of the overlapping joints of the skeleton are averaged to form the action sequence of the digital human. Step 4.3: Upsample the generated action sequence and iterate the above steps to make the difference between the generated action sequence and the sequence distribution of the sample small enough and no longer have obvious improvement.

6. The method for generating digital human motion based on sample synthesis according to claim 4, characterized in that: In step 3.1, the reduction process is as follows: (3.11) Define X as the set of action subsequences extracted from the initial action sequence N, and Y as the set of action subsequences extracted from the sample sequence T; (3.12) Calculate the MSE distance between each query sequence X in N and each action sample sequence Y in T.

7. The method for generating digital human motion based on sample synthesis according to claim 4, characterized in that: In step 3.2, the reduction process is as follows: (3.21) For each element of the distance matrix, calculate the normalized score S, and use the score to replace the corresponding element of the distance matrix to obtain the normalized distance matrix adjusted by the control coefficient. For each query action sequence X, find the action sequence with the minimum normalized score. When there are multiple identical element values, use average voting to merge to obtain the matching action sequence; (3.22) For each partial body motion sequence obtained, the sequence average of the overlapping parts of the skeleton is calculated to form the final result.

Images