A human motion sequence generation model training method, generation method and system

Abstract

The application provides a human motion sequence generation model training method, generation method and system. A training sample set trains an initial human motion sequence generation model. The sample contains a human motion sequence and text description data thereof, and a corresponding matchstick figure is generated based on a preset matchstick figure generation algorithm. Noise is added to the human motion sequence to obtain a noise motion sequence. A motion vector, a matchstick figure vector and a text vector are input into a multi-layer multi-condition fusion module, and the matchstick figure vector and the text vector are at least divided and combined to obtain a plurality of combined input vectors. An offset is obtained by fusing the motion vector through an attention mechanism. The target motion vector is obtained by adding the offset to the motion vector. After re-encoding, the target motion vector is input into a backbone network and output to obtain a predicted motion noise and a position index prediction value. The predicted motion noise is removed to obtain a human motion sequence prediction value. The initial human motion sequence generation model is updated in parameters according to a total loss function to obtain a human motion sequence generation model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human motion sequence generation, and in particular to a human motion sequence generation model training method, a human motion sequence generation method and a system. BACKGROUND

[0002] Human motion sequence generation technology is widely used in the fields of film and television production, virtual reality simulation and game industry, and the text-driven human motion sequence generation technology can generate natural human motion sequences according to language descriptions, thereby reducing the burden of 3D animators manually setting key frames.

[0003] In the prior art, a description of human motion, such as "run a few steps and then crouch down", is input into a deep learning-based model, and a human motion sequence is output, which can be represented by 3D joint coordinates or joint rotation matrices; pure text description data is often difficult to accurately capture complex and detailed human motion in the user's mind, such as the simple description "kick the leg forward", which cannot clearly indicate the specific posture of the arms. Previous research has focused on generating fine-grained motion through complex text description data, and Flame allows additional text description data to modify the motion sequence of the character based on the diffusion model; FineMoGen controls each body part of the 3D character through detailed description, but these methods require users to provide more detailed text description data to generate the desired motion sequence. In terms of multi-condition fusion and data processing, the existing methods also have shortcomings. When processing multiple condition inputs, previous work achieves condition combination by using a mask operation in the self-attention module, but calculating attention with a mask label introduces unnecessary computational complexity, and the inconsistency between the stick figure and the text can lead to a decrease in the performance of generating human motion sequences. SUMMARY

[0004] In view of this, the embodiments of the present application provide a human motion sequence generation model training method, a human motion sequence generation method and a system to eliminate or improve one or more defects in the prior art, and solve the problem that the prior art cannot quickly generate natural and accurate human motion sequences.

[0005] One aspect of the present application provides a human motion sequence generation model training method, which comprises the following steps:

[0006] Obtain a training sample set, the training sample set comprising a plurality of samples, each sample containing a human motion sequence for a sample action and text description data thereof; generate a corresponding stick figure for the human motion sequence in each sample based on a preset stick figure generation algorithm; the human motion sequence is labeled with a human motion sequence true value; and the stick figure is labeled with a position index true value;

[0007] The initial human motion sequence generation model includes a diffusion forward noise adding module, a motion encoder, a stick figure encoder, a text encoder, a multi-layer multi-condition fusion module, a backbone network, and a diffusion reverse noise removing module. The multi-layer multi-condition fusion module layer includes a first feature decoder, a second feature decoder, and a latent encoder. The diffusion forward noise adding module receives the human motion sequence and gradually adds noise to obtain a noisy motion sequence. The motion encoder encodes the noisy motion sequence into a motion vector. The stick figure encoder encodes the stick figure into a stick figure vector. The text encoder encodes the text description data into a text vector. The multi-layer multi-condition fusion module at least divides and combines the stick figure vector and the text vector in batches to obtain a plurality of combined input vectors and integrates the motion vector. The first combined input vector at least contains the text vector. The second combined input vector at least contains the text vector and the stick figure vector. The third combined input vector at least contains the stick figure vector. The fourth combined input vector does not contain the text vector and the stick figure vector. The first feature decoder at least receives the first combined input vector and the second combined input vector. The second feature decoder at least receives the third combined input vector and the fourth combined input vector and integrates the motion vector through an attention mechanism to obtain a text-influenced motion offset and a stick figure-influenced motion offset. The sum of the motion vector and the offset is a target motion vector. The target motion vector is re-encoded by the latent encoder and input into the backbone network. The output is a predicted motion noise and a position index prediction value of the stick figure. The diffusion reverse noise removing module gradually removes the predicted motion noise to obtain a human motion sequence prediction value.

[0008] A motion loss function is constructed by the human motion sequence prediction value and the human motion sequence true value. An index loss function is constructed by the position index prediction value and the position index true value. The motion loss function and the index loss function are fused to obtain a total loss function. The motion encoder, the multi-layer multi-condition fusion module, the backbone network, and the diffusion reverse noise removing module are parameter updated to construct the human motion sequence generation model.

[0009] In some embodiments, the index loss function The motion loss function And the total loss function The expression is:

[0010]

[0011] Wherein, M represents a preset weight coefficient, L represents a sequence length, represents the position index prediction value of the l-th frame of the stick figure in the human motion sequence prediction value, represents the human motion sequence generated under the second combined input vector and the third combined input vector, represents the human motion sequence generated under all condition combinations, l represents the action of the l-th frame in the human motion sequence true value, i represents the action of the stick figure generated from the real motion sequence by the stick figure generation algorithm.

[0012] In some embodiments, the method further comprises supervising the motion loss by adopting a classifier-free diffusion guided algorithm, and the supervision expression is:

[0013]

[0014] wherein, represents the expectation value calculated for the noise true value ∈ t , time step t and initial motion x0; ∈ θ represents the predicted motion noise; C(stick) represents the input text vector; C(stick) represents the input stick figure vector.

[0015] In some embodiments, the expression of the predicted motion noise when the human motion sequence prediction value is obtained by step-by-step removal of the predicted motion noise by the diffusion reverse denoising module is:

[0016]

[0017] when t ∈ [T, T / 10],

[0018] when t ∈ [T / 10, 0], (w1 = 1; w2, w3, w4 = 0);

[0019] wherein, stick represents a stick figure, text represents a text, w and is a self-defined weight parameter.

[0020] In some embodiments, the process of generating a corresponding stick figure for the human motion sequence in each sample based on a preset stick figure generation algorithm comprises:

[0021] The lengths of the arms and legs of the human body are used as the denominator to normalize the 3D joint coordinates in the preset joint coordinate dataset, and the 2D normalized coordinates are obtained by observing the front view posture in which the line of sight is perpendicular to the target human pelvis plane;

[0022] generating a standard circle and cutting off the circle at a random position to obtain a head line, connecting the 2D normalized coordinates to obtain limb lines and a spine line;

[0023] adding a first preset number of interpolation coordinate points representing the trajectory of the line to the head line, the limb line and the spine line, accumulating and adding a randomly generated noise array with the same shape and uniform distribution to the line to obtain a noise-added line, smoothing the noise-added line by Gaussian filtering, aligning the noise-added line with the original head line, limb line and spine line by affine transformation, adding a small-amplitude uniformly distributed noise array to obtain a hand-drawn style line;

[0024] After obtaining the head hand-drawn style line, the spine hand-drawn style line and the limb hand-drawn style line, the head hand-drawn style line and the spine hand-drawn style line are reasonably combined according to the neck direction, and the arm hand-drawn style line is connected to the head of the spine hand-drawn style line, and the leg hand-drawn style line is connected to the tail of the spine hand-drawn style line to obtain an initial stick figure, the position of each line of the initial stick figure is disturbed by generating two noise values and adding them to the horizontal and vertical coordinates of the line, and each line of the initial stick figure is equally spaced sampled according to a certain distance interval to unify the number of coordinate points of each line to obtain a stick figure.

[0025] In some embodiments, the noise motion sequence is encoded into a motion vector by a motion encoder, the stick figure is encoded into a stick figure vector by a stick figure encoder, and the text description data is encoded into a text vector by a text encoder, including:

[0026] The motion encoder adopts a simple linear layer to encode the noise motion sequence to obtain a motion vector;

[0027] The stick figure encoder adopts a transform encoder to encode each line of the stick figure to obtain a stick figure vector;

[0028] The text encoder adopts CLIP ViT-B / 32 to encode the text description data to obtain a text vector.

[0029] In another aspect, the present application also provides a human motion sequence generation method, which is performed in the human motion sequence generation model obtained by the human motion sequence generation model training method of any one of the above, and the method comprises the following steps:

[0030] Inputting the randomly generated noise motion sequence and the stick figure and text description data about the target action into the human motion sequence generation model, and outputting the human motion sequence of the target action.

[0031] In another aspect, the present application also provides a human motion sequence generation system, comprising a processor, a memory and a computer program / instruction stored on the memory, wherein the processor is configured to execute the computer program / instruction, and when the computer program / instruction is executed, the system implements the steps of any one of the above-mentioned methods.

[0032] In another aspect, the present application also provides a computer readable storage medium, wherein a computer program / instruction is stored on the computer readable storage medium, and when the computer program / instruction is executed by a processor, the steps of any one of the above-mentioned methods are implemented.

[0033] In another aspect, the present application also provides a computer program product, comprising a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of any one of the above-mentioned methods are implemented.

[0034] In the training method of the human motion sequence generation model based on the matchstick man according to the present application, the matchstick man corresponding to the human motion sequence is generated according to a preset matchstick man generation algorithm, the motion details are controlled by introducing the matchstick man, the demand for complex text description data is reduced, and the consistency between the generated motion and the user's imagination is improved. The noise motion sequence, the matchstick man and the text description data are input into an initial human motion sequence generation model, the matchstick man is encoded by a matchstick man encoder, the embedding representation of the matchstick man is obtained through simple transformation of the encoder interaction, the calculation demand is reduced, the recognition accuracy is improved, the multi-layer multi-condition fusion module has lower calculation complexity and better performance, the problem of inconsistency between the matchstick man and the text representation is effectively solved, and high-quality motion sequences are better generated by fusing multiple input conditions. The first feature decoder receives at least the first combined input vector and the second combined input vector, the second feature decoder receives at least the third combined input vector and the fourth combined input vector, and the motion vector is fused through an attention mechanism to obtain a text influence motion offset and a matchstick man influence motion offset. After the motion vector is added, the target motion vector is obtained, the target motion vector is input into a latent encoder for re-encoding and then input into a backbone network, a predicted motion noise and a position index prediction value of the matchstick man are output, the noise in the predicted noise is removed step by step to obtain a human motion sequence prediction value, the multi-condition fusion performance is improved, the generation process is optimized, more natural and accurate human motion sequence generation is realized, and a method for more simply and quickly obtaining ideal human motion sequence is provided.

[0035] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will become apparent to those skilled in the art upon examination of the following detailed description and drawings in which:

[0036] Those skilled in the art will appreciate that the objects and advantages of the application can be obtained by specific structures as specifically described herein below, and that the said and other objects and advantages of the application can be obtained from the practice of the application as described below. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description, serve to explain the principles of the application.

[0038] Figure 1 Flow chart of the training method of the human motion sequence generation model according to an embodiment of the present application.

[0039] Figure 2 Structure diagram of the training method of the human motion sequence generation model according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the embodiments and drawings. Herein, the illustrative embodiments of the present application and their descriptions are used to explain the present application, but are not used to limit the present application.

[0041] It should be noted that, in order to avoid the present application being obscured by unnecessary details, only the structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0042] It should be emphasized that the terms "comprises / comprising" when used in this specification, specify the presence of stated features, elements, steps or components, but do not preclude the presence or addition of one or more other features, elements, steps or components.

[0043] It should be noted that, if not specifically stated, the term "connected" herein can not only mean direct connection, but also mean indirect connection with an intermediate.

[0044] In the following, embodiments of the present application will be described with reference to the drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0045] In the prior art, a description of human body movement such as "run a few steps and squat down" is input into a deep learning-based model, and a human body movement sequence is output. Purely relying on text description data often cannot accurately capture complex and detailed human body movements in the user's mind, such as the simple description of "kicking the leg forward", which cannot clearly indicate the specific posture of the arm. Previous studies have focused on generating fine-grained movements through complex text description data. Flame allows additional text description data to modify the movement sequence of the character based on the diffusion model. FineMoGen controls each body part of the 3D character through detailed description, but these methods require users to provide more detailed text description data to generate the desired movement sequence. In terms of multi-condition fusion and data processing, existing methods also have shortcomings. When processing multiple condition inputs, previous work uses a mask operation in the self-attention module to achieve condition combination, but calculating attention with a mask tag will introduce unnecessary computational complexity, and the inconsistency between the stick figure and the text will cause performance degradation. The present application provides a human body movement sequence generation model training method, a generation method and a system. Each sample in the training sample set contains a human body movement sequence for the sample action and its text description data. The human body movement sequence in each sample is generated based on a preset stick figure generation algorithm to generate a corresponding stick figure. The training sample set is used to train the initial human body movement sequence generation model. Noise is added to the human body movement sequence to obtain a noise movement sequence. The noise movement sequence is encoded into a movement vector by a movement encoder, the stick figure is encoded into a stick figure vector by a stick figure encoder, and the text description data is encoded into a text vector by a text encoder. A multi-layer multi-condition fusion module at least divides and combines the stick figure vector and the text vector into multiple combined input vectors and integrates the movement vector. The first combined input vector at least contains the text vector, the second combined input vector at least contains the text vector and the stick figure vector, the third combined input vector at least contains the stick figure vector, and the fourth combined input vector does not contain the text vector and the stick figure vector. The first feature decoder receives at least the first combined input vector and the second combined input vector, the second feature decoder receives at least the third combined input vector and the fourth combined input vector, and the movement vector is integrated through an attention mechanism to obtain a text-affected movement offset and a stick figure-affected movement offset. The target movement vector is obtained by adding the movement vector. The latent encoder is re-encoded and input into the backbone network to output the predicted movement noise and the position index prediction value of the stick figure. The human body movement sequence prediction value is obtained by gradually removing the predicted movement noise through the diffusion reverse denoising module.A motion loss function is constructed by a human motion sequence prediction value and a human motion sequence true value, an index loss function is constructed by a position index prediction value and a position index true value, a total loss function is obtained by fusing the motion loss function and the index loss function, and parameter updating is performed on the motion encoder, the multi-layer multi-condition fusion module, the backbone network and the diffusion reverse denoising module to construct a human motion sequence generation model.

[0046] Figure 1 The flowchart of the training method of the human motion sequence generation model is described in an embodiment of the present application. Specifically, one aspect of the present application provides a training method of a human motion sequence generation model, which comprises the following steps S101-S103:

[0047] Step S101: obtaining a training sample set, the training sample set comprising a plurality of samples, each sample containing a human motion sequence for a sample action and text description data thereof; generating a corresponding matchstick man for the human motion sequence in each sample based on a preset matchstick man generation algorithm; the human motion sequence taking a human motion sequence true value as a label; and the matchstick man taking a position index true value as a label.

[0048] Step S102: training an initial human motion sequence generation model using the training sample set; the initial human motion sequence generation model comprising a diffusion forward noise adding module, a motion encoder, a matchstick man encoder, a text encoder, a multi-layer multi-condition fusion module, a backbone network and a diffusion reverse denoising module; the multi-layer multi-condition fusion module layer comprising a first feature decoder, a second feature decoder and a latent encoder; the diffusion forward noise adding module receiving the human motion sequence and gradually adding noise to obtain a noisy motion sequence, encoding the noisy motion sequence into a motion vector by the motion encoder, encoding the matchstick man into a matchstick man vector by the matchstick man encoder, encoding the text description data into a text vector by the text encoder, at least batch dividing and combining the matchstick man vector and the text vector by the multi-layer multi-condition fusion module to obtain a plurality of combined input vectors and fuse the motion vector, the first combined input vector containing at least the text vector, the second combined input vector containing at least the text vector and the matchstick man vector, the third combined input vector containing at least the matchstick man vector, and the fourth combined input vector not containing the text vector and the matchstick man vector, obtaining a text influenced motion offset and a matchstick man influenced motion offset by at least receiving the first combined input vector and the second combined input vector by the first feature decoder, at least receiving the third combined input vector and the fourth combined input vector by the second feature decoder and fusing the motion vector by an attention mechanism, adding the motion vector to obtain a target motion vector, re-encoding by the latent encoder and inputting into the backbone network, outputting a predicted motion noise and a position index prediction value of the matchstick man; and obtaining a human motion sequence prediction value by gradually removing the predicted motion noise by the diffusion reverse denoising module.

[0049] Step S103: constructing a motion loss function by the human motion sequence predicted value and the human motion sequence true value, constructing an index loss function by the position index predicted value and the position index true value, fusing the motion loss function and the index loss function to obtain a total loss function, and taking minimizing the total loss function as a target to update parameters of the motion encoder, the multi-layer multi-condition fusion module, the backbone network and the diffusion reverse denoising module and construct a human motion sequence generation model.

[0050] In step S101, the pre-set motion sequence dataset contains multiple human motion sequences, and the human motion sequence is a series of action sets arranged in time sequence, which is commonly used to describe the motion change of a person or object in a period of time. The stickman generation algorithm automatically generates a stickman based on the 3D joint coordinates in the existing joint coordinate dataset. The stickman algorithm converts the 3D joint coordinates into 2D stroke coordinate sequence data, and represents the stickman through a series of coordinate sequences. The stroke smoothness, position deviation and scaling difference are considered in the stickman generation process. In some embodiments, the process of generating a corresponding stickman for each human motion sequence in the sample based on the pre-set stickman generation algorithm includes steps S1011-S1014:

[0051] Step S1011: normalizing the 3D joint coordinates in the pre-set joint coordinate dataset by taking the length of the human arm and leg as the denominator, and obtaining 2D normalized coordinates through front view observation posture with the line of sight perpendicular to the target human pelvis plane.

[0052] Step S1012: generating a standard circle, randomly extending the head line after cutting the circle at a random position, and connecting the 2D normalized coordinates to obtain the limb line and the spine line.

[0053] Step S1013: adding a first pre-set number of interpolation coordinate points representing the trajectory of the line to the head line, the limb line and the spine line, then adding a randomly generated noise array with the same shape as the line and uniformly distributed to the line to obtain a noisy line, and adding a small amplitude uniformly distributed noise array to the generated hand-drawn style line after smoothing the noisy line through Gaussian filtering and aligning it with the original head line, limb line and spine line through affine transformation.

[0054] Step S1014: After obtaining the head hand-drawn style line, the spine hand-drawn style line and the limb hand-drawn style line, the head hand-drawn style line and the spine hand-drawn style line are reasonably combined according to the neck direction, and the arm hand-drawn style line is connected to the head of the spine hand-drawn style line, and the leg hand-drawn style line is connected to the tail of the spine hand-drawn style line to obtain an initial stick figure. The position of each line of the initial stick figure is disturbed by adding two noise values to the horizontal and vertical coordinates of the line, and each line of the initial stick figure is equidistantly sampled according to a certain distance interval to unify the number of coordinate points of each line to obtain a stick figure.

[0055] In step S102, the diffusion forward noise adding module adds Gaussian noise to the human motion sequence in the diffusion forward process of the diffusion model, and gradually adds Gaussian noise ∈ to the initial actual motion x0 from the initial time t=0 to the final time t=T t ~N(0,I), the Gaussian noise obeys the normal distribution with mean 0 and covariance matrix I, and the weight of the added noise is an increasing sequence β t The probability distribution formula of the forward process is Where q(x 1:T |x0) represents the joint probability of a series of noisy motion sequences from x1 to x T given the initial motion x0. is a continuous multiplication symbol, indicating that the probability q(x t |x t-1 ) of each time step from t=1 to t=T is multiplied; It is shown that, given the motion x t-1 at the previous time, the motion x t at the current time obeys the normal distribution, whose mean is and the variance is (1-α t )I. Here, α t is related to the weight β t , α t =1-β t ; the formula is equivalent to Where is the product of all α s from 1 to t.

[0056] In some embodiments, encoding the noisy motion sequence into a motion vector by a motion encoder, encoding the stick figure into a stick figure vector by a stick figure encoder, and encoding the text description data into a text vector by a text encoder, comprises: the motion encoder employs a simple linear layer to encode the noisy motion sequence to obtain the motion vector. The stick figure encoder employs a transformer encoder to encode each line of the stick figure to obtain the stick figure vector. The text encoder employs CLIPViT-B / 32 to encode the text description data to obtain the text vector. Specifically, the noisy motion sequence, the stick figure, and the text description data are respectively encoded into [L m ,E]、[L s ,E] and [L t ,E] shaped vectors, where L m ,L s ,L t represents the sequence length of each input, and E represents the dimension of the input encoding; the stick figure encoder uses a transformer encoder to separately encode each line of the six lines representing the head, torso, and limbs of the stick figure, and then obtains the embedding representation of the stick figure through the encoder interaction, which reduces the computational demand and improves the accuracy.

[0057] Further, each layer in the multi-layer multi-condition fusion module divides and combines the stick figure vector and the text vector of the current batch input according to the batch dimension to obtain multiple input condition combinations: a first combined input vector B1, a second combined input vector B2, a third combined input vector B3, and a fourth combined input vector B4; the condition combination input containing the text vector in the four input condition combinations only considers the first feature decoder (Feat Decoder) of the text vector, the condition combination input containing the stick figure vector only considers the second feature decoder of the stick figure vector as a value vector and a key vector, and is fused with the motion vector as a query vector through a self-attention mechanism to obtain a text influence motion offset representing the direction and degree of adjustment of the motion vector by the text vector, and a stick figure influence motion offset representing the direction and degree of adjustment of the motion vector by the stick figure vector; the target motion vector is re-encoded by a latent encoder to further fuse information; the backbone network adopts a transformer; the human motion sequence prediction value is obtained by gradually removing the motion noise of the noisy motion sequence; the human motion sequence is visualized according to the 3D coordinates; the position index score is used to represent the position of the action represented by the input stick figure in the noisy motion sequence, which helps the user to determine whether the output human motion sequence meets the requirements of the stick figure and the text description data.

[0058] In some embodiments, when the human motion sequence prediction value is obtained by gradually removing the predicted motion noise through the diffusion reverse denoising module, the expression of the predicted motion noise is:

[0059]

[0060] When t∈[T,T / 10],

[0061] When t∈[T / 10,0], (w1=1; w2,w3,w4=0);

[0062] Where stick represents a stick figure, text represents text, and w and These are custom weight parameters.

[0063] Specifically, during the training of the human motion sequence generation model, the conditional mixture weights w1, w2, w3, and w4 are adjusted to bias the generated human motion sequences towards different conditional combinations. A classifier-free free diffusion guided algorithm is used to balance the output human motion sequence between stick figure vectors and text vectors. The motion loss is supervised by calculating the expected value based on predicted motion noise and the actual noise value. The probability of the stick figure vector input condition and the probability of the text vector input condition appearing during the predicted motion noise generation process are both 0.7. During the diffusion-backward denoising module, the conditional mixture weights w1, w2, w3, and w4 are adjusted such that w1 + w2 + w3 + w4 = 1 to bias the output predicted motion noise towards different combinations of input vectors. Weight allocation is defined for the initial and final stages to optimize the predicted noise; (stick, text) represents the combined input vector containing stick figure vectors and text vectors. This represents a combined input vector containing stick figure vectors. This represents a combined input vector that includes text vectors. This represents a combined input vector that does not contain stick figure vectors or text vectors.

[0064] In step S103, the motion loss function, the index loss function, and the fusion loss function are constructed. In some embodiments, the index loss function... Motion loss function Sum of total loss functions The expression is:

[0065]

[0066] Where M represents the preset weight coefficient, and L represents the sequence length. This represents the predicted position index of the stick figure in the l-th frame of the human motion sequence prediction value. This represents the human motion sequence generated under the second and third combined input vectors. x represents the sequence of human motion generated under all combinations of conditions. lx l represents the action of the l-th frame in the real value of the human motion sequence i x l represents the action of the l-th frame in the real value of the human motion sequence

[0067] In some embodiments, the method further comprises supervising the motion loss with a classifier-free diffusion guided algorithm, and the supervision expression is:

[0068]

[0069] wherein, x l represents the action of the l-th frame in the real value of the human motion sequence t , a time step t and an initial motion x 0 to calculate the expected value; x l represents the action of the l-th frame in the real value of the human motion sequence θ x l represents the action of the l-th frame in the real value of the human motion sequence

[0070] In another aspect, the present application also provides a human motion sequence generation method, which is performed in the human motion sequence generation model obtained by the training method of any one of the above-mentioned human motion sequence generation models, and the method comprises the following steps:

[0071] A randomly generated noise motion sequence and a matchstick and text description data about a target action are input into the human motion sequence generation model, and a human motion sequence of the target action is output.

[0072] Specifically, a noise motion sequence is obtained by adding Gaussian noise to a series of action sets arranged in time sequence through a diffusion forward process, and a target action is selected from the noise motion sequence, wherein the target action is a 3D human posture randomly selected from the vicinity of the start, middle and end positions of the noise motion sequence. The matchstick of the target action and the text description data representing the action of the matchstick are input into the trained human motion sequence generation model, and a human motion sequence is output.

[0073] In another aspect, the present application also provides a human motion sequence generation system, comprising a processor, a memory and a computer program / instruction stored on the memory, wherein the processor is configured to execute the computer program / instruction, and the system implements the steps of any one of the above-mentioned methods when the computer program / instruction is executed.

[0074] In another aspect, the present application also provides a computer readable storage medium, which stores a computer program / instruction, and wherein the computer program / instruction is executed by a processor to implement the steps of any one of the above-mentioned methods.

[0075] In another aspect, the present application also provides a computer program product, which comprises a computer program / instruction, and wherein the computer program / instruction is executed by a processor to implement the steps of any one of the above-mentioned methods.

[0076] The application will be described below in connection with a specific embodiment:

[0077] Figure 2 The structure diagram of the training method of the human motion sequence generation model of an embodiment of the application is shown. The application proposes a human motion sequence generation method based on stickman conditions. By introducing a stickman as a condition for controlling human motion details, the dependence on complex text is reduced by combining text description data. At the same time, an efficient multi-condition fusion module and a dynamic supervision strategy are designed to improve the multi-condition fusion performance, optimize the generation process, and realize more natural and accurate human motion generation, thereby providing a simpler and faster method for creators to obtain ideal human motion.

[0078] 1. Stickman generation algorithm: Since there is a lack of hand-drawn stickmen in existing training data sets, a stickman generation algorithm can be used to automatically generate stickmen. An example of the stickman generation algorithm is a simple genetic algorithm (SGA: Simple Genetic Algorithm). This algorithm automatically generates stickmen based on the 3D coordinates of human joints in the existing joint coordinate data set. The generation process takes into account factors such as stroke smoothness, position deviation, and scaling differences. The specific generation process includes:

[0079] (1) 3D human pose coordinate normalization: The length of the arm and leg is used as the denominator to normalize the 3D joint coordinate, and the 2D normalized coordinate is obtained by observing the human pose from the front, with the line of sight perpendicular to the pelvis plane of the pose; connecting each 2D normalized coordinate obtains the limb lines and spine lines.

[0080] (2) Head line: First, a standard circle is generated, and the circle is truncated at a random position and randomly extended as the intersection of the hand-drawn circle.

[0081] (3) After obtaining the head line, spine line and limb line, uniform hand-drawn style generation is performed on the lines. The detailed steps include: 1) N times of interpolation is performed on the line composed of fewer coordinate points. When the arm is composed of 3 coordinate points, the number of coordinate points representing the line trajectory is increased, including but not limited to N coordinate points are interpolated between the shoulder and the elbow, improving the line granularity, for accepting the hand-drawn jitter noise in the next step. 2) A noise array of the same shape as the line is randomly generated, and the distribution is uniform distribution. The array is accumulated from the beginning to simulate the large error of hand-drawing. 3) The accumulated noise array is added to the interpolated line, and the noisy line is smoothed using Gaussian filtering. 4) The processed line is aligned with the original line through affine transformation. 5) Finally, a small amount of uniformly distributed noise array is generated again and added to the line to simulate the small jitter of hand-drawing.

[0082] (4) Line combination and sampling: after obtaining the hand-drawn style lines of the head, spine and limbs, the lines need to be combined and a certain disturbance is added to the combined position to represent the hand-drawing error, and finally equidistant sampling is performed. The specific steps include: 1) The head line and the spine line are reasonably combined according to the direction of the neck vector, and the arm line and the leg line are connected to the head and tail of the spine line respectively. 2) Then, a whole position offset disturbance is added to each combined line, that is, two noise values are added to the horizontal and vertical coordinates of the line to randomly offset the line, representing the hand-drawing positioning error. 3) Equidistant sampling is performed on each line, that is, the lines are sampled at a certain distance interval to unify the number of coordinate points of each line.

[0083] (5) Information encoding: in order to balance user convenience and computing efficiency, the user only needs to draw six one-stroke lines representing the head, torso and limbs. After each line is encoded separately, the embedding representation of the stick figure is obtained through the interaction of a simple Transformer encoder, reducing the computing demand and improving the recognition accuracy.

[0084] 2. Diffusion-based motion generation: a diffusion model is used to decompose the task into forward and reverse processes.

[0085] 2.1 Forward process: gradually add Gaussian noise ∈ t ~ N(0, I) to the actual motion x0 from t = 0 to t = T. Here, t represents the time step, which gradually advances from the initial time 0 to the final time T; x0 is the initial actual motion sequence; ∈ t is the Gaussian noise added at time step t, which follows a normal distribution with mean 0 and covariance matrix I. The weight of the added noise is an increasing sequence β t . The probability distribution formula of the forward process is where q(x 1:T |x0) represents the probability distribution of x1 to xT The joint probability of this series of noisy motion sequences; It is a multiplication symbol, representing the probability q(x) at each time step from t=1 to t=T. t |x t-1 Multiply by . This indicates that the motion x at the previous moment is known. t-1 In the case of current motion x t It follows a normal distribution with a mean of 1 / 2. The variance is (1-α) t I. Here, α t With weight β t Related, α t =1-β t This formula is equivalent to in It is all α from time 1 to time t s The product of the noisy motion x and the initial motion x0, and the noise ∈ t The relationship between them is used for model training.

[0086] 2.2 Reverse Process: Gradually remove noise X using the trained model. T The formula for obtaining the actual motion sequence is: Where x t-1 The motion sequence from the previous moment to be recovered; x t It is the noisy motion sequence at the current moment; α t Consistent with the meaning in the forward process; ∈ θ It is the noise predicted by the model, output by the trained model; σ t It is weighted Gaussian noise, used to appropriately introduce randomness during the denoising process to help the model better recover the real motion sequence.

[0087] 2.3 Condition Mixing: To balance the impact of stick figure and text conditions on the final generation result during the reverse process, classifier-free diffusion guidance is adopted in the forward process (training process). The supervision formula is as follows: in This represents the expectation, here for noise ∈ t Find the expected value of the time step t and the initial motion x0; 2 It is to find the square norm; ∈ t It is real noise; ∈ θ (x t ,t,L,C(stick),C(text)) is a model based on the current noisy motion sequence x t, time step t, generated action sequence length L, stickman condition C(stick) and text condition C(text) predict motion noise, at training time Here and respectively represent the probability of the occurrence of the stickman condition and the text condition, both set to 0.7.

[0088] In the reverse process (generation process), by adjusting the condition mixing weight (w1, w2, w3, w4), satisfying w1+w2+w3+w4=1, the generation result is biased towards different condition combinations, i.e. four conditions. In the initial stage (t∈[T,T / 10]), the condition mixing formula is (w1=w; w4=1-2·w), here w and are self-defined weight parameters, and through such weight distribution, the influence of different conditions is balanced in the initial stage; in the final stage (t∈[T / 10,0]), set (w1=1; w2, w3, w4=0) to further optimize the result, i.e. the final stage mainly relies on the conditions (sitck, text) to perform final optimization on the result.

[0089] 3.3D human motion sequence generation method structure diagram.

[0090] 3.1 Input encoding: the input data includes noise motion sequence (Motion), stickman (Stickman) and text description data (Text), which are respectively encoded into [L m ,E]、[L s ,E] and [L t ,E] shaped vectors, where L m ,L s ,L t represent the sequence length of each input, and E represents the dimension of the input. The noise motion sequence is encoded using a simple linear layer; the text description data is encoded using CLIP ViT-B / 32 (containing 154M parameters); the stickman is encoded through a standard Transformer encoder and decoded through a decoder to restore the 3D pose to realize loss supervision. The above text encoder and stickman encoder will be frozen during training, i.e. not involved in the training process of the human motion sequence generation model.

[0091] 3.2 Multi-Condition Module (MCM): The Multi-Condition Module is used to fuse conditions, which fuses the stick figure vector and the text vector into the motion vector in latent space through the Multi-Condition Module, and then re-encodes it with the latent encoder to further fuse information. The data is divided into four parts (B1, B2, B3, B4) in the batch dimension, representing the four combinations of text vector and stick figure vector conditions; two standard Transformer decoder layers: the feature decoder (Feat Decoder) considers the text input and the stick figure input respectively, and obtains the new motion vector by adding it to the corresponding motion vector, which reduces the computational complexity and improves the performance. At the same time, an efficient attention mechanism is used in the feature decoder and the latent encoder to further reduce the computational load.

[0092] 3.3 Output: The human motion sequence generation model is composed of multiple layers of multi-condition fusion modules, and outputs the predicted noise ∈ of the noise motion sequence θ and the position index score of the input stick figure The index score is used for training supervision and user interaction, which supervises the distance between the dynamically assigned pose and the stick figure during training, and indicates the position of the stick figure in the motion sequence during generation, helping the user to decide whether to adjust the generated result. The predicted noise ∈ θ is used for the reverse process of the diffusion model, i.e. removing the noise from the input noise to restore the final human motion sequence. The index score is used for training supervision and user interaction, which supervises the distance between the dynamically assigned pose and the stick figure during training, and indicates the position of the stick figure in the motion sequence during generation, helping the user to decide whether to adjust the generated result.

[0093] 3.4 Model training: The overall training process of the human motion sequence generation model includes obtaining 3D human motion sequences from the training data set, selecting 3 3D human poses near the beginning, middle and end of the sequence and converting them to 2D hand-drawn style stick figures as input through the stick figure generation algorithm. At this time, the stick figure, the noise motion sequence after adding noise and the corresponding text description data are input into the human motion sequence generation model as three inputs, and the final output is the predicted noise ∈ θ and the position index score of the input stick figure position

[0094] 4. Dynamic supervision. To solve the problem of determining the index of the stick figure in the generated motion sequence, the user only needs to specify the approximate position of the stick figure at the beginning, middle or end, and the network automatically adjusts the index near the specified position to optimize the naturalness of the generated result and the fit with the text description data. By randomly sampling human poses at the beginning, middle and end of the motion sequence and creating a mask to simulate user random input, the loss function is divided into index loss function motion loss function and total loss function are respectively where, is the predicted index score of the l-th frame of the stick figure in the generated motion sequence, is the generated motion sequence under the (stick figure, text) and (stick figure, empty) condition combinations, is the predicted pose of the l-th frame under all condition combinations, x l is the pose of the l-th frame of the real motion sequence, x i is the pose of the l-th frame of the stick figure generated from the real motion sequence by the stick figure generation algorithm.

[0095] In summary, the present application provides a human motion sequence generation model training method, generation method and system. The training method comprises: obtaining a training sample set comprising a human motion sequence for a sample action and text description data of the human motion sequence; generating a matchstick man corresponding to the human motion sequence based on a preset matchstick man generation algorithm; training an initial human motion sequence generation model using the training sample set; a diffusion forward noise adding module receives the human motion sequence and gradually adds noise to obtain a noisy motion sequence, encodes the noisy motion sequence into a motion vector using a motion encoder, encodes the matchstick man into a matchstick vector using a matchstick encoder, and encodes the text description data into a text vector using a text encoder; a multi-layer multi-condition fusion module at least divides and combines the matchstick vector and the text vector in batches to obtain a plurality of combined input vectors and fuses the motion vector, wherein the first combined input vector at least contains the text vector, the second combined input vector at least contains the text vector and the matchstick vector, the third combined input vector at least contains the matchstick vector, and the fourth combined input vector does not contain the text vector and the matchstick vector; a first feature decoder receives at least the first combined input vector and the second combined input vector, a second feature decoder receives at least the third combined input vector and the fourth combined input vector, and a motion vector is fused through an attention mechanism to obtain a text-affected motion offset and a matchstick-affected motion offset, which are added to the motion vector to obtain a target motion vector, which is input into a backbone network after being re-encoded, and a predicted motion noise and a position index prediction value of the matchstick man are output; a diffusion reverse de-noising module gradually removes the predicted motion noise to obtain a human motion sequence prediction value; a motion loss function is constructed based on the human motion sequence prediction value and a human motion sequence true value, an index loss function is constructed based on the position index prediction value and a position index true value, the motion loss function and the index loss function are fused to obtain a total loss function, and the motion encoder, the multi-layer multi-condition fusion module, the backbone network and the diffusion reverse de-noising module are parameter updated to construct the human motion sequence generation model.

[0096] Corresponding to the above method, the present application also provides a human motion sequence generation system comprising a processor, a memory and a computer program / instruction stored on the memory, wherein the processor is configured to execute the computer program / instruction, and when the computer program / instruction is executed, the system implements the steps of the above method.

[0097] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the foregoing edge computing server deployment method. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0098] Those skilled in the art should understand that the exemplary components, systems and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software or a combination thereof. The exact implementation depends on the specific application and design constraints imposed on the overall system. Those skilled in the art can use various methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link.

[0099] It should be noted that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of known methods are omitted. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the present application.

[0100] In the present application, the features described and / or illustrated for one embodiment can be used in the same way or in a similar way in one or more other embodiments, and / or in combination with or instead of features of other embodiments.

[0101] The above description is merely preferred embodiments of the present application, and is not intended to limit the present application. The embodiments of the present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the present application.

Claims

1. A method for training a human motion sequence generation model, characterized in that, The method comprises the following steps: Obtain a training sample set, which comprises a plurality of samples, each sample containing a human motion sequence for a sample action and its text description data; generate a corresponding matchstick man for the human motion sequence in each sample based on a preset matchstick man generation algorithm; the human motion sequence is labeled with a human motion sequence ground truth; the matchstick man is labeled with a position index ground truth; Train an initial human motion sequence generation model using the training sample set; the initial human motion sequence generation model comprises a diffusion forward noise adding module, a motion encoder, a matchstick man encoder, a text encoder, a multi-layer multi-condition fusion module, a backbone network and a diffusion reverse noise removing module; the multi-layer multi-condition fusion module layer comprises a first feature decoder, a second feature decoder and a latent encoder; the diffusion forward noise adding module receives the human motion sequence and gradually adds noise to obtain a noisy motion sequence, encodes the noisy motion sequence into a motion vector through the motion encoder, encodes the matchstick man into a matchstick man vector through the matchstick man encoder, encodes the text description data into a text vector through the text encoder, and the multi-layer multi-condition fusion module at least divides and combines the matchstick man vector and the text vector in batches to obtain a plurality of combined input vectors and integrates the motion vector, wherein the first combined input vector at least contains the text vector, the second combined input vector at least contains the text vector and the matchstick man vector, the third combined input vector at least contains the matchstick man vector, and the fourth combined input vector does not contain the text vector and the matchstick man vector; the first feature decoder at least receives the first combined input vector and the second combined input vector, the second feature decoder at least receives the third combined input vector and the fourth combined input vector, and after integrating the motion vector through an attention mechanism, text-influenced motion offsets and matchstick man-influenced motion offsets are obtained, which are added to the motion vector to obtain a target motion vector, which is re-encoded by the latent encoder and input into the backbone network to output predicted motion noise and position index prediction value of the matchstick man; the diffusion reverse noise removing module gradually removes the predicted motion noise to obtain a human motion sequence prediction value; Construct a motion loss function based on the human motion sequence prediction value and the human motion sequence ground truth, construct an index loss function based on the position index prediction value and the position index ground truth, fuse the motion loss function and the index loss function to obtain a total loss function, and update the parameters of the motion encoder, the multi-layer multi-condition fusion module, the backbone network and the diffusion reverse noise removing module to minimize the total loss function to obtain the human motion sequence generation model.

2. The method of training a human motion sequence generation model according to claim 1, wherein, The index loss function The motion loss function And the total loss function The expression is: wherein M represents a preset weight coefficient, and L represents a sequence length, represents the position index prediction value of the l-th frame of the stick figure in the human motion sequence prediction value, represents the human motion sequence generated under the second combined input vector and the third combined input vector, represents the human motion sequence generated under all condition combinations, l represents the action of the l-th frame in the human motion sequence ground truth value, i represents the action of the stick figure generated from the real motion sequence by the stick figure generation algorithm.

3. The method of training a human motion sequence generation model according to claim 2, wherein, The method further comprises supervising the motion loss using a classifier-free diffusion guided algorithm, and the supervision expression is: wherein, represents the noise true value ∈ t , time step t and initial motion x0calculate the expected value; ∈ θ represents the predicted motion noise; C(stick) represents the input text vector; C(stick) represents the input stick figure vector.

4. The method of training a human motion sequence generation model according to claim 1, wherein, The expression of the predicted motion noise when the diffusion reverse noise removing module gradually removes the predicted motion noise to obtain a human motion sequence prediction value is: t∈[T, T / 10], w1 = w; w4 = 1 - 2 - w; t∈[T / 10,0] when w1=1; w2,w3,w4=0; where stick represents a stick figure, text represents text, w and are custom weight parameters.

5. The method of training a human motion sequence generation model according to claim 1, wherein, The process of generating a corresponding matchstick man for each sample human motion sequence based on a preset matchstick man generation algorithm comprises: The length of the human arm and leg is used as a denominator to normalize the 3D joint coordinates in the preset joint coordinate dataset, and the 2D normalized coordinates are obtained by observing the front view posture in which the line of sight is perpendicular to the target human pelvis plane; A standard circle is generated, and the head line is obtained by randomly extending the circle after truncation at a random position. The limbs line and spine line are obtained by connecting the 2D normalized coordinates; After adding a first preset number of interpolation coordinate points representing the line trajectory to the head line, the limbs line and the spine line, a randomly generated noise array with the same shape as the line and uniformly distributed is accumulated and added to the line to obtain a noisy line. After smoothing the noisy line by Gaussian filtering, the initial head line, the limbs line and the spine line are aligned through affine transformation. A small amplitude uniformly distributed noise array is added to obtain a hand-drawn style line; After obtaining the head hand-drawn style line, the spine hand-drawn style line and the limbs hand-drawn style line, the head hand-drawn style line and the spine hand-drawn style line are reasonably combined according to the neck direction, and the arm hand-drawn style line is connected to the head of the spine hand-drawn style line, and the leg hand-drawn style line is connected to the tail of the spine hand-drawn style line to obtain an initial matchstick man. The position of each line of the initial matchstick man is disturbed by adding two noise values to the horizontal and vertical coordinates of the line, and each line of the initial matchstick man is equally spaced sampled according to a certain distance interval to unify the number of coordinate points of each line to obtain a matchstick man.

6. The method of training a human motion sequence generation model according to claim 1, wherein, The motion encoder encodes the noise motion sequence into a motion vector, the matchstick man encoder encodes the matchstick man into a matchstick man vector, and the text encoder encodes the text description data into a text vector, comprising: The motion encoder uses a simple linear layer to encode the noise motion sequence to obtain a motion vector; The matchstick man encoder uses a transform encoder to encode each line of the matchstick man to obtain a matchstick man vector; The text encoder uses CLIP ViT-B / 32 to encode the text description data to obtain a text vector.

7. A method of generating a human motion sequence, characterized by, The method is performed in the human motion sequence generation model obtained by the training method of the human motion sequence generation model in any one of claims 1 to 6, comprising the following steps: The randomly generated noise motion sequence, the matchstick man and the text description data about the target action are input into the human motion sequence generation model, and the human motion sequence of the target action is output.

8. A human motion sequence generation system comprising a processor, a memory, and a computer program / instructions stored on the memory, wherein, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method of any one of claims 1 to 7.

9. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the method of any one of claims 1 to 7.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the steps of the method of any one of claims 1 to 7.

Images