A training method for clothing deformation prediction model and animation production method

Abstract

The present invention provides a method for training a clothing deformation prediction model and a method for producing an animation. The training method comprises: obtaining multiple training samples, each training sample comprising a motion posture sequence corresponding to a character and a clothing deformation label sequence; wherein the motion posture sequence comprises multiple temporally consecutive motion postures, each motion posture in the motion posture sequence corresponding to a clothing deformation label in the clothing deformation label sequence; and training the clothing deformation prediction model using the multiple training samples until convergence. The technical solution of the present invention utilizes a model capable of processing sequence-to-sequence to learn the mapping relationship between a complete motion posture sequence and a corresponding clothing deformation sequence. This allows the prediction of a corresponding clothing deformation effect based on a motion posture to fully consider the previous pair of motion postures and clothing deformation effects, thereby ensuring the continuity of the generated clothing deformation effect.

Description

Technical Field

[0001] The present invention relates to the technical field of clothing animation, and more particularly to a method for training a clothing deformation prediction model and a method for producing animation. Background Art

[0002] With the rapid development of CG (computer graphics) technology, clothing animation technology plays an increasingly important role in the fields of film, animation, etc. The core problem lies in how to generate a continuous and stable clothing deformation sequence based on the human body's motion posture sequence. That is: the generated clothing deformation sequence must not only keep the timing consistent with the human body movement, but its fold details must also correspond to the human body posture characteristics.

[0003] Currently, clothing animation creation is mainly based on physical simulation method and instance data driven method.

[0004] Among them, the physical simulation method has high computational cost and requires setting complex simulation parameters. The generated effect is difficult to control and requires artists to make continuous manual iterative adjustments, which is inefficient.

[0005] The instance-driven data method aims to learn the correlation between the human body and clothing deformation from existing clothing deformation data. However, because the instance-driven data method uses multiple pairs of human posture (from posture A to posture B) and clothing deformation (clothing deformation data a to clothing deformation data b) to train a mathematical function during the learning process, the clothing deformation results predicted by the instance-driven data method deviate significantly from the actual results. When the animation frames generated by the clothing deformation results predicted by the instance-driven data method are spliced ​​together, the clothing shape of adjacent frames will produce discontinuous jumps. As a result, it is difficult to ensure the continuity of the generated clothing deformation effect when using the clothing deformation results predicted by the instance-driven data method to create clothing animation. Summary of the Invention

[0006] Therefore, the purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a training method for a clothing deformation prediction model and a method for producing animation.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] According to a first aspect of the present invention, a method for training a clothing deformation prediction model is provided, wherein the clothing deformation prediction model is a sequence-to-sequence model for predicting clothing deformation of a character-customized clothing, the method comprising:

[0009] Acquire multiple training samples, each training sample including a motion posture sequence and a clothing deformation label sequence corresponding to a character; wherein the motion posture sequence includes multiple temporally continuous motion postures, and each motion posture in the motion posture sequence corresponds to a clothing deformation label in the clothing deformation label sequence;

[0010] The clothing deformation prediction model is trained using the multiple training samples to output a clothing deformation sequence according to a motion posture sequence, and the parameters of the clothing deformation prediction model are updated according to a loss value calculated based on the output clothing deformation sequence and a clothing deformation label sequence.

[0011] In some embodiments of the present invention, the motion gestures in the motion gesture sequence are represented by gesture parameters expressing the motion states of the character's joints or by vertex coordinates expressing the positions of various parts of the character;

[0012] The clothing deformation labels in the clothing deformation label sequence are represented by the vertex coordinates of the clothing mesh model corresponding to the clothing or by the deformation gradient matrix of the clothing mesh model, wherein the vertex coordinates of the clothing mesh model indicate the positions of multiple preset vertices that control the change of the clothing shape; and the deformation gradient matrix indicates the deformation amount of the clothing mesh model corresponding to the clothing from an initial state to a target state.

[0013] In some embodiments of the present invention, the training samples are obtained in the following manner:

[0014] Obtain multiple two-dimensional garment pieces of clothing customized for the character, and sew the multiple two-dimensional garment pieces onto the character model in a T-pose to obtain a clothing template corresponding to the clothing;

[0015] Obtain the character's motion posture sequence, let the character model move according to the motion posture sequence, so that the clothing template produces trend deformation, and obtain the clothing deformation sequence of the simulation state corresponding to the character wearing the customized clothing and moving according to the motion posture sequence;

[0016] According to predetermined rules, motion posture sequences and corresponding simulated clothing deformation sequences are intercepted to obtain multiple training samples.

[0017] In some embodiments of the present invention, during the training of the clothing deformation prediction model, the loss value is determined by a vertex error value and a Laplace error value; wherein the vertex error value is determined by the difference value of the clothing vertices in the output clothing deformation sequence and the corresponding clothing deformation label sequence, and the Laplace error value is determined by the difference value of the clothing vertices in the output clothing deformation sequence and the corresponding clothing deformation label sequence after Laplace transformation.

[0018] In some embodiments of the present invention, the loss value is determined as follows:

[0019]

[0020] Among them, L represents the loss value, V t represents the clothing vertex set of the t-th frame in the clothing deformation label sequence, represents the set of clothing vertices in the t-th frame in the output clothing deformation sequence, Δ(·) represents the Laplacian operator, and α represents the weight.

[0021] In some embodiments of the present invention, the clothing deformation prediction model is a Transformer model, a GRU model, or a RNN model.

[0022] In some embodiments of the present invention, the Transformer model includes:

[0023] An encoding component, comprising a plurality of encoders, and configured to encode a motion posture sequence and output an encoding information matrix corresponding to the motion posture sequence;

[0024] The decoding component includes a plurality of decoders and is used to decode the result of clothing deformation using the coding information matrix corresponding to the motion posture sequence to obtain the corresponding clothing deformation sequence.

[0025] In some embodiments of the present invention, the number of the encoders and the decoders is set to 2-4.

[0026] In some embodiments of the present invention, the encoding component includes: an encoding vector mapping module, a position encoding module and multiple encoders connected in sequence, each encoder includes a multi-head attention module, a first residual and layer normalization module, a feedforward network module and a second residual and layer normalization module connected in sequence, wherein the first residual and layer normalization module of the encoder performs layer normalization processing after performing residual addition on the input and output of the multi-head attention module to which it is connected, and the second residual and layer normalization module of the encoder performs layer normalization processing after performing residual addition on the input and output of the feedforward network module to which it is connected,

[0027] The decoding component includes: a decoding vector mapping module, a position encoding module, multiple decoders and a linear module connected in sequence, wherein the decoder includes a masked multi-head attention module, a first residual and layer normalization module, a multi-head attention module, a second residual and layer normalization module, a feedforward network module and a third residual and layer normalization module connected in sequence, the first residual and layer normalization module of the decoder performs residual addition on the input and output of the masked multi-head attention module connected to it and then performs layer normalization processing, the second residual and layer normalization module of the decoder performs residual addition on the input and output of the multi-head attention module connected to it and then performs layer normalization processing, and the third residual and layer normalization module of the decoder performs residual addition on the input and output of the feedforward network module connected to it and then performs layer normalization processing.

[0028] In some embodiments of the present invention, the encoder's multi-head attention module, the decoder's masked multi-head attention module, and the multi-head attention module are all configured with 4 heads.

[0029] According to a second aspect of the present invention, there is provided a method for producing an animation, comprising:

[0030] Get the sequence of motion poses specified for the character in the animation;

[0031] Inputting the motion posture sequence formulated for the character into a clothing deformation prediction model trained using the method described in the first aspect, and outputting a predicted clothing deformation sequence;

[0032] Determining a character motion frame according to each motion posture in the character's motion posture sequence, and determining a clothing deformation frame of a clothing deformation customized for the character according to each clothing deformation in the predicted clothing deformation sequence; each motion posture in the motion posture sequence corresponds to each clothing deformation in the clothing deformation sequence in a one-to-one sequence;

[0033] Each character motion frame is fused with its corresponding clothing deformation frame to obtain an animation frame. All animation frames constitute the animation of the character wearing clothing and moving.

[0034] In some embodiments of the present invention, when there are animation frames in which clothing is sunken into the character's outline in the animation, the animation frames in which clothing is sunken into the character's outline are corrected to obtain a final animation in which the character wears clothing.

[0035] In some embodiments of the present invention, when correcting an animation frame in which clothing is sunken into a character's outline, a target value for the correction is determined by a Laplace penalty value and a minimum distance between the clothing's sunken position and the character's outline; wherein the Laplace penalty value is determined by the difference in Laplace transform values ​​of clothing vertices before and after the correction in the animation frame in which clothing is sunken into the character's outline.

[0036] In some embodiments of the present invention, the target value is determined as follows:

[0037]

[0038] Where H represents the target value, is the Laplace penalty term, is the correction term, Δ(·) represents the Laplace operator, V * represents the corrected clothing vertex set, represents the clothing vertex set before correction, The set of clothing vertices representing where the clothing is indented into the character's outline, The set of clothing vertices whose edges are inset into the character's outline, v k represents the kth invaginated point, Indicates the distance v k The nearest character vertex, τ represents the correction step size, and ω represents the weight.

[0039] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. The computer program can be executed by a processor to implement the steps of the method described in the first aspect or the second aspect.

[0040] According to a fourth aspect of the present invention, there is provided an electronic device, comprising:

[0041] one or more processors; and

[0042] a memory, wherein the memory is used to store executable instructions;

[0043] The one or more processors are configured to implement the steps of the method of the first aspect or the second aspect by executing the executable instructions.

[0044] Compared with the prior art, the advantages of the present invention are:

[0045] The embodiment of the present application adopts a sequence-to-sequence model to learn the mapping relationship between a complete motion posture sequence and a corresponding clothing deformation sequence, so that during the learning process, it is possible to learn the mapping relationship between the current motion posture and the corresponding clothing deformation, and to perform prediction and adjustment based on the mapping relationship between the previous pair of motion postures and clothing deformations to ensure the smoothness or coherence between the predicted clothing deformation and the previous clothing deformation, so that when clothing animation is subsequently produced according to the predicted clothing deformation sequence and the corresponding motion posture sequence, the changes in clothing form in adjacent frames are more coherent, which can effectively ensure the continuity of the generated clothing deformation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0047] Figure 1 A technical roadmap for creating clothing animation for the embodiments of this application;

[0048] Figure 2 This is a comparison chart of the effects of constructing clothing deformation prediction models based on different models in the embodiment of the present application;

[0049] Figure 3 This is a schematic diagram comparing the effects of clothing reconstruction based on different methods in the embodiment of the present application;

[0050] Figure 4 A flowchart of a method for producing clothing animation according to an embodiment of the present application;

[0051] Figure 5 Generate renderings for the clothing animation of the embodiment of this application;

[0052] Figure 6 This is a schematic diagram of the deformation effects of various types of clothing in the embodiments of this application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] As mentioned in the background technology section, the current clothing animation creation is mainly based on physical simulation method and instance data driven method, among which the physical simulation method is less efficient; the instance data driven method is committed to learning the correlation between the human body and clothing deformation from the existing clothing deformation data, but it is difficult to ensure the continuity of the generated clothing deformation effect. In order to solve the above technical problems, the embodiment of the present application adopts a sequence-to-sequence model to learn the mapping relationship between a complete motion posture sequence and a corresponding clothing deformation sequence, so that in the learning process, it is possible to learn the mapping relationship between the current motion posture and the corresponding clothing deformation, and to make predictions and adjustments based on the mapping relationship between the previous pair of motion postures and clothing deformations to ensure the smoothness or coherence between the predicted clothing deformation and the previous clothing deformation, so that when the clothing animation is subsequently produced according to the predicted clothing deformation sequence and the corresponding motion posture sequence, the changes in the clothing form of adjacent frames are more coherent, which can effectively ensure the continuity of the generated clothing deformation effect.

[0055] According to one embodiment of the present invention, the technical solution can be as follows Figure 1 As shown, Figure 1 (a) represents the constructed training samples, which include the same body shape parameters β and different posture parameters θi Constructed multiple motion postures, where i∈(1, T), T represents the number of motion postures, and also includes clothing templates G corresponding to multiple motion postures i , i∈∈(1,T). Figure 1 (b) represents the training process of the clothing deformation prediction model, where (θ1, θ2, ..., θ T ) represents the motion posture sequence, Figure 1 Posture parameters θ corresponding to multiple motion postures in (a) i is composed and input into the encoder module of the clothing deformation prediction model; (V1, V2, ..., V T ) represents the clothing deformation sequence, Figure 1 The vertex coordinates of the clothing mesh model corresponding to multiple clothing templates in (a) are composed and input into the decoder module of the clothing deformation prediction model. Figure 1 (c) represents the generation process of clothing animation, which is achieved by fusing each character motion frame with its corresponding clothing deformation frame to obtain an animation of the character moving in clothing, where each character motion frame is determined by each motion posture in the motion posture sequence, and each clothing deformation frame is determined by each clothing deformation in the predicted clothing deformation sequence.

[0056] In order to illustrate the technical solution of the present invention in more detail, the following is an introduction from five aspects: role introduction, usage model, training samples, training process and application scenarios.

[0057] 1. Role Introduction

[0058] As mentioned in the background technology section, the role played by clothing animation technology in the fields of movies, cartoons, etc. is becoming increasingly important. Its application is mainly reflected in the design of clothes with corresponding deformations according to the movement postures of the characters in movies, cartoons or games, so that the movement effects of the characters are more realistic. Therefore, the characters mentioned in the embodiments of the present application refer to any objects wearing clothes, for example, they can be cartoon characters in cartoons, or they can be game heroes controlled by players in games. In addition, in the field of clothing design, it is also possible to customize some clothes for people (characters) of different body shapes, observe the animation effects of the clothes worn on the characters as the characters move, so as to show them to designers for adjusting the design of clothes, or show them to customers for evaluating whether the design effects are satisfactory, etc.

[0059] 2. Using the Model

[0060] In order to implement the above technical solution, according to one embodiment of the present invention, the clothing deformation prediction model can be implemented using a Transformer model, a GRU model, a LSTM model, or a RNN model. The inventors trained each of the above models and found that the Transformer model based on the output of the motion posture sequence has the best effect in predicting clothing deformation sequences. The training results are as follows: Figure 2 As shown, therefore, as a preferred embodiment, the clothing deformation prediction model is implemented based on the Transformer model. For example, a Transformer model including 6 encoders and 6 decoders is used to implement prediction, that is, the encoding component of the Transformer model includes 6 encoders connected in sequence, and the decoding component of the Transformer model includes 6 decoders connected in sequence; for another example, a Transformer model including 8 encoders and 8 decoders is used to implement prediction, that is, the encoding component of the Transformer model includes 8 encoders connected in sequence, and the decoding component of the Transformer model includes 8 decoders connected in sequence. At the same time, in order to achieve better prediction results and improve the prediction accuracy of the Transformer model, the inventors adjusted the parameters of the Transformer model (the number of encoders, the number of decoders, and the number of heads in the multi-head attention module, etc.) many times and conducted relevant prediction experiments. It was found that the best effect was achieved when a Transformer model including 2 encoders and 2 decoders was used to implement prediction, that is, the encoding component of the Transformer model includes 2 encoders connected in sequence, and the decoding component of the Transformer model includes 2 decoders connected in sequence. The following is an explanation of the Transformer model including 2 encoders and 2 decoders:

[0061] The Transformer model used in the embodiment of the present application includes an encoding component (or motion posture encoding component) for encoding a motion posture sequence and outputting an encoding information matrix corresponding to the motion posture sequence, and a decoding component (or clothing deformation decoding component) for decoding the result of clothing deformation using the encoding information matrix corresponding to the motion posture sequence to obtain a corresponding clothing deformation sequence;

[0062] According to one embodiment of the present invention, the encoding component includes an encoding vector mapping module (InputEmbedding_encoder), a position encoding module (Positional Encoding) and two encoders (Encoder) connected in sequence, and each encoder includes a multi-head attention module (Multi-Head Attention), a first residual and layer normalization module (Add&Norm), a feedforward network module (Feed Forward) and a second residual and layer normalization module (Add&Norm) connected in sequence, wherein the encoding vector mapping module extracts the features corresponding to each motion posture in the motion posture sequence and maps all the extracted features into a vector matrix X1 (wherein, each row in the vector matrix X1 represents a feature vector corresponding to a motion posture), the position encoding module performs position encoding according to the vector matrix X1 to obtain the feature vector X2 corresponding to the motion posture sequence, the multi-head attention module calculates the attention intensity matrix Z1 between the motion postures according to the input feature vector X2, and the first residual and layer normalization module performs residual addition on the input and output of the multi-head attention module to which it is connected and performs layer normalization processing before transmitting. To the feedforward network module for feature extraction, the second residual and layer normalization module of the encoder adds the residuals of the input and output of the feedforward network module to which it is connected, and then performs layer normalization processing to obtain the output matrix C1 of the first encoder, and the output matrix C1 is transmitted to the second encoder and processed by the multi-head attention module, the first residual and layer normalization module, the feedforward network module and the second residual and layer normalization module to obtain the encoding information matrix; according to one embodiment of the present invention, the multi-head attention module includes 4 heads, which does not change the dimension of the feature and is used to establish the temporal dependency relationship between different data frames of the sequence using the self-attention mechanism; the feedforward network module is a fully connected network including 1 hidden layer, and the corresponding dimension of the hidden layer is d ff_size , used to integrate the features learned by different heads; residual addition is used to prevent the gradient vanishing problem caused by the network being too deep; layer normalization is used to normalize the features of different dimensions of the same data;

[0063] According to one embodiment of the present invention, the decoding component includes a decoding vector mapping module (InputEmbedding_decoder), a positional encoding module (Positional Encoding), a plurality of decoders (Decoder) and a linear module (Linear) connected in sequence, wherein the decoder includes a masked multi-head attention module (MaskedMulti-Head Attention), a first residual and layer normalization module (Add&Norm), a multi-head attention module (Multi-Head Attention), a second residual and layer normalization module (Add&Norm), a feedforward network module (Feed Forward) and the third residual and layer normalization module (Add&Norm), wherein the decoding vector mapping module extracts the features corresponding to the input information (wherein, during the training process, the input information is a clothing deformation sequence corresponding to the human body movement posture; after the training is completed, the input information of the initial stage is the input start symbol, and the subsequent input information is the output of the transformer at the previous moment) and maps all the extracted features to the vector matrix X3 (wherein, each row in the vector matrix X3 represents a feature vector corresponding to a clothing deformation), the position encoding module performs position encoding according to the vector matrix X3 to obtain the feature vector X4, the masked multi-head attention module calculates the attention intensity matrix Z2 according to the feature vector X4, and then the first residual and layer normalization module is connected to the input and output of the masked multi-head attention module After residual addition and layer normalization processing, it is transmitted to the multi-head attention module. The multi-head attention module calculates the attention intensity matrix Z3 between clothing deformations based on the input. The second residual and layer normalization module performs residual addition on the input and output of the multi-head attention module to which it is connected and performs layer normalization processing, and then transmits it to the feedforward network module for feature extraction. The third residual and layer normalization module performs residual addition on the input and output of the feedforward network module to which it is connected and performs layer normalization processing to obtain the output matrix C3 of the decoding component. The output matrix C3 is transmitted to the next encoder and processed by the masked multi-head attention module, the first residual and layer normalization module, the multi-head attention module, the second residual and layer normalization module, the feedforward network module and the third residual and layer normalization module, and then transmitted to the linear module, thereby mapping the output of the encoding component to the corresponding vector.According to one embodiment of the present invention, the masked multi-head attention module and the multi-head attention module both include 4 heads, and the masked multi-head attention module shields the influence of subsequent frames through a lower triangular mask matrix, so that the multi-head attention module can only establish the dependency relationship between the existing clothing vertices and the encoding information matrix. For example, if the current clothing vertex is the hth (1≤h<T) frame, the multi-head attention module can only learn the relationship between the clothing vertices of the {1,…,h} frame and Z, and the subsequent {h+1,…,T} frame clothing vertices will be shielded. According to one embodiment of the present invention, the feedforward network module, residual module, and layer normalization module in the decoding component have the same structure as the corresponding feedforward network module, residual module, and layer normalization module in the encoding component.

[0064] Among them, it is worth mentioning that one improvement of the Transformer model in the embodiment of the present application is that the number of encoders and decoders in the Transformer model in the embodiment of the present application is set to be different from the number of encoders and decoders in the common Transformer model. Therefore, the model's processing process of input data is no different from the existing model's processing process of input data. Therefore, the embodiment of the present application does not explain the model's specific processing process of input data.

[0065] 3. Training Samples

[0066] After the network structure parameters of the clothing deformation prediction model are set, in order to use the clothing deformation prediction model to predict the clothing deformation sequence according to the motion posture sequence corresponding to the character, it is also necessary to construct training samples and use the constructed training samples to train the clothing deformation prediction model, so as to obtain the optimal parameters of the clothing deformation prediction model through continuous iterative training, so that the clothing deformation prediction model can predict accurate clothing deformation sequences according to the input motion posture sequence.

[0067] According to one embodiment of the present invention, in order to use the clothing deformation prediction model to predict the clothing deformation sequence based on the motion posture sequence corresponding to the character, any training sample constructed includes the motion posture sequence corresponding to the character and the clothing deformation label sequence; wherein the motion posture sequence includes multiple temporally continuous motion postures, and each motion posture in the motion posture sequence corresponds to a clothing deformation label in the clothing deformation label sequence; in addition, considering the diversity of sports types, in order to be able to predict the clothing deformation corresponding to the character under different sports types, according to one embodiment of the present invention, the posture sequences corresponding to the characters of different training samples are different. For example, the motion posture sequences in some training samples are motion posture sequences corresponding to running, the motion posture sequences in some training samples are motion posture sequences corresponding to walking, and the motion posture sequences in some training samples are motion posture sequences corresponding to jumping. After determining the training samples, in order to input the training samples into the clothing deformation prediction model for training, it is also necessary to set the expression method of the training samples. According to one embodiment of the present invention, any one of the motion postures in the motion posture sequence is represented by the posture parameters that express the motion state of the character's joints or by the vertex coordinates that express the positions of the various parts of the character; the clothing deformation label in the clothing deformation label sequence is represented by the vertex coordinates of the clothing mesh model corresponding to the clothing or by the deformation gradient matrix of the clothing mesh model, wherein the vertex coordinates of the clothing mesh model indicate the positions of multiple preset vertices that control the change of clothing shape; the deformation gradient matrix indicates the deformation amount of the clothing mesh model corresponding to the clothing from the initial state to the target state. It is worth noting that the initial state and the target state of the clothing mesh model can be set according to actual conditions and do not specifically refer to the clothing mesh model in a certain state. In the embodiment of the present application, in order to quickly obtain training samples, the posture parameter θ is used, (The posture of the human body is represented by a 72-dimensional vector) to represent the movement posture, with the clothing vertex V, (where |V j | represents the number of clothing vertices corresponding to the clothing. In theory, |V j | can be equal to any value, but if |V j If it is too small, the clothes will look rough. j | is too large, which will increase the computational cost, so |V j Generally, we set the value to an appropriate size based on experience. (3 represents the x, y, and z dimensions) to represent clothing deformation. After determining the training samples and their representation, we need to obtain a large number of training samples to train the model. The following describes how to obtain training samples, including:

[0068] Step a: Obtain multiple two-dimensional garment pieces for a custom costume for the character, and sew the multiple two-dimensional garment pieces onto a T-pose character model to obtain a corresponding garment template. According to one embodiment of the present invention, the T-pose is constructed based on the SMPL model, which is a differentiable function controlled by a body shape parameter β and a posture parameter θ. When generating the T-pose, the posture parameters corresponding to the T-pose and the body shape parameters corresponding to the character are input into the SMPL model to generate the T-pose character model. To simplify the complexity of the character model, the character's body shape parameter β is preferably set to 0. According to one embodiment of the present invention, the multiple two-dimensional garment pieces are designed based on the body shape characteristics of different parts of the generated human body model using professional clothing design software (Marvelous Designer). Generally, different parts of the human body have different requirements for the comfort of garment pieces. Therefore, when designing two-dimensional garment pieces, it's necessary to measure body parameters of different parts of the human body, and then iteratively modify the piece sizes. For example, when designing a two-dimensional garment piece, a loose fit can easily create an appearance of bulk, while a tight fit can easily highlight excess fat. Therefore, when designing these two-dimensional garment pieces, the waist circumference must first be measured. Based on the measured data, the width of the two-dimensional garment piece is then designed, with a certain amount of clearance reserved, to create a two-dimensional garment piece that fits the waist. According to one embodiment of the present invention, after obtaining multiple two-dimensional garment pieces and a character model in a preset pose, the multiple two-dimensional garment pieces are sewn onto the character model in the preset pose using a physical simulation method to create the corresponding garment. Specifically, multiple sets of virtual seams are first set based on the type of two-dimensional garment piece, and simulator parameters, such as the material of the two-dimensional garment piece, gravity, and friction, are set. A certain seam force is then applied to each set of seams. Finally, under the action of the seam force, the seams continuously contract until they overlap, so that the multiple two-dimensional garment pieces are ultimately attached to the character model, creating the corresponding garment.

[0069] Step b: Obtain the character's motion posture sequence, let the character model move according to the motion posture sequence, so that the clothing template produces a trend deformation, and obtain the clothing deformation sequence of the simulation state corresponding to the character wearing the customized clothing and moving according to the motion posture sequence. According to one embodiment of the present invention, the motion posture sequence is constructed based on the SMPL model. Since the SMPL model requires the posture parameter θ when constructing the human body posture, therefore, in a specific implementation, different human body posture parameter sequences can be first selected from the AMASS human body data set, and then the posture parameter θ and the body shape parameter β (body shape parameter β is set to 0) in the human body posture parameter sequence are sequentially input into the SMPL model to generate the corresponding motion posture sequence, and then the motion posture sequence is imported into the clothing design software (Marvelous Designer) to cause the clothing to deform, thereby generating the clothing deformation label sequence corresponding to the character wearing the corresponding clothing in the motion posture sequence.

[0070] Step c: intercepting the motion posture sequence and its corresponding simulated clothing deformation sequence according to predetermined rules to obtain multiple training samples.

[0071] Considering that in the specific implementation process, since the input size of the Transformer model is fixed, the size (sequence length) of the generated motion posture sequence and clothing deformation label sequence is likely to be larger than the input size of the Transformer model, and the sizes of any two motion posture sequences or clothing deformation label sequences may also be different. Therefore, in order to enable the training samples to be input into the Transformer model and to be able to predict any input motion posture sequence, it is also necessary to intercept the motion posture sequence and its corresponding simulated clothing deformation sequence according to predetermined rules, so that any training sample has the same sequence length. For example, the complete sequence length corresponding to a complete motion posture sequence is 32, that is: L = {θ1, θ2, θ3, ...θ 31 ,θ 32}, the size of the training sample is set to 6, then when obtaining the training sample, the complete motion posture sequence can be cut with the sequence length 6 as a cutting point, so as to obtain 5 training samples of specified length, namely, including: L1 = {θ1, θ2, θ3, θ4, θ5, θ6}, L2 = {θ7, θ8, θ9, θ 10 ,θ 11 ,θ 12}……L5={θ 25 ,θ 26 ,θ 27 ,θ 28 ,θ 29 ,θ 30}, similarly, the corresponding clothing deformation label sequences include: G1 = {V1, V2, V3, V4, V5, V6}, G2 = {V7, V8, V9, V 10 、V 11 、V 12}……G5={V 25 、V 26 、V 27 、V 28 、V 29 、V 30}; As an optimal method, in order to increase the number of training samples as much as possible, a sliding window strategy with a window size of T and a sliding step size of d can be adopted to divide the training samples. For example, the complete sequence length corresponding to a complete motion posture sequence is 20, that is: L = {θ1, θ2, θ3, ... θ 19 ,θ 20}, the window size is set to 5, and the sliding step is set to 1. The motion posture sequence after cutting includes: L1 = {θ1, θ2, θ3, θ4, θ5}, L2 = {θ2, θ3, θ4, θ5, θ6}, L3 = {θ3, θ4, θ5, θ6, θ7}...L 16 ={θ 16 ,θ 17 ,θ 18 ,θ 19 ,θ 20}, similarly, the corresponding clothing deformation label sequences include G1 = {V1, V2, V3, V4, V5}, G2 = {V2, V3, V4, V5, V6}, G3 = {V3, V4, V5, V6, V7}...G 16 ={V 16 、V 17 、V 18 、V 19 、V 20 It should be noted that the above settings of sequence length and sliding step size are only for illustration, so as to intuitively understand the technical solution of the present invention. In actual application, they can be set as needed according to the needs of the application site and the specific conditions of the equipment.

[0072] 4. Training Process

[0073] After the network structure parameters and training samples of the clothing deformation prediction model are constructed, in order to use the clothing deformation prediction model to predict the accurate clothing deformation sequence based on the corresponding motion posture sequence of the character, it is necessary to train the clothing deformation prediction model through training samples to obtain the optimal parameters of the clothing deformation prediction model. The following describes the training process of the clothing deformation prediction model:

[0074] When training a clothing deformation prediction model, the clothing deformation prediction model is trained using multiple training samples to output a clothing deformation sequence based on a motion posture sequence, and the parameters of the clothing deformation prediction model are updated based on the loss value calculated from the output clothing deformation sequence and the clothing deformation label sequence. It is worth noting that, similar to the existing Transformer model training process, during the training process, the input clothing deformation sequence includes a start marker (BOS) and the output clothing deformation sequence includes an end marker (END). Therefore, the actual sequence length of the input and output clothing deformation sequences is Y+1, where Y represents the sequence length of the clothing deformation sequence that does not include the start marker (BOS) or the end marker (END). When calculating the loss value, the last Y frames of the input clothing deformation sequence and the first Y frames of the output clothing deformation sequence are used to calculate the loss value. According to one embodiment of the present invention, during the training of the clothing deformation prediction model, the loss value is determined by a vertex error value and a Laplace error value; wherein the vertex error value is determined by the difference between the clothing vertices in the output clothing deformation sequence and the corresponding clothing deformation label sequence, and the Laplace error value is determined by the difference between the clothing vertices in the output clothing deformation sequence and the corresponding clothing deformation label sequence after Laplace transformation. According to one embodiment of the present invention, the loss value is determined by the following formula:

[0075]

[0076] Among them, L represents the loss value, is the vertex error term, which represents the difference between the predicted clothing vertices and the original sample vertices. This error will prompt the network to learn in the direction of "reconstructing clothing". Represents the Laplace error term, which represents the difference between the predicted clothing vertex after Laplace coordinate transformation and the original sample vertex after Laplace coordinate transformation. This error is used to ensure that the network prediction result has the same or similar wrinkle details as the original clothing sample. t represents the clothing vertex set of the t-th frame in the clothing deformation label sequence, Denotes the set of clothing vertices in the t-th frame of the output clothing deformation sequence, Δ(·) denotes the Laplacian operator, and α denotes the weight. Schematically, the Adam optimizer can be used for network training and ReLU can be selected as the activation function. The hyperparameters are set as follows: the Laplacian loss term weight is set to 0.5, and the weight decay is set to weight_decay = 10 -5 , dropout is set to 0.1, batch size is set to 50, 100, 150, and 200, and changes every 8000 rounds until it reaches 200 and then does not change; for the learning rate, it is dynamically adjusted according to the current number of training rounds based on the warm-up strategy. The adjustment strategy is as follows:

[0077]

[0078] Among them, d model =256, indicating the dimension of the model processing data, current_step indicates the current number of training rounds, and warmup_steps=4000 indicates the number of learning rate warm-up rounds.

[0079] It is worth noting that this is only an illustrative description. In specific implementation, the implementer can set the weight corresponding to the Laplace error term as needed according to the data set used in the implementation or the specific circumstances of the actual application. The present invention does not impose any restrictions on this.

[0080] After training the clothing deformation prediction model, in order to verify the feasibility of the solution provided by the embodiment of the present application, the inventor used the prediction of skirt deformation under different postures as a comparison object, and compared the prediction results obtained by this solution with the prediction results obtained based on other different methods. The comparison results are as follows: Figure 3 As shown, in the vertical (a)-(f), (a) represents the label of the original sample, (b) represents the clothing prediction result of the method of the present invention based on the original sample, (c) to (f) are respectively the clothing prediction effects of the existing methods based on the original sample, and the horizontal (1)-(4) rows respectively represent schematic diagrams comparing the prediction effects of different methods on the same posture, wherein different rows correspond to different postures, and the postures in the same row are the same, wherein the square blocks represent the local amplification of the corresponding clothing parts in the prediction effects of different methods, in order to better compare the effects of different prediction methods. Figure 3 It can be seen from the figure that the clothing predicted based on the solution adopted by the present invention does not produce obvious artifacts, can reduce the deviation from the actual results, and has better reconstruction performance.

[0081] In summary, by using the above-mentioned clothing deformation prediction model and training samples and training the constructed clothing deformation prediction model based on the constructed training samples until convergence, the deviation between the predicted clothing deformation result and the actual result can be reduced, so that the predicted clothing deformation has better smoothness or coherence with the previous clothing deformation.

[0082] 5. Application Scenarios

[0083] As mentioned in the background technology section, clothing animation technology plays an increasingly important role in the fields of film, animation, etc., and its core problem is how to generate a continuous and stable clothing deformation sequence based on the human body's motion posture sequence. According to the description of the first to third aspects, the clothing deformation prediction model can output a corresponding clothing deformation sequence based on the motion posture sequence, so the clothing deformation prediction model provided by the aforementioned embodiment can be used for clothing animation generation. According to one embodiment of the present invention, a method for producing clothing animation based on the clothing deformation sequence output by the clothing deformation prediction model and the corresponding motion posture sequence is provided, such as Figure 4 As shown, including:

[0084] S1: Get the motion posture sequence formulated for the character in the animation;

[0085] Taking into account that in the specific implementation process, there is a situation where the motion posture sequence formulated for the character is larger than the input size of the Transformer model. Therefore, in order to enable the Transformer model to predict the corresponding clothing deformation sequence for the formulated motion posture sequence, according to one embodiment of the present invention, it is necessary to intercept the motion posture sequence according to a predetermined rule so that the intercepted motion posture sequence can be input into the Transformer model. At the same time, in order to ensure the continuity of the clothing animation subsequently generated according to the motion posture sequence and the predicted clothing deformation sequence, the continuity of the intercepted motion posture sequence must also be ensured when the motion posture sequence is intercepted. According to one embodiment of the present invention, the formulated motion posture sequence is divided using a sliding window strategy with a window size of T and a sliding interval of T. For example, the complete sequence length corresponding to a complete motion posture sequence is 30, that is: L = {θ1, θ2, θ3, ...θ 29 ,θ 30}, the window size is set to 6, the sliding interval is set to 6, and the motion posture sequence after cutting includes: L1 = {θ1, θ2, θ3, θ4, θ5, θ6}, L2 = {θ7, θ8, θ9, θ 10 ,θ 11 ,θ 12}……L5={θ 25 ,θ 26 ,θ 27 ,θ 28 ,θ 29 ,θ 30 If a complete motion posture sequence has a motion posture sequence that is less than the window size after being divided, the motion posture sequence that is less than the window size is discarded. For example, the complete sequence length corresponding to a complete motion posture sequence is 34, that is: L = {θ1, θ2, θ3, ... θ 33 ,θ 34}, the window size is set to 6, the sliding interval is set to 6, and the motion posture sequence after cutting includes: L1 = {θ1, θ2, θ3, θ4, θ5, θ6}, L2 = {θ7, θ8, θ9, θ 10 ,θ 11 ,θ 12}……L5={θ 25 ,θ 26 ,θ 27 ,θ 28 ,θ 29 ,θ 30}、L6={θ 31 ,θ 32 ,θ 33 ,θ 34}, then during prediction, only the motion posture sequences of L1-L5 are used, and the corresponding motion postures in L6 are discarded. It should be noted that the above settings of sequence length and sliding window are only for illustration purposes to facilitate an intuitive understanding of the technical solution of the present invention. In actual application, they can be set as needed based on the needs of the application site and the specific conditions of the equipment.

[0086] S2: inputting the motion posture sequence formulated for the character into the clothing deformation prediction model trained by the method of the aforementioned embodiment, and outputting the predicted clothing deformation sequence;

[0087] S3: Determine a character motion frame according to each motion posture in the character's motion posture sequence, and determine a clothing deformation frame of clothing deformation customized for the character according to each clothing deformation in the predicted clothing deformation sequence; each motion posture in the motion posture sequence corresponds to each clothing deformation in the clothing deformation sequence in a one-to-one sequence; according to one embodiment of the present invention, when determining any character motion frame, first select the corresponding posture parameters from the motion posture sequence, and then input the posture parameters and the character's body parameters into the SMPL model to obtain the corresponding character motion frame; according to one embodiment of the present invention, when determining any clothing deformation frame, first select the vertex coordinates corresponding to the clothing from the clothing deformation sequence, and then simulate the clothing structure based on the position of the vertex coordinates to obtain the corresponding clothing deformation frame.

[0088] S4: Each character motion frame is fused with its corresponding clothing deformation frame to obtain an animation frame. All animation frames constitute the animation of the character wearing clothing and moving. According to one embodiment of the present invention, the fusion of the character motion frame and the corresponding clothing deformation frame is achieved by coordinate transformation. The fusion effect is as follows: Figure 5 and Figure 6 As shown, Figure 5 For a continuous clothing motion animation, Figure 6 In order to predict the schematic diagram of a certain frame of clothing animation effect corresponding to different clothing based on this method, Figure 5 and Figure 6 It can be seen that the technical solution provided in the embodiment of the present application can not only generate realistic, delicate, and stable continuous clothing deformation effects, but can also be applied to different types of clothing and has good generalization.

[0089] Furthermore, considering that the clothing deformation in the clothing deformation sequence is based on the approximate regression of non-penetrating physical simulation data, there may be a penetration area (the area where the clothing is sunken into the character's outline) after the character motion frame and the clothing deformation frame are fused, that is: the clothing was originally on the outer surface of the human skin, but now appears inside the human body. Therefore, it is necessary to perform a penetration correction on this part of the penetrating area so that the clothing is completely attached to the outer surface of the human skin. According to one embodiment of the present invention, when correcting the area where the clothing is sunken into the character's outline, the target value of the correction is determined by the Laplace penalty value and the minimum distance between the clothing sunken position and the character's outline; wherein the Laplace penalty value is determined by the difference between the clothing vertices in the area where the clothing is sunken into the character's outline before and after the correction after Laplace transformation. According to one embodiment of the present invention, the target value is determined by the following formula:

[0090]

[0091] Where H represents the target value, is the Laplace penalty term, which ensures that the wrinkle details of the clothing are not lost during the correction process. is a correction term that makes the inward-facing clothing vertex move in a specific direction according to a certain step size, thereby eliminating the penetration area. Δ(·) represents the Laplace operator, and V * represents the corrected clothing vertex set, represents the clothing vertex set before correction, The set of clothing vertices representing where the clothing is indented into the character's outline, The set of clothing vertices whose edges are inset into the character's outline, v k represents the kth invaginated point, Indicates the distance v k The nearest character vertex, τ represents the correction step, ω represents the weight, for example, τ can be set to 0.1, ω can be set to 0.5. It should be understood that this is only an illustrative description. In specific implementation, the implementer can set the weight value and the correction step as needed according to the data set used in the implementation or the specific situation of the actual application. The present invention does not impose any restrictions on this.

[0092] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0093] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0094] Computer-readable storage media can be a tangible device that holds and stores the instructions used by an instruction execution device. Computer-readable storage media can, for example, include, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a raised structure in a groove on which instructions are stored, for example, and any suitable combination thereof.

[0095] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a clothing deformation prediction model, wherein: The clothing deformation prediction model is a sequence-to-sequence model, and the method includes: Acquire multiple training samples, each training sample including a motion posture sequence and a clothing deformation label sequence corresponding to a character; wherein the motion posture sequence includes multiple temporally continuous motion postures, and each motion posture in the motion posture sequence corresponds to a clothing deformation label in the clothing deformation label sequence; The clothing deformation prediction model is trained using the multiple training samples to output a clothing deformation sequence according to the motion posture sequence, and the parameters of the clothing deformation prediction model are updated according to the loss value calculated based on the output clothing deformation sequence and the clothing deformation label sequence, wherein the clothing deformation prediction model is a Transformer model, and the Transformer model includes: an encoding component, which includes multiple encoders and is used to encode the motion posture sequence and output an encoding information matrix corresponding to the motion posture sequence; a decoding component, which includes multiple decoders and is used to decode the result of clothing deformation using the encoding information matrix corresponding to the motion posture sequence to obtain a corresponding clothing deformation sequence; the encoding component includes: a coding vector mapping module, a position coding module and multiple encoders connected in sequence, each encoder includes a multi-head attention module, a first residual and layer normalization module, a feedforward network module and a second residual and layer normalization module connected in sequence, wherein the first residual and layer normalization module of the encoder is connected to the multi-head attention module connected to it. The input and output of the module are residually added and then layer normalization processing is performed. The second residual and layer normalization module of the encoder performs residual addition on the input and output of the feedforward network module to which it is connected and then layer normalization processing is performed. The decoding component includes: a decoding vector mapping module, a position encoding module, multiple decoders and a linear module connected in sequence, wherein the decoder includes a masked multi-head attention module, a first residual and layer normalization module, a multi-head attention module, a second residual and layer normalization module, a feedforward network module and a third residual and layer normalization module connected in sequence. The first residual and layer normalization module of the decoder performs residual addition on the input and output of the masked multi-head attention module to which it is connected and then layer normalization processing is performed. The second residual and layer normalization module of the decoder performs residual addition on the input and output of the multi-head attention module to which it is connected and then layer normalization processing is performed. The third residual and layer normalization module of the decoder performs residual addition on the input and output of the feedforward network module to which it is connected and then layer normalization processing is performed.

2. The method according to claim 1, characterized in that The motion postures in the motion posture sequence are represented by posture parameters that express the motion state of the character's joints or by vertex coordinates that express the position of each part of the character; The clothing deformation labels in the clothing deformation label sequence are represented by the vertex coordinates of the clothing mesh model corresponding to the clothing or by the deformation gradient matrix of the clothing mesh model, wherein the vertex coordinates of the clothing mesh model indicate the positions of multiple preset vertices that control the change of the clothing shape; and the deformation gradient matrix indicates the deformation amount of the clothing mesh model corresponding to the clothing from an initial state to a target state.

3. The method according to claim 1, characterized in that The training samples are obtained in the following manner: Obtain multiple two-dimensional garment pieces of clothing customized for the character, and sew the multiple two-dimensional garment pieces onto the character model in a T-pose to obtain a clothing template corresponding to the clothing; Obtain the character's motion posture sequence, let the character model move according to the motion posture sequence, so that the clothing template produces trend deformation, and obtain the clothing deformation sequence of the simulation state corresponding to the character wearing the customized clothing and moving according to the motion posture sequence; According to predetermined rules, motion posture sequences and corresponding simulated clothing deformation sequences are intercepted to obtain multiple training samples.

4. The method according to any one of claims 1 to 3, characterized in that During the training of the clothing deformation prediction model, the loss value is determined by the vertex error value and the Laplace error value; wherein the vertex error value is determined by the difference value of the clothing vertices in the output clothing deformation sequence and the corresponding clothing deformation label sequence, and the Laplace error value is determined by the difference value of the clothing vertices in the output clothing deformation sequence and the corresponding clothing deformation label sequence after Laplace transformation.

5. The method according to claim 4, characterized in that The loss value is determined as follows: ； in, represents the loss value, Indicates the first The frame's clothing vertex set, Indicates the output clothing deformation sequence The frame's clothing vertex set, represents the Laplace operator, Represents weight.

6. The method according to claim 1, characterized in that The number of the encoders and decoders is set to 2-4.

7. The method according to claim 1, characterized in that The encoder's multi-head attention module, the decoder's masked multi-head attention module, and the multi-head attention module are all set with 4 heads.

8. A method for producing an animation, characterized in that: include: Get the sequence of motion poses specified for the character in the animation; inputting the motion posture sequence formulated for the character into a clothing deformation prediction model trained by the method according to any one of claims 1 to 7, and outputting a predicted clothing deformation sequence; determining a character motion frame according to each motion pose in the character's motion pose sequence, and determining a clothing deformation frame of a clothing deformation customized for the character according to each clothing deformation in the predicted clothing deformation sequence; Each motion posture in the motion posture sequence corresponds to each clothing deformation in the clothing deformation sequence in a one-to-one sequence; Each character motion frame is fused with its corresponding clothing deformation frame to obtain an animation frame. All animation frames constitute the animation of the character wearing clothing and moving.

9. The method according to claim 8, characterized in that When there are animation frames in which the clothing is sunken into the character's outline in the animation, the animation frames in which the clothing is sunken into the character's outline are corrected to obtain a final animation in which the character wears the clothing.

10. The method according to claim 9, characterized in that When correcting animation frames in which clothing is sunken into the character's outline, the target value of the correction is determined by a Laplace penalty value and a minimum distance between the location of the clothing sunken and the character's outline; wherein the Laplace penalty value is determined by the difference in Laplace transform values ​​of clothing vertices before and after the correction in the animation frame in which the clothing is sunken into the character's outline.

11. The method according to claim 10, characterized in that The target value is determined according to the following formula: in, represents the target value, is the Laplace penalty term, is the correction term, represents the Laplace operator, represents the corrected clothing vertex set, represents the clothing vertex set before correction, The set of clothing vertices representing where the clothing is indented into the character's outline, The set of clothing vertices whose edges are inset into the character's outline. Indicates the An indentation point, Indicates distance The most recent character apex, represents the correction step size, Represents weight.

12. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 7 or 8 to 11.

13. An electronic device, characterized in that: include: one or more processors; as well as a memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1-7 or 8-11 by executing the executable instructions.

Images