3D animation generation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of 3D animation generation, and discloses a 3D animation generation method and device, equipment and a storage medium. The 3D animation generation method comprises the following steps: acquiring a variable-length continuous image sequence from a target video source, inputting the acquired continuous image sequence into a trained 3D animation generation model for 3D attitude extraction, and outputting a 3D animation sequence with a corresponding length. The 3D animation generation model is trained again by using the trained animation diffusion model after being pre-trained, so that the generation effect of the 3D animation generation model is optimized, and the generation effect of the 3D animation sequence can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D animation generation, and in particular to a 3D animation generation method, device, equipment and storage medium. Background Art

[0002] 3D animation, also known as three-dimensional animation, is a stereoscopic animation effect created using computer technology. Currently, the more mature 3D animation generation methods usually use motion capture and light capture systems, which have high implementation costs. The low-cost generation method mainly recognizes the target action posture from the video and synthesizes the motion sequence. The latter lacks the high-precision sensors of the former and relies entirely on visual recognition, resulting in poor 3D animation effects. It is easy to have problems such as motion jitter, continuity and unrealistic motion, resulting in poor generated 3D animation effects and poor user experience. Summary of the invention

[0003] The main purpose of the present invention is to provide a 3D animation generation method, device, equipment and storage medium, aiming to solve the technical problem of how to improve the generation effect of 3D animation in a low-cost manner.

[0004] A first aspect of the present invention provides a 3D animation generation method, the 3D animation generation method comprising:

[0005] Obtaining a variable length continuous image sequence from a target video source;

[0006] Inputting the continuous image sequence into a trained 3D animation generation model to extract 3D postures, and outputting a 3D animation sequence of corresponding length;

[0007] After the 3D animation generation model is pre-trained, it is trained again using the trained animation diffusion model to optimize the generation effect of the 3D animation generation model.

[0008] Optionally, in a first implementation of the first aspect of the present invention, the 3D animation generation model is obtained by using the following training method:

[0009] Obtain continuous image sequences and corresponding real 3D animation sequences as training samples;

[0010] Input each frame of the continuous image sequence into the convolutional layer of the 3D animation generation model to be trained for downsampling and feature extraction, and output the shallow feature map corresponding to each frame of the image;

[0011] Flatten the shallow feature map corresponding to each frame of the image to form a one-dimensional vector, and obtain the feature sequence corresponding to the continuous image sequence;

[0012] Inputting the feature sequence into a visual model of a 3D animation generation model to be trained to extract 3D postures, and outputting a 3D animation sequence corresponding to a continuous image sequence;

[0013] Based on the real 3D animation sequence and the 3D animation sequence output by the model, the 3D animation generation model is trained to obtain a pre-trained 3D animation generation model.

[0014] Optionally, in a second implementation of the first aspect of the present invention, the animation diffusion model is obtained by using the following training method:

[0015] Get 3D animation sequences and use them as training samples;

[0016] Based on the forward diffusion process, the first noise used in the current time step is added to the 3D animation sequence to obtain the first noisy animation sequence of the current time step;

[0017] Input the first noisy animation sequence based on the time step, the time code corresponding to the time step, and the position code corresponding to each frame of data in the first noisy animation sequence into the animation diffusion model to be trained for noise prediction, and output the second noise corresponding to the time step;

[0018] Based on the first noise and the second noise, an animation diffusion model is trained to obtain a trained animation diffusion model.

[0019] Optionally, in a third implementation of the first aspect of the present invention, the trained animation diffusion model uses the following training method to train the pre-trained 3D animation generation model:

[0020] Acquire variable length continuous image sequences;

[0021] Input the continuous image sequence into the pre-trained 3D animation generation model to extract the 3D posture and output a 3D animation sequence of corresponding length;

[0022] Based on the forward diffusion process, the third noise used in the current time step is added to the 3D animation sequence output by the pre-trained 3D animation generation model to obtain the second noisy animation sequence of the current time step;

[0023] Input the second noisy animation sequence based on the time step, the time code corresponding to the time step, and the position code corresponding to each frame of data in the second noisy animation sequence into the trained animation diffusion model for noise prediction, and output the fourth noise corresponding to the time step;

[0024] Based on the third noise and the fourth noise, the pre-trained 3D animation generation model is trained to obtain a final trained 3D animation generation model.

[0025] Optionally, in a fourth implementation manner of the first aspect of the present invention, the training of a pre-trained 3D animation generation model based on the third noise and the fourth noise to obtain a final trained 3D animation generation model includes:

[0026] During the training of the pre-trained 3D animation generation model, the weights of the frozen trained animation diffusion model are not updated;

[0027] A noise loss function is constructed based on the third noise and the fourth noise, and based on the noise loss function, gradient calculation and weight update are performed on the pre-trained 3D animation generation model to obtain a final trained 3D animation generation model.

[0028] Optionally, in a fifth implementation of the first aspect of the present invention, the gradient of the noise loss function relative to the model parameters of the pre-trained 3D animation generation model is:

[0029]

[0030] in, represents the gradient of the noise loss function relative to the model parameters of the pre-trained 3D animation generation model, L 3D represents the noise loss function used when optimizing the pre-trained 3D animation generation model, θ represents the model parameters of the 3D animation generation model, represents the parameters related to the time step t, D(Z t |t) represents the noise predicted by the trained animation diffusion model, D represents the animation diffusion model, Z represents the output of the pre-trained 3D animation generation model, and Z t represents the noisy animation sequence obtained by adding noise to Z at time step t, e t represents the noise satisfying Gaussian distribution added at the tth time step;

[0031] Optionally, in a sixth implementation of the first aspect of the present invention, the following formula is used to update the weight of the pre-trained 3D animation generation model:

[0032]

[0033] Among them, θ′ represents the model parameters after the weight update of the 3D animation generation model, β represents the learning rate, and ||D(Z t |t)-e t || represents the error term of noise, represents the gradient of the animated diffusion model and takes 1, Represents the gradient of the 3D animation generative model.

[0034] A second aspect of the present invention provides a 3D animation generation device, the 3D animation generation device comprising:

[0035] An acquisition module, used for acquiring a variable-length continuous image sequence from a target video source;

[0036] A generation module, used for inputting the continuous image sequence into a trained 3D animation generation model to perform 3D posture extraction and output a 3D animation sequence of corresponding length;

[0037] After the 3D animation generation model is pre-trained, it is trained again using the trained animation diffusion model to optimize the generation effect of the 3D animation generation model.

[0038] Optionally, in a first implementation of the second aspect of the present invention, the 3D animation generating device further includes:

[0039] The first training module is used to obtain a continuous image sequence and a corresponding real 3D animation sequence as training samples; input each frame of the continuous image sequence into the convolution layer of the 3D animation generation model to be trained for downsampling and feature extraction, and output a shallow feature map corresponding to each frame of the image; flatten the shallow feature map corresponding to each frame of the image to form a one-dimensional vector, and obtain a feature sequence corresponding to the continuous image sequence; input the feature sequence into the visual model of the 3D animation generation model to be trained for 3D posture extraction, and output a 3D animation sequence corresponding to the continuous image sequence; based on the real 3D animation sequence and the 3D animation sequence output by the model, train the 3D animation generation model to obtain a pre-trained 3D animation generation model.

[0040] Optionally, in a second implementation of the second aspect of the present invention, the 3D animation generating device further includes:

[0041] The second training module is used to obtain a 3D animation sequence as a training sample; based on the forward diffusion process, the first noise used in the current time step is added to the 3D animation sequence to obtain the first noisy animation sequence of the current time step; the first noisy animation sequence based on the time step, the time code corresponding to the time step, and the position code corresponding to each frame of data in the first noisy animation sequence are input into the animation diffusion model to be trained for noise prediction, and the second noise corresponding to the time step is output; based on the first noise and the second noise, the animation diffusion model is trained to obtain a trained animation diffusion model.

[0042] Optionally, in a third implementation of the second aspect of the present invention, the 3D animation generating device further includes:

[0043] The third training module is used to obtain a continuous image sequence of variable length; input the continuous image sequence into a pre-trained 3D animation generation model for 3D posture extraction, and output a 3D animation sequence of corresponding length; based on the forward diffusion process, add the third noise used in the current time step to the 3D animation sequence output by the pre-trained 3D animation generation model to obtain a second noisy animation sequence for the current time step; input the second noisy animation sequence based on the time step, the time code of the corresponding time step, and the position code corresponding to each frame of data in the second noisy animation sequence into the trained animation diffusion model for noise prediction, and output a fourth noise corresponding to the time step; based on the third noise and the fourth noise, train the pre-trained 3D animation generation model to obtain a final trained 3D animation generation model.

[0044] Optionally, in a fourth implementation of the second aspect of the present invention, the third training module is specifically used to:

[0045] During the training process of the pre-trained 3D animation generation model, the weights of the trained animation diffusion model are frozen and not updated; a noise loss function is constructed based on the third noise and the fourth noise, and based on the noise loss function, the gradient calculation and weight update of the pre-trained 3D animation generation model are performed to obtain the final trained 3D animation generation model.

[0046] Optionally, in a fifth implementation of the second aspect of the present invention, the gradient of the noise loss function relative to the model parameters of the pre-trained 3D animation generation model is:

[0047]

[0048] in, represents the gradient of the noise loss function relative to the model parameters of the pre-trained 3D animation generation model, L 3D represents the noise loss function used when optimizing the pre-trained 3D animation generation model, θ represents the model parameters of the 3D animation generation model, represents the parameters related to the time step t, D(Z t |t) represents the noise predicted by the trained animation diffusion model, D represents the animation diffusion model, Z represents the output of the pre-trained 3D animation generation model, and Z t represents the noisy animation sequence obtained by adding noise to Z at time step t, e t represents the noise satisfying Gaussian distribution added at the tth time step;

[0049] Optionally, in a sixth implementation of the second aspect of the present invention, the following formula is used to update the weight of the pre-trained 3D animation generation model:

[0050]

[0051] Among them, θ′ represents the model parameters after the weight update of the 3D animation generation model, β represents the learning rate, and ||D(Z t |t)-e t || represents the error term of noise, represents the gradient of the animated diffusion model and takes 1, Represents the gradient of the 3D animation generative model.

[0052] A third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the above-mentioned 3D animation generation method.

[0053] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned 3D animation generation method.

[0054] The embodiment of the present invention provides a 3D animation generation method, device, equipment and storage medium, firstly obtains a variable length continuous image sequence from a target video source, then inputs the obtained continuous image sequence into a trained 3D animation generation model for 3D posture extraction, and outputs a 3D animation sequence of corresponding length. Since the 3D animation generation model is trained again using a trained animation diffusion model after pre-training to optimize the generation effect of the 3D animation generation model, the generation effect of the 3D animation sequence is significantly improved. In this embodiment, a continuous image sequence (i.e., a video frame) is used as a control signal to generate a corresponding animation sequence based on a trained 3D animation generation model, and the trained animation diffusion model is only used as a discriminator to provide motion prior knowledge, and does not perform animation generation tasks, so that the 3D animation generation model of the traditional video extraction 3D animation sequence can be improved by combining the motion prior knowledge of high-quality actions integrated in the animation diffusion model, so as to promote the latter to improve the realism of the 3D animation, thereby ensuring the generation effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of an embodiment of a 3D animation generation method in an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of a first embodiment flow chart of a network model training method in an embodiment of the present invention;

[0057] Figure 3 This is a flow chart of a second embodiment of the network model training method in an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of a third embodiment of the flow chart of the network model training method in an embodiment of the present invention;

[0059] Figure 5 A schematic diagram of an embodiment of a 3D animation generating device in an embodiment of the present invention;

[0060] Figure 6 FIG. 1 is a schematic diagram of an embodiment of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0062] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the 3D animation generation method in the embodiment of the present invention includes:

[0063] 101. Acquire a variable length continuous image sequence from a target video source;

[0064] This embodiment is mainly applied to 3D animation generation scenarios, so the target video source used to generate 3D animation needs to contain motion pictures of the target object, such as motion pictures of people. By decoding the target video source, continuous video frames can be formed, and the posture of the target object can be identified from the continuous video frames and the motion sequence can be synthesized to obtain 3D animation.

[0065] In this embodiment, the variable-length continuous image sequence refers to an image (i.e., video frame) whose number (i.e., sequence length) is not fixed, and these images are continuous in content, and are usually used to represent dynamic scenes or the motion process of objects. Different from a fixed-length image sequence, the length of a variable-length sequence can be adjusted according to actual conditions, so it is more flexible and practical.

[0066] 102. Input the continuous image sequence into a trained 3D animation generation model to perform 3D posture extraction, and output a 3D animation sequence of corresponding length; wherein, after the 3D animation generation model is pre-trained, it is trained again using the trained animation diffusion model to optimize the generation effect of the 3D animation generation model.

[0067] In this embodiment, the 3D animation generation model is used to recognize the posture of the target object from the continuous image sequence and synthesize the motion sequence, so as to obtain a 3D animation sequence, and the length of the 3D animation sequence is the same as the length of the input image.

[0068] In this embodiment, although the 3D animation generation model obtained through pre-training can generate a 3D animation sequence, in order to further improve the effect of the 3D animation sequence generated by the 3D animation generation model, such as image clarity, etc., this embodiment further optimizes the 3D animation generation model, specifically: a network model based on a diffusion model is selected for training to obtain an animation diffusion model, and then the trained animation diffusion model is used as a discriminator to identify the effect of the 3D animation sequence output by the 3D animation generation model based on prior knowledge of high-quality actions, thereby indirectly optimizing the 3D animation generation model and improving the effect and quality of the generated animation.

[0069] In this embodiment, a variable length continuous image sequence is first obtained from the target video source, and then the obtained continuous image sequence is input into the trained 3D animation generation model for 3D posture extraction, and a 3D animation sequence of corresponding length is output. Since the 3D animation generation model is trained again using the trained animation diffusion model after pre-training, the generation effect of the 3D animation generation model is optimized, thereby significantly improving the generation effect of the 3D animation sequence. In this embodiment, a continuous image sequence (i.e., video frame) is used as a control signal to generate a corresponding animation sequence based on the trained 3D animation generation model, and the trained animation diffusion model only serves as a discriminator that provides motion prior knowledge and does not perform animation generation tasks, so that the 3D animation generation model of the traditional video extraction 3D animation sequence can be improved by combining the motion prior knowledge of high-quality actions integrated in the animation diffusion model, so that the latter is prompted to improve the realism of the 3D animation, thereby ensuring the generation effect of the model.

[0070] The embodiment of the present invention also provides a network model training method for training a 3D animation generation model used in a 3D animation generation method. The network model training method of the embodiment of the present invention requires that the training of the 3D animation generation model be completed in advance before executing the 3D animation generation task. Figure 2 , Figure 2This is a flow chart of a first embodiment of the network model training method in an embodiment of the present invention. The steps of training the 3D animation generation model in this embodiment include:

[0071] 201. Obtain a continuous image sequence and a corresponding real 3D animation sequence as training samples;

[0072] In this embodiment, the training samples used by the 3D animation generation model are: a continuous image sequence and a real 3D animation sequence generated by the continuous image sequence (such as generated by a motion capture or light capture system). The 3D animation generation model is mainly based on the continuous image sequence of the video source I = {I1 I2 ... I N Extract the action posture in each frame image to form a 3D animation sequence.

[0073] 202. Input each frame of the continuous image sequence into the convolution layer of the 3D animation generation model to be trained for downsampling and feature extraction, and output a shallow feature map corresponding to each frame of the image;

[0074] In this embodiment, since the recognition granularity of the target object action in each frame image is relatively large, each frame image I can be i First, downsampling is performed to reduce the amount of calculation, and then shallow features of each frame of the image are extracted based on several convolutional layers of the initial 3D animation generation model used in training and a shallow feature map corresponding to each frame of the image is formed. The initial 3D animation generation model contains several convolutional layers.

[0075] Downsampling is a common technique in image processing, mainly used to reduce the resolution or size of an image. In deep learning and computer vision, downsampling is often used to reduce the amount of model computation, improve model efficiency, or as part of feature extraction. Downsampling methods mainly include:

[0076] (1) Average Pooling: In average pooling, the pixel values ​​in each pooling window are replaced with the average of all pixel values ​​within the window. This helps reduce noise and details in the image while preserving the overall structure.

[0077] (2) Max Pooling: In max pooling, the pixel value in each pooling window is replaced by the maximum pixel value within the window. This helps capture salient features in the image, such as edges and textures.

[0078] (3) Strided Convolution: By setting the step size of the convolution operation to be greater than 1, image downsampling can be achieved. Strided convolution can perform feature extraction and downsampling at the same time, which is a flexible and efficient method.

[0079] The convolutional layers of the 3D animation generation model are mainly used to extract features from the input image data. When processing image data, these features are usually abstracted from pixel-level information and can describe patterns such as edges, corners, and textures in the image. For video data, although the video is composed of multiple frames (i.e., a temporally continuous sequence of images), in most cases, the convolutional layer is applied independently to each frame. That is, for each frame in the continuous image sequence, the convolutional layer performs the same operation to extract shallow features of the frame.

[0080] 203. Flatten the shallow feature map corresponding to each frame of the image to form a one-dimensional vector, and obtain a feature sequence corresponding to the continuous image sequence;

[0081] In this embodiment, after obtaining the shallow feature map corresponding to each frame of image in step 202, in order to facilitate subsequent processing, the feature map corresponding to each frame of image needs to be flattened to form a one-dimensional vector, denoted as y∈R d , d represents the vector dimension, then the entire feature sequence can be expressed as Y = {y1 y2...y N}∈R N×d ,

[0082] Flattening feature maps to form a one-dimensional vector is a common operation in deep learning, especially when transitioning from convolutional layers to fully connected layers. This process usually occurs in convolutional layers to convert multi-dimensional feature maps into one-dimensional feature vectors so that subsequent fully connected layers or classifiers can process them. In convolutional neural networks, batch_size, channels, height, and width are usually known. Before flattening the feature map, the shape of the feature map is usually [batch_size, channels, height, width]. For example, when using the view() method to flatten the feature map, all dimensions except batch_size can be merged into one dimension, resulting in a one-dimensional vector with a shape of [batch_size,-1]. The -1 here is a placeholder indicating that the size of this dimension is automatically calculated from the sizes of other dimensions.

[0083] 204. Input the feature sequence into a visual model of a 3D animation generation model to be trained to perform 3D posture extraction, and output a 3D animation sequence corresponding to the continuous image sequence;

[0084] In this embodiment, the 3D animation generation model also includes a visual model for extracting corresponding 3D gestures from the feature sequence. The visual model used in this embodiment is a visual model based on the Transformer architecture, such as the VIT model. The basic structure of the VIT (Vision Transformer) model consists of a Transformer encoder, which contains multiple Transformer encoding layers. Each encoding layer captures global context information and long-term dependencies through a self-attention mechanism, thereby extracting high-level feature representations. The VIT model first divides the input image into patches of fixed size (e.g., blocks of 16x16 pixels), then flattens each patch into a one-dimensional vector, and maps the vector to the embedding space of the model through a linear transformation to obtain an embedded representation of each patch. The VIT model directly uses the Encoder in the Transformer in the image feature extraction part. The Encoder is stacked by multiple identical layers, each of which contains a self-attention mechanism and a feedforward neural network. By stacking multiple such layers, the VIT model can gradually extract higher-level image features.

[0085] In this embodiment, the feature sequence Y is input into the visual model to be trained (such as the VIT model), and each feature vector y in the feature sequence Y i That is, a token. The visual model outputs a vector sequence, where each vector in the output vector sequence represents the 3D posture vector corresponding to the frame. For example, the feature vector corresponding to the i-th frame image is y i , then the 3D posture output by the visual model is represented by z i ∈R 3(J+1) , where the animation data of each frame is represented by the movement speed and rotation speed of the root joint and the spatial position of other joints relative to the root joint. Assuming the total number of joints is J, the data dimension of each frame of animation is 3(J+1). Assuming the length of the continuous image sequence used for training is N, the final 3D posture sequence is represented as Z = {z1 z2 ... z N}∈R N×3(J+1) , which is the 3D animation sequence corresponding to the continuous image sequence output by the model.

[0086] 205. Based on the real 3D animation sequence and the 3D animation sequence output by the model, train the 3D animation generation model to obtain a pre-trained 3D animation generation model.

[0087] In this embodiment, the 3D animation generation model uses an open source dataset including continuous image sequences and real 3D animation sequences. During the training process, the continuous image sequence is used as the model input, and the model outputs the corresponding 3D animation sequence. The 3D animation sequence output by the model is then compared with the real 3D animation sequence to calculate the L2 loss for model parameter optimization. The optimized 3D animation generation model can continuously recognize the 3D postures in different image frames, and finally a trained 3D animation generation model is obtained.

[0088] The embodiment of the present invention also provides a network model training method for training the animation diffusion model used in the 3D animation generation method. The network model training method of the embodiment of the present invention needs to complete the training of the animation diffusion model in advance before executing the 3D animation generation task. Figure 3 , Figure 3 1 is a flow chart of a second embodiment of the network model training method in an embodiment of the present invention. The steps of training the animation diffusion model in this embodiment include:

[0089] 301. Obtain a 3D animation sequence as a training sample;

[0090] In this embodiment, the animation diffusion model preferably uses the encoder part of the Transformer model. Each token represents the animation data of a frame of animation. The attention mechanism is used for information exchange within the animation sequence. The animation data of each frame is represented by the movement speed and rotation speed of the root joint and the spatial position of other joints relative to the root joint. Let the total number of joints be J, then the data dimension of each frame of animation is 3(J+1), let the sequence length be N, then a 3D animation sequence can be represented as X∈R N×3(J+1) , and used as the original unnoised training sample.

[0091] 302. Based on the forward diffusion process, the first noise used in the current time step is added to the 3D animation sequence to obtain a first noisy animation sequence in the current time step;

[0092] In this embodiment, the 3D animation sequence X is noised according to the setting of the forward diffusion process. If the noise et∈R that satisfies the Gaussian distribution is added in the tth step, N×3(J+1) , then the noise-added animation sequence after the t-th step of noise addition is where x i ∈R 3(J+1) Represents the single frame animation data of the i-th frame of 3D animation in the 3D animation sequence. It represents the parameters related to the time step t, which is automatically controlled and calculated by the diffusion process.

[0093] 303. Input the first noisy animation sequence based on the time step, the time code corresponding to the time step, and the position code corresponding to each frame of data in the first noisy animation sequence into the animation diffusion model to be trained for noise prediction, and output the second noise corresponding to the time step;

[0094] In this embodiment, since the animation diffusion model adopts the transformer structure, the input model data (the first noisy animation sequence) also needs to add the corresponding position code P (using sin-cos coding) to the animation data of each frame to obtain a new sequence vector, that is, X t =X t +{p1 p2...p N}, the new sequence vector and the time encoding of time step t (using sin-cos encoding) are used as the final input data to input the animation diffusion model (transformer encoder). The vector sequence output by the animation diffusion model is the predicted noise corresponding to each time step t The noise sequence.

[0095] 304. Train an animation diffusion model based on the first noise and the second noise to obtain a trained animation diffusion model.

[0096] In this embodiment, during the training of the animation diffusion model, it is necessary to optimize the parameters of the animation diffusion model. Specifically, the noise predicted by the animation diffusion model And the L2 loss of the noise et added to the sequence As the optimization target of the animation diffusion model parameters, an animation diffusion model that learns more prior knowledge is obtained.

[0097] In this embodiment, after the model training is completed, the model effect can be further verified by the decoder part of the Transformer model, specifically: based on the noise sample X in step t t and time step t, predicting the noise Then the noise sample in step t-1 is in are all parameters automatically controlled by time step t. Similarly, based on the denoised sample X t-1 and time step t-1 to obtain the next noise And the denoised result X t-2 , and so on.

[0098] Based on the above derivation process, the 3D animation sequence can be added with the noise sample X after adding the standard Gaussian noise from the initial time step T TStart denoising. If the result after denoising at step 0 is a completely noise-free sample X, that is, the 3D animation sequence used before denoising, then it can be considered that the currently trained animation diffusion model has prior knowledge of the relevant 3D animation sequence. Otherwise, the model training needs to continue.

[0099] The embodiment of the present invention also provides a network model training method for training a 3D animation generation model using an animation diffusion model. The network model training method of the embodiment of the present invention requires that the training of the 3D animation generation model be completed in advance before executing the 3D animation generation task. Figure 4 , Figure 4 The third embodiment of the network model training method in the embodiment of the present invention is a flow chart. The steps of training the pre-trained 3D animation generation model based on the trained animation diffusion model in this embodiment include:

[0100] 401. Acquire a variable length continuous image sequence;

[0101] 402. Input the continuous image sequence into a pre-trained 3D animation generation model to extract 3D postures, and output a 3D animation sequence of corresponding length;

[0102] In this embodiment, the training samples used for the optimization of the 3D animation generation model are continuous image sequences of varying lengths. The continuous image sequence is first input into the trained 3D animation generation model for 3D posture extraction, and a 3D animation sequence of the same length as the continuous image sequence is output.

[0103] 403. Based on the forward diffusion process, the third noise used in the current time step is added to the 3D animation sequence output by the pre-trained 3D animation generation model to obtain a second noisy animation sequence in the current time step;

[0104] First, the 3D animation sequence X output by the pre-trained 3D animation generation model is denoised according to the setting of the forward diffusion process. If the noise et∈R satisfying the Gaussian distribution is added in the tth step N×3(J+1) , then the noise-added animation sequence after the t-th step of noise addition is where x i ∈R 3(J+1) Represents the single frame animation data of the i-th frame of 3D animation in the 3D animation sequence. It represents the parameters related to the time step t, which is automatically controlled and calculated by the diffusion process.

[0105] 404. Input the second noisy animation sequence based on the time step, the time code corresponding to the time step, and the position code corresponding to each frame of data in the second noisy animation sequence into the trained animation diffusion model for noise prediction, and output the fourth noise corresponding to the time step;

[0106] In this embodiment, since the animation diffusion model adopts the transformer structure, the data input to the model (the second noisy animation sequence) also needs to add the corresponding position code (using sin-cos coding) to the animation data of each frame, and also add the time code corresponding to the time step t (using sin-cos coding) as the final input data to input the trained animation diffusion model (transformer encoder). The vector sequence output by the trained animation diffusion model is the predicted noise corresponding to each time step t. The noise sequence.

[0107] 405. Based on the third noise and the fourth noise, train the pre-trained 3D animation generation model to obtain a final trained 3D animation generation model.

[0108] In this embodiment, the animation diffusion model realizes the generation of high-quality animation sequences from Gaussian noise, which contains prior knowledge of high-quality animation sequences, such as strong kinematic and dynamic priors and motion continuity guarantees. However, the trained 3D animation generation model directly obtains 3D animation sequences from videos, and it is difficult to ensure the motion continuity and kinematic and dynamic characteristics between 3D animation sequences. Therefore, this step further optimizes the effect of the 3D animation generation model based on the prior of the animation diffusion model.

[0109] In this embodiment, for a continuous image sequence I with a length of N as input, the 3D animation generation model outputs a 3D animation sequence Z∈R N×3(J+1) is regarded as a sample without noise, so standard Gaussian distribution noise and position encoding are added to it. Then, at step t, it can be known that the input of the animation diffusion model is the noisy animation sequence sample at this time and time step t, predict the added noise Similarly, using prediction noise And the added noise et calculates the L2 loss as optimization goal.

[0110] In one embodiment, the above step 405 further includes:

[0111] 4051. During the training of the pre-trained 3D animation generation model, the weights of the frozen trained animation diffusion model are not updated;

[0112] 4052. Construct a noise loss function based on the third noise and the fourth noise, and based on the noise loss function, perform gradient calculation and weight update on the pre-trained 3D animation generation model to obtain a final trained 3D animation generation model.

[0113] In this embodiment, since the trained animation diffusion model is used to train the pre-trained 3D animation generation model (equivalent to model optimization), during the entire model optimization process, the weights of the animation diffusion model need to be frozen and not updated, that is, the original prior knowledge of the animation diffusion model is not changed, and only the gradient calculation and weight update of the pre-trained 3D animation generation model are performed, thereby obtaining a 3D animation generation model with better animation generation effect.

[0114] In one embodiment, the gradient of the noise loss function relative to the model parameters of the pre-trained 3D animation generation model is:

[0115]

[0116] in, represents the gradient of the noise loss function relative to the model parameters of the pre-trained 3D animation generation model, L 3D represents the noise loss function used when optimizing the pre-trained 3D animation generation model, θ represents the model parameters of the 3D animation generation model, represents the parameters related to the time step t, D(Z t |t) represents the noise predicted by the trained animation diffusion model, D represents the animation diffusion model, Z represents the output of the pre-trained 3D animation generation model, and Z t represents the noisy animation sequence obtained by adding noise to Z at time step t, and et represents the noise that satisfies the Gaussian distribution added at the tth time step;

[0117] In one embodiment, the following formula is further used to update the weight of the pre-trained 3D animation generation model:

[0118]

[0119] Among them, θ′ represents the model parameters after the weight update of the 3D animation generation model, β represents the learning rate, and ||D(Z t |t)-e t || represents the error term of noise, represents the gradient of the animated diffusion model and takes 1, Represents the gradient of the 3D animation generative model.

[0120] After fully optimizing the 3D animation generation model based on the above-mentioned weight updating algorithm, the animation sequence output by the 3D animation generation model can fall into the data domain composed of a high-quality action library, which can meet the relevant kinematic and dynamic characteristics, thereby achieving the effect of improving the realism of the animation.

[0121] This embodiment summarizes that the trained animation diffusion model is used as a discriminator, and the effect of the animation sequence output by the 3D animation generation model is identified based on the prior knowledge of high-quality actions. The 3D animation sequence output by the 3D animation generation model is denoised according to the diffusion process to obtain a noisy sample, and together with the position encoding and time step encoding, it is used as the input of the animation diffusion model, and the 3D animation generation model is indirectly optimized based on the error of the predicted noise and the added noise, thereby improving the effect and quality of the generated animation. After the 3D animation generation model is optimized, a variable length continuous image sequence (the sequence length is not necessarily a fixed value N) can be input into the 3D animation generation model, and the animation generation model can output a 3D animation sequence of corresponding length that meets the realism requirements. The entire 3D animation generation process uses video frames (i.e., continuous image sequences) as control signals, and generates corresponding 3D animation sequences based on the 3D animation generation model, while the animation diffusion model only serves as a discriminator that provides prior knowledge of motion and does not participate in the execution of the animation generation task.

[0122] The above describes the 3D animation generation method in the embodiment of the present invention. The following describes the 3D animation generation device in the embodiment of the present invention. Figure 5 In one embodiment of the present invention, a 3D animation generating device includes:

[0123] An acquisition module 501 is used to acquire a variable length continuous image sequence from a target video source;

[0124] A generation module 502 is used to input the continuous image sequence into a trained 3D animation generation model to perform 3D posture extraction and output a 3D animation sequence of corresponding length;

[0125] After the 3D animation generation model is pre-trained, it is trained again using the trained animation diffusion model to optimize the generation effect of the 3D animation generation model.

[0126] Optionally, in one embodiment, the 3D animation generating device further includes:

[0127] The first training module 503 is used to obtain a continuous image sequence and a corresponding real 3D animation sequence as training samples; input each frame of the continuous image sequence into the convolution layer of the 3D animation generation model to be trained for downsampling and feature extraction, and output a shallow feature map corresponding to each frame of the image; flatten the shallow feature map corresponding to each frame of the image to form a one-dimensional vector, and obtain a feature sequence corresponding to the continuous image sequence; input the feature sequence into the visual model of the 3D animation generation model to be trained for 3D posture extraction, and output a 3D animation sequence corresponding to the continuous image sequence; based on the real 3D animation sequence and the 3D animation sequence output by the model, train the 3D animation generation model to obtain a pre-trained 3D animation generation model.

[0128] Optionally, in one embodiment, the 3D animation generating device further includes:

[0129] The second training module 504 is used to obtain a 3D animation sequence as a training sample; based on the forward diffusion process, the first noise used in the current time step is added to the 3D animation sequence to obtain the first noisy animation sequence of the current time step; the first noisy animation sequence based on the time step, the time code corresponding to the time step, and the position code corresponding to each frame of data in the first noisy animation sequence are input into the animation diffusion model to be trained for noise prediction, and the second noise corresponding to the time step is output; based on the first noise and the second noise, the animation diffusion model is trained to obtain a trained animation diffusion model.

[0130] Optionally, in one embodiment, the 3D animation generating device further includes:

[0131] The third training module 505 is used to obtain a continuous image sequence of variable length; input the continuous image sequence into a pre-trained 3D animation generation model to extract 3D postures, and output a 3D animation sequence of corresponding length; based on the forward diffusion process, add the third noise used in the current time step to the 3D animation sequence output by the pre-trained 3D animation generation model to obtain a second noisy animation sequence for the current time step; input the second noisy animation sequence based on the time step, the time code corresponding to the time step, and the position code corresponding to each frame of data in the second noisy animation sequence into the trained animation diffusion model for noise prediction, and output a fourth noise corresponding to the time step; based on the third noise and the fourth noise, train the pre-trained 3D animation generation model to obtain a final trained 3D animation generation model.

[0132] Optionally, in one embodiment, the third training module 505 is specifically used for:

[0133] During the training process of the pre-trained 3D animation generation model, the weights of the trained animation diffusion model are frozen and not updated; a noise loss function is constructed based on the third noise and the fourth noise, and based on the noise loss function, the gradient calculation and weight update of the pre-trained 3D animation generation model are performed to obtain the final trained 3D animation generation model.

[0134] Optionally, in one embodiment, the gradient of the noise loss function relative to the model parameters of the pre-trained 3D animation generation model is:

[0135]

[0136] in, represents the gradient of the noise loss function relative to the model parameters of the pre-trained 3D animation generation model, L3D represents the noise loss function used when optimizing the pre-trained 3D animation generation model, θ represents the model parameters of the 3D animation generation model, represents the parameters related to the time step t, D(Z t |t) represents the noise predicted by the trained animation diffusion model, D represents the animation diffusion model, Z represents the output of the pre-trained 3D animation generation model, and Z t represents the noisy animation sequence obtained by adding noise to Z at time step t, e t represents the noise satisfying Gaussian distribution added at the tth time step;

[0137] Optionally, in one embodiment, the following formula is used to update the weight of the pre-trained 3D animation generation model:

[0138]

[0139] Among them, θ′ represents the model parameters after the weight update of the 3D animation generation model, β represents the learning rate, and ||D(Z t |t)-e t || represents the error term of noise, represents the gradient of the animated diffusion model and takes 1, Represents the gradient of the 3D animation generative model.

[0140] Since the embodiments of the device part correspond to the embodiments of the above method, please refer to the above method embodiments for the introduction of the 3D animation generation device provided by the present invention. The present invention will not be repeated here, and it has the same beneficial effects as the above 3D animation generation method.

[0141] above Figure 5 The 3D animation generating apparatus in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the computer device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0142] Figure 66 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention. The computer device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 610 (for example, one or more processors) and a memory 620, and one or more storage media 630 (for example, one or more mass storage devices) storing application programs 633 or data 632. Among them, the memory 620 and the storage medium 630 can be short-term storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device 600. Furthermore, the processor 610 can be configured to communicate with the storage medium 630 to execute a series of instruction operations in the storage medium 630 on the computer device 600.

[0143] The computer device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input and output interfaces 660, and / or one or more operating systems 631, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 6 The illustrated computer device structure does not constitute a limitation on the computer device, and may include more or fewer components than illustrated, or combine certain components, or arrange the components differently.

[0144] The present invention also provides a computer device, which includes a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor executes the steps of the 3D animation generation method in the above embodiments.

[0145] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the 3D animation generation method.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0148] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D animation generation method, characterized in that: The 3D animation generation method comprises: Obtaining a variable length continuous image sequence from a target video source; Inputting the continuous image sequence into a trained 3D animation generation model to extract 3D postures, and outputting a 3D animation sequence of corresponding length; After the 3D animation generation model is pre-trained, it is trained again using the trained animation diffusion model to optimize the generation effect of the 3D animation generation model.

2. The 3D animation generation method according to claim 1, characterized in that: The 3D animation generation model is obtained by the following training method: Obtain continuous image sequences and corresponding real 3D animation sequences as training samples; Input each frame of the continuous image sequence into the convolutional layer of the 3D animation generation model to be trained for downsampling and feature extraction, and output the shallow feature map corresponding to each frame of the image; Flatten the shallow feature map corresponding to each frame of the image to form a one-dimensional vector, and obtain the feature sequence corresponding to the continuous image sequence; Inputting the feature sequence into a visual model of a 3D animation generation model to be trained to extract 3D postures, and outputting a 3D animation sequence corresponding to a continuous image sequence; Based on the real 3D animation sequence and the 3D animation sequence output by the model, the 3D animation generation model is trained to obtain a pre-trained 3D animation generation model.

3. The 3D animation generation method according to claim 1, characterized in that: The animation diffusion model is obtained by the following training method: Get 3D animation sequences and use them as training samples; Based on the forward diffusion process, the first noise used in the current time step is added to the 3D animation sequence to obtain the first noisy animation sequence of the current time step; Input the first noisy animation sequence based on the time step, the time code corresponding to the time step, and the position code corresponding to each frame of data in the first noisy animation sequence into the animation diffusion model to be trained for noise prediction, and output the second noise corresponding to the time step; Based on the first noise and the second noise, an animation diffusion model is trained to obtain a trained animation diffusion model.

4. The 3D animation generation method according to claim 2, characterized in that: The trained animation diffusion model uses the following training method to train the pre-trained 3D animation generation model: Acquire a continuous image sequence of varying length; Input the continuous image sequence into the pre-trained 3D animation generation model to extract the 3D posture and output a 3D animation sequence of corresponding length; Based on the forward diffusion process, the third noise used in the current time step is added to the 3D animation sequence output by the pre-trained 3D animation generation model to obtain the second noisy animation sequence of the current time step; Input the second noisy animation sequence based on the time step, the time code corresponding to the time step, and the position code corresponding to each frame of data in the second noisy animation sequence into the trained animation diffusion model for noise prediction, and output the fourth noise corresponding to the time step; Based on the third noise and the fourth noise, the pre-trained 3D animation generation model is trained to obtain a final trained 3D animation generation model.

5. The 3D animation generation method according to claim 4, characterized in that: The training of the pre-trained 3D animation generation model based on the third noise and the fourth noise to obtain a final trained 3D animation generation model comprises: During the training of the pre-trained 3D animation generation model, the weights of the frozen trained animation diffusion model are not updated; A noise loss function is constructed based on the third noise and the fourth noise, and based on the noise loss function, gradient calculation and weight update are performed on the pre-trained 3D animation generation model to obtain a final trained 3D animation generation model.

6. The 3D animation generation method according to claim 5, characterized in that: The gradient of the noise loss function relative to the model parameters of the pre-trained 3D animation generation model is: in, represents the gradient of the noise loss function relative to the model parameters of the pre-trained 3D animation generation model, L 3D represents the noise loss function used when optimizing the pre-trained 3D animation generation model, θ represents the model parameters of the 3D animation generation model, represents the parameters related to the time step t, D(Z t |t) represents the noise predicted by the trained animation diffusion model, D represents the animation diffusion model, Z represents the output of the pre-trained 3D animation generation model, and Z t represents the noisy animation sequence obtained by adding noise to Z at time step t, e t Represents the noise that satisfies the Gaussian distribution added at the tth time step.

7. The 3D animation generation method according to claim 6, characterized in that: The following formula is used to update the weights of the pre-trained 3D animation generation model: Among them, θ′ represents the model parameters after the weight update of the 3D animation generation model, β represents the learning rate, and ||D(Z t |t)-e t || represents the error term of noise, represents the gradient of the animated diffusion model and takes 1, Represents the gradient of the 3D animation generative model.

8. A 3D animation generating device, characterized in that: The 3D animation generating device comprises: An acquisition module, used for acquiring a variable-length continuous image sequence from a target video source; A generation module, used for inputting the continuous image sequence into a trained 3D animation generation model to extract 3D postures and output a 3D animation sequence of corresponding length; After the 3D animation generation model is pre-trained, it is trained again using the trained animation diffusion model to optimize the generation effect of the 3D animation generation model.

9. A computer device, characterized in that: The computer device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the computer device to execute the 3D animation generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, a 3D animation generation method according to any one of claims 1 to 7 is implemented.

Images