A text sketch guided 3D cartoon character video generation method and system and a medium

Abstract

The application discloses a 3D cartoon character motion video generation method oriented to a text sketch guide, takes a text prompt and a sketch as explicit input, takes noise input and a time step as implicit input, and correspondingly obtains a text vector and a motion feature; the noise input, the time step, the text vector and the motion feature are taken as input of a diffusion model to obtain noise of a next time step; after denoising by one step, a motion feature obtained from the sketch is used to calculate a key frame posture loss through a regressor, so that the key frame posture is more suitable for the sketch posture; a natural guide module is responsible for adjusting the natural degree of the key frame and adjacent frame postures to obtain denoised noise input; and after T-step continuous denoising, a final motion sequence is obtained. Through sketch feature extraction and sketch control information fusion branches, the application realizes the guiding function of the sketch on human motion sequence generation, and through the natural guide module, realizes the natural and smooth adjacent frame postures and motion diversity in the human motion sequence.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D reconstruction technology, and in particular to a method, system and medium for generating 3D animation character motion videos guided by text sketches. Background Technology

[0002] With the rapid iteration of computer vision and graphics, 3D human motion sequence generation technology has gradually matured, showing broad application prospects in virtual reality, digital entertainment, personalized fashion design, and other fields. Although 3D human motion video generation technology has made breakthrough progress in recent years, the currently generated motion sequences still have obvious limitations, especially in terms of the strong randomness and insufficient stability of human postures. Sketches, as an intuitive and convenient artistic expression medium, have irreplaceable advantages over direct motion input: they can accurately capture the core structural features of human motion and abstract and simplify redundant and complex details, thus becoming an ideal medium for rapid prototyping of 3D human motion and exploration of creative solutions, providing a new approach to solving the problem of posture randomness.

[0003] However, the expressive characteristics of sketches and the complexity of cross-modal information conversion present a dual challenge to sketch-guided 3D human motion generation: On the one hand, there is the inherent technical bottleneck of 3D human motion generation itself. Even with the introduction of guiding signals, existing methods still struggle to balance the temporal continuity and posture accuracy of motion. In particular, joint drift and disordered movement rhythm are prone to occur in long sequence generation. Furthermore, for dynamic movements with large amplitudes such as running and jumping, it is difficult to ensure the physical rationality of limb movements, and the movement trajectories of local joints (such as wrists and ankles) often deviate from the laws of natural movement. On the other hand, there is the modal adaptation problem of sketches as a guiding medium. As a 2D abstract expression, sketches have significant individual differences in the degree of line simplification and drawing style (such as simple outlines and detailed strokes). It is difficult for models to uniformly extract the core features of motion. At the same time, 2D sketches lack depth information, and when mapped to 3D motion space, problems such as confusion of limb front-back relationships and distortion of posture scale are prone to occur. More importantly, existing methods often treat sketches as global guiding signals, failing to establish a precise correlation between local sketch lines (such as torso curves and limb outlines) and the corresponding joint movements of the human body. This results in insufficient fine-grained control of the sketch over posture, making it impossible to achieve targeted optimization of motion posture by adjusting local sketch details. These challenges are intertwined, making it difficult for existing sketch guidance methods to accurately correlate sketch semantics with 3D motion posture. Consequently, they cannot guarantee the stability and continuity of motion, nor can they fully leverage the precise control advantages of sketches.

[0004] Regarding technical solutions for motion generation, the goal of human motion generation tasks is to generate motion based on conditional signals. Various conditions have been explored, such as partial poses, trajectories, images, music, text, objects, action labels, or no conditions. Recently, many diffusion-based motion generation models have been proposed and have demonstrated better quality compared to alternative models such as Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs). By employing CLIP models, these models have shown significant improvements in challenging text-to-motion generation tasks, while also being able to condition with partial motion. However, they do not support conditional signals that have not been specifically trained, such as following keyframe positions or avoiding obstacles.

[0005] Motion synthesis can be broadly categorized into two types: autoregressive methods and sequence-level methods. Autoregressive methods utilize information from past motion to recursively generate the current motion frame by frame. These methods are primarily suitable for real-time scenarios. In contrast, sequence-level methods aim to generate complete, fixed-length motion sequences. Due to this inherent characteristic, they can be seamlessly integrated with existing generative models, such as variational autoencoders (VAEs) and diffusion models, enabling various cueing functions that can originate from diverse external sources.

[0006] Regarding motion generation schemes based on diffusion models, these generative models have attracted widespread attention due to their impressive performance in image generation. Diffusion models are well-suited for control and condition setting, and depending on the different control and condition settings, they can generally be categorized into methods such as interpolation and inpainting, classifier-guided, classifier-free guided, and neural network-guided methods using ControlNet.

[0007] (1) Imputation and repair use observation data to fill in missing parts of the data so that the filled content is visually consistent with the surrounding area. However, this becomes difficult when the observation data and the filled part are in different spaces, such as when generating an image from a semantic map.

[0008] (2) Classifier guidance improves the conditional diffusion generative model by training a single classifier.

[0009] (3) Classifier-free joint training of conditional and unconditional diffusion models is combined to achieve a balance between sample quality and diversity. GLIGEN adds a trainable gated self-attention layer to each Transformer block to absorb new grounded inputs.

[0010] (4) ControlNet introduces a neural network designed to control large image diffusion models, which can quickly adapt to control signals for specific tasks with minimal data and training.

[0011] These control methods are not mutually exclusive, and using only one of them may not achieve the desired results. Summary of the Invention

[0012] The technical problem to be solved by the embodiments of the present invention is to provide a method for generating 3D animation character motion videos guided by text sketches. This method can combine a control branch based on the control signal of the generated sketch features and a noise reduction main branch to form a dual-branch architecture, which can effectively integrate sketch features and local motion features.

[0013] To address the aforementioned technical problems, embodiments of the present invention provide a method for generating 3D animation character motion videos guided by text sketches, comprising the following steps: S1: Display text prompts ,sketch The input is explicit, and then noise is input. And time steps as implicit input; S2: The text prompt p is encoded to obtain the corresponding text vector, while the sketch Action features are obtained through a feature extractor; S3: Input the noise The time step, the text vector, and the action features are used as inputs to the diffusion model, and the noise of the next time step is obtained. After each step of denoising, the action features obtained from the sketch are used by a regressor to calculate the keyframe pose loss. S4: The natural guidance module is responsible for adjusting the pose naturalness of keyframes and adjacent frames, ultimately obtaining the denoised noise input. After T steps of continuous denoising, the final motion sequence is obtained. .

[0014] Furthermore, S3 includes a noise-adding step, and includes the following steps: S311: Define the accumulation of noise during the noise addition process as... ,in It's clean input. , It is a noise scheduler used to control the noise intensity at each step; the result after adding noise is... ; S312: Will have trainable parameters The denoising model uses an approximate posterior distribution. This indicates that during the training phase, the noise generated by the noise-adding process... and corresponding clean data As a training pair, Input into a neural network, learn to predict To make the prediction results as close as possible to reality. Optimize parameters through directional propagation Until the model can stably process noisy data. To restore a clearer image ,go through After continuous denoising, the diffusion model will have a prior distribution Mapping to target data distribution ; S313: When the noisy time step t reaches the maximum total number of steps, the data is completely degraded into random noise from the prior distribution. Take one noise start, Represented as complete noise, during the noise reduction process, for different time steps ,sampling ,until ,in

[0015] in, It is the selected variance scheduler. This is the prediction result of the denoising model. When the denoising iteration reaches... At that time, the model outputs the final clean data.

[0016] Furthermore, S3 includes a denoising step, and includes denoising using one of the following methods: directly predicting clean input. of Model and prediction mixed with noise of Model, mean of predicted denoising distribution of Model; in: The model uses squared loss with clean input. Training was conducted, among which This indicates that the model input contains noisy data. Clean data for the final output prediction; The model uses squared loss. Training was conducted, among which This indicates that the model input contains noisy data. The predicted noise is mixed into the output. This refers to the actual noise mixed in during the forward noise addition process; The model uses squared loss. Training was conducted, among which The model input is noisy data. The mean of the predicted denoising distribution in the output. This represents the true mean of the denoised distribution.

[0017] Furthermore, S3 also includes using an analytical function. Evaluate the spatial position of the joints in the generated motion relative to the desired sketch pose. The alignment degree is determined, and the spatial position of the pose joints corresponding to the sketch is obtained through the pose generator, so that the keyframe pose in the 3D human body sequence is aligned with the 3D human body pose in the sketch.

[0018] Furthermore, the pose generator performs the following steps: S331: Input a hand-drawn sketch. Using a deep residual network ResNet50 as the backbone encoder, perform multi-layer convolution and feature aggregation on the sketch to extract image features of size 7×7×2048. Then, it is compressed into a 2048-dimensional global sketch feature vector through pooling operations. ; S332: Optimized features after residual connection Based on this, combined with the initial attitude parameters of the SMPL model Feature concatenation is performed to supplement the prior structural information of human pose. The concatenated feature vector has a dimension of 2048 + 72 = 2120. The concatenated feature vector is input into the MLP module of the pose decoder. Through multi-layer fully connected and non-linear mapping, the regression from sketch features to SMPL pose parameters is completed, finally obtaining 72-dimensional SMPL pose parameters. This pose parameter corresponds to the three-dimensional rotation of the 23 human joints in the SMPL model and the global spatial position of the model. S333: Input the generated 72-dimensional SMPL attitude parameters into the SMPL model, and perform forward computation. Obtain the 3D spatial coordinates of 23 joints ,in For SMPL shape parameters, the default mean value is used. , This is a function for calculating joint positions in the SMPL model. Following the mapping rules between HumanML3D and SMPL, two joints without corresponding entries in HumanML3D are removed from SMPL. The 3D coordinates of the remaining 21 joints are then rearranged according to the index order in HumanML3D, resulting in... The rearranged joint coordinates are then transformed to obtain the final 3D joint coordinates.

[0019] Furthermore, the analytical function The gradient is used to guide the generated motion in the desired direction, and spatial guidance is employed in each denoising step. Perturb the predicted mean: ; in The original mean value for denoising in step t of the diffusion model. For loss function right The gradient is used to represent the spatial position that makes the generated result more consistent with the desired sketch pose. The direction that needs to be adjusted is... Control the intensity of guidance, Measuring the relationship between joint positions and spatial constraints in generating motion distance: ; in It is a binary value representing the sketch control signal. In frame For joints Does it contain valid values? The joint positions are output by the model. Represents the nth frame, the nth frame Target constraint position of the joint. Represents the nth frame, the nth frame Global location for joint generation Convert the local position of the joint to the global absolute position.

[0020] Furthermore, it also includes determining the global position of the pelvis in a frame by accumulating and aggregating the rotations and translations of all previous frames, and determining the positions of the other joints by aggregating the position of the pelvis with the relative positions of the other joints.

[0021] Furthermore, the position of the pelvic bone is determined by the following method: S341: Define the basic coordinate system and single-frame transformation, the local coordinate system with the pelvis itself as the origin, and the rotation matrix of the pelvis in the nth frame relative to the global coordinate system of the pelvis in the previous frame. Translation vector ; S342: Calculate the global position frame by frame, setting the global position of the pelvis in the first frame's pose as absolute coordinates. This represents the corresponding coordinate point in the absolute coordinate system, and sets the initial rotation matrix. Subsequent frames can be obtained through recursive calculation. ,in It is from the first frame to the... The cumulative rotation matrix of the frame is used to convert the local translation of the pelvis in the nth frame into a global translation in the absolute coordinate system, and then superimpose the global position of the previous frame to obtain the global position of the pelvis in the current frame.

[0022] Furthermore, the positions of the other joints are determined by the following method: For any joint Global position in frame n Based on the global position of the pelvis and joints Relative position of the pelvis calculate By accumulating rotation and translation, the predicted local joint positions in the diffusion model are converted into global absolute positions in the world coordinate system.

[0023] Furthermore, the natural guidance module performs the following steps: S41: Naturally guided reception of text prompts is the same as the action diffusion model. and sketch control signals Each Transformer layer in the original model is connected to a new trainable copy via a linear layer whose weights and biases are initialized to zero. As training progresses, realism guides the model to learn spatial constraints and adds the learned feature corrections to the corresponding layers of the action diffusion model to implicitly correct the generated actions. S42: Keyframe valid values ​​output by the sketch guidance module To determine the constraint frames, obtain the corresponding control signals, distinguish between control frames and invalid frames, encode the control signals, and mask the invalid frames. At this point, the sketch control signal characteristics of the keyframes are obtained. ,Will The text vectors and other data are input into the trainable copy of the Transformer, and the final output is the feature correction value of each layer. A linear mapping is performed on each layer to ensure that the correction value and the feature dimension of the original model are completely matched. Finally, the correction value is superimposed on the original feature to achieve implicit adjustment. Finally, the corrected features are used to continue to complete the subsequent calculations of the denoising step.

[0024] This invention also provides a system for generating 3D animation character motion videos guided by text sketches, the system being obtained using the text sketch-guided 3D animation character motion video generation method described in this invention.

[0025] This invention provides a computer-readable storage medium including a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the text-based sketch-guided 3D animation character motion video generation method as described in this invention.

[0026] Implementing the embodiments of the present invention has the following beneficial effects: The present invention introduces a pre-trained sketch feature extraction and sketch control information fusion branch, which realizes the sketch's guiding function for the generation of human motion sequences and contributes to the diversity of action sequence postures. Through the natural guidance module, the postures of adjacent frames in the human motion sequence are natural and smooth, reducing the generation of unnatural actions and improving action diversity. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the framework of the sketch guidance module of the present invention; Figure 2 It is a natural guidance module design framework; Detailed Implementation To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0028] The present invention provides a method for generating 3D animation character motion videos guided by text sketches, which is implemented through the following method.

[0029] like Figure 1 As shown, the method proposed in this invention is used to integrate sketch constraints into the human motion generation process. (Given text prompts) and additional sketch input (3×224×224), the goal of this invention is to generate a human motion sequence. .in It is the length of the motion sequence. In the HumanML3D dataset, the dimension of human pose representation is the sketch control signal. Obtained from the sketch-guided module.

[0030] The main steps include: using text prompts Sketch (PNG image) The input is explicit, and then noise is input. The time step is used as an implicit input. The text prompt is processed by the Encoder to obtain the corresponding text vector, while the Sketch is processed by the Feature Extractor to obtain motion features. At this point, noise is input, and the time step, text vector, and motion features are used together as input to the Diffusion Model to obtain the noise for the next time step. This is also called one-step denoising. After each denoising step, the motion features obtained from the Sketch are used by the regressor to calculate the keyframe pose loss, i.e., SketchGuidance, to make the keyframe pose more consistent with the sketch pose. NatureGuidance is responsible for adjusting the pose naturalness of the keyframe and neighboring frames, and finally, the denoised noise input is obtained. After T steps of continuous denoising, the final motion sequence is obtained. .

[0031] The diffusion model is used to perform the following steps: Diffusion-based probabilistic generative models (DPMs) are a class of generative models that learn from inputs. The denoising process at different noise levels T. To provide a clear learning objective for the denoising model training, a controllable and reversible "clean data to noisy data" process needs to be constructed. Since the denoising process is gradual, not a one-time denoising, this invention needs to design the noise addition process as gradual as well. Therefore, the noise addition process of DPM is cumulatively defined as... That is, in Under the conditions The probability distribution can also be expressed as from The result obtained after t-step noise addition ,in It's clean input. , It is a noise scheduler used to control the noise intensity at each step; the result after adding noise is... It is from the distribution The data obtained after noise addition is obtained from mid-sample. Here, it's important to clarify the difference between different noise levels T and the number of noise addition steps t. Noise level T characterizes the actual noise intensity at a specific noise addition time step t, while the number of noise addition steps t only represents an abstract index of the noise addition iteration time step. In this invention, it is guaranteed that the noise added at each step... While all conform to a Gaussian distribution, Cumulative calculations and clean data A mathematical connection was established.

[0032] With trainable parameters The denoising model uses an approximate posterior distribution. It is said to be composed of a neural network, in which These are the weights and biases of the network. During the training phase, they are generated using the noise-adding process. and corresponding clean data As a training pair, Input into a neural network, learn to predict To make the prediction results as close as possible to reality. Optimize parameters through directional propagation Until the model can stably process noisy data. To restore a clearer image .go through After continuous denoising, DPMs can reduce the prior distribution Mapping to target data distribution .

[0033] In the diffusion model, when the number of time steps t is reached (the maximum total number of steps), the data will completely degenerate into random noise, and the noise distribution at this point will be a standard normal distribution. Therefore, in order to draw samples from the DPM, this invention starts from the prior distribution. Take one noise start, This is represented as complete noise, i.e., t reaches its maximum value, at which point no noise reduction step is performed. Then, during the noise reduction process, for different time steps... The present invention samples ,until ,in (1) It is the selected variance scheduler, typically

[46] . In equation (1) This is the prediction result of the denoising model, for Model, Finally, when the denoising iteration reached... At that time, the final clean data output by the model is... ,at this time With the model continuously predicting during the denoising process Totally consistent.

[0034] The denoising model can predict a variety of objects, including directly predicting clean input. of Model and prediction mixed with noise of Model and mean of the predicted denoised distribution of Model.

[0035] The model uses squared loss with clean input. Training was conducted, among which This indicates that the model input contains noisy data. Clean data for the final output prediction; The model uses squared loss. Training was conducted, among which This indicates that the model input contains noisy data. The predicted noise is mixed into the output. This refers to the actual noise mixed in during the forward noise addition process; The model uses squared loss. Training was conducted, among which The model input is noisy data. The mean of the predicted denoising distribution in the output. The true mean of the denoised distribution is given (derived from diffusion model theory). Of the three schemes mentioned above, this scheme is preferred. Model, i.e., input noise Predicting noise during the denoising process Then, noise is gradually removed to finally obtain clean data. .

[0036] The following steps are performed using the sketch-guided module: The core of sketching guidance is an analytical function. This is used to evaluate the spatial position of the joints in the generated motion relative to the desired sketch pose. To address the alignment issue, this invention, based on the analytical function, adds a Sketchtopose pose generator to correspond the sketch to its 3D spatial position. This allows the spatial position of the pose joints corresponding to the sketch to be obtained, thus aligning the keyframe poses in the 3D human sequence with the 3D human pose in the sketch. Therefore, the analytical function is responsible for evaluating the alignment, while the pose generator provides the analytical function with the desired spatial position corresponding to the sketch.

[0037] The Sketchtopose pose generator works by extracting image features from a sketch to create corresponding action poses. It first takes a hand-drawn sketch as input, then uses a ResNet50 deep residual network as the backbone encoder to perform multi-layer convolutions and feature aggregation on the sketch, extracting image features of size 7×7×2048. Then, it is compressed into a 2048-dimensional global sketch feature vector through pooling operations. This feature vector integrates information such as the global outline of the sketch, key lines, and human anatomy.

[0038] For 2048-dimensional sketch feature vectors The pose decoder uses a multi-head self-attention mechanism to globally enhance features, focusing them on key pose-related information such as human joints and limb alignment in the sketch. This enhancement is achieved through linear layers. Mapped to a query vector specific to the pose branch Key vector Value vector Enhanced features for pose branches are obtained using a multi-head self-attention mechanism. , original features With enhanced features Perform residual connections to obtain optimized global pose features. .

[0039] Optimization features after residual connection Based on this, combined with the initial attitude parameters of the SMPL model Feature concatenation is performed to supplement prior structural information of the human pose, resulting in a concatenated feature vector with a dimension of 2048 + 72 = 2120. This concatenated feature vector is then input into the MLP module of the pose decoder. Through multi-layer fully connected layers and non-linear mapping, regression from sketch features to SMPL pose parameters is completed, ultimately yielding 72-dimensional SMPL pose parameters. This pose parameter corresponds to the three-dimensional rotation of the 23 human joints in the SMPL model and the global spatial position of the model.

[0040] Input the 72-dimensional SMPL pose parameters generated by SketchtoPose into the SMPL model, and perform forward computation. Obtain the 3D spatial coordinates of 23 joints (in world coordinates), where For SMPL shape parameters, the default mean value is used. , This is a function for calculating joint positions in the SMPL model. Following the mapping rules between HumanML3D and SMPL, two joints without corresponding entries in HumanML3D are removed from SMPL. The 3D coordinates of the remaining 21 joints are then rearranged according to the index order in HumanML3D, resulting in... The rearranged joint coordinates are then transformed to obtain the final 3D joint coordinates.

[0041] analytic function The gradient is used to guide the generated motion in the desired direction. This invention employs spatial guidance in each denoising step. Perturb the predicted mean: (2) in The original mean value for denoising in step t of the diffusion model. For loss function right The gradient is used to represent the spatial position that makes the generated result more consistent with the desired sketch pose. The direction that needs to be adjusted is... Control the intensity of guidance. Measuring the relationship between the joint position that generates the motion and the desired spatial position distance: (3) in It is a binary value representing the sketch control signal. In frame For joints Does it contain valid values? The joint positions are output by the model. Represents the nth frame, the nth frame The target constraint position of the joint (from the sketch control signal). Represents the nth frame, the nth frame Global location for joint generation The local position of the joint is converted into a global absolute position. For simplicity, the diffusion denoising step is omitted here. In this context, the global position of the pelvis in a given frame can be determined by accumulating and aggregating the rotations and translations of all previous frames, and the positions of other joints can also be determined by aggregating the relative positions of the pelvis and other joints.

[0042] The pelvis is the root joint (often referred to as the pelvic joint) of human movement. The global position of all other joints is based on the pelvis. The cumulative aggregation of its global position is performed frame by frame from the first frame of the motion sequence to the current frame, accumulating rotation and translation transformations. First, a basic coordinate system and single-frame transformations are defined. This method uses an absolute coordinate system (global coordinates) with the XZ plane as the ground and the positive Y-axis as the top. Simultaneously, a local coordinate system for the pelvis is designed, i.e., a relative coordinate system, with the pelvis itself as the origin. The rotation matrix of the pelvis in the nth frame relative to the global coordinate system of the pelvis in the previous frame is... (3×3, used to describe orientation changes) Translation vector (3×1, describing spatial positional movement), these two quantities are the core motion information contained in the diffusion model (predicted local joint positions including root joint rotation and translation). Then, the global position is calculated frame by frame, setting the global position of the pelvis in the first frame's pose as the absolute coordinates. This represents the corresponding coordinate point in the absolute coordinate system, and sets the initial rotation matrix. (Identity matrix, no initial rotation), subsequent frames (frame n) It can be obtained through recursive calculation. ,in It is from the first frame to the... The cumulative rotation matrix of the frame is used to convert the local translation of the pelvis in the nth frame into a global translation in the absolute coordinate system, and then superimpose the global position of the previous frame to obtain the global position of the pelvis in the current frame.

[0043] For global position calculations of other joints, for any joint Global position in frame n Based on the global position of the pelvis and joints Relative position of the pelvis calculate That is, through the cumulative rotation and translation described above, the predicted local joint positions in the diffusion model are converted into global absolute positions in the world coordinate system.

[0044] The key advantage of the sketch-guided approach proposed in this invention in terms of control accuracy lies in the fact that the gradient is calculated relative to the predicted mean µt, which only requires backpropagation through the lightweight function in equation (3). Simultaneously, this invention can perturb the generated motion sequence multiple times, thereby significantly improving control accuracy. Specifically, in the denoising step… This invention applies formula (2) iteratively. The next disturbance , The value is dynamically set to balance control precision and inference speed; its core function is for each denoising step. The formula for assigning the number of iterations for gradient perturbation is as follows: (4) in This represents the total number of denoising steps in the diffusion model. The denoising process is divided into later stages by setting thresholds for each stage. ) and the early stage ( ), corresponding For the number of later iterations, This refers to the number of iterations in the early stages, which varies depending on the time period. Different iteration numbers are obtained. The dynamic K value improves the efficiency of "sketching constraint guidance": after the attitude generator provides attitude data, the analytical function G calculates the alignment between the current attitude data and the attitude data generated by the diffusion model, and the gradient... The guide direction is given, and the K value determines the degree of adjustment along that guide direction, so that the generated pose (joint space position) fits the sketch, while ensuring natural movement and acceptable inference speed.

[0045] The Natural Boots module is used to perform the following: While sketch guidance can effectively prompt controlled joints to follow input control signals, it may leave other joints unchanged, leading to unrealistic movements. Furthermore, because the perturbed position is only a small part of the overall movement, motion diffusion models may ignore changes from spatial guidance and fail to make appropriate adjustments to other joints, resulting in disjointed movements and foot slippage.

[0046] To address issues such as unrealistic joint control, neglect of local spatial changes, disjointed movements, and foot slippage caused by sketch guidance, this invention proposes Natural Guidance. By implicitly modifying features of the motion diffusion model, it allows all joints to naturally adjust according to sketch constraints. Specifically, it is a trainable copy of the Transformer encoder in the motion diffusion model, used to learn how to enforce spatial constraints. Natural Guidance receives the same text prompts as the motion diffusion model. (e.g., “Jackiswalking”, provided by user input or dataset) and sketch control signals. (That is, whether the frame's pose is controlled by a sketch, user input, or a dataset). Each Transformer layer in the original model is connected to a new trainable copy via a linear layer whose weights and biases are initialized to zero, so they have no control at the beginning. As training progresses, realism guides the model to learn spatial constraints and adds the learned feature corrections to the corresponding layers of the action diffusion model to implicitly correct the generated actions.

[0047] like Figure 2 As shown, this invention uses a sketch control signal. In the The specific joint constraints on a frame are extracted from the complete action sequence by time frame. It is a binary label, which is in equation (3) The aggregation, therefore if If any of them is 1, then It should be 1 (valid); otherwise, =0 (invalid). These are the features of the sketch control signal in the nth frame, and these features are input into a trainable copy of the Transformer. This helps subsequent attention layers understand the location of effective sketch control signals, thereby refining the corresponding features.

[0048] Specifically, this is manifested in the keyframe valid values ​​output by the sketch guidance module. (Obtained from SketchImage and SketchMask via the sketch guidance module), to determine the constraint frames, obtain the corresponding control signals, distinguish between control frames and invalid frames, and then encode the control signals and mask the invalid frames. At this point, the sketch control signal characteristics of the keyframes are obtained. At this time Along with text vectors, these are input into a trainable copy of the Transformer (8 layers in total). The final output is the feature correction value for each layer. A linear mapping is performed on each layer to ensure that the correction value and the feature dimension of the original model are perfectly matched. Finally, the correction value is superimposed on the original features to achieve implicit adjustment. The corrected features are then used to continue the subsequent calculations for the denoising step. After all denoising steps are completed, the final natural motion sequence is output.

[0049] Accordingly, embodiments of the present invention also disclose a system for generating 3D animation character motion videos guided by text sketches, wherein the system is obtained by the method for generating 3D animation character motion videos guided by text sketches according to embodiments of the present invention.

[0050] This invention provides a computer-readable storage medium including a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the text-based sketch-guided 3D animation character motion video generation method as described in this invention.

[0051] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0052] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0053] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0054] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created based on the use of the mobile terminal, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0055] Wherein, if the modules / units integrated in the terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0056] To evaluate the performance of this invention, two datasets, HumanML3D and KIT-ML, were used respectively. The control accuracy was measured by comparing the naturalness of the generated motion (FID), the relevance of the generated motion to its text prompt (R-precision), the variability within the generated motion (Diversity), foot slip rate (FootSkating), trajectory error, and the average error of the controlled joint position in keyframes.

[0057] HumanML3D, with its rich sample library covering various types of movements such as daily life, sports, and dance, and its accurate text-action pairing annotations, provides reliable support for naturalness and text relevance assessment. KIT-ML, with its diverse human skeleton dimension data and action sequences in complex scenes, strengthens the testing effectiveness of variability and control precision indicators. Specifically, FID intuitively reflects naturalness by calculating the distance between the generated action distribution and the real action distribution; R-precision assesses relevance by judging the ranking of the corresponding action in the candidate set; Diversity quantifies the degree of difference using the distribution entropy in the action feature space; FootSkating measures realism by the position offset rate of the foot contact frame with the ground; trajectory error focuses on the deviation of the motion path of key human body parts from the expected value; and the average error of the controlled joint position in key frames accurately captures the control precision of the core joints at key time points. The combination of multiple dimensions of indicators enables a comprehensive and detailed consideration of the overall performance of the model.

[0058] To verify the performance of this invention, comparative experiments were conducted with recent state-of-the-art methods, including MDM, PriorMDM, and GMD. It is worth noting that GMD only processes the position of the human pelvis on the ground (xz coordinates), therefore this invention retrains GMD to handle the complete position.

[0059]

[0060] Table 1 compares various motion generation models on the HumanML3D and KIT-ML datasets. As shown in Table 1, the SketchControlDiffusion method proposed in this invention achieves the best performance across all metrics. The data in the table shows that the method of this invention consistently outperforms all existing methods in pelvic control across all metrics, both in terms of realism and control precision. In particular, compared to PriorMDM, the method of this invention significantly improves FID (Film Indicator Distortion), demonstrating that the hybrid guidance proposed in this invention generates more realistic movements. The method of this invention also surpasses the previous state-of-the-art method, GMD (Foot Mass Deposition Method). Furthermore, compared to all other methods, the method of this invention exhibits the lowest foot slippage rate.

[0061] More importantly, unlike traditional methods that can only control the pelvic area individually and in isolation, the model proposed in this invention can precisely control all joints in the human posture corresponding to any specified frame in a motion video using a single model architecture. This groundbreaking design simplifies the operation process and adaptation threshold in practical application scenarios, and provides unprecedented flexibility and exploration space for precise control, style adjustment, and creative expansion in human motion generation tasks.

[0062] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for generating 3D animation character motion videos guided by text sketches, characterized in that, Includes the following steps: S1: Display text prompts ,sketch The input is explicit, and then noise is input. And time steps as implicit input; S2: The text prompt p is encoded to obtain the corresponding text vector, while the sketch Action features are obtained through a feature extractor; S3: Input the noise The time step, the text vector, and the action features are used as inputs to the diffusion model, and the noise of the next time step is obtained. After each step of denoising, the action features obtained from the sketch are used by a regressor to calculate the keyframe pose loss. S4: The natural guidance module is responsible for adjusting the pose naturalness of keyframes and adjacent frames, ultimately obtaining the denoised noise input. After T steps of continuous denoising, the final motion sequence is obtained. .

2. The method for generating 3D animation character motion video guided by text sketches according to claim 1, characterized in that, S3 includes a noise-adding step, and includes the following steps: S311: Define the accumulation of noise during the noise addition process as... ,in It's clean input. , It is a noise scheduler used to control the noise intensity at each step; the result after adding noise is... ; S312: Will have trainable parameters The denoising model uses an approximate posterior distribution. This indicates that during the training phase, the noise generated by the noise-adding process... and corresponding clean data As a training pair, Input into a neural network, learn to predict To make the prediction results as close as possible to reality. Optimize parameters through directional propagation Until the model can stably process noisy data. To restore a clearer image ,go through After continuous denoising, the diffusion model will have a prior distribution Mapping to target data distribution ; S313: When the noisy time step t reaches the maximum total number of steps, the data is completely degraded into random noise from the prior distribution. Take one noise start, Represented as complete noise, during the noise reduction process, for different time steps ,sampling ,until ,in in, It is the selected variance scheduler. This is the prediction result of the denoising model. When the denoising iteration reaches... At that time, the model outputs the final clean data.

3. The method for generating 3D animation character motion video guided by text sketches according to claim 2, characterized in that, S3 includes a denoising step, and includes denoising using one of the following methods: directly predicting clean input. of Model and prediction mixed with noise of Model, mean of predicted denoising distribution of Model; in: The model uses squared loss with clean input. Training was conducted, among which This indicates that the model input contains noisy data. Clean data for the final output prediction; The model uses squared loss. Training was conducted, among which This indicates that the model input contains noisy data. The predicted noise is mixed into the output. This refers to the actual noise mixed in during the forward noise addition process; The model uses squared loss. Training was conducted, among which The model input is noisy data. The mean of the predicted denoising distribution in the output. This represents the true mean of the denoised distribution.

4. The method for generating 3D animation character motion video guided by text sketches according to claim 3, characterized in that, S3 also includes the use of analytical functions. Evaluate the spatial position of the joints in the generated motion relative to the desired sketch pose. The alignment degree is determined, and the spatial position of the pose joints corresponding to the sketch is obtained through the pose generator, so that the keyframe pose in the 3D human body sequence is aligned with the 3D human body pose in the sketch.

5. The method for generating 3D animation character motion video guided by text sketches according to claim 4, characterized in that, The attitude generator performs the following steps: S331: Input a hand-drawn sketch. Using a deep residual network ResNet50 as the backbone encoder, perform multi-layer convolution and feature aggregation on the sketch to extract image features of size 7×7×2048. Then, it is compressed into a 2048-dimensional global sketch feature vector through pooling operations. ; S332: Optimized features after residual connection Based on this, combined with the initial attitude parameters of the SMPL model Feature concatenation is performed to supplement the prior structural information of human pose. The concatenated feature vector has a dimension of 2048 + 72 = 2120. The concatenated feature vector is input into the MLP module of the pose decoder. Through multi-layer fully connected and non-linear mapping, the regression from sketch features to SMPL pose parameters is completed, finally obtaining 72-dimensional SMPL pose parameters. This pose parameter corresponds to the three-dimensional rotation of the 23 human joints in the SMPL model and the global spatial position of the model. S333: Input the generated 72-dimensional SMPL attitude parameters into the SMPL model, and perform forward computation. Obtain the 3D spatial coordinates of 23 joints ,in For SMPL shape parameters, the default mean value is used. , This is a function for calculating joint positions in the SMPL model. Following the mapping rules between HumanML3D and SMPL, two joints without corresponding entries in HumanML3D are removed from SMPL. The 3D coordinates of the remaining 21 joints are then rearranged according to the index order in HumanML3D, resulting in... The rearranged joint coordinates are then transformed to obtain the final 3D joint coordinates.

6. The method for generating 3D animation character motion video guided by text sketches according to claim 5, characterized in that, The analytical function The gradient is used to guide the generated motion in the desired direction, and spatial guidance is employed in each denoising step. Perturb the predicted mean: ； in The original mean value for denoising in step t of the diffusion model. For loss function right The gradient is used to represent the spatial position that makes the generated result more consistent with the desired sketch pose. The direction that needs to be adjusted is... Control the intensity of guidance, Measuring the relationship between joint positions and spatial constraints in generating motion distance: ； in It is a binary value representing the sketch control signal. In frame For joints Does it contain valid values? The joint positions are output by the model. Represents the nth frame, the nth frame Target constraint position of the joint. Represents the nth frame, the nth frame Global location for joint generation Convert the local position of the joint to the global absolute position.

7. The method for generating 3D animation character motion video guided by text sketches according to claim 6, characterized in that, It also includes determining the global position of the pelvis in a frame by accumulating and aggregating the rotations and translations of all previous frames, and determining the positions of the other joints by aggregating the position of the pelvis with the relative positions of the other joints; The position of the pelvis is determined by the following method: S341: Define the basic coordinate system and single-frame transformation, the local coordinate system with the pelvis itself as the origin, and the rotation matrix of the pelvis in the nth frame relative to the global coordinate system of the pelvis in the previous frame. Translation vector ; S342: Calculate the global position frame by frame, setting the global position of the pelvis in the first frame's pose as absolute coordinates. This represents the corresponding coordinate point in the absolute coordinate system, and sets the initial rotation matrix. Subsequent frames can be obtained through recursive calculation. ,in It is from the first frame to the... The cumulative rotation matrix of the frame is used to convert the local translation of the pelvis in the nth frame into a global translation in the absolute coordinate system, and then superimpose the global position of the previous frame to obtain the global position of the pelvis in the current frame. The positions of the other joints are determined by the following method: For any joint Global position in frame n Based on the global position of the pelvis and joints Relative position of the pelvis calculate By accumulating rotation and translation, the predicted local joint positions in the diffusion model are converted into global absolute positions in the world coordinate system.

8. The method for generating 3D animation character motion video guided by text sketches according to any one of claims 1-7, characterized in that, The natural guidance module performs the following steps: S41: Naturally guided reception of text prompts is the same as the action diffusion model. and sketch control signals Each Transformer layer in the original model is connected to a new trainable copy via a linear layer whose weights and biases are initialized to zero. As training progresses, realism guides the model to learn spatial constraints and adds the learned feature corrections to the corresponding layers of the action diffusion model to implicitly correct the generated actions. S42: Keyframe valid values ​​output by the sketch guidance module To determine the constraint frames, obtain the corresponding control signals, distinguish between control frames and invalid frames, encode the control signals, and mask the invalid frames. At this point, the sketch control signal characteristics of the keyframes are obtained. ,Will The text vectors and other data are input into the trainable copy of the Transformer, and the final output is the feature correction value of each layer. A linear mapping is performed on each layer to ensure that the correction value and the feature dimension of the original model are completely matched. Finally, the correction value is superimposed on the original feature to achieve implicit adjustment. Finally, the corrected features are used to continue to complete the subsequent calculations of the denoising step.

9. A system for generating 3D animation character motion video guided by text sketches, characterized in that, The system is obtained by the text-oriented sketch-guided 3D animation character motion video generation method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the text-based sketch-guided 3D animation character motion video generation method as described in any one of claims 1 to 8.

Images