Digital human gesture style quantification method and system based on kinematics and spatial distribution decomposition, terminal and medium
By constructing a style feature extractor and loss function based on kinematics and spatial distribution decomposition, the style of digital human gestures is quantified, which solves the problem of unstable style in the generation of new characters in the existing technology and realizes the personalization and accurate replication of digital human gestures.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG LAB OF ARTIFICIAL INTELLIGENCE & DIGITAL ECONOMY (SZ)
- Filing Date
- 2026-02-13
- Publication Date
- 2026-06-26
AI Technical Summary
In existing technologies, when generating digital human gestures based on labels or pre-trained models, it is impossible to understand and generate new character styles outside the training set. Furthermore, the features extracted by the pre-trained model are mixed with various factors, resulting in unstable generation and poor stylized action effects.
A style feature extractor and loss function based on kinematic and spatial distribution decomposition are constructed to extract objective style features from a single reference video. The gestures are quantified through two dimensions: kinematic style and spatial style, which guides the generative model to produce gestures that conform to the speech rhythm and have the unique style of the target person.
It achieves objective quantification and precise replication of personalized styles in digital human motion generation, improves the generation effect of stylized motion, and makes the generated motion highly consistent with the "expression" and "body posture" of the target person.
Smart Images
Figure CN121708174B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of personalized motion generation technology for digital humans, and in particular to a method, system, terminal, and medium for quantifying digital human gesture styles based on kinematics and spatial distribution decomposition. Background Technology
[0002] Digital human motion generation refers to the technology of synthesizing natural and expressive body movements (such as gestures and postures) for digital humans (virtual characters) under voice-driven conditions. The core objective of this task is to break down the separation between sound and image, ensuring a high degree of consistency between the visual performance (motions) and auditory input (speech) in rhythm, semantics, and personality style. In voice-driven digital human generation tasks, the lack of an effective method for objectively quantifying human speech styles leads to insufficient personalization of the generated movements, making it difficult to depict and reproduce the unique speech and behavior characteristics of different individuals. In other words, the generated movements are often monotonous and fail to reflect the unique personal characteristics of the target individual. Existing personalized digital human motion generation technologies mainly include: 1. Differentiating different styles based on labels within the dataset: This type of method typically uses discrete labels (such as speaker IDs, emotion labels, or preset style categories) to control the generation results during the training phase. The model distinguishes styles by learning the mapping relationship between specific labels and action patterns. 2. Style extraction based on large-scale multimodal pre-trained models: This type of method uses multimodal models pre-trained on large-scale data (such as CLIP, Contrastive Language-Image Pre-training) to extract features from reference videos and injects them as style conditions into generative models (such as Diffusion Model).
[0003] The two existing technologies described above have the following main drawbacks: 1. First type of method: a) Poor generalization ability: This type of method is limited by the boundaries of the training dataset and can only generate styles that already exist in the training set. It cannot handle the styles of unseen speakers. b) Coarse control granularity: Labels (such as "happy", "ID_001") can only represent a broad category and cannot capture the subtle habits that constitute personal style (such as specific finger movements, habitual gesture amplitude, etc.), making it difficult to achieve true personalized replication. 2. Second type of method: a) Style feature entanglement: The features extracted by the pre-trained model often contain irrelevant information such as semantics and background, making it difficult to decouple the pure "action style" and resulting in unstable generation results. a) Lack of specific quantitative indicators: This type of method usually treats style as a black box vector and lacks explicit definition and quantification of the essence of style (such as the speed of movement, the size of the range of motion). This makes it difficult for the model to quickly and accurately lock the movement patterns of the target person from a single short video (One-shot), resulting in generated actions that are semantically related but not similar to the target person in terms of "expression" and "body posture".
[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention
[0005] The main purpose of this application is to provide a method, system, terminal and medium for quantifying digital human gesture styles based on kinematics and spatial distribution decomposition. It aims to solve the problems in the existing technology of generating digital human gestures based on labels or pre-trained models. These problems are limited by the label system, which cannot understand and generate the style of new characters outside the training set, or the features extracted by the pre-trained model are mixed with multiple factors, making it difficult to separate pure and transferable action style components, resulting in unstable generation and poor effect of stylized action generation.
[0006] The first aspect of this application provides a method for quantifying the style of digital human gestures based on kinematics and spatial distribution decomposition. The method includes the following steps:
[0007] Acquire multiple action sequences of digital human gestures and corresponding reference speech data, and generate objective labels based on the multiple action sequences;
[0008] A style feature extractor is constructed based on multiple action sequences and objective labels, and the multiple action sequences are input into the style feature extractor to output a reference style vector.
[0009] Construct an action generation model, and train the action generation model based on multiple action sequences, the reference style vector, and all the reference speech data to obtain a trained action generation model;
[0010] The system acquires video and audio data of the target person, inputs the video data into the style feature extractor to obtain the target style vector, and inputs the target style vector and the audio data into the trained motion generation model to output digital human gesture actions.
[0011] Optionally, in one embodiment of this application, the objective label includes a speed label and an amplitude label;
[0012] The generation of objective labels based on the multiple action sequences specifically includes:
[0013] Calculate kinematic quantitative indicators based on the described action sequences;
[0014] Based on the aforementioned kinematic quantification indicators, velocity and amplitude labels are generated.
[0015] Optionally, in one embodiment of this application, the action sequence includes multiple hand keypoint coordinates corresponding to each frame of the action sequence, and the kinematic quantization index includes a global velocity scalar value and a global amplitude scalar value;
[0016] The calculation of kinematic quantification indicators based on the multiple action sequences specifically includes:
[0017] Based on the coordinates of multiple hand key points corresponding to each frame of the action sequence, the displacement between adjacent frames of each joint is calculated, the average absolute value of the displacement of the corresponding joint in the action sequence is calculated based on the displacement, and the global velocity scalar value is obtained based on the average absolute value of the displacement of all joints.
[0018] Based on the coordinates of multiple hand key points corresponding to each frame of the action sequence, the spatial center point of the hand in each frame of the action sequence is calculated, the distance from each joint to the spatial center point of the hand is calculated, and a global amplitude scalar value is obtained based on the distance.
[0019] Optionally, in one embodiment of this application, the construction of a style feature extractor based on the action sequence and the objective labels specifically includes:
[0020] Construct a backbone network based on RVQ-VAE;
[0021] The multiple action sequences are input into the backbone network to obtain quantized latent vectors;
[0022] Based on the latent vector, the velocity label, and the amplitude label, a multi-task loss is calculated, and the backbone network is updated based on the multi-task loss to obtain a style feature extractor.
[0023] Optionally, in one embodiment of this application, training the action generation model based on multiple action sequences, the reference style vector, and all the reference speech data to obtain a trained action generation model specifically includes:
[0024] Calculate spatial distribution statistics based on the multiple action sequences described above;
[0025] The action generation model is jointly optimized based on multiple action sequences, spatial distribution statistics, reference style vectors, and all reference speech data to obtain the trained action generation model.
[0026] Optionally, in one embodiment of this application, the spatial distribution statistics include the spatial distribution mean and the spatial distribution covariance matrix;
[0027] The calculation of spatial distribution statistics based on the multiple action sequences specifically includes:
[0028] The coordinates of multiple hand key points corresponding to each frame of the action sequence are used as a three-dimensional point cloud.
[0029] Calculate the spatial distribution mean and spatial distribution covariance matrix corresponding to the three-dimensional point cloud.
[0030] Optionally, in one embodiment of this application, the reference speech data includes reference audio features and reference text features;
[0031] The step of jointly optimizing the action generation model based on multiple action sequences, spatial distribution statistics, reference style vectors, and all reference speech data to obtain a trained action generation model specifically includes:
[0032] The predicted noise is obtained based on the action sequence, the reference audio features, the reference text features, and the reference style vector;
[0033] The basic generation loss is calculated based on the predicted noise, and the spatial distribution constraint loss is calculated based on the spatial distribution mean and the spatial distribution covariance matrix.
[0034] The trained action generation model is obtained by jointly optimizing the basic generation loss and the spatial distribution constraint loss.
[0035] A second aspect of this application also provides a digital human gesture style quantification system based on kinematics and spatial distribution decomposition, wherein the kinematics and spatial distribution decomposition-based digital human gesture style quantification system is used to implement the kinematics and spatial distribution decomposition-based digital human gesture style quantification method described in any of the above solutions; the kinematics and spatial distribution decomposition-based digital human gesture style quantification system includes:
[0036] The training set acquisition and label generation module is used to acquire multiple action sequences of digital human gestures and corresponding reference speech data, and generate objective labels based on the multiple action sequences.
[0037] An extractor construction module is used to construct a style feature extractor based on multiple action sequences and the objective labels, and input the multiple action sequences into the style feature extractor to output a reference style vector;
[0038] The generator construction and training module is used to construct an action generation model and train the action generation model based on multiple action sequences, the reference style vector, and all the reference speech data to obtain the trained action generation model.
[0039] The gesture action output module is used to acquire video data and voice data of the target person, input the video data into the style feature extractor to obtain the target style vector, input the target style vector and the voice data into the trained action generation model, and output digital human gesture actions.
[0040] A third aspect of this application also provides a terminal, wherein the terminal includes: a memory, a processor, and a digital human gesture style quantization program based on kinematics and spatial distribution decomposition stored in the memory and executable on the processor, wherein when the digital human gesture style quantization program based on kinematics and spatial distribution decomposition is executed by the processor, it implements the steps of the digital human gesture style quantization method based on kinematics and spatial distribution decomposition as described above.
[0041] A fourth aspect of this application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a digital human gesture style quantization program based on kinematics and spatial distribution decomposition, and the digital human gesture style quantization program based on kinematics and spatial distribution decomposition, when executed by a processor, implements the steps of the digital human gesture style quantization method based on kinematics and spatial distribution decomposition as described above.
[0042] Beneficial effects: This application provides a method, system, terminal, and medium for quantifying the style of digital human gestures based on kinematic and spatial distribution decomposition. By constructing a style extractor and loss function based on two quantification indicators, kinematic style and spatial style, this invention can extract objective style features from a single reference video, guide the generation model to produce gestures that conform to the rhythm of speech and have the unique style of the target person, and enable the objective quantification and accurate replication of personalized styles in digital human action generation, thereby improving the generation effect of stylized digital human actions. Attached Figure Description
[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1 This is a flowchart of a preferred embodiment of the digital human gesture style quantification method based on kinematics and spatial distribution decomposition in this application;
[0045] Figure 2 This is a schematic diagram illustrating the training of the style extractor in a preferred embodiment of the digital human gesture style quantification method based on kinematics and spatial distribution decomposition in this application.
[0046] Figure 3 This is a schematic diagram illustrating the training of a conditional action generation model in a preferred embodiment of the digital human gesture style quantification method based on kinematics and spatial distribution decomposition in this application.
[0047] Figure 4 This is a structural diagram of a preferred embodiment of the digital human gesture style quantification system based on kinematics and spatial distribution decomposition of this application;
[0048] Figure 5 This is a structural diagram of a preferred embodiment of the terminal of this application.
[0049] Explanation of reference numerals in the attached figures:
[0050] 100. Training set acquisition and label generation module; 200. Extractor construction module; 300. Generator construction and training module; 400. Gesture and action output module. Detailed Implementation
[0051] To make the objectives, technical solutions, and effects of this application clearer and more explicit, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only possible technical implementations of this application and not all possible implementations. Based on the embodiments in this application, those skilled in the art can obtain other embodiments without creative effort, and these embodiments are also within the protection scope of this application.
[0052] To address the problem that existing technologies cannot transform subjective, abstract "style" into objective, calculable "mathematical features," thus failing to objectively describe personalized style, this invention provides a digital human gesture style quantification method based on kinematic and spatial distribution decomposition. This method orthogonally decomposes abstract style into two dimensions: 1. Kinematic Style: quantifies "how to move," including the speed and amplitude of gestures. 2. Spatial Style: quantifies "where to move," including the spatial distribution and range of gesture habits.
[0053] This invention, by constructing a style extractor and loss function based on the aforementioned quantitative indicators, can extract objective style features from a single reference video, guiding the generative model to generate gestures that both conform to the speech rhythm and possess the unique style of the target person. This invention can automatically and objectively extract the unique gesture style of a target person from a single reference video, and guide the model to generate personalized movements that not only conform to the speech content but also highly replicate the person's "expression" and "body posture."
[0054] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0055] The preferred embodiment of this application describes a digital human gesture style quantification method based on kinematics and spatial distribution decomposition, such as... Figure 1 As shown, the digital human gesture style quantification method based on kinematics and spatial distribution decomposition includes the following steps:
[0056] In step S101, multiple action sequences of digital human gestures and corresponding reference speech data are obtained, and objective labels are generated based on the multiple action sequences.
[0057] It is worth noting that this invention decouples abstract style into computable objective indicators, extracts the style features of the target person from a single reference video, and guides the generation model to synthesize a digital human action sequence that is highly consistent with the style of the reference video.
[0058] In one possible implementation, the objective labels include speed labels and amplitude labels.
[0059] Step S101 specifically includes: calculating kinematic quantification indicators based on the multiple action sequences; and generating velocity and amplitude labels based on the kinematic quantification indicators.
[0060] In one possible implementation, the action sequence includes multiple hand keypoint coordinates corresponding to each frame of the action sequence, and the kinematic quantization index includes a global velocity scalar value and a global amplitude scalar value.
[0061] The specific implementation of the step of calculating the kinematic quantization index is as follows: based on the coordinates of the multiple hand key points corresponding to each frame of the action sequence, calculate the displacement between adjacent frames of each joint, calculate the average absolute value of the displacement of the corresponding joint in the action sequence based on the displacement, and obtain the global velocity scalar value based on the average absolute value of the displacement of all joints; based on the coordinates of the multiple hand key points corresponding to each frame of the action sequence, calculate the hand space center point of each frame of the action sequence, calculate the distance from each joint to the hand space center point, and obtain the global amplitude scalar value based on the distance.
[0062] Specifically, step S101 involves objectively quantifying and defining style and labeling data. To enable computers to understand and learn "style," this invention first constructs an objective style quantification system and preprocesses and labels the training dataset accordingly. Two core style dimensions are defined: Dimension A: Kinematic Style – quantifying "how to move." Speed: calculates the average displacement of hand joints per unit time, representing the speed of the movement; Amplitude: calculates the degree of deviation of hand joints relative to the center of motion, representing the tension of the movement. Dimension B: Spatial Style – quantifying "where to move." Spatial Distribution: by calculating the mean and covariance matrix of hand position points, it represents the habitual activity area of the gesture (e.g., chest, sides) and the shape range of the activity space (e.g., flat, spherical).
[0063] The specific steps for implementing dataset labeling are as follows: For each action segment in the large-scale action dataset, the program automatically calculates the specific values of "velocity" and "amplitude". Based on the distribution of values throughout the dataset, the continuous values are discretized into specific style labels.
[0064] This invention transforms subjective style descriptions into computer-recognizable objective labels (Ground Truth), providing a solid data foundation for the supervised training of subsequent style extractors.
[0065] Furthermore, step S101 specifically includes:
[0066] Step S11: Input the 3D hand joint motion sequence from a large-scale human motion dataset. Each frame of data contains the coordinates of multiple hand key points (e.g., 21 joint points); perform necessary smoothing filtering on the coordinate sequence to eliminate the high-frequency noise inherent in motion capture and ensure the stability of subsequent calculations.
[0067] Step S12: Calculate the core quantitative indicators. This can be understood as being done independently for each action segment (e.g., a 10-second video clip of speaking).
[0068] A. Kinematic Style Calculation. 1. Velocity Calculation: For a sequence of motion with T frames, calculate the average velocity of all hand joints across all time steps. Specifically: For each joint, calculate the displacement between its adjacent frames, and calculate the average absolute value of the joint's displacement throughout the entire sequence; average the average displacement of all joints again to obtain a global velocity scalar value representing the rhythm of the gesture throughout the segment. The larger the global velocity scalar value, the faster the average hand movement when speaking, and the more "rapid" or "active" the style. 2. Amplitude Calculation: Calculate the degree of deviation of the hand joint position relative to its own center of motion. Specifically: Calculate a hand spatial center point for each frame (e.g., the geometric mean of all hand joint positions); calculate the distance from each hand joint to the center point of each frame throughout the sequence; calculate the standard deviation (or mean absolute deviation) of all these distances over the entire sequence and time to obtain a global amplitude scalar value. The larger the global amplitude scalar value, the greater the range of motion of the gesture away from the body's center, and the more "exaggerated" or "tensiony" the style; conversely, the smaller the value, the more "convergent" or "reserved."
[0069] B. Spatial Style Calculation. The coordinates of all hand joints (across all frames) in the entire sequence are treated as a point cloud, and statistical analysis is performed on it. Specifically: 1. Spatial Distribution Mean: The geometric center of this point cloud in 3D space is calculated. This vector directly indicates the habitual activity area of the person's gestures. For example: high vertical value - gestures are habitually performed at a higher position (e.g., above the shoulder); positive and large horizontal value - gestures are habitually performed to the right side of the body (for right-handed individuals). 2. Spatial Distribution Covariance Matrix: The 3x3 covariance matrix of this point cloud is calculated. This matrix describes the range, direction, and shape of the gesture activity; the magnitude of the matrix's eigenvalues indicates the degree of extension (range of activity) in the three principal axes; the direction of the matrix's eigenvectors indicates the main orientation of the activity space (e.g., more forward / backward swinging or more left / right swaying); a flat covariance matrix (one large eigenvalue and two small eigenvalues) means the gesture activity is on a plane (e.g., mainly swaying horizontally in front of the chest); a spherical covariance matrix (three similar eigenvalues) means the range of gesture activity is relatively balanced in all directions.
[0070] Step S13, Automated Label Generation: The calculated continuous values (global velocity scalar value and global amplitude scalar value) are converted into discrete class labels for subsequent supervised learning by the classifier. Specifically, the distribution of velocity values and amplitude values are statistically analyzed across all segments of the entire dataset. Quantile segmentation (e.g., ternary or quintile) is used to determine the thresholds. For example, the velocity distribution is divided into three equal parts: if the velocity value < 33% quantile - label "low speed"; 33% ≤ velocity value ≤ 66% quantile - label "medium speed"; velocity value > 66% quantile - label "high speed". The same operation is performed on amplitude A to generate labels such as Amplitude_Small, Medium, Large, etc. It is understandable that spatial style statistics are usually used directly as continuous conditions in the subsequent loss function without discretization.
[0071] In step S102, a style feature extractor is constructed based on the multiple action sequences and the objective labels, and the multiple action sequences are input into the style feature extractor to output a reference style vector.
[0072] Step S102 specifically includes: constructing a backbone network based on RVQ-VAE; inputting multiple action sequences into the backbone network to obtain quantized latent vectors; calculating multi-task loss based on the latent vectors, the velocity labels, and the amplitude labels, and updating the backbone network based on the multi-task loss to obtain a style feature extractor.
[0073] In this embodiment of the application, step S102 involves training the style extractor. For example... Figure 2 As shown, this step aims to train a neural network model that can extract decoupled style feature vectors from any input video.
[0074] The model architecture uses a backbone network based on RVQ-VAE (Residual Vector Quantization Variational Autoencoder), which includes an encoder and a decoder to map action sequences to the latent space.
[0075] Multi-task learning training strategy: During training, in addition to completing the basic action reconstruction task, the model also includes a style classification task. Two classifier branches are introduced into the model's latent layer, using the "velocity label" and "amplitude label" generated in step one as supervision signals, forcing the model to predict the velocity and amplitude levels of the input video. Simultaneously, a triplet loss is introduced: constructing sample pairs with "same velocity but different amplitude" or "same amplitude but different velocity" for comparative learning forces the model to completely separate velocity features from amplitude features in the feature space. After training, the encoder part of this extractor can serve as a general-purpose style analysis tool.
[0076] Specifically, step S102 includes:
[0077] Step S21: Input the action sequence (raw, unprocessed spatiotemporal coordinates of hand joints, velocity labels, and amplitude labels, providing the model with a "set of exercises" (action sequence) and "standard answers" (labels) for learning.
[0078] Step S22, Model Forward Propagation: Encoding and Multi-Task Inference. This is the core computational flow of a single training iteration. Specifically, it includes: Step S221, Feature Encoding: Inputting the action sequence into the RVQ-VAE encoder, outputting a continuous latent vector, which is a preliminary compressed representation of the input action. Step S222, Vector Quantization: Passing the above continuous latent vector through the RVQ quantizer, outputting a quantized latent vector. Discretizing the continuous features forces the model to learn a more representative and robust "style prototype," avoiding an overly smooth feature space and helping to capture the style's category attributes. Step S223, Multi-Task Branch Parallel Computation: The quantized vector simultaneously flows to three downstream tasks: Branch A: Action Reconstruction, quantized vector - decoder - reconstructed action sequence, ensuring the encoder doesn't "slack off" and retains enough original action information. Branch B: Style Classification, quantized vector - velocity classifier - predicted velocity level; quantized vector - amplitude classifier - predicted amplitude level. Directly using the "standard answer" (label) from Step S101 for supervision, forcing the quantized vector to contain clear, classifiable velocity and amplitude information. Branch C: Feature Decoupling (Triple Comparison). In a training batch, triples (Anchor, Positive, Negative) are automatically constructed based on velocity / amplitude labels. The quantized vectors of these three elements are then fed into the triple loss function, aiming to bring the Anchor and Positive closer together and widen the distance between the Anchor and Negative.
[0079] Step S23, Loss Calculation and Backpropagation: Multi-objective optimization. Specifically, this includes: Step S231: Calculating multi-task losses: Reconstruction loss: Comparing the difference between the reconstructed action and the original action; Classification loss: Calculating the cross-entropy loss between the predicted velocity / amplitude values and the true labels; Triple loss: Calculating the contrastive loss based on the constructed triples. Step S232: Aggregating losses and updating the model: Calculating the gradient of the total loss with respect to all model parameters (encoder, quantizer, decoder, classifier), and updating these parameters through backpropagation.
[0080] Step S24: Iterative Training and Model Convergence. Repeat steps S22 and S23 until the model performance stabilizes. The convergence state occurs when the model has learned to generate a special quantization vector that allows: the original action to be reconstructed in the decoder (satisfying reconstruction loss); the classifier to easily determine its velocity and amplitude (satisfying classification loss); and velocity and amplitude information encoded in nearly orthogonal directions in the vector space (satisfying triplet loss). Operation: After training, discard the decoder and two classifiers used only for training, retaining the encoder and RVQ quantizer. This is the trained style extractor; inputting any new action sequence will output a decoupled style feature vector.
[0081] In step S103, an action generation model is constructed, and the action generation model is trained based on multiple action sequences, the reference style vector, and all the reference speech data to obtain a trained action generation model.
[0082] Step S103 specifically includes: calculating spatial distribution statistics based on multiple action sequences; jointly optimizing the action generation model based on the multiple action sequences, the spatial distribution statistics, the reference style vector, and all the reference speech data to obtain the trained action generation model.
[0083] In one possible implementation, the spatial distribution statistics include the spatial distribution mean and the spatial distribution covariance matrix.
[0084] The specific implementation of the step of calculating spatial distribution statistics is as follows: taking the coordinates of multiple hand key points corresponding to each frame of the action sequence as a three-dimensional point cloud; calculating the spatial distribution mean and spatial distribution covariance matrix corresponding to the three-dimensional point cloud.
[0085] In one possible implementation, the reference speech data includes reference audio features and reference text features.
[0086] The specific implementation of the step of obtaining the trained action generation model is as follows: obtaining prediction noise based on the action sequence, the reference audio features, the reference text features, and the reference style vector; calculating the basic generation loss based on the prediction noise, and calculating the spatial distribution constraint loss based on the spatial distribution mean and the spatial distribution covariance matrix; and performing joint optimization based on the basic generation loss and the spatial distribution constraint loss to obtain the trained action generation model.
[0087] In this embodiment of the application, step S103 involves training a conditional action generation model. For example... Figure 3 As shown, this step trains the final action generator, enabling it to generate actions with a specific style based on speech content.
[0088] Model Input and Architecture: A diffusion model or a stream matching model is used as the generation base. Conditional Input: The model receives three parts of input: audio features (speech content); text features (semantic content); and style vectors (Style Code), which are extracted from the user-provided reference video by the extractor trained in step S102.
[0089] Introducing Spatial Constraints: While style vectors control the speed and amplitude of movements, this invention introduces a spatial distribution loss function during generator training to replicate the unique spatial habits (i.e., "spatial style") of the target character. The mean and covariance matrices of the generated action sequences are calculated and forced to conform to the statistical distribution of the reference video.
[0090] Specifically, step S103 includes:
[0091] Step S31: Input data includes: First, a speech-action pairing dataset, i.e., a large number of paired (audio, transcribed text, action sequence) data. Second, style reference: For each (audio, action) pair in the dataset, its style conditions need to be provided, which have two sources: Style vector: extracted from the action sequence itself or from other videos of the same speaker using the style extractor trained in S102; Spatial statistics: calculated according to the formula in S101, using the mean and covariance matrix of the spatial distribution of the action sequence.
[0092] Step S32: Build the generator architecture, using a diffusion model or a flow matching model as the generation foundation. These models generate high-quality samples through a gradual denoising process "from noise to data". The core of the generator (such as U-Net) is designed to handle multiple input conditions simultaneously, incorporating conditional information into each step of the denoising prediction through cross-attention layers or feature concatenation.
[0093] Step S33, Training Process: Generation and Constraints Under Multiple Conditions, aiming to teach the generator to create under multiple constraints. Specifically, this includes:
[0094] Step S331: For each training sample, the model needs to receive an instruction package containing three types of information: audio features: extracted from the original speech waveform, providing temporal information such as rhythm, stress, and intonation; text features: extracted from the speech-transcribed text, providing semantic and lexical information; style vector: extracted from the action of the current sample itself or other videos of the same speaker using a trained style extractor, providing kinematic style instructions on "how to move".
[0095] Step S332: Conditional Injection within the Diffusion Generation Framework. Taking the diffusion model as an example, its training process is a "destruction-reconstruction" game: Forward Destruction Process: For a real action sequence, noise is gradually added to obtain a series of noisy versions. Conditional Reconstruction Process (Core Training Objective): In each step of denoising prediction, the model not only needs to consider the current noise state but also simultaneously refer to the instruction packet, which means that the model can determine what action should be recovered under the current noise state based on all conditional information.
[0096] Step S333: Calculate the basic generation loss. Loss function: Basic generation loss = (True noise - Predicted noise)². Ensure the model can accurately reproduce actions that meet all conditions.
[0097] Step S334: Calculate the spatial distribution constraint loss. In addition to generating the action, a hard constraint is imposed on the geometric properties of the action space. Specifically, this includes: Step S3341: For the current training sample, directly calculate the spatial distribution mean and covariance matrix of its true action sequence. Step S3342: During training, after the model predicts noise based on conditions, it can derive the clean action predicted in the current step. Calculate the spatial distribution mean and covariance matrix of the clean action. Step S3343: Calculate the spatial alignment loss, forcing the activity center of the generated action to be consistent with the target, and forcing the shape of the activity range of the generated action to be consistent with the target.
[0098] Step S335, Joint Optimization and Backpropagation. Calculate the total loss. Backpropagation calculates the gradient of the total loss with respect to all generator parameters and updates the model. The model is forced to simultaneously optimize two objectives: accurately predict noise to reconstruct actions that meet the conditional instructions; and ensure that the spatial statistics of the generated actions are consistent with the reference.
[0099] Step S34: Iterative training and model convergence. Repeat step S33 until convergence. Convergence state: The generator has learned the internal mapping rules: given (audio, text, style vector), it can generate an action that matches the speech content naturally, is consistent with the style vector in terms of movement rhythm / amplitude, and is aligned with the spatial distribution mean and spatial distribution covariance matrix in terms of spatial location.
[0100] In step S104, video data and voice data of the target person are acquired, and the video data is input into the style feature extractor to obtain the target style vector. The target style vector and the voice data are input into the trained motion generation model to output digital human gesture actions.
[0101] In this invention, the trained generator is capable of accepting a new audio input and generating a new sequence of movements that is highly consistent with the style of the reference video (containing the same speed / amplitude rhythm and moving within the same spatial area) based on a reference video (One-shot) provided by the user.
[0102] Specifically, after training, the inputs are: a new target audio clip and its transcribed text; and a reference video of the target person. Processing: The reference video is processed using the style extractor from step S102 to obtain a style vector; the spatial statistics of the reference video are calculated using the formula from step S101, and the (new audio features, new text features, style vector) are input into the trained generator. The output is a completely new action sequence synchronized with the input audio and with a style highly consistent with the person in the reference video.
[0103] This invention employs a two-dimensional orthogonal style quantification system, abandoning the ambiguous style labels (such as ID or sentiment) in existing technologies. It innovatively constructs a set of quantification standards, decomposing abstract personal style into two orthogonal dimensions: kinematic style and spatial style. A decoupling extraction strategy based on procedural labeling is used, employing an explicit style-supervised training method. Unlike implicit learning, this scheme automatically calculates the velocity and amplitude values of the data as objective labels (Ground Truth) before training. Simultaneously, a triplet loss is combined to construct comparative samples, forcing the model to completely decouple velocity and amplitude features in the feature space, achieving precise style decoupling. This invention employs a spatial constraint generation mechanism based on statistical distribution. It introduces spatial distribution loss in the action generation stage. Unlike the traditional frame-by-frame position alignment (MSE, Mean Squared Error), this mechanism constrains the mean position and spatial covariance matrix of the generated action to be consistent with the reference video. This ensures that the generated digital human action maintains temporal freedom (adapting to new speech) while strictly replicating the spatial occupancy habits of the target person.
[0104] Next, referring to the accompanying drawings, a digital human gesture style quantization system based on kinematics and spatial distribution decomposition proposed according to an embodiment of this application is described, which is used to implement the digital human gesture style quantization method based on kinematics and spatial distribution decomposition as described in any of the above schemes.
[0105] Figure 4 This is a structural diagram of a digital human gesture style quantification system based on kinematics and spatial distribution decomposition, according to an embodiment of this application.
[0106] like Figure 4 As shown, the digital human gesture style quantification system based on kinematics and spatial distribution decomposition includes: a training set acquisition and label generation module 100, an extractor construction module 200, a generator construction and training module 300, and a gesture action output module 400.
[0107] Specifically, the training set acquisition and label generation module 100 is used to acquire multiple action sequences of digital human gestures and corresponding reference speech data, and generate objective labels based on the action sequences;
[0108] Extractor construction module 200 is used to construct a style feature extractor based on the action sequence and the objective label, and input the action sequence into the style feature extractor to output a reference style vector;
[0109] The generator construction and training module 300 is used to construct an action generation model and train the action generation model based on the action sequence, the reference style vector and the reference speech data to obtain the trained action generation model.
[0110] The gesture action output module 400 is used to acquire video data and voice data of the target person, input the video data into the style feature extractor to obtain the target style vector, input the target style vector and the voice data into the trained action generation model, and output digital human gesture actions.
[0111] Figure 5 A structural diagram of a terminal provided in an embodiment of this application. The terminal may include:
[0112] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.
[0113] When the processor 502 executes the program, it implements the digital human gesture style quantization method based on kinematics and spatial distribution decomposition provided in the above embodiments.
[0114] Furthermore, the terminal also includes:
[0115] Communication interface 503 is used for communication between memory 501 and processor 502.
[0116] The memory 501 is used to store computer programs that can run on the processor 502.
[0117] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0118] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EIS) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0119] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.
[0120] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.
[0121] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described digital human gesture style quantization method based on kinematics and spatial distribution decomposition.
[0122] One embodiment of this application provides a computer program product, including a computer program that, when executed by a processor, implements the features described in this application. Figure 1 The corresponding embodiments provide a digital human gesture style quantification method based on kinematics and spatial distribution decomposition.
[0123] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0124] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0125] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0126] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable storage medium could be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.
[0127] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0128] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0129] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0130] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.
[0131] It should be understood that the application of this application is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A method for quantifying the gesture style of digital humans based on kinematics and spatial distribution decomposition, characterized in that, The digital human gesture style quantification method based on kinematics and spatial distribution decomposition includes: Acquire multiple action sequences of digital human gestures and corresponding reference speech data, and generate objective labels based on the multiple action sequences; A style feature extractor is constructed based on multiple action sequences and objective labels, and the multiple action sequences are input into the style feature extractor to output a reference style vector. Construct an action generation model, and train the action generation model based on multiple action sequences, the reference style vector, and all the reference speech data to obtain a trained action generation model; The system acquires video and audio data of the target person, inputs the video data into the style feature extractor to obtain the target style vector, and inputs the target style vector and the audio data into the trained motion generation model to output digital human gesture actions. The objective labels include speed labels and amplitude labels; The generation of objective labels based on the multiple action sequences specifically includes: Calculate kinematic quantitative indicators based on the described action sequences; Based on the aforementioned kinematic quantification indicators, generate velocity and amplitude labels; The multiple segments construct a style feature extractor based on the action sequence and the objective labels, specifically including: Construct a backbone network based on RVQ-VAE; The multiple action sequences are input into the backbone network to obtain quantized latent vectors; Based on the latent vector, the velocity label, and the amplitude label, a multi-task loss is calculated, and the backbone network is updated based on the multi-task loss to obtain a style feature extractor.
2. The digital human gesture style quantification method based on kinematics and spatial distribution decomposition according to claim 1, characterized in that, The motion sequence includes the coordinates of multiple hand key points corresponding to each frame of the motion sequence, and the kinematic quantization index includes a global velocity scalar value and a global amplitude scalar value. The calculation of kinematic quantification indicators based on the multiple action sequences specifically includes: Based on the coordinates of multiple hand key points corresponding to each frame of the action sequence, the displacement between adjacent frames of each joint is calculated, the average absolute value of the displacement of the corresponding joint in the action sequence is calculated based on the displacement, and the global velocity scalar value is obtained based on the average absolute value of the displacement of all joints. Based on the coordinates of multiple hand key points corresponding to each frame of the action sequence, the spatial center point of the hand in each frame of the action sequence is calculated, the distance from each joint to the spatial center point of the hand is calculated, and a global amplitude scalar value is obtained based on the distance.
3. The digital human gesture style quantification method based on kinematics and spatial distribution decomposition according to claim 2, characterized in that, The step of training the action generation model based on multiple action sequences, the reference style vector, and all the reference speech data to obtain the trained action generation model specifically includes: Calculate spatial distribution statistics based on the multiple action sequences described above; The action generation model is jointly optimized based on multiple action sequences, spatial distribution statistics, reference style vectors, and all reference speech data to obtain the trained action generation model.
4. The digital human gesture style quantification method based on kinematics and spatial distribution decomposition according to claim 3, characterized in that, The spatial distribution statistics include the spatial distribution mean and the spatial distribution covariance matrix; The calculation of spatial distribution statistics based on the multiple action sequences specifically includes: The coordinates of multiple hand key points corresponding to each frame of the action sequence are used as a three-dimensional point cloud. Calculate the spatial distribution mean and spatial distribution covariance matrix corresponding to the three-dimensional point cloud.
5. The digital human gesture style quantification method based on kinematics and spatial distribution decomposition according to claim 4, characterized in that, The reference speech data includes reference audio features and reference text features; The step of jointly optimizing the action generation model based on multiple action sequences, spatial distribution statistics, reference style vectors, and all reference speech data to obtain a trained action generation model specifically includes: The predicted noise is obtained based on the action sequence, the reference audio features, the reference text features, and the reference style vector; The basic generation loss is calculated based on the predicted noise, and the spatial distribution constraint loss is calculated based on the spatial distribution mean and the spatial distribution covariance matrix. The trained action generation model is obtained by jointly optimizing the basic generation loss and the spatial distribution constraint loss.
6. A digital human gesture style quantification system based on kinematics and spatial distribution decomposition, characterized in that, The kinematics and spatial distribution decomposition-based digital human gesture style quantization system is used to implement the kinematics and spatial distribution decomposition-based digital human gesture style quantization method according to any one of claims 1-5. The kinematics and spatial distribution decomposition-based digital human gesture style quantization system includes: The training set acquisition and label generation module is used to acquire multiple action sequences of digital human gestures and corresponding reference speech data, and generate objective labels based on the multiple action sequences. An extractor construction module is used to construct a style feature extractor based on multiple action sequences and the objective labels, and input the multiple action sequences into the style feature extractor to output a reference style vector; The generator construction and training module is used to construct an action generation model and train the action generation model based on multiple action sequences, the reference style vector, and all the reference speech data to obtain the trained action generation model. The gesture action output module is used to acquire video data and voice data of the target person, input the video data into the style feature extractor to obtain the target style vector, input the target style vector and the voice data into the trained action generation model, and output digital human gesture actions.
7. A terminal, characterized in that, The terminal includes: a memory, a processor, and a digital human gesture style quantization program based on kinematics and spatial distribution decomposition stored in the memory and executable on the processor. When the digital human gesture style quantization program based on kinematics and spatial distribution decomposition is executed by the processor, it implements the steps of the digital human gesture style quantization method based on kinematics and spatial distribution decomposition as described in any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a digital human gesture style quantization program based on kinematics and spatial distribution decomposition, which, when executed by a processor, implements the steps of the digital human gesture style quantization method based on kinematics and spatial distribution decomposition as described in any one of claims 1-5.