Controllable dance motion generation method and system based on standardized generation flow

Abstract

The application provides a controllable dance action generation method based on a standardized generation flow, comprising: constructing a dance action generation model based on a standardized generation flow, and training the dance action generation model using an open-source audio-dance dataset; setting a key frame of a target dance action sequence, generating a control signal from the key frame and target audio, and using the dance action generation model to implicitly reflect an original dance action sequence into original implicit variables in a Gaussian space; and using the control signal to conditionally generate the target dance action sequence from target implicit variables sampled from the Gaussian space by the dance action generation model. The application also provides a controllable dance action generation system based on a standardized generation flow, and a data processing device for realizing controllable dance action generation based on a standardized generation flow.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics technology, specifically relating to a method and system for generating controllable dance movements based on a standardized generation flow. Background Technology

[0002] Existing work utilizes network structures such as recurrent neural networks, generative adversarial networks, and variational autoencoders to encode dance movements of the human skeleton. An encoder is designed to encode the audio signal, and the resulting latent variables are used by a motion generator to generate dance movement sequences, thus realizing audio-driven movement.

[0003] For example, Taoran Tang et al. published in Proceedings of the 26th National Congress of the Communist Party of China in 2018. th The ACM International Conference on Multimedia's paper "Dance with melody: An LSTM-autoencoder approach to music-oriented dance synthesis" uses recurrent neural networks to encode dance movements. Hsin-Ying Lee et al.'s 2019 paper "Dancing to Music," published at the Conference on Neural Information Processing Systems, uses adversarial neural network structures to generate music-consistent dance movements. Joao P Ferreira et al.'s 2021 paper "Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio," published at Computers & Graphics, uses graph convolutional networks to encode dance movement information.

[0004] The main technical shortcomings of existing methods include: insufficient diversity of generated motion results, insufficient realism of generated motion results, insufficient consistency between generated motion results and musical rhythm, and lack of flexible control over generated motion. The insufficient diversity of generated results is related to the diversity of the dataset itself and the choice of network structure. Low realism of motion is manifested in small motion amplitude or large motion jitter, caused by network design and data errors. The low consistency with musical rhythm is due to different processing of the music signal resulting in different results. Existing work focuses on single-point control of the music, without considering user interaction and more comprehensive control. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a controllable dance movement generation method based on a standardized generation flow, comprising: constructing a dance movement generation model based on a standardized generation flow; training the dance movement generation model using an open-source audio-dance dataset; setting keyframes for a target dance movement sequence; generating a control signal using the keyframes and target audio; mapping the original dance movement sequence into original latent variables in Gaussian space using the dance movement generation model; and generating the target dance movement sequence conditionally using the target latent variables sampled from the Gaussian space by the dance movement generation model and the control signal.

[0006] The controllable dance movement generation method of the present invention obtains the original latent variable z by mapping the original dance movement sequence x to the Gaussian space Z based on the control signal s:

[0007]

[0008] The control signal s includes keyframe information and audio information. The keyframe information includes the motion embedding parameter e and the time embedding parameter E of the keyframe. The audio information includes the Mel-Cepstral Coefficient signal features m extracted from the target audio.

[0009] The controllable dance motion generation method of the present invention, wherein the target latent variable z′ obtained from Z is used to generate the target dance motion sequence x′ based on s:

[0010] x'=f(z'|s=f1(f2(…f K (z'|s)))

[0011] For the target dance movement x′ generated at time t t Using the target dance move x′ at time τ τ As an autoregressive signal, time τ is prior to time t.

[0012] The controllable dance motion generation method of the present invention, wherein the loss function of the dance motion generation model is L = L NF +L key_pose L NF L is the normalized generation flow loss in this dance motion generation model. key_pose For the keyframe control loss in the dance motion generation model, for the control signal s at time t... t The corresponding action embedding parameter e t ,

[0013]

[0014]

[0015] L key_pose=||FK(x′) t )-FK(e t )||2

[0016] z0 is the first original latent variable, δ K,d δ0 represents the scaling parameter of the d-dimensional data of the K-th layer network of the dance motion generation model, where D represents the dimension of x, μ0 represents the offset parameter of the first layer network of the dance motion generation model, and δ0 represents the scaling parameter of the first layer network. FK(·) represents random sampling within the neighborhood of the 0 vector, and FK(·) represents the forward kinematic function.

[0017] This invention also proposes a controllable dance motion generation system based on a standardized generation flow, comprising: a model building and training module for building a dance motion generation model based on a standardized generation flow and training the model using an open-source audio-dance dataset; a raw dance motion mapping module for setting keyframes of a target dance motion sequence, generating control signals using the keyframes and target audio, and mapping the raw dance motion sequence to raw latent variables in Gaussian space using the dance motion generation model; and a target dance motion generation module for generating the sequence using the target latent variables sampled from the Gaussian space by the dance motion generation model and the control signals.

[0018] The controllable dance motion generation system of the present invention includes an original dance motion mapping module comprising: mapping the original dance motion sequence x to a Gaussian space Z based on the control signal s to obtain the original latent variable z.

[0019]

[0020] The control signal s includes keyframe information and audio information. The keyframe information includes the motion embedding parameter e and the time embedding parameter E of the keyframe. The audio information includes the Mel-Cepstral Coefficient signal features m extracted from the target audio.

[0021] The controllable dance motion generation system of the present invention includes a target dance motion generation module comprising: obtaining the target latent variable z′ from Z sampling, and generating the target dance motion sequence x′ based on s.

[0022] x'=f(z'|s=f1(f2(…f K (z'|s)))

[0023] For the target dance movement x′ generated at time t t Using the target dance move x′ at time τ τ As an autoregressive signal, time τ is prior to time t.

[0024] The controllable dance motion generation system of the present invention, wherein in the model construction and training module, the loss function L = L of the dance motion generation model NF +L key_pose L NF L is the normalized generation flow loss in this dance motion generation model. key_pose For the keyframe control loss in the dance motion generation model, for the control signal s at time t... t The corresponding action embedding parameter e t ,

[0025]

[0026]

[0027] L key_pose =||FK(x′) t )-FK(e t )||2

[0028] z0 is the first original latent variable, δ K,d δ0 represents the scaling parameter of the d-dimensional data of the K-th layer network of the dance motion generation model, where D represents the dimension of x, μ0 represents the offset parameter of the first layer network of the dance motion generation model, and δ0 represents the scaling parameter of the first layer network. FK(·) represents random sampling within the neighborhood of the 0 vector, and FK(·) represents the forward kinematic function.

[0029] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that, when the computer-executable instructions are executed, the controllable dance motion generation method based on standardized generation flow as described above is implemented.

[0030] The present invention also proposes a data processing apparatus, including the computer-readable storage medium as described above, wherein when the processor of the data processing apparatus retrieves and executes computer-executable instructions in the computer-readable storage medium, it performs controllable dance motion generation based on a standardized generation stream. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the controllable dance motion generation system of the present invention.

[0032] Figure 2 This is a schematic diagram of the standardized flow model structure of the present invention.

[0033] Figure 3 This is a schematic diagram illustrating a user usage example of an embodiment of the present invention.

[0034] Figure 4A , 4BThis is a schematic diagram of the diversity generation results of the present invention.

[0035] Figure 5 This is a comparison diagram of the controllable dance motion generation method of the present invention and the effects of existing technologies.

[0036] Figure 6 This is a schematic diagram of the data processing device of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0038] While conducting research on skeleton motion generation, the inventors discovered that the insufficient diversity and low realism of generated results in existing technologies were due to network structure design. Existing methods often employ recurrent neural networks, graph convolutions, and generative adversarial networks, failing to consider the complexity of the motion data itself. The standardized generation flow proposed in this solution uses a 16-layer flow structure to map complex motion data to a Gaussian space, effectively learning the data distribution. The inverse mapping reduces independent generator training steps, improving the realism of the generated results. Due to the randomness of latent space sampling, the generated results also exhibit diversity.

[0039] While researching dance movement control, the inventors discovered a lack of flexible control modules in existing technologies. Most current work only considers the influence of audio rhythm on dance movements. This invention, combined with actual choreography processes, proposes a keyframe control scheme. Users can select joint frames and specify the timing of keyframe movements; this scheme can automatically generate reasonable and diverse transitions that conform to the audio rhythm.

[0040] To achieve accurate keyframe control, the inventors proposed an audio and keyframe information (keyframe action and keyframe time) controller, which serves as a mapping condition for the standardized generation stream during training and controls the latent variables to generate the corresponding action results during the generation phase.

[0041] The controllable dance motion generation method based on standardized generation flow of the present invention includes:

[0042] 1. Dance motion generation based on standardized generation streams; the core idea of ​​standardized generation streams is to map complex data distributions to a simple standard Gaussian distribution through a set of continuous invertible transformations, thereby learning the probability distribution of complex samples. The inventors applied standardized generation streams to dance motion generation, achieving diverse and realistic dance motion generation.

[0043] 2. Flexible dance generation control; Technical effect: The keyframe controller of this invention includes audio information and keyframe information. User input of control conditions can generate high-quality dance sequences that meet the control requirements. Inputting a piece of music can generate dance movements that match the rhythm of the music. Inputting multiple keyframes and specifying their occurrence times can generate natural and realistic dance sequences that complete the user-specified movements;

[0044] 3. Training constraints for standardized generated streams; Technical effect: During training, the inverse mapping of standardized generated streams is used to constrain the keyframes of the generated segments to be consistent with the keyframes in the control signal;

[0045] Figure 1 This is a schematic diagram of the controllable dance motion generation system of the present invention. Figure 1 As shown, the controllable dance motion generation system of the present invention, by inputting keyframe information and music signals, constrains the dance motions to be mapped to Gaussian space, and randomly samples in Gaussian space to generate new dance motion sequences.

[0046] Specifically, the network structure of the controllable dance motion generation system of the present invention includes:

[0047] 1. Standardized Generation Flow

[0048] The dance movement sequence x, under the control signal s, is mapped to the original latent variable z in the Gaussian space Z via a normalizing flow:

[0049]

[0050] The control signal s consists of keyframe information and audio information. The audio signal is extracted using Mel-frequency cepstral coefficients (MFCC) signal features m, and the keyframe information consists of keyframe action embedding e and keyframe time embedding E.

[0051] During the generation phase, the target latent variable z′ obtained from latent space sampling is used to generate the target motion sequence x′ under control signal conditions:

[0052] x'=f(z'|s=f1(f2(…f K (z'|s)))

[0053] Action x at time t during encoding and generation t In order to maintain the continuity of the action, this invention uses the action at time τ before time t as an autoregressive signal to make the generated action at time t smoother.

[0054] Figure 2The normalized flow model is shown in detail. The normalized flow structure consists of 16 layers of flow structures, each with a consistent structure and all capable of invertible transformations. In each layer of the normalizing flow structure, x... t First, divide it into two parts: h hi and h lo h hi Combined with the control signal, encoded by LSTM, and then with h lo Couple the input to the next layer.

[0055] 2. Keyframe Controller

[0056] Keyframe information consists of keyframe action embedding *e* and keyframe temporal embedding *E*. The keyframe action embedding *e* is represented by the root node position, relative rotation of each joint, orientation of the root node, and velocity. For the keyframe temporal embedding *E*, to represent time continuously, this scheme uses position encoding of the Transformer result to process the temporal information.

[0057]

[0058]

[0059] Here, nt represents the time interval from the initial frame to the target keyframe, which can be timed forward or backward. Experiments have shown that sampling with backward timing generates better results. D represents the dimension of the input action, l varies between 0 and D / 2, and basis is set to 10000, which is the recommended setting for the Transformer architecture. The purpose of encoding here is to make the temporal embedding more smooth. l covers all dimensions of the data from 0 to D / 2, and smooth embedding is performed on each dimension. basis is a basic parameter of positional encoding used to calculate the encoding result, and its value is determined empirically.

[0060] The training process of the controllable dance movement generation system of the present invention includes:

[0061] 1. Data Processing

[0062] This invention uses the open-source audio-dance dataset AIST++ for network training, validation, and testing. The dataset contains 1408 music tracks and their corresponding dances, across ten style categories. This method divides the motion sequences into 40-frame intervals, resulting in 14645 motion segments. To construct a suitable test set, this method selects one style from the ten styles, randomly selects two dancers for each style, and then randomly selects two dance segments for each dancer. This results in 40 different segments for testing, with each segment containing 320 frames.

[0063] 2. Loss Function

[0064] The training process of this invention mainly involves two loss functions: one is the loss function for standardizing the generated stream, and the other is the loss function for keyframe control.

[0065] (1) Standardized generation flow loss

[0066] Given an observable variable x and a latent variable z, p(x, z) represents their joint probability distribution. A generative model learns a parameterized model of this distribution. Given a dataset X = {x1, x2, ..., xn}, its maximum likelihood function is:

[0067]

[0068] The inference model is q(z|x), and the lower bound of the logical likelihood can be obtained:

[0069] logp(x) ≥ logp(x) - D KL (q(z|x)||p(z|x))=L(x;θ)

[0070] Where D KL (q(z|x)||p(z|x)) is the KL divergence. By minimizing the KL divergence, L(x;θ) approaches logp(x).

[0071] To improve the speed and accuracy of the calculation, it is necessary to find a suitable method to calculate the posterior probability distribution q(z|x) so that it is consistent with the prior distribution.

[0072] Based on the above introduction to the standardized generation flow, without considering the control signal, the Kth original hidden variable z K :

[0073] z K =f1(f2(...f K (z0, x)))

[0074] Where z0 represents the first primitive latent variable, z0 = q(z0|x);

[0075] Therefore, z K The logical likelihood can be expressed as:

[0076]

[0077] To reduce computational load, z K With z K-1 The relationship is represented as follows:

[0078] z K =μ K +δK⊙z K-1

[0079] μ K =(1-δK )⊙a K

[0080] δ K =sigmoid(b K )

[0081] Where sigmoid is the activation function, a K b K It is obtained from the LSTM module in each layer of the flow structure. can be Calculation. Considering the control signal s, the normalized generated flow loss is:

[0082]

[0083]

[0084] δ K，d Let represent the scaling parameter (scale) of the Kth layer network, d represent the dimension of the input action (there are D dimensions in total), μ0 represent the bias parameter (bias) of the first layer network, δ0 represent the scaling parameter (scale) of the first layer network, and FK(·) represent the forward kinematics, which can be based on the action parameters at a certain time (e.g., x). t The coordinates of the action at the current moment are calculated and used for subsequent loss function calculation. t ' represents the action generated at time t. This represents random sampling within the neighborhood of vector 0, and N(0, I) indicates that the sampling follows a normal distribution.

[0085] (2) Keyframe control loss

[0086] Given keyframe motion information, embed e t Given the control signal s at this time t It can generate a new action sequence x t '.in:

[0087] x' t =f(z'|s t )=f1(f2(…f K (z'|s t )))

[0088] According to x′ t With e t The distance constraint generates keyframe motion consistency. Since Euclidean distance based on rotation cannot accurately represent the proximity of motions, this invention applies forward kinematics—calculating the current keypoint positions based on the rotation angle and the initial skeleton pose—to the loss calculation.

[0089] L key_pose =||FK(x′)t )-FK(e t )||2

[0090] In summary, the training loss function for the network structure in this invention is:

[0091]

[0092] The following are user examples of the controllable dance motion generation system of the present invention:

[0093] like Figure 3 As shown, this invention provides a user example. Users can input any number of keyframes and specify their target time. The input for the keyframes is 3D human skeleton information, which can be extracted from a single image. Similarly, by inputting any piece of music and extracting MFCC features, along with the keyframe information, the network can be fed with diverse and realistic dance movement sequences (the red skeleton represents the input keyframe movements, and the blue skeleton represents the generated intermediate transition movements).

[0094] Figure 3 For example, the user inputs an image containing human movements, extracts a 3D skeleton as a keyframe signal, and inputs music to obtain a complete dance movement sequence.

[0095] Figure 4A , 4B The diversity of the results generated by this invention is demonstrated. In the figure, the red skeleton represents keyframe actions, and the blue skeleton represents generated actions. Figure 4A The results shown are the different results generated under different keyframes. Figure 4B The results shown are different outcomes given the same keyframes. This demonstrates that the present invention can generate diverse and realistic motion sequences.

[0096] Figure 5 This paper demonstrates a comparison between the present invention and other existing works, which only consider music-driven dance movements. Therefore, the present invention only compares the control effect of music signals, removing the keyframe control module. The images show two results generated by the four methods under the same music. It can be seen that some existing works suffer from unrealistic motion (small variation), poor rhythm matching (line graph below the skeleton diagram, blue dashed line represents music rhythm, yellow dashed line represents movement rhythm, red circle indicates matching rhythm points), and poor diversity (little variation between the two results). This indicates that the present invention optimizes the problems existing in existing works, achieving industry-leading performance.

[0097] Figure 6 This is a schematic diagram of the data processing apparatus of the present invention. Figure 6As shown, embodiments of the present invention also provide a computer-readable storage medium and a data processing apparatus. The computer-readable storage medium of the present invention stores computer-executable instructions. When these computer-executable instructions are executed by the processor of the data processing apparatus, the aforementioned controllable dance motion generation method based on a standardized generation flow is implemented. Those skilled in the art will understand that all or part of the steps in the above method can be implemented by a program instructing related hardware (e.g., processor, FPGA, ASIC, etc.), and the program can be stored in a readable storage medium, such as a read-only memory, a disk, or an optical disk. All or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiments can be implemented in hardware, for example, by using integrated circuits to implement its corresponding function, or it can be implemented as a software functional module, for example, by a processor executing a program / instruction stored in memory to implement its corresponding function. Embodiments of the present invention are not limited to any particular combination of hardware and software.

[0098] This invention presents a multimodal 3D human dance control generation method based on standardized generation flow. It addresses the problems of low motion realism, poor matching of movements to music rhythm, and insufficient control flexibility in existing methods. This method maps complex dance movement posture data distribution to a simple Gaussian distribution through a series of invertible functions. Arbitrary sampling is performed within the Gaussian distribution, using input music features as control signals, and their inverse functions are used to generate dance sequences. To achieve flexible control of the generated results, this method proposes a keyframe-based driving approach. Users can freely input dance movement postures as keyframes for the generated results. The network encodes the position and posture signals of the keyframes to generate accurate key postures. Using the publicly available dataset AIST++, this method surpasses existing open-source work in terms of audio matching accuracy, motion realism, and diversity of generated results, achieving industry-leading performance.

[0099] The above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention, and the patent protection scope of the present invention should be defined by the claims.

Claims

1. A controllable dance movement generation method based on standardized generation flow, characterized in that, include: A dance motion generation model based on a standardized generation stream was constructed and trained using an open-source audio-dance dataset; the loss function of this dance motion generation model is L=L NF +L key_pose L NF L is the normalized generation flow loss in this dance motion generation model. key_pose For the keyframe control loss in the dance motion generation model, for the control signal at time t Corresponding action embedding parameters , , , , δ is the first original latent variable. K,d The scaling parameter for the d-dimensional data of the K-th layer network of the model that generates this dance movement, where D represents the dimension of the original dance movement sequence x. This represents the offset parameters of the first layer of the network in the dance motion generation model. This represents the scaling parameters of the first-layer network. This represents random sampling within the neighborhood of the 0 vector, and FK(·) represents the forward kinematic function; Keyframes for the target dance sequence are defined, and control signals are generated using these keyframes and the target audio. The dance sequence generation model then converts the original dance sequence into a single sequence. Based on this control signal The implicit variable is mapped to the primitive latent variable of the Gaussian space Z. , The control signal It includes keyframe information and audio information. The keyframe information includes the action embedding parameter e and the time embedding parameter E of the keyframe. The audio information includes the Mel-Cepstral Coefficient signal features m extracted from the target audio. The target latent variable is obtained by sampling from the Gaussian space Z by the dance motion generation model. Through this control signal Conditional generation of the target dance movement sequence , For the target dance movement generated at time t Using the target dance movement at time τ As an autoregressive signal, time τ is prior to time t.

2. A controllable dance motion generation system based on standardized generation flow, characterized in that, include: The model building and training module is used to construct a dance motion generation model based on a standardized generation stream. This model is trained using an open-source audio-dance dataset. The loss function of this dance motion generation model is L=L NF +L key_pose L NF L is the normalized generation flow loss in this dance motion generation model. key_pose For the keyframe control loss in the dance motion generation model, for the control signal at time t Corresponding action embedding parameters , , , , δ is the first original latent variable. K,d The scaling parameter for the d-dimensional data of the K-th layer network of the model that generates this dance movement, where D represents the dimension of the original dance movement sequence x. This represents the offset parameters of the first layer of the network in the dance motion generation model. This represents the scaling parameters of the first-layer network. This represents random sampling within the neighborhood of the 0 vector, and FK(·) represents the forward kinematic function; The original dance motion mapping module is used to set keyframes for the target dance motion sequence. Control signals are generated using these keyframes and the target audio, and the dance motion generation model then maps the original dance motion sequence... Based on this control signal The implicit variable is mapped to the primitive latent variable of the Gaussian space Z. , The control signal It includes keyframe information and audio information. The keyframe information includes the action embedding parameter e and the time embedding parameter E of the keyframe. The audio information includes the Mel-Cepstral Coefficient signal features m extracted from the target audio. The target dance motion generation module is used to sample the target latent variables obtained by the dance motion generation model from the Gaussian space Z. Through this control signal The sequence is generated under certain conditions. , For the target dance movement generated at time t Using the target dance movement at time τ As an autoregressive signal, time τ is prior to time t.

3. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, the controllable dance motion generation method based on standardized generation flow as described in claim 1 is implemented.

4. A data processing apparatus comprising the computer-readable storage medium as claimed in claim 3, wherein when a processor of the data processing apparatus retrieves and executes computer-executable instructions in the computer-readable storage medium, it generates controllable dance movements based on a standardized generation stream.

Images