Controllable music-driven three-dimensional dance motion generation model training method, generation method and device

Abstract

The invention provides a controllable music-driven three-dimensional dance motion generation model training method, a generation method and a device. The method comprises the following steps: constructing a training set containing music-three-dimensional dance motion data and two-dimensional key point data; constructing an initial generation model which comprises a backbone network and a control network; the backbone network takes music and three-dimensional dance motion data as input and outputs predicted motion data; the control network takes music, three-dimensional dance motion data and two-dimensional key point data as input and outputs motion characteristics; splicing the motion features with intermediate features extracted by the backbone network to control predicted motion data; and carrying out two-stage training on the initial generation model by adopting the training set, and finally obtaining a three-dimensional dance motion generation model. The three-dimensional dance motion generation model obtained through training can be used for generating three-dimensional dance motion data matched with music, and the controllable generation capability is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of music-driven dance action choreography methods, and in particular to a controllable music-driven three-dimensional dance action generation model training method, generation method and device. Background Art

[0002] The music-driven dance movement generation model aims to automatically generate high-quality and diverse dance movements given a piece of music. It can eventually be presented in the form of a two-dimensional video, or it can generate three-dimensional movement data and render it through tools such as post-production modeling software to obtain the visual effects people need.

[0003] This type of AI-generated model not only has broad application prospects in film and television and video games, but can also be used as a tool to assist choreographers or animators to improve their production efficiency. For example, game production may require a large number of characters and character movements, while filming of film and television works may require the production of task movements in special scenes.

[0004] According to current technology, there are generally two solutions for computer character motion production. One is to hire professionals and motion capture technology teams to use motion capture technology to collect motion data, and then process it with computers in the later stage, and finally apply it to actual scenes; the other is to hire professional animation production personnel to use relevant computer software to produce the character model's motion frame by frame. However, no matter which solution is used, it will consume a lot of manpower, material resources and financial resources. Therefore, if it can be automatically generated using artificial intelligence technology, it will save a lot of time and cost and improve production efficiency.

[0005] The diffusion model is a generative neural network model. Due to its powerful generation ability, it is also widely used in dance movement generation. Among them, the classic work of the diffusion model in three-dimensional dance movement generation comes from the EDGE model proposed by Tseng et al. It designed a model for predicting and generating dance movement sequences, which can predict and generate a three-dimensional dance movement data of the same length based on the input music features. However, this music-driven dance movement generation scheme cannot control the generated three-dimensional dance movements. Since the diffusion model generation process uses random sampling from noise, the final generated results are random, and it is difficult to control the generation of the same three-dimensional dance movements. It can be seen that the existing music-driven dance movement generation scheme lacks the ability to control the generation results and it is difficult to control the specific motion trajectory of the generated human joints. Summary of the invention

[0006] In view of this, the embodiments of the present invention provide a controllable music-driven three-dimensional dance movement generation model training method, generation method and device to eliminate or improve one or more defects existing in the prior art.

[0007] In one aspect, the present invention provides a controllable music-driven three-dimensional dance movement generation model training method, the method comprising the following steps:

[0008] Constructing a first training sample set, wherein the first training sample set includes a plurality of samples, each sample includes music and its corresponding three-dimensional dance movement data; constructing a second training sample set, wherein the second training sample set includes a plurality of samples, each sample includes the music, the three-dimensional dance movement data corresponding to the music, and the two-dimensional key point data of the human joints corresponding to the three-dimensional dance movement data;

[0009] Constructing an initial generation model, the initial generation model includes a backbone network and a control network; the backbone network includes multiple linear layers and Transformer decoder layers, and the control network includes multiple linear layers, Transformer decoder layers and zero linear layers; the backbone network takes the music and its corresponding three-dimensional dance motion data as input, extracts the features of the three-dimensional dance motion data, and outputs predicted motion data; the control network takes the music, the three-dimensional dance motion data and its corresponding two-dimensional key point data as input, extracts the features of the two-dimensional key point data, and outputs motion features; the motion features are input into the backbone network, and spliced ​​with the intermediate features extracted by the backbone network to control the predicted motion data;

[0010] The diffusion model is used to train the initial generation model, and noise is added to the three-dimensional dance motion data of the first training sample set and the second training sample set to obtain noise motion data; the first training sample set is used to train the backbone network, the loss of the predicted motion data and the three-dimensional dance motion data is constructed, and the parameters of the backbone network are updated; the structure and parameters of the trained backbone network are fixed, and the structure and parameters are applied to the control network, the second training sample set is used to train the initial generation model, the loss of the predicted motion data and the three-dimensional dance motion data generated by the control network is constructed, the parameters of the control network are updated, and finally the three-dimensional dance motion generation model composed of the backbone network and the control network is trained.

[0011] In some embodiments of the present invention, the motion features are input into the backbone network and concatenated with the intermediate features extracted by the backbone network, including:

[0012] Outputting the motion features extracted by the control network to the backbone network via the zero linear layer;

[0013] The output of the zero linear layer is added to the output of the corresponding Transformer decoder layer in the backbone network; wherein the zero linear layer represents a linear layer whose parameters are initialized to zero.

[0014] In some embodiments of the present invention, the backbone network is a U-shaped structure, the number of Transformer decoder layers on the left and right arms of the U-shaped structure is the same, and the bottom layer has a Transformer decoder layer, and the method further includes:

[0015] Input the music into the Transformer decoder layers on the left and right arms of the U-shaped structure, and input the noise motion data into the Transformer decoder layer of the left arm via the linear layer of the left arm;

[0016] In each layer, the output of the left-arm Transformer decoder layer is concatenated with the output of the right-arm Transformer decoder layer of the next layer according to the feature dimension, and after the feature dimension size is adjusted by the linear layer of the current layer, it is input into the right-arm Transformer decoder layer of the current layer; finally, the right-arm linear layer outputs the predicted action data.

[0017] In some embodiments of the present invention, constructing the loss of the predicted motion data and the three-dimensional dance motion data includes:

[0018] Extracting predicted attribute information and real attribute information from the predicted motion data and the three-dimensional dance motion data respectively; the predicted attribute information includes predicted coordinates, predicted speed and predicted acceleration of human joints; the real attribute information includes real coordinates, real speed and real acceleration of human joints;

[0019] Calculate the root mean square error of each attribute, and perform weighted summation of the root mean square error of each attribute to obtain the final loss function value.

[0020] In some embodiments of the present invention, constructing the loss of the predicted motion data and the three-dimensional dance motion data generated by the control network includes:

[0021] Extracting predicted attribute information and real attribute information from the predicted motion data generated by the control network and the three-dimensional dance motion data; the predicted attribute information includes predicted coordinates, predicted speed and predicted acceleration of human joints; the real attribute information includes real coordinates, real speed and real acceleration of human joints;

[0022] Calculate the root mean square error of each attribute; and perform weighted summation of the root mean square error of each attribute to obtain the final loss function value.

[0023] In another aspect, the present invention provides a controllable music-driven three-dimensional dance movement generation method, the method comprising:

[0024] Obtaining music for the three-dimensional dance movements to be generated and two-dimensional key point data for controlling the generated movements;

[0025] The music and the two-dimensional key point data are input into a three-dimensional dance movement generation model trained by any of the controllable music-driven three-dimensional dance movement generation model training methods mentioned above to control the generation of three-dimensional dance movement data.

[0026] In some embodiments of the present invention, the music and the two-dimensional key point data are input into a three-dimensional dance movement generation model trained based on any of the controllable music-driven three-dimensional dance movement generation model training methods mentioned above, and the three-dimensional dance movement data is controlled to be generated, including:

[0027] Inputting the music, the two-dimensional key point data and the random noise motion data generated based on Gaussian distribution into the three-dimensional dance motion generation model, and obtaining first predicted motion data after denoising the random noise motion data;

[0028] adding noise to the first predicted motion data using a diffusion model to obtain first noise motion data;

[0029] Inputting the music, the two-dimensional key point data and the first noise motion data into the three-dimensional dance motion generation model, and obtaining second predicted motion data after denoising the first noise motion data;

[0030] Repeat the above steps for a preset number of times to obtain the final predicted action data.

[0031] In some embodiments of the present invention, the controllable music-driven three-dimensional dance movement generation method further includes:

[0032] The three-dimensional dance movement data is converted into a three-dimensional modeling data format for rendering and display in three-dimensional modeling software.

[0033] On the other hand, the present invention also provides a controllable music-driven three-dimensional dance movement generation device, comprising a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of any of the methods mentioned above.

[0034] On the other hand, the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of any of the methods mentioned above when executed by a processor.

[0035] The present invention provides a controllable music-driven three-dimensional dance movement generation model training method, generation method and device, including: constructing a training set containing music-three-dimensional dance movement data and two-dimensional key point data; constructing an initial generation model, including a backbone network and a control network; the backbone network takes music and three-dimensional dance movement data as input, and outputs predicted movement data; the control network takes music, three-dimensional dance movement data and two-dimensional key point data as input, and outputs motion features; splicing the motion features with the intermediate features extracted by the backbone network to control the predicted movement data; using the training set to perform two-stage training on the initial generation model, and finally obtaining a three-dimensional dance movement generation model. The three-dimensional dance movement generation model obtained by the training of the present invention can be used to generate three-dimensional dance movement data matching music, and has controllable generation capability.

[0036] Furthermore, the backbone network is designed as a "U"-shaped structure. The connection method of the left and right arms of the U-shaped structure can not only make full use of the time information before and after the sequence, but also is compatible with the design of the control network, thereby achieving a controllable generation effect.

[0037] Furthermore, based on the connection method between the control network and the backbone network, and the control network copying the neural network structure and parameters corresponding to the left arm in the backbone network, and combining the zero linear layer method, the two-dimensional key point data can be converted into deviation values ​​for adjusting three-dimensional dance movements.

[0038] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0039] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0041] Figure 1 Schematic diagram of the steps of a method for training a controllable music-driven three-dimensional dance movement generation model in one embodiment of the present invention.

[0042] Figure 2 Schematic diagram of the structure of the initial generation model (three-dimensional dance movement generation model) in one embodiment of the present invention.

[0043] Figure 3 The figure is a schematic diagram of the connection between the control network and the backbone network in one embodiment of the present invention.

[0044] Figure 4 Schematic diagram of the backbone network structure in one embodiment of the present invention.

[0045] Figure 5 This is a schematic diagram of the backbone network training process in one embodiment of the present invention.

[0046] Figure 6 The figure is a schematic diagram of the steps of a method for generating controllable music-driven three-dimensional dance movements in one embodiment of the present invention.

[0047] Figure 7 Schematic diagram of the principle of a controllable music-driven three-dimensional dance movement generation method in one embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0049] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0050] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0051] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection with an intermediate.

[0052] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0053] In order to solve the problem that the existing music-driven dance generation scheme lacks the ability to control the generation results and is difficult to control the specific motion trajectory of the human body joints, the present invention provides a controllable music-driven three-dimensional dance movement generation model training method, such as Figure 1 As shown, the method includes the following steps S101 to S103:

[0054] Step S101: construct a first training sample set, the first training sample set includes multiple samples, each sample includes music and its corresponding three-dimensional dance movement data; construct a second training sample set, the second training sample set includes multiple samples, each sample includes music, three-dimensional dance movement data corresponding to the music, and two-dimensional key point data of human joints corresponding to the three-dimensional dance movement data.

[0055] Step S102: construct an initial generation model, wherein the initial generation model includes a backbone network and a control network, the backbone network includes multiple linear layers and a Transformer decoder layer, and the control network includes multiple linear layers, a Transformer decoder layer, and a zero linear layer. The backbone network takes music and its corresponding three-dimensional dance motion data as input, extracts the features of the three-dimensional dance motion data, and outputs predicted motion data; the control network takes music, three-dimensional dance motion data, and its corresponding two-dimensional key point data as input, extracts the features of the two-dimensional key point data, and outputs motion features; the motion features are input into the backbone network, and spliced ​​with the intermediate features extracted by the backbone network to control the predicted motion data.

[0056] Step S103: Use the diffusion model to train the initial generation model, and add noise to the three-dimensional dance motion data of the first training sample set and the second training sample set to obtain noise motion data. Use the first training sample set to train the backbone network, construct the loss of the predicted motion data and the three-dimensional dance motion data, and update the parameters of the backbone network; fix the structure and parameters of the trained backbone network, and apply the structure and parameters to the control network, use the second training sample set to train the initial generation model, construct the loss of the predicted motion data and the three-dimensional dance motion data generated by the control network, update the parameters of the control network, and finally train to obtain a three-dimensional dance motion generation model composed of the backbone network and the control network.

[0057] In step S101, firstly, a first training sample set and a second training sample set are constructed for subsequent model training. The first training sample set is a large-scale sample, each sample contains music and three-dimensional dance movement data corresponding to the music, and the two correspond one to one. The second training sample set is a small-scale sample, each sample contains music, three-dimensional dance movement data corresponding to the music, and two-dimensional key point data of human joints corresponding to the three-dimensional dance movement data, and the three correspond one to one. Among them, the two-dimensional key point data represents human motion data, and the key points therein correspond to some joints of the human body, thereby describing the motion trajectory of the human body.

[0058] In some embodiments, two-dimensional key points in the COCO format disclosed by Microsoft are used to represent human motion, and this format can also be compatible with existing human pose estimation methods. Human pose estimation refers to a task of automatically identifying and extracting human motion postures from data such as images and videos. Therefore, two-dimensional key point data can be extracted by image and video extraction, manual editing, etc.

[0059] In the present invention, the model is trained in a supervised learning manner using the noise diffusion principle of the diffusion model, so that the model has the ability to predict the actual motion data, and then in the denoising generation process of the diffusion model, the noise samples are gradually denoised to restore the predicted actual motion data. Therefore, in step S101, noise is added to the three-dimensional dance motion data using the diffusion model denoising process to obtain noise motion data.

[0060] In the present invention, for the real music in the sample, preset music feature extraction tools, models, etc. are used to extract the features of the real music, such as volume, duration, pitch, spectrum, etc.

[0061] In step S102, it is considered that the prior art generates three-dimensional dance movement data that matches the music by inputting music data. Since random sampling of samples from noise is adopted in the diffusion model generation process, the final generated result is random, and it is difficult to control the generation of the same three-dimensional dance movement. On the other hand, in actual application scenarios in the fields of film, television, games, etc., users often hope to control the generated three-dimensional dance movement to run according to the motion trajectory they designed. However, it is difficult for the prior art to provide such a function, and to accept the input of the user's text, image, etc. for controlling the motion trajectory, so as to achieve the control generation result in accordance with the user design. Therefore, the present invention designs a control network in view of the above technical problems.

[0062] Construct an initial generative model, such as Figure 2 As shown, the initial generation model includes a backbone network and a control network. The backbone network includes multiple linear layers and Transformer decoder layers, and the control network includes multiple linear layers, Transformer decoder layers and zero linear layers.

[0063] The backbone network takes music and its corresponding 3D dance action data as input, extracts the features of the 3D dance action data, and outputs the predicted action data. Therefore, the backbone network learns the relationship between the two modalities of music and 3D dance action data from a large-scale music-3D dance action dataset, thus possessing rich prior knowledge of 3D dance action generation.

[0064] The control network takes music, three-dimensional dance movement data and its corresponding two-dimensional key point data as input, extracts features from the two-dimensional key point data, and converts them into deviation values ​​of the corresponding motion trajectory of the three-dimensional dance movement, that is, the motion features mentioned above. The motion features are then input into the backbone network and spliced ​​with the intermediate features extracted by the backbone network, so as to control the generated predicted motion data to be similar to the motion trajectory of the two-dimensional key points, thereby achieving the effect of controllable predicted motion data.

[0065] exist Figure 2 In the figure, the dotted arrows and ellipsis in the backbone network and the control network indicate that the repeated model structure is omitted. The black arrows indicate that the output of the previous layer structure is input to the next layer structure. The gray arrows indicate that they are input to the corresponding structure as conditions. The specific connection method of the red arrows can be as follows: Figure 3 As shown, the output of the zero linear layer of the control network is added to the output of the Transformer decoder layer in the corresponding backbone network, where the zero linear layer represents a linear layer whose parameters are initialized to zero, which is mainly used to convert the extracted features into deviation values ​​corresponding to the motion trajectory of the three-dimensional dance movement.

[0066] In some embodiments, in order for the backbone network to have stronger sequence modeling capabilities and to be compatible with the design of the control network, Figure 4 As shown in , the overall structure is a "U"-shaped structure, with the same number of Transformer decoder layers on the left and right arms of the U-shaped structure, and an additional Transformer decoder layer at the bottom. In order to make full use of the information before and after the sequence, as Figure 4 As shown, the output of the left decoder layer is spliced ​​and input into the corresponding decoder layer on the right. Specifically: the music is input into the Transformer decoder layers on the left and right arms of the U-shaped structure, and the noise action data is input into the Transformer decoder layer of the left arm through the linear layer of the left arm; in each layer, the output of the left-arm Transformer decoder layer is spliced ​​with the output of the right-arm Transformer decoder layer of the next layer according to the feature dimension, and after the feature dimension size is adjusted through the linear layer of the current layer, it is input into the right-arm Transformer decoder layer of the current layer; finally, the right-arm linear layer outputs the predicted action data. Figure 4 In the figure, the black arrow indicates that the output of the previous layer structure is input into the next layer structure, the gray arrow indicates that it is input into the corresponding structure as a condition, and the dotted arrow indicates that the repeated model structure is omitted.

[0067] In step S103, the diffusion model is used to train the initial generation model, and noise is added to the three-dimensional dance motion data of the first training sample set and the second training sample set to obtain noise motion data. In the present invention, the training process is divided into two stages, the first stage only trains the backbone network, and the second stage mainly trains the control network.

[0068] In the first stage, the backbone network is supervised and trained using the first training sample set, namely the large-scale music-3D dance action dataset. Figure 5 As shown, a diffusion model is used to add noise to the three-dimensional dance motion data to obtain noise motion data, which is input into the backbone network together with the music features, and finally the predicted motion data is output. A loss function between the predicted motion data and the real motion data (three-dimensional dance motion data) is constructed to update the parameters of the backbone network.

[0069] In some embodiments, constructing the root mean square error loss of the predicted motion data and the three-dimensional dance motion data includes: extracting predicted attribute information and real attribute information from the predicted motion data and the three-dimensional dance motion data, respectively. The predicted attribute information includes predicted coordinates, predicted speed, and predicted acceleration of human joints; the real attribute information includes real coordinates, real speed, and real acceleration of human joints. Calculate the root mean square error of each attribute, and perform weighted summation on the root mean square error of each attribute to obtain the final loss function value.

[0070] In the second stage, the trained backbone network parameters are loaded and the parameters of this part of the structure are fixed so that they will not be updated during the second stage of training. After the backbone network parameters are loaded, the control network will copy the structure and parameters of the corresponding backbone network. The parameters of this part of the structure will not be fixed and will be updated as the data is input during the second stage of training. Figure 2 Corresponding annotations are also made in the . Figure 2 The structure marked with a snowflake pattern in the figure indicates that its parameters are fixed and will not be updated during training. The structure without this mark will update its parameters during training. The "Transformer Decoder Layer (Trainable Copy)" in the control network indicates that the structure is the same as the corresponding "Transformer Decoder Layer" in the backbone network, and the parameters are also the same as the corresponding decoder layer. "Trainable" means that the parameters of this structure are not fixed and can be updated during training.

[0071] The entire initial generation model (mainly the control network) is trained using the second training sample set, namely the music-three-dimensional dance action data and its corresponding two-dimensional key point data set. The diffusion model is used to add noise to the three-dimensional dance action data to obtain noise action data, which is input into the initial generation model together with the music features and two-dimensional key point data, and finally the predicted action data is output. The loss function between the predicted action data generated by the control network and the three-dimensional dance action data is constructed to update the parameters of the control network.

[0072] In some embodiments, constructing the root mean square error loss of the predicted motion data and the three-dimensional dance motion data generated by the control network includes: extracting predicted attribute information and real attribute information from the predicted motion data and the three-dimensional dance motion data generated by the control network. The predicted attribute information includes the predicted coordinates, predicted speed and predicted acceleration of the human joints; the real attribute information includes the real coordinates, real speed and real acceleration of the human joints. The root mean square error of each attribute is calculated, and the root mean square error of each attribute is weighted and summed to obtain the final loss function value.

[0073] Corresponding to the above controllable music-driven three-dimensional dance movement generation model training method, the present invention also provides a controllable music-driven three-dimensional dance movement generation method, such as Figure 6 As shown, the method includes the following steps S201-S202:

[0074] Step S201: Obtain music for the three-dimensional dance movements to be generated and two-dimensional key point data for controlling the generated movements.

[0075] Step S202: Figure 7 As shown, music and two-dimensional key point data are input into the three-dimensional dance movement generation model trained based on the controllable music-driven three-dimensional dance movement generation model training method described above, and the three-dimensional dance movement data is controlled to be generated.

[0076] In some embodiments, multiple denoising methods are used to improve the quality of the generated three-dimensional dance motion data, including:

[0077] Music, two-dimensional key point data and random noise motion data generated based on Gaussian distribution are input into a three-dimensional dance motion generation model, and the first predicted motion data is obtained after denoising the random noise motion data.

[0078] A diffusion model is used to add noise to the first predicted motion data to obtain first noise motion data.

[0079] The music, the two-dimensional key point data and the first noise motion data are input into the three-dimensional dance motion generation model, and the second predicted motion data is obtained after the first noise motion data is denoised.

[0080] The above steps are repeated for a preset number of times to obtain the final predicted motion data, that is, the final generated three-dimensional dance motion data.

[0081] In some embodiments, the generated three-dimensional dance motion data is converted into a three-dimensional modeling data format, such as FBX, OBJ, GLTF, etc., so as to be rendered and displayed in three-dimensional modeling software.

[0082] Corresponding to the above method, the present invention also provides a controllable music-driven three-dimensional dance movement generation device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0083] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0084] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0085] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0086] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A controllable music-driven three-dimensional dance movement generation model training method, characterized in that: The method comprises the following steps: Constructing a first training sample set, wherein the first training sample set includes a plurality of samples, each sample includes music and its corresponding three-dimensional dance movement data; constructing a second training sample set, wherein the second training sample set includes a plurality of samples, each sample includes the music, the three-dimensional dance movement data corresponding to the music, and the two-dimensional key point data of the human joints corresponding to the three-dimensional dance movement data; Constructing an initial generation model, the initial generation model includes a backbone network and a control network; the backbone network includes multiple linear layers and Transformer decoder layers, and the control network includes multiple linear layers, Transformer decoder layers and zero linear layers; the backbone network takes the music and its corresponding three-dimensional dance motion data as input, extracts the features of the three-dimensional dance motion data, and outputs predicted motion data; the control network takes the music, the three-dimensional dance motion data and its corresponding two-dimensional key point data as input, extracts the features of the two-dimensional key point data, and outputs motion features; the motion features are input into the backbone network, and spliced ​​with the intermediate features extracted by the backbone network to control the predicted motion data; The diffusion model is used to train the initial generation model, and noise is added to the three-dimensional dance motion data of the first training sample set and the second training sample set to obtain noise motion data; the first training sample set is used to train the backbone network, the loss of the predicted motion data and the three-dimensional dance motion data is constructed, and the parameters of the backbone network are updated; the structure and parameters of the trained backbone network are fixed, and the structure and parameters are applied to the control network, the second training sample set is used to train the initial generation model, the loss of the predicted motion data and the three-dimensional dance motion data generated by the control network is constructed, the parameters of the control network are updated, and finally the three-dimensional dance motion generation model composed of the backbone network and the control network is trained.

2. The controllable music-driven three-dimensional dance movement generation model training method according to claim 1, characterized in that: Inputting the motion features into the backbone network and concatenating them with the intermediate features extracted by the backbone network, including: Outputting the motion features extracted by the control network to the backbone network via the zero linear layer; The output of the zero linear layer is added to the output of the corresponding Transformer decoder layer in the backbone network; wherein the zero linear layer represents a linear layer whose parameters are initialized to zero.

3. The controllable music-driven three-dimensional dance movement generation model training method according to claim 1, characterized in that: The backbone network is a U-shaped structure, the number of Transformer decoder layers on the left and right arms of the U-shaped structure is the same, and the bottom layer has a Transformer decoder layer, and the method further includes: Input the music into the Transformer decoder layers on the left and right arms of the U-shaped structure, and input the noise motion data into the Transformer decoder layer of the left arm via the linear layer of the left arm; In each layer, the output of the left-arm Transformer decoder layer is concatenated with the output of the right-arm Transformer decoder layer of the next layer according to the feature dimension, and after the feature dimension size is adjusted by the linear layer of the current layer, it is input into the right-arm Transformer decoder layer of the current layer; finally, the right-arm linear layer outputs the predicted action data.

4. The controllable music-driven three-dimensional dance movement generation model training method according to claim 1, characterized in that: Constructing the loss of the predicted motion data and the three-dimensional dance motion data, comprising: Extracting predicted attribute information and real attribute information from the predicted motion data and the three-dimensional dance motion data respectively; the predicted attribute information includes predicted coordinates, predicted speed and predicted acceleration of human joints; the real attribute information includes real coordinates, real speed and real acceleration of human joints; Calculate the root mean square error of each attribute, and perform weighted summation of the root mean square error of each attribute to obtain the final loss function value.

5. The controllable music-driven three-dimensional dance movement generation model training method according to claim 1, characterized in that: Constructing the loss of the predicted motion data and the three-dimensional dance motion data generated by the control network, including: Extracting predicted attribute information and real attribute information from the predicted motion data generated by the control network and the three-dimensional dance motion data; the predicted attribute information includes predicted coordinates, predicted speed and predicted acceleration of human joints; the real attribute information includes real coordinates, real speed and real acceleration of human joints; Calculate the root mean square error of each attribute, and perform weighted summation of the root mean square error of each attribute to obtain the final loss function value.

6. A controllable music-driven three-dimensional dance movement generation method, characterized in that: The method comprises: Obtaining music for the three-dimensional dance movements to be generated and two-dimensional key point data for controlling the generated movements; The music and the two-dimensional key point data are input into a three-dimensional dance movement generation model trained by the controllable music-driven three-dimensional dance movement generation model training method according to any one of claims 1 to 5 to control the generation of three-dimensional dance movement data.

7. The controllable music-driven three-dimensional dance movement generation model training method according to claim 6, characterized in that: Inputting the music and the two-dimensional key point data into a three-dimensional dance movement generation model trained by the controllable music-driven three-dimensional dance movement generation model training method according to any one of claims 1 to 5, and controlling the generation of three-dimensional dance movement data, comprising: Inputting the music, the two-dimensional key point data and the random noise motion data generated based on Gaussian distribution into the three-dimensional dance motion generation model, and obtaining first predicted motion data after denoising the random noise motion data; adding noise to the first predicted motion data using a diffusion model to obtain first noise motion data; Inputting the music, the two-dimensional key point data and the first noise motion data into the three-dimensional dance motion generation model, and obtaining second predicted motion data after denoising the first noise motion data; Repeat the above steps for a preset number of times to obtain the final predicted action data.

8. The controllable music-driven three-dimensional dance movement generation method according to claim 6, characterized in that: The method further comprises: The three-dimensional dance movement data is converted into a three-dimensional modeling data format for rendering and display in three-dimensional modeling software.

9. A controllable music-driven three-dimensional dance movement generation device, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions. When the computer program / instructions are executed, the device implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images