Model training method, dance generation method, device, and medium

Abstract

The application provides a model training method, a dance generation method, equipment and a medium. The model training method comprises: obtaining music features of sample music and action features of sample dances; calling a first mapping network module to fuse and process the music features and the action features to obtain fused features and process the fused features to obtain a first multi-element dance category; calling a coding and decoding network module to code and decode the fused features to obtain decoded features, calling a second mapping network module to classify the decoded features to obtain a second multi-element dance category; determining a target loss based on a first loss between the fused features and the decoded features and a second loss between the first multi-element dance category and the second multi-element dance category, adjusting model parameters of an initial neural network model based on the target loss to obtain a dance generation model. The application embodiment reduces the difficulty of cross-modal learning and improves the accuracy of generated dance actions.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, a dance generation method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Dance is a universal language that effectively shares emotions. It can be shared and disseminated through platforms such as social media and short video applications. However, the creation of dance is a highly creative and artistic process, so choreographers typically undergo extensive professional training to reach a stage where they can choreograph and generate dances. Furthermore, different choreographers have their own habits and styles. With the development of artificial intelligence and computer technology, not only can intelligent tools generate choreography, but they can also blend the styles of different choreographers to create dances that integrate their influences. For example, based on music provided by a user, an intelligent tool can automatically generate a dance corresponding to the human body movements.

[0003] Currently, there are two main categories of methods for music-driven dance generation: the first category generates dances based on a given initial sequence of dance movements; the second category generates dances based on a choreographed version of a piece of music. However, both categories require cross-modal learning between music features and dance movement features. This is because cross-modal learning involves inter-domain differences between music features and dance movement features, causing the generated dance movements to be inconsistent with the music's rhythm, resulting in low accuracy.

[0004] Therefore, improving the accuracy of generated dance moves is an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a model training method, a dance generation method, an electronic device, and a computer-readable storage medium, which can reduce the difficulty of cross-modal learning between music features and dance features and improve the accuracy of generated dance movements.

[0006] On one hand, embodiments of this application provide a model training method, which includes: acquiring musical features of sample music and motion features of sample dance, wherein there is a matching relationship between the sample music and the sample dance; calling a first mapping network module included in an initial neural network model to fuse the musical features and motion features to obtain fused features, and processing the fused features to obtain a first multi-dimensional dance category, wherein the first multi-dimensional dance category includes at least one dance category label; calling an encoding / decoding network module included in the initial neural network model to encode and decode the fused features to obtain decoded features, and calling a second mapping network module included in the initial neural network model to classify the decoded features to obtain a second multi-dimensional dance category, wherein the second multi-dimensional dance category includes at least one dance category label; determining a target loss based on a first loss between the fused features and the decoded features, and a second loss between the first multi-dimensional dance category and the second multi-dimensional dance category, and adjusting the model parameters of the initial neural network model based on the target loss to obtain a dance generation model.

[0007] In one optional implementation, obtaining the musical features of the sample music and the motion features of the sample dance includes: dividing the sample music into multiple music segments based on the drum beats of the sample music, and dividing the sample dance into multiple dance movements, with a one-to-one correspondence between the multiple music segments and the multiple dance movements; calling the feature extraction network module included in the initial neural network model to extract features from the multiple music segments to obtain the musical features of the sample music, and to extract features from the multiple dance movements to obtain the motion features of the sample dance.

[0008] In one optional implementation, the first mapping network module included in the initial neural network model is invoked to process the music features and motion features to obtain fused features and a first multi-dimensional dance category. This includes: invoking the feature fusion layer of the first mapping network module to fuse the music features and motion features to obtain fused features; and invoking the classification layer of the first mapping network module to classify the fused features to obtain the first multi-dimensional dance category.

[0009] In one optional implementation, the feature fusion layer of the first mapping network module included in the initial neural network model is invoked to perform fusion processing on music features and action features to obtain fused features. This includes: invoking the feature fusion layer included in the first mapping network module to combine music features and latent encoding to obtain target music features; the latent encoding is used to reduce noise in the music features; and invoking the feature fusion layer to perform fusion processing on the target music features and action features to obtain fused features.

[0010] In one optional implementation, determining a target loss based on a first loss between fused features and decoded features, and a second loss between a first multi-dimensional dance category and a second multi-dimensional dance category, includes: determining a first loss based on difference data between fused features and decoded features; determining a second loss based on difference data between the first multi-dimensional dance category and the second multi-dimensional dance category; and determining a target loss based on the first loss and the second loss.

[0011] In one optional implementation, determining a second loss based on the difference data between a first multi-dimensional dance category and a second multi-dimensional dance category includes: determining at least one difference data between the dance category labels included in the first multi-dimensional dance category and the dance category labels included in the second multi-dimensional dance category; and determining the second loss based on the deviation information between the at least one difference data.

[0012] In one optional implementation, the model parameters of the initial neural network model are adjusted based on the target loss to obtain a dance generation model, including: adjusting the model parameters of the initial neural network model based on the target loss to obtain an adjusted initial neural network model; updating the target loss based on the adjusted initial neural network model until the updated target loss satisfies the prediction convergence condition to obtain a trained initial neural network model; and determining the dance generation model based on the feature extraction network module and the encoding / decoding network module in the trained initial neural network model.

[0013] On one hand, embodiments of this application provide a dance generation method, which includes: acquiring target music; inputting the target music into a dance generation model to obtain a target dance movement output by the dance generation model, wherein the target dance movement includes matching the target music; wherein the dance generation model is trained based on the model training method in the previous aspect.

[0014] On one hand, embodiments of this application provide a model training apparatus, the apparatus comprising:

[0015] The system comprises the following components: an acquisition unit, which acquires the musical features of sample music and the motion features of sample dance, with a matching relationship between the sample music and the sample dance; a processing unit, which calls the first mapping network module included in the initial neural network model to fuse the musical features and motion features, obtain fused features, and processes the fused features to obtain a first multi-dimensional dance category, which includes at least one dance category label; a further processing unit, which calls the encoding / decoding network module included in the initial neural network model to encode and decode the fused features, obtain decoded features, and calls the second mapping network module included in the initial neural network model to classify the decoded features, obtain a second multi-dimensional dance category, which includes at least one dance category label; and a final processing unit, which determines a target loss based on a first loss between the fused features and the decoded features, and a second loss between the first multi-dimensional dance category and the second multi-dimensional dance category, and adjusts the model parameters of the initial neural network model based on the target loss to obtain a dance generation model.

[0016] On one hand, embodiments of this application provide a model training apparatus, the apparatus comprising:

[0017] The acquisition unit is used to acquire target music; the processing unit is used to input the target music into the dance generation model to obtain the target dance movement output by the dance generation model, wherein the target dance movement matches the target music; wherein the dance generation model is trained based on the model training method in the previous aspect.

[0018] On one hand, embodiments of this application provide an electronic device, including: a processor, a user interface, a communication interface, and a memory, wherein the processor, the user interface, the communication interface, and the memory are interconnected, wherein the memory stores executable program code, and the processor is used to call the executable program code to execute the method provided in embodiments of this application.

[0019] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, implement the method provided in embodiments of this application.

[0020] Accordingly, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the method provided in embodiments of this application.

[0021] In this embodiment, the musical features of sample music and the motion features of sample dance are obtained, and there is a correspondence between the sample music and the sample dance. The first mapping network module of the initial neural network model is called to process the musical features and motion features to obtain fused features and a first multi-dimensional dance category, which includes at least one dance category label. The encoding / decoding network module of the initial neural network model is called to process the fused features to obtain decoded features, and the second mapping network module of the initial neural network model is called to process the decoded features to obtain a second multi-dimensional dance category, which also includes at least one dance category label. A target loss is determined based on a first loss between the fused features and the decoded features, and a second loss between the first and second multi-dimensional dance categories. The model parameters of the initial neural network model are adjusted based on the target loss to obtain a dance generation model. On the one hand, the dance generation model determined by this method can perform fused learning of musical features and motion features, reducing the difficulty of cross-modal learning and thus improving the accuracy of the generated dance movements. On the other hand, the multi-dimensional dance categories corresponding to the fused features are also trained during the training process of the dance generation model, making the dance categories corresponding to the generated dance movements more diversified. Attached Figure Description

[0022] To more clearly illustrate the technical methods in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of a dance movement provided in an embodiment of this application;

[0024] Figure 2 This is a schematic flowchart of a model training method provided in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of an initial neural network model provided in an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of a dance generation method provided in an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of a model training device provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The technical methods in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0030] To better understand the embodiments of this application, some terms involved in the embodiments of this application will be introduced below:

[0031] Dance movements: These refer to the 24 three-dimensional (3D) keypoints of the human body, represented as a 24*3 matrix in the network, and temporally as T*24*3 (meaning there are T dance movements within a given timeframe). Please see [link / reference]. Figure 1 , Figure 1 This is a schematic diagram of a dance movement provided in an embodiment of this application, such as... Figure 1 As shown, there are 24 key points on the human body image on the left. The coordinate position of each dance move changes as the 24 key points change. The right side of the image shows T (T is 7) dance moves, which constitute a dance move library, represented as 7*24*3.

[0032] Concat: Connects two feature vectors into a longer feature.

[0033] Fully connected layer (FC): In deep learning, the fully connected layer acts as a "classifier" in the entire convolutional neural network.

[0034] To improve the accuracy of generated dance movements, the following methods are currently used:

[0035] Method 1: In the training phase, the feature learning network module is first used to learn dance movement features and music features separately. Then, the feature learning network module is used to perform cross-modal learning of the dance movement features and music features, ultimately obtaining the mapping relationship between the dance movement features and music features. In the usage phase, the feature learning network module is used to learn the music features, and then the Long Short-Term Memory (LSTM) network module is used to decode the music features to obtain the corresponding dance movement sequence.

[0036] Method 2: Utilize a two-stage learning approach to learn music features and dance movement features. In the first stage, only the model of dance movements needs to be trained; in the second stage, the dance movements from the first stage are replaced with music input, thus ensuring that a corresponding dance sequence can be obtained for any input music.

[0037] Method 3: Utilize the feature similarity between dance movement features and music features to identify N optimal matching dance movements, and then choreograph these dance movements to obtain the required dance movement sequence.

[0038] However, due to inherent problems in their implementation, each of the above choreography methods produces dance movements with low accuracy, failing to meet user needs. Specifically:

[0039] In Method 1, there is no direct constraint between dance movement features and music features, making it difficult to generate high-quality dance movement sequences (due to low accuracy of dance movements).

[0040] Method 2 directly replaces the dance movement generation network with a music-generated dance movement network, resulting in less information between dance movement features and music features. Consequently, the dance movement features obtained in the first stage cannot express more information about the music features, making it difficult to generate high-quality dance movement sequences (low accuracy of dance movements).

[0041] While Method 3 focuses on the mapping relationship between dance movement features and music features, it neglects the effective extraction of music features, leading to choreographic dance movements that are prone to style confusion and inaccurate timing (low accuracy of dance movements). Furthermore, this method is based on movements in a database and cannot generate new dance movements.

[0042] Therefore, to improve the accuracy of generated dance movements, this application provides a model training method. In this method, musical features of sample music and motion features of sample dance are obtained, and a matching relationship exists between the sample music and sample dance. The first mapping network module of the initial neural network model is invoked to fuse the musical features and motion features to obtain fused features, and the fused features are processed to obtain a first multi-dimensional dance category, which includes at least one dance category label. The encoding / decoding network module of the initial neural network model is invoked to encode and decode the fused features to obtain decoded features, and the second mapping network module of the initial neural network model is invoked to classify the decoded features to obtain a second multi-dimensional dance category, which includes at least one dance category label. A target loss is determined based on a first loss between the fused features and the decoded features, and a second loss between the first and second multi-dimensional dance categories. The model parameters of the initial neural network model are adjusted based on the target loss to obtain a dance generation model. On the one hand, the dance generation model determined by this method can fuse music features and motion features to obtain fused features, and learn the fused features, which reduces the difficulty of cross-modal learning and thus improves the accuracy of generated dance movements. On the other hand, during the training process of the dance generation model, the multi-dance categories corresponding to the fused features and the multi-dance categories corresponding to the decoded features are jointly trained, making the generated dance movements more diversified in terms of dance categories.

[0043] It should be noted that, in specific implementations, the dance generation model in the above method can be an electronic device or a module of an electronic device. This electronic device can be a terminal or a server; the terminal can be a smartphone, tablet, laptop, desktop computer, smartwatch, smart vehicle terminal, etc., but is not limited to these. The server can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, but is not limited to these. When the electronic device is a server, the above method processes data through a server backend, resulting in high processing efficiency and fast operation.

[0044] The model training method will be explained in detail below with reference to the accompanying diagram.

[0045] Please see Figure 2 , Figure 2 This is a schematic flowchart of a model training method provided in an embodiment of this application. Figure 2 As shown, the model training method may include, but is not limited to, the following steps:

[0046] S201. Obtain the musical features of the sample music and the movement features of the sample dance.

[0047] There is a matching relationship between the sample music and the sample dance. That is, the sample music and the sample dance are paired together.

[0048] In one optional implementation, obtaining the musical features of the sample music and the motion features of the sample dance includes: dividing the sample music into multiple musical segments based on the drum beats of the sample music, and dividing the sample dance into multiple dance movements, with a one-to-one correspondence between the multiple musical segments and the multiple dance movements; calling the feature extraction network module included in the initial neural network model to extract features from the multiple musical segments to obtain musical features of the sample music, such as Mel spectrogram features, and to extract features from the multiple dance movements to obtain motion features of the sample dance.

[0049] For example, a sample piece of music has 5 drum beats. The sample piece of music is divided into 6 music segments according to the 5 drum beats. At the same time, the sample dance corresponding to the sample music is also divided into 6 dance movement sequences according to the 5 drum beats. There is a one-to-one correspondence between the 6 music segments and the 6 dance movement sequences. Each dance movement sequence is represented as T*24*3 in the temporal dimension, where T represents that there are T dance movements in the temporal sequence. The feature extraction network module included in the initial neural network model is called to extract features from the 6 music segments and the 6 dance movement sequences to obtain the music features of the sample music and the movement features of the sample dance.

[0050] It should be noted that in this embodiment, the musical features of the sample music and the movement features of the sample dance are represented in matrix form, and both are feature information encoded through encoding operations (such as linear projection), which is beneficial for cross-modal learning of the musical features of the sample music and the movement features of the sample dance. Furthermore, the training samples used in this embodiment come from different choreographers, including sample music and sample dances of various styles and rhythms. The dance generation model obtained using these training samples has better adaptability and can generate dance movements of different types with high accuracy based on the music.

[0051] S202. Call the first mapping network module included in the initial neural network model to perform fusion processing on music features and motion features to obtain fused features, and process the fused features to obtain the first multi-dimensional dance category.

[0052] To distinguish it from other multi-dance categories described below, this multi-dance category can be referred to as the first multi-dance category. The first multi-dance category includes at least one dance category label. For example, dance movement category labels could include label 1 for jazz, label 2 for street dance, label 3 for Latin dance, label 4 for breakdancing, and so on. Therefore, the dance category labels included in the first multi-dance category can be any combination of label 1, label 2, label 3, and label 4. This embodiment does not limit the dance category labels.

[0053] The initial neural network model includes a mapping network module, which can be referred to as the first mapping network model for easy distinction from other mapping network modules. In an optional implementation, the first mapping network module included in the initial neural network model is invoked to fuse music features and motion features to obtain fused features, and the fused features are then processed to obtain a first multi-dimensional dance category. This includes: invoking the feature fusion layer of the first mapping network module to fuse music features and motion features to obtain fused features; and invoking the classification layer of the first mapping network module to classify the fused features to obtain the first multi-dimensional dance category.

[0054] In this embodiment, the feature fusion layer in the first mapping network module is used to fuse music features and action features to obtain fused features. These fused features can represent both music features and action features, reducing the difficulty of cross-modal learning between music features and action features.

[0055] In this embodiment, the first mapping network module includes four fully connected (FC) layers. These four FC layers are used to classify the fused features to obtain a first multi-dimensional dance category. This embodiment does not limit the number of FC layers.

[0056] Optionally, the feature fusion layer of the first mapping network module is invoked to fuse the music features and action features to obtain fused features. This includes: invoking the feature fusion layer included in the first mapping network module to combine the music features and latent encoding to obtain target music features; the latent encoding is used to reduce noise in the music features; and invoking the feature fusion layer to fuse the target music features and action features to obtain fused features.

[0057] In this context, latent encoding refers to feature data obtained by dimensionality reduction or compression of the original data. It can express the essence of the original data with less information, and the original data can be completely recovered based on the latent encoding. For example, in the embodiments of this application, a portion of music features can be used as the original data.

[0058] In this embodiment, latent coding is used to reduce noise, and fusion features are information with latent coding added. This information has less noise, and the features learned using this information are more accurate.

[0059] S203. Call the encoding / decoding network module included in the initial neural network model to encode and decode the fused features to obtain decoded features, and call the second mapping network module included in the initial neural network model to classify the decoded features to obtain the second multi-dimensional dance category.

[0060] To distinguish it from the multi-dimensional dance categories obtained above, the multi-dimensional dance categories here can be referred to as the second multi-dimensional dance categories. The second multi-dimensional dance categories include at least one dance category label. To distinguish it from the mapping network module mentioned above, the mapping network module here can be referred to as the second mapping network model.

[0061] For details regarding the second mapping network module, please refer to the relevant content of the first mapping network module mentioned above. For details regarding the second multi-dimensional dance category, please refer to the relevant content of the first multi-dimensional dance category mentioned above. These details will not be elaborated upon here.

[0062] S204. Determine the target loss based on the first loss between the fused features and the decoded features, and the second loss between the first multi-dimensional dance category and the second multi-dimensional dance category. Adjust the model parameters of the initial neural network model based on the target loss to obtain the dance generation model.

[0063] In this embodiment, the target loss is obtained through a first loss and a second loss, meaning the target loss takes into account the meanings of both the first and second losses. For example, the first loss represents the first difference data between the input fused features and the output decoded features; the smaller the value of the first difference data, the smaller the first loss (a smaller first loss indicates more accurate output decoded features). The second loss represents the second difference data between the first multivariate dance category obtained at input and the second multivariate dance category obtained at output; the smaller the value of the second difference data, the smaller the second loss (a smaller second loss indicates more accurate predicted dance category).

[0064] In one alternative implementation, determining a target loss based on a first loss between fused features and decoded features, and a second loss between a first multi-variety dance category and a second multi-variety dance category, includes: determining a first loss based on difference data between fused features and decoded features; determining a second loss based on difference data between the first multi-variety dance category and the second multi-variety dance category; and determining a target loss based on the first loss and the second loss.

[0065] Optionally, a second loss is determined based on the difference data between the first multi-dimensional dance category and the second multi-dimensional dance category, including: obtaining the labels of each dance category included in the first multi-dimensional dance category and the labels of each dance category included in the second multi-dimensional dance category; determining at least one difference data between the labels of each dance category included in the first multi-dimensional dance category and the labels of each dance category included in the second multi-dimensional dance category; and determining the second loss based on the deviation information between the at least one difference data.

[0066] Optionally, the first loss and the second loss can be mean squared error (L2 loss) values, which are calculated using the following formula:

[0067]

[0068] Among them, y i This represents the predicted value, such as the values ​​corresponding to the decoded features and the second multivariate dance category mentioned above; f(x) i ) represents the true value of the sample, such as the values ​​corresponding to the fusion feature and the first multivariate dance category mentioned above; y i -f(x i ) represents the bias information (i.e., the difference between the predicted value and the true value of the sample); n represents the number of samples.

[0069] Optionally, the method for adjusting model parameters in this embodiment is as follows: Adjusting the model parameters of the initial neural network model based on the target loss to obtain a dance generation model. Specifically, the process is as follows: Adjusting the model parameters of the initial neural network model based on the target loss to obtain an adjusted initial neural network model; updating the target loss based on the adjusted initial neural network model until the updated target loss meets the prediction convergence condition, thus obtaining a trained initial neural network model; determining the dance generation model based on the feature extraction network module and the encoding / decoding network module in the trained initial neural network model. Alternatively, if the updated target loss does not meet the prediction convergence condition, the initial neural network model can continue to be trained using the method described in steps S201 to S204.

[0070] As can be seen, the dance generation model determined in this application embodiment includes a first mapping network module and an encoding / decoding network module. Compared with existing dance generation models (such as recurrent neural networks (RNNs) or self-attention models (Transformers)) that use fixed initial action sequences to generate dance action sequences for music, the dance generation model in this application embodiment can generate high-quality dance action sequences based on multiple dance categories corresponding to the music. Furthermore, the dance generation model in this application embodiment is trained by fusing music features and action features, which reduces the difficulty of cross-modal learning between music and action when generating dance action sequences, thus improving the quality of the dance action sequences.

[0071] Please see Figure 3 , Figure 3 This is a schematic diagram of an initial neural network model provided in an embodiment of this application. For example... Figure 3 As shown, the dance movements (x1, x2, ..., x...) are input to the feature extraction network module. m ) and musical fragments (y1, y2, ..., y nThe dance movements are all divided based on the drumbeats of the music, meaning there is a one-to-one correspondence between dance movements and music segments. The feature extraction network module extracts features from the dance movements and music segments (see step S201 above for relevant details), obtaining movement features and music features (such as Mel spectral features). These features are then input into the first mapping network module, where a feature fusion layer fuses the music and movement features to obtain fused features (latent encoding can also be added here; see step S202 above for relevant details). The fused features are then classified using four fully connected (fc) layers in the classification layer to obtain the first multi-dimensional dance category (see step S202 above for details on the first multi-dimensional dance category). The encoding / decoding network module encodes and decodes the fused feature information (this module includes convolutional processing and spectral norm regularization) to obtain decoded features. Finally, the classification layer in the second mapping network module... The four fully connected layers (fc) classify the decoded features to obtain the second multivariate dance category (please refer to the relevant content in step S203 above for the relevant content of the second multivariate dance category); the first loss is determined based on the difference data between the fused features and the decoded features; the second loss is determined based on the difference data between the first multivariate dance category and the second multivariate dance category; the target loss is determined based on the first loss and the second loss, and the model parameters of the initial neural network model are adjusted based on the target loss to obtain the adjusted initial neural network model; the target loss is updated based on the adjusted initial neural network model until the updated target loss meets the prediction convergence condition to obtain the trained initial neural network model.

[0072] Please see Figure 4 , Figure 4 This is a schematic diagram of a dance generation method provided in an embodiment of this application. Figure 4 As shown, inputting target music into the dance generation model can output the target dance movements for that music. This dance generation model is trained using the aforementioned model training method. It is evident that this dance generation method can generate target dance movements from target music, thereby reducing the manual cost in the target dance movement generation process. Furthermore, since the dance generation model is trained by fusing music features and motion features, it reduces the difficulty of cross-modal learning between music and motion when determining the target dance movements for target music, thus improving the quality of the target dance movements.

[0073] Please see Figure 5 , Figure 5 This is a schematic diagram of a model training device provided in an embodiment of this application. Figure 5As shown, the model training device may include, but is not limited to:

[0074] The acquisition unit 501 is used to obtain the musical features of the sample music and the motion features of the sample dance, and there is a matching relationship between the sample music and the sample dance.

[0075] Processing unit 502 is used to call the first mapping network module included in the initial neural network model, perform fusion processing on music features and motion features to obtain fused features, and process the fused features to obtain a first multi-dimensional dance category, the first multi-dimensional dance category including at least one dance category label;

[0076] The processing unit 502 is also used to call the encoding and decoding network module included in the initial neural network model to encode and decode the fused features to obtain decoded features, and call the second mapping network module included in the initial neural network model to classify the decoded features to obtain a second multi-dimensional dance category, wherein the second multi-dimensional dance category includes at least one dance category label.

[0077] The processing unit 502 is also used to determine the target loss based on the first loss between the fused features and the decoded features, and the second loss between the first multi-dimensional dance category and the second multi-dimensional dance category, and to adjust the model parameters of the initial neural network model based on the target loss to obtain a dance generation model.

[0078] In one optional implementation, the acquisition unit 501 acquires the musical features of the sample music and the motion features of the sample dance. Specifically, the processing unit 502 divides the sample music into multiple music segments based on the drum beats of the sample music and divides the sample dance into multiple dance movements, with a one-to-one correspondence between the multiple music segments and the multiple dance movements. The processing unit 502 calls the feature extraction network module included in the initial neural network model to extract features from the multiple music segments to obtain the musical features of the sample music and to extract features from the multiple dance movements to obtain the motion features of the sample dance.

[0079] In one optional implementation, the processing unit 502 calls the first mapping network module included in the initial neural network model to perform fusion processing on music features and motion features to obtain fused features, and processes the fused features to obtain a first multi-dimensional dance category. Specifically, it is used to: call the feature fusion layer of the first mapping network module to perform fusion processing on music features and motion features to obtain fused features; and call the classification layer of the first mapping network module to perform classification processing on the fused features to obtain a first multi-dimensional dance category.

[0080] In one optional implementation, the processing unit 502 calls the feature fusion layer of the first mapping network module to perform fusion processing on the music features and action features to obtain fused features. Specifically, it is used to: call the feature fusion layer of the first mapping network module to combine the music features and latent encoding to obtain target music features; the latent encoding is used to reduce the noise of the music features; and call the feature fusion layer to perform fusion processing on the target music features and action features to obtain fused features.

[0081] In one optional implementation, the processing unit 502 determines a target loss based on a first loss between the fused features and the decoded features, and a second loss between the first multi-dimensional dance category and the second multi-dimensional dance category. Specifically, it is used to: determine the first loss based on the difference data between the fused features and the decoded features; determine the second loss based on the difference data between the first multi-dimensional dance category and the second multi-dimensional dance category; and determine the target loss based on the first loss and the second loss.

[0082] In one optional implementation, the processing unit 502 determines a second loss based on the difference data between the first multi-dimensional dance category and the second multi-dimensional dance category, specifically: the processing unit 502 determines at least one difference data between the dance category labels included in the first multi-dimensional dance category and the dance category labels included in the second multi-dimensional dance category; the processing unit 502 determines the second loss based on the deviation information between the at least one difference data.

[0083] In one optional implementation, the processing unit 502 adjusts the model parameters of the initial neural network model based on the target loss to obtain a dance generation model. Specifically, it is used to: adjust the model parameters of the initial neural network model based on the target loss to obtain an adjusted initial neural network model; update the target loss based on the adjusted initial neural network model until the updated target loss meets the prediction convergence condition to obtain a trained initial neural network model; and determine the dance generation model based on the feature extraction network module and the encoding / decoding network module in the trained initial neural network model.

[0084] Optionally, embodiments of this application also provide a dance generation device, which is used to acquire target music; input the target music into a dance generation model to obtain a target dance movement output by the dance generation model, wherein the target dance movement matches the target music; wherein the dance generation model is... Figure 2 The dance generation model in the model training method shown.

[0085] It is understood that the specific implementation of each module in the model training device described in the embodiments of this application and the beneficial effects that can be achieved can be referred to the description of the foregoing method embodiments, and will not be repeated here.

[0086] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device described in this embodiment includes: a processor 601, a user interface 602, a communication interface 603, and a memory 604. The processor 601, user interface 602, communication interface 603, and memory 604 can be connected via a bus or other means; this embodiment uses a bus connection as an example.

[0087] The processor 601 (or CPU, Central Processing Unit) is the computing and control core of the electronic device. It can parse various instructions and process various data within the electronic device. For example, the CPU can parse power-on / off commands sent by an object to the electronic device and control the device to perform power-on / off operations. It can also transmit various interactive data between internal structures of the electronic device. The user interface 602 is the medium for user interaction and information exchange with the electronic device. Specifically, it can include a display screen for output and a keyboard for input. It should be noted that the keyboard can be a physical keyboard, a touchscreen virtual keyboard, or a combination of both. The communication interface 603 can optionally include standard wired interfaces and wireless interfaces (such as Wi-Fi, mobile communication interfaces, etc.), controlled by the processor 601 for sending and receiving data. The memory 604 is the storage device in the electronic device, used to store programs and data. It can be understood that the memory 604 can include the built-in memory of the electronic device, or it can include extended memory supported by the electronic device. The memory 604 provides storage space for storing the operating system of the electronic device, which may include, but is not limited to, Android, iOS, Windows Phone, etc. This application does not limit this.

[0088] In this embodiment of the application, the processor 601 performs the following operations by running the executable program code in the memory 604:

[0089] The process involves acquiring musical features from sample music and motion features from sample dance, establishing a matching relationship between the two. The first mapping network module of the initial neural network model is then invoked to fuse the musical and motion features, resulting in a fused feature. This fused feature is further processed to obtain a first multi-dimensional dance category, which includes at least one dance category label. The encoding / decoding network module of the initial neural network model is then invoked to encode and decode the fused feature, resulting in a decoded feature. The second mapping network module of the initial neural network model is then invoked to process the decoded feature, resulting in a second multi-dimensional dance category, which also includes at least one dance category label. A target loss is determined based on a first loss between the fused and decoded features, and a second loss between the first and second multi-dimensional dance categories. The model parameters of the initial neural network model are then adjusted based on the target loss to obtain the dance generation model.

[0090] In one optional implementation, the processor 601 acquires the musical features of the sample music and the motion features of the sample dance, specifically for: dividing the sample music into multiple musical segments based on the drum beats of the sample music, and dividing the sample dance into multiple dance movements, with a one-to-one correspondence between the multiple musical segments and the multiple dance movements; calling the feature extraction network module included in the initial neural network model to extract features from the multiple musical segments to obtain the musical features of the sample music, and to extract features from the multiple dance movements to obtain the motion features of the sample dance.

[0091] In one optional implementation, the processor 601 calls the first mapping network module included in the initial neural network model to perform fusion processing on music features and motion features to obtain fused features, and processes the fused features to obtain a first multi-dimensional dance category. Specifically, it is used to: call the feature fusion layer of the first mapping network module to perform fusion processing on music features and motion features to obtain fused features; and call the classification layer of the first mapping network module to perform classification processing on the fused features to obtain a first multi-dimensional dance category.

[0092] In one optional implementation, the processor 601 calls the feature fusion layer of the first mapping network module to perform fusion processing on the music features and action features to obtain fused features. Specifically, the processor 601 calls the feature fusion layer of the first mapping network module to combine the music features and latent coding to obtain target music features; the latent coding is used to reduce the noise of the music features; and the feature fusion layer is called to perform fusion processing on the target music features and action features to obtain fused features.

[0093] In one alternative implementation, the processor 601 determines a target loss based on a first loss between fused features and decoded features, and a second loss between a first multi-dimensional dance category and a second multi-dimensional dance category. Specifically, the processor 601 determines the first loss based on the difference data between the fused features and decoded features; determines the second loss based on the difference data between the first multi-dimensional dance category and the second multi-dimensional dance category; and determines the target loss based on the first loss and the second loss.

[0094] In one alternative implementation, the processor 601 determines a second loss based on the difference data between the first multi-dimensional dance category and the second multi-dimensional dance category, specifically by: determining at least one difference data between the dance category labels included in the first multi-dimensional dance category and the dance category labels included in the second multi-dimensional dance category; and determining the second loss based on the deviation information between the at least one difference data.

[0095] In one optional implementation, the processor 601 adjusts the model parameters of the initial neural network model based on the target loss to obtain a dance generation model. Specifically, the processor 601 adjusts the model parameters of the initial neural network model based on the target loss to obtain an adjusted initial neural network model; updates the target loss based on the adjusted initial neural network model until the updated target loss meets the prediction convergence condition to obtain a trained initial neural network model; and determines the dance generation model based on the feature extraction network module and the encoding / decoding network module in the trained initial neural network model.

[0096] In specific implementations, the processor 601, user interface 602, communication interface 603, and memory 604 described in the embodiments of this application can execute the implementation of the electronic device described in the model training method provided in the embodiments of this application, or they can execute the implementation of the model training device provided in the embodiments of this application, which will not be repeated here.

[0097] In this embodiment of the application, the processor 601, by running the executable program code in the memory 604, also performs the following operations:

[0098] Obtain the target music; input the target music into the dance generation model to obtain the target dance address output by the dance generation model, and the target dance address matches the target music; the dance generation model is trained by the model training method in the previous section.

[0099] In specific implementations, the processor 601, user interface 602, communication interface 603, and memory 604 described in the embodiments of this application can execute the implementation of the electronic device described in the dance generation method provided in the embodiments of this application, or the implementation of the dance generation device provided in the embodiments of this application, which will not be repeated here.

[0100] This application also provides a computer-readable storage medium storing a computer program, which includes program instructions. When the program instructions are executed by a processor, they implement the dance generation method provided in this application. For details, please refer to the implementation methods provided in the above steps, which will not be repeated here.

[0101] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the audio quality assessment method provided in this application. Specific implementation details can be found in the foregoing description and will not be repeated here.

[0102] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0103] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0104] The above-disclosed embodiments are only some of the embodiments of this application, and should not be construed as limiting the scope of this application. Therefore, any equivalent changes made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A model training method, characterized in that, The method comprises: obtaining music features of sample music and action features of sample dance, the sample music and the sample dance having a matching relationship; calling a first mapping network module included in an initial neural network model to perform fusion processing on the music features and the action features to obtain fusion features and to process the fusion features to obtain a first multi-element dance category, the first multi-element dance category including at least one dance category label; calling a coding and decoding network module included in the initial neural network model to perform coding and decoding processing on the fusion features to obtain decoding features, and calling a second mapping network module included in the initial neural network model to perform classification processing on the decoding features to obtain a second multi-element dance category, the second multi-element dance category including at least one dance category label; determining a target loss based on a first loss between the fusion features and the decoding features and a second loss between the first multi-element dance category and the second multi-element dance category, and adjusting model parameters of the initial neural network model based on the target loss to obtain a dance generation model.

2. The method of claim 1, wherein, The method comprises: dividing the sample music into a plurality of music segments and the sample dance into a plurality of dance actions based on drum points of the sample music, the plurality of music segments and the plurality of dance actions corresponding to each other in a one-to-one manner; calling a feature extraction network module included in the initial neural network model to perform feature extraction on the plurality of music segments to obtain the music features of the sample music and to perform feature extraction on the plurality of dance actions to obtain the action features of the sample dance.

3. The method of claim 1, wherein, The method comprises: calling a feature fusion layer of the first mapping network module to perform fusion processing on the music features and the action features to obtain fusion features; calling a classification layer of the first mapping network module to perform classification processing on the fusion features to obtain a first multi-element dance category.

4. The method of claim 3, wherein, The method comprises: calling a feature fusion layer included in the first mapping network module to combine the music features and latent encoding to obtain target music features; the latent encoding is used to reduce noise of the music features; the latent encoding is feature data obtained by reducing dimension or compressing original data, the original data including part of the music features; calling the feature fusion layer to perform fusion processing on the target music features and the action features to obtain fusion features.

5. The method according to any one of claims 1 to 4, characterized in that, The method comprises: determining a first loss based on difference data between the fusion features and the decoding features; determine a second loss based on difference data between the first multi-dance category and the second multi-dance category; determine a target loss based on the first loss and the second loss.

6. The method of claim 5, wherein, The determining of the second loss based on the difference data between the first multi-dance category and the second multi-dance category comprises: determining at least one difference data between each dance category label included in the first multi-dance category and each dance category label included in the second multi-dance category; determining a second loss based on deviation information between the at least one difference data.

7. The method of claim 6, wherein, The adjusting of the model parameters of the initial neural network model based on the target loss to obtain a dance generation model comprises: adjusting the model parameters of the initial neural network model based on the target loss to obtain an adjusted initial neural network model; updating the target loss based on the adjusted initial neural network model until an updated target loss meets a predicted convergence condition to obtain a trained initial neural network model; determining a dance generation model based on a feature extraction network module and a coding and decoding network module in the trained initial neural network model.

8. A dance generation method characterized by comprising: The method comprises: obtaining target music; inputting the target music into the dance generation model to obtain target dance actions output by the dance generation model, the target dance actions matching the target music; wherein the dance generation model is trained based on the model training method of any one of claims 1 to 7.

9. An electronic device, comprising: comprise: a processor, a communication interface and a memory, which are connected to each other, wherein the memory stores executable program codes, the processor is configured to invoke the executable program codes to execute the method of any one of claims 1 to 7, or execute the method of claim 8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, the computer program comprises program instructions, and the program instructions make the processor execute the method of any one of claims 1 to 7 or execute the method of claim 8 when executed by the processor.

Images