Dance generation method and device

Abstract

The embodiment of the invention provides a dance generation method. The method comprises the following steps: learning according to an input dance action to obtain a dance style signal; music features of the input music are obtained; and generating dance according to the music features and the dance style signal. According to the dance generation method provided by the embodiment of the invention, the action characteristics of the dance action can be learned, the dance of different dance styles can be generated for the same music according to the single model, and the control on the generated dance styles can be realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a dance generation method, device, computer equipment and storage medium. Background Art

[0002] Dance generation refers to the generation of dance based on the input music-driven model, thereby increasing people's entertainment methods, improving the efficiency of choreographers, and reducing choreography costs.

[0003] In the related art, a model is usually used to directly establish the relationship between music and the same dance style, and a dance of the same dance style is generated according to the input music. Since the model can only output a dance of one dance style according to the music, if it is necessary to generate dances of multiple different dance styles, it is necessary to train multiple different models to generate dances of different dance styles. A single model cannot generate dances of different dance styles for the same music; and, in the related art, only the characteristics of the input music itself are often focused on, and the movement characteristics implied in the dance movements themselves are ignored, so it is impossible to control the generated dance style. Summary of the invention

[0004] The purpose of the present application is to provide a dance generation method, apparatus, computer equipment and storage medium, which are used to solve the technical problem in the related art that a single model cannot generate dances of different dance styles based on the same music and cannot achieve control over the generated dance style.

[0005] One aspect of an embodiment of the present application provides a dance generation method, comprising: obtaining a dance style signal according to input dance movements; acquiring music features of input music; and generating a dance according to the music features and the dance style signal.

[0006] Optionally, the learning and obtaining a dance style signal according to the input dance movements includes: extracting features of the input dance movements to obtain dance movement features; and obtaining a dance style signal according to a latent space matrix and the dance movement features.

[0007] Optionally, the dance movement features include spatial features and temporal features, and the step of extracting features from the input dance movement to obtain the dance movement features includes: extracting the spatial features from the input dance movement based on a convolutional neural network, and extracting features from the input dance movement based on a long short-term memory network.

[0008] Optionally, obtaining the dance style signal according to the latent space matrix and the dance movement features includes: obtaining a low-dimensional latent vector according to the latent space matrix and the dance movement features through a multi-head self-attention mechanism, and using the latent vector as the dance style signal.

[0009] Optionally, obtaining a low-dimensional latent vector according to the latent space matrix and the dance movement features through a multi-head self-attention mechanism includes: obtaining the latent vector according to the latent space matrix and the dance movement features through a multi-head self-attention mechanism and contrastive learning loss.

[0010] Optionally, generating a dance according to the music features and the dance style signal includes: encoding the music features to obtain encoded music features; obtaining starting action features; splicing the encoded music features, the starting action features and the dance style signal to obtain a spliced ​​vector; decoding the spliced ​​vector to obtain a dance action sequence corresponding to the dance style signal.

[0011] Optionally, the music feature includes at least one of a Mel-cephalometric coefficient feature, a beat feature, a root mean square energy feature, a constant Q transform feature, and a fundamental frequency feature.

[0012] One aspect of an embodiment of the present application provides a dance generation device, including: a learning module, used to learn a dance style signal based on input dance movements; an acquisition module, used to acquire music features of input music; and a generation module, used to generate a dance based on the music features and the dance style signal.

[0013] One aspect of an embodiment of the present application provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the steps of the above-mentioned dance generation method when executing the computer program.

[0014] One aspect of an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned dance generation method.

[0015] The dance generation method, apparatus, computer device, and storage medium provided in the embodiments of the present application have the following advantages:

[0016] By learning the input dance movements to obtain the dance style signal, the music features of the input music are obtained, and the dance is generated according to the obtained music features and the learned dance style signal. The movement features of the dance movements themselves can be learned to obtain different dance style signals, so that dances of different dance styles can be generated for the same music according to a single model, and there is no need to train multiple models to generate dances of different dance styles. At the same time, since the dance style signal is obtained according to the movement features of the input dance movements themselves, the style of the generated dance movements can be clearly determined according to the dance style signal, thereby effectively realizing the control of the generated dance style, reducing the noise of the generated dance movements, and making the generated dance more realistic and diverse. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The flowchart of the dance generation method of the first embodiment of the present application is schematically shown;

[0018] Figure 2 for Figure 1 Flow chart of the sub-steps of step S110;

[0019] Figure 3 Schematic diagram of the distribution of dance style signals in contrastive learning;

[0020] Figure 4 for Figure 1 Flow chart of the sub-steps of step S130;

[0021] Figure 5 Example graphs of dances generated by dance generation methods;

[0022] Figure 6 An example diagram of the principle of the dance generation method;

[0023] Figure 7 The block diagram of the dance generation device of the second embodiment of the present application is schematically shown;

[0024] Figure 8 The hardware architecture diagram of the computer device of the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0026] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0027] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0028] The following is an explanation of the terms involved in this application:

[0029] Latent Space is a representation of compressed data. Its function is to learn data features and simplify data representation in order to find patterns. Data compression refers to encoding information with fewer bits than the original representation, such as reducing a 19-dimensional data to 9 dimensions. The purpose of data compression is to learn the more important information in the data.

[0030] Attention is a technique in artificial neural networks that mimics cognitive attention. This mechanism can increase the weight of some parts of the neural network input data while weakening the weight of other parts, so as to focus the network's attention on the most important small part of the data. Which parts of the data are more important than other parts depends on the context. The attention mechanism can be trained by gradient descent.

[0031] Self-attention is a variant of the attention mechanism that reduces dependence on external information and is better at capturing the internal correlation of data or features.

[0032] The multi-head self-attention mechanism works by implementing multiple attention modules in parallel using multiple different versions of the same query. The idea is to linearly transform the query using different weight matrices to obtain multiple queries. Each newly formed query essentially requires different types of relevant information, allowing the attention model to introduce more information in the context vector calculation.

[0033] Manifold is a topological space that can be locally Euclidean. It is a generalization of the concepts of curves and surfaces in Euclidean space.

[0034] Contrastive learning requires learning a representation learning model by automatically constructing similar instances and dissimilar instances. Through this model, similar instances are closer in the projected space, while dissimilar instances are farther apart in the projected space.

[0035] Convolutional neural network (CNN) is a feedforward neural network whose artificial neurons can respond to surrounding units within a certain coverage area and performs well in large image processing.

[0036] Long Short-Term Memory (LSTM) is a time-recurrent neural network that is specially designed to solve the long-term dependency problem of general RNNs (recurrent neural networks). All RNNs have a chain form of repeated neural network modules.

[0037] The Transformer model (literally "transformer") is a deep learning model that uses a self-attention mechanism, which can assign different weights to different parts of the input data according to their importance. This model is mainly used in the fields of natural language processing (NLP) and computer vision (CV).

[0038] Mel-frequency cepstral coefficients (MFCC) are the coefficients that make up the Mel-frequency cepstral, derived from the cepstral of an audio clip. The difference between the cepstral and the Mel-frequency cepstral is that the frequency bands of the Mel-frequency cepstral are equally spaced on the Mel scale, which better approximates the human auditory system than the linearly spaced bands used in the normal logarithmic cepstral. Such nonlinear representations can lead to better representations of sound signals in many fields, such as in audio compression.

[0039] Root-mean-square energy expresses the comprehensive information of all sample points in a frame, and can also express loudness information.

[0040] Constant Q transform refers to a filter group whose center frequencies are distributed according to an exponential law, with different filter bandwidths, but the ratio of center frequency to bandwidth is a constant Q.

[0041] Fundamental frequency (pitch), when the sound-generating body makes a sound due to vibration, the sound can generally be decomposed into many simple sine waves, that is to say, all natural sounds are basically composed of many sine waves with different frequencies, among which the sine wave with the lowest frequency is the fundamental frequency, and the other sine waves with higher frequencies are overtones. In music performance or singing, the fundamental frequency is the main element to distinguish the pitch and determine the melody. The overtones determine the timbre of the instrument or the human voice.

[0042] In the related art, only one model can be built for one dance style, and each model is only responsible for generating dances of a single dance style. A single model cannot generate dances of different dance styles for the same music, and multiple models need to be trained to generate dances of different dance styles. For example, if you need to generate a ballet-style dance, you need to train a separate model; if you need to generate a jazz-style dance, you need to train another separate model, and so on; and the related art often only focuses on the characteristics of the input music itself, ignoring the movement characteristics implied in the dance movements themselves, so it is impossible to control the generated dance style.

[0043] The dance generation method of the embodiment of the present application can generate dances of different dance styles for the same music according to a single model, and can obtain the dance style according to the movement characteristics of the dance movements themselves, thereby effectively realizing the control of the generated dance style.

[0044] It should be noted that the execution subject of the dance generation method of the embodiment of the present application can be a client or a server, wherein the client can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server can be implemented as an independent server or a server cluster composed of multiple servers. In actual application scenarios, the client can also send a request to generate a dance to the server, and after receiving the request, the server generates dances of different styles according to the request of the client.

[0045] The dance generation scheme of the embodiment of the present application will be introduced through several embodiments below. For ease of understanding, the following exemplary description will be given with the server as the execution subject.

[0046] Embodiment 1

[0047] Figure 1 The flowchart of the dance generation method of the first embodiment of the present application is schematically shown. As shown in the figure, it can include steps S110 to S130, which are specifically described as follows:

[0048] Step S110, learning and obtaining a dance style signal according to the input dance movements.

[0049] Among them, dance style can be understood as dance types, such as locking, pop dance, ballet, jazz, house dance, street dance, square dance and other dance types.

[0050] Specifically, a preset artificial neural network model can be used to learn the dance style of the input dance movements, thereby obtaining a dance style signal. During learning, a supervised method can be used, such as labeling the input dance movements with dance style labels, so that the model learns the corresponding dance style signal; an unsupervised method can also be used for learning, and the specific method is not limited here.

[0051] The input dance moves can come from dance videos of various dance styles. For example, each frame of an image is obtained from a dance video of a popular dance style, and then each frame of the image is input into the model for learning.

[0052] It should be understood that the dance style signals corresponding to the same dance style can be the same or different. For example, for ballet style, different dance video clips of ballet styles are input, and the dance style signals obtained are different. However, in the trained model, the dance style signals corresponding to the same dance style should be close or similar. In addition, in the trained model, the dance style signals corresponding to different dance styles should be obviously different, so as to facilitate the distinction of dances of different dance styles. For example, the dance style signal obtained by inputting a ballet style dance video clip should be obviously different from the dance style signal obtained by inputting a popular dance style dance video clip.

[0053] Since the dance style signals corresponding to the same dance style can be different, in practical applications, the scheme of step S110 to step S130 can be: obtain the target dance style signal according to the video clip of the input dance action, or directly sample / interpolate the target dance style signal from the trained dance style signal, and then directly control the dance style of the generated dance according to the target dance style signal. For example, a ballet dance style signal can be generated by inputting a video clip of a ballet dance, or directly sample / interpolate the ballet style signal from the trained ballet style signal, and generate a ballet style dance according to the ballet dance style signal and the input music. It can be understood that when the method of directly sampling the target dance style signal directly from the trained dance style signal is adopted, since the dance style signal sampled each time will be different, the dance generated each time will also be different, which can increase the diversity of the dances generated with the same dance style. Among them, the interpolation method refers to selecting two known dance style signals from the trained dance style signal, interpolating between the two dance style signals (vectors) to obtain a new vector, and using the new vector as the target dance style signal.

[0054] Step S120, obtaining music features of input music.

[0055] The input music is the music corresponding to the dance that needs to be generated.

[0056] In an exemplary embodiment, the music features may include Mel cepstral coefficient features, beat features, root mean square energy features, constant Q transform features, and fundamental frequency features. The specific music features obtained can be set according to actual needs, and are not specifically limited here. When obtaining the music features of the input music, each music feature can be extracted to obtain it. It is understandable that the more types of features the music features include, the more realistic and diverse the generated dance is.

[0057] Step S130: generating a dance according to the music features and the dance style signal.

[0058] After obtaining the dance style signal, a dance generator with an encoding-decoding structure can be used to generate the dance. Specifically, the acquired music features and the dance style signal can be input into the dance generator, and the dance generator can be used to generate a dance action sequence corresponding to the dance style signal. The encoder and decoder specifically used by the dance generator can be set according to actual needs, and no specific restrictions are made here.

[0059] The dance generation method of the embodiment of the present application obtains a dance style signal by learning the input dance movements, obtains the music features of the input music, and generates a dance according to the obtained music features and the learned dance style signal. The movement features of the dance movements themselves can be learned to obtain different dance style signals, so that dances of different dance styles can be generated for the same music according to a single model, and there is no need to train multiple models to generate dances of different dance styles. At the same time, since the dance style signal is obtained according to the movement features of the input dance movements themselves, the style of the generated dance movements can be clearly determined according to the dance style signal, thereby effectively realizing the control of the generated dance style, reducing the noise of the generated dance movements, and making the generated dance more realistic and diverse. In addition, since the generated dance is obtained according to the music features and the dance style signal, the same dance style signal can also be used to generate different dances of the same dance style according to different input music.

[0060] In an exemplary embodiment, Figure 2 As shown, step S110 may also include steps S111 to S112, which are specifically as follows:

[0061] Step S111, extracting features of the input dance movements to obtain dance movement features.

[0062] Optionally, the dance movement features may include spatial features and temporal features, and step S111 may specifically include: extracting spatial features of the input dance movement according to a convolutional neural network (CNN), and extracting temporal features of the input dance movement according to a long short-term memory network (LSTM). That is, CNN-LSTM may be used to extract features of the input dance movement, CNN is used to extract spatial features of each movement of the input dance movement, and LSTM is used to extract temporal features of each movement. Among them, the spatial features may include the relationship between human body joints in each dance movement; and the temporal features may include the association relationship between each dance movement in a dance sequence.

[0063] Step S112, obtaining a dance style signal according to the latent space matrix and the dance movement features.

[0064] The latent space matrix is ​​a learnable latent space matrix (denoted as W), which is obtained by learning the input dance movement samples. The basis vectors in the latent space matrix W can learn the input dance movement samples, and each basis vector can learn a basic feature of a global movement.

[0065] When obtaining the dance style signal based on the latent space matrix W and the dance movement features, the dance movement features can be similarly calculated with the basis vectors in the latent space matrix W to obtain the weight value, and then the basis vectors in W are weightedly summed with the weight value to finally obtain the style vector corresponding to the dance style signal.

[0066] When calculating the similarity of dance movement features, methods such as dot-product attention and location-based attention can be used for calculation. In an exemplary embodiment, step S112 may include: obtaining a low-dimensional latent vector through a multi-head self-attention mechanism according to the latent space matrix and the dance movement features, and using the obtained low-dimensional latent vector as a dance style signal. That is, the similarity between the dance movement features and the basis vectors in the latent space matrix W is calculated through a multi-head self-attention mechanism. Specifically, in self-attention, the dance movement features can be used as query vectors, and the decomposition space W composed of the decomposition basis vectors in the latent space matrix W will participate in the operation as both Key and Value vectors. The specific self-attention operation is shown in the following formula:

[0067]

[0068]

[0069] In the above formula, attn h represents the attention value, FC represents the fully connected operation, Indicates dance movement characteristics, d gru represents the scaling factor (which can be combined with z motion The feature dimension of FC is consistent), the superscript h of FC represents different heads, h∈{1,…,H} (there are H heads in total, that is, the self-attention mechanism of H heads, so H different full connections are needed to map the features to different mapping spaces), and the subscripts z and w are to indicate whether to process the full connection of Z or the full connection of W; s Represents the latent vector corresponding to the dance style signal, and Mean represents the average value.

[0070] It should be understood that in the operation of self-attention, When using the multi-head attention mechanism, Due to the characteristics of dance movements motion As q(query) vector, so The calculation method of K and V is similar to that of Q, so the above formula is obtained

[0071]

[0072] Compared with other methods of calculating similarity, the multi-head self-attention mechanism can calculate similarity in multiple different mapping spaces, and can obtain more effective similarity valuation, thereby making the obtained dance style signal more accurate and effective.

[0073] In this embodiment, the dance movement features are obtained by extracting features from the input dance movements, and the dance style signal is obtained according to the latent space matrix and the dance movement features. The input high-dimensional dance movement vector can be decomposed into low-dimensional manifolds in the latent space through the latent space matrix for representation, thereby removing noise in the input dance movement sequence and retaining the most important features as the dance style signal.

[0074] In an exemplary embodiment, in the above steps, a low-dimensional latent vector is obtained according to the latent space matrix and the dance movement features through a multi-head self-attention mechanism, and it can also include: obtaining a latent vector according to the latent space matrix, the dance movement features, through a multi-head self-attention mechanism and contrastive learning loss.

[0075] As mentioned above, the dance style signals corresponding to the same dance style can be diverse. Through contrastive learning, different dance style signals can form clustering characteristics in the latent space, which facilitates the selection of dance style signals and increases the diversity of dances generated in the same dance style.

[0076] Among them, the formula for contrastive learning loss is as follows:

[0077]

[0078] Among them, L C (e si ,e sj ) represents the contrastive learning loss between the latent vector corresponding to dance style i and the latent vector corresponding to dance style j, Dis represents e si , e sj This is the Euclidean distance of the hidden vector, m represents the boundary distance, which is an empirical value; in contrastive learning loss, if the hidden vector e si , e sj The corresponding dance styles are the same, then Y = 0; if the hidden vector e si , e sj If the corresponding dance styles are different, then Y=1.

[0079] Please refer to Figure 3 , which is a schematic diagram of the distribution of dance style signals in contrastive learning; Figure 3 It can be seen that by using contrastive learning, the latent vectors corresponding to the same dance style signal can be clustered together, while the latent vectors corresponding to different dance style signals are separated from each other. For example, the latent vectors corresponding to the popping dance style, locking dance style, and anime dance style represented in the figure are clustered together, while different dance styles are separated from each other.

[0080] In this embodiment, the latent vector corresponding to the dance style signal is obtained through a multi-head self-attention mechanism according to the latent space matrix and the dance movement characteristics, and the distribution of the latent vector in the latent space is controlled through contrastive learning loss, so that the latent vectors corresponding to different dance styles are as far apart as possible, and the latent vectors corresponding to the same dance style are as close as possible. Therefore, through the supervision of contrastive loss, the latent vectors corresponding to the dance style signal are clustered in the latent space according to the style label, so that the obtained latent vector can be used as a dance style signal to control the dance style of the generated dance.

[0081] In an exemplary embodiment, Figure 4 As shown, step S130 may also include steps S131 to S134, which are specifically as follows:

[0082] Step S131, encode the music features to obtain the encoded music features.

[0083] Specifically, a preset encoder may be used to encode the music feature. Exemplarily, the music feature may be encoded through multiple transformer layers to obtain the encoded music feature.

[0084] Step S132, obtaining the starting action feature.

[0085] Step S133, concatenating the encoded music features, starting action features and dance style signals to obtain a concatenated vector.

[0086] Step S134: Decode the concatenated vectors to obtain a dance action sequence corresponding to the dance style signal.

[0087] Specifically, the concatenated vector can be sent to a decoder for decoding, such as an autoregressive LSTM decoder, so as to obtain a dance action sequence corresponding to the dance style signal.

[0088] Please refer to Figure 5 , which is an example diagram of a dance generated by a dance generation method. As shown in the figure, if the dance style signal corresponds to the Anime Dance style, a dance corresponding to the Anime Dance style is generated; if the dance style signal corresponds to the Locking Dance style, a dance corresponding to the Locking Dance style is generated; if the dance style signal corresponds to the Popping Dance style, a dance corresponding to the Popping Dance style is generated.

[0089] In this embodiment, by encoding the music features to obtain the encoded music features, obtaining the starting action features, splicing the encoded music features, the starting action features and the dance style signal to obtain a spliced ​​vector, decoding the spliced ​​vector to obtain a dance action sequence corresponding to the dance style signal, and then generating a dance corresponding to the dance style signal according to the input music; by introducing different dance style signals, different styles of dance can be generated for the same music, and at the same time, the dance style signal can reduce the noise of the generated dance, making the generated dance more realistic and diverse.

[0090] In order to more clearly illustrate the dance generation method of the embodiment of the present application, a specific example is used below for illustration.

[0091] Please refer to Figure 6 , which is an example diagram of the principle of the dance generation method. As shown in the figure, it can be divided into two parts: style signal generator and dance generator. The principles of the two parts are explained below:

[0092] 1. Style signal generator part:

[0093] In the style signal generator, the input dance movements can be feature extracted (Motion Encoder) to obtain dance movement features Then, the multi-head self-attention mechanism is used to calculate the similarity with the basis vectors in the latent space matrix W to obtain the weight value attn, and the weighted sum is used to obtain the latent vector e corresponding to the dance style signal; through contrast learning loss, the latent vectors corresponding to different dance styles are made as far apart as possible, and the latent vectors corresponding to the same dance style are made as close as possible. The latent vector corresponding to the obtained dance style signal will be used in the dance generator.

[0094] 2. Dance generator part:

[0095] In the dance generator, the Mel-cephalometric coefficient features, beat features, root mean square energy features, constant Q transform features and fundamental frequency features can be extracted from the input music; then the extracted music features are encoded according to multiple Transformers; then the encoded results, the starting action and the latent vector obtained by the style signal generator are concatenated and sent to the autoregressive LSTM for decoding, and finally a dance action sequence corresponding to the dance style corresponding to the latent vector is generated.

[0096] Embodiment 2

[0097] Figure 7 The block diagram of the dance generation device 200 according to the second embodiment of the present application is schematically shown. The dance generation device 200 can be divided into one or more program modules, and one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment.

[0098] like Figure 7 As shown, the dance generation device 200 may include a learning module 210, an acquisition module 220 and a generation module 230, wherein:

[0099] A learning module 210, configured to learn and obtain a dance style signal according to input dance movements;

[0100] An acquisition module 220, used to acquire music features of input music;

[0101] The generating module 230 is used to generate a dance according to the music feature and the dance style signal.

[0102] In an exemplary embodiment, the learning module 210 is further configured to extract features from the input dance movements to obtain dance movement features; and obtain a dance style signal according to the latent space matrix and the dance movement features.

[0103] In an exemplary embodiment, the dance movement features include spatial features and temporal features, and the learning module 210 is further used to extract the spatial features of the input dance movement according to a convolutional neural network, and to extract features of the input dance movement according to a long short-term memory network.

[0104] In an exemplary embodiment, the learning module 210 is further configured to obtain a low-dimensional latent vector according to the latent space matrix and the dance movement features through a multi-head self-attention mechanism, and use the latent vector as the dance style signal.

[0105] In an exemplary embodiment, the learning module 210 is further configured to obtain the latent vector according to the latent space matrix and the dance movement features through a multi-head self-attention mechanism and contrastive learning loss.

[0106] In an exemplary embodiment, the generation module 230 is also used to encode the music feature to obtain the encoded music feature; obtain the starting action feature; splice the encoded music feature, the starting action feature and the dance style signal to obtain a spliced ​​vector; decode the spliced ​​vector to obtain a dance action sequence corresponding to the dance style signal.

[0107] In an exemplary embodiment, the music feature includes at least one of a Mel-cephalometric coefficient feature, a beat feature, a root mean square energy feature, a constant Q transform feature, and a fundamental frequency feature.

[0108] Embodiment 3

[0109] Figure 8 The hardware architecture diagram of a computer device 300 suitable for the dance generation method according to the third embodiment of the present application is schematically shown. The computer device 300 can be a device that can automatically perform numerical calculations and / or data processing according to pre-set or stored instructions. For example, it can be a rack server, a blade server, a tower server or a cabinet server (including an independent server, or a server cluster composed of multiple servers), a gateway, etc. Figure 8 As shown, the computer device 300 includes at least but is not limited to: a memory 310, a processor 320, and a network interface 330 which can be interconnected through a system bus. Among them:

[0110] The memory 310 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage module of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, referred to as SMC), a secure digital (Secure Digital, referred to as SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage module of the computer device 300 and its external storage device. In this embodiment, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as the program code of the dance generation method, etc. In addition, the memory 310 can also be used to temporarily store various types of data that have been output or are to be output.

[0111] In some embodiments, the processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 320 is generally used to control the overall operation of the computer device 300, such as performing control and processing related to data interaction or communication with the computer device 300. In this embodiment, the processor 320 is used to run the program code stored in the memory 310 or process data.

[0112] The network interface 330 may include a wireless network interface or a wired network interface, and the network interface 330 is generally used to establish a communication link between the computer device 300 and other computer devices. For example, the network interface 330 is used to connect the computer device 300 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 300 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0113] It should be pointed out that Figure 8 Only a computer device having components 310 - 330 is shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0114] In this embodiment, the dance generation method stored in the memory 310 may also be divided into one or more program modules and executed by one or more processors (processor 320 in this embodiment) to complete the embodiment of the present application.

[0115] Embodiment 4

[0116] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the dance generation method in the embodiment are implemented.

[0117] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, referred to as SMC), a secure digital (Secure Digital, referred to as SD) card, a flash card, etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the dance generation method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0118] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0119] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A dance generation method, characterized in that: include: The dance style signal is obtained based on the input dance movements; Obtaining music features of input music; A dance is generated according to the music feature and the dance style signal.

2. The dance generation method according to claim 1, characterized in that: The step of learning a dance style signal according to the input dance movements includes: Extracting features of the input dance movements to obtain dance movement features; A dance style signal is obtained according to the latent space matrix and the dance movement features.

3. The dance generation method according to claim 2, characterized in that: The dance movement features include spatial features and temporal features. The feature extraction of the input dance movement to obtain the dance movement features includes: The spatial features of the input dance movements are extracted according to a convolutional neural network, and the temporal features of the input dance movements are extracted according to a long short-term memory network.

4. The dance generation method according to claim 3, characterized in that: The step of obtaining a dance style signal according to the latent space matrix and the dance movement features includes: A low-dimensional latent vector is obtained according to the latent space matrix and the dance movement features through a multi-head self-attention mechanism, and the latent vector is used as the dance style signal.

5. The dance generation method according to claim 4, characterized in that: The method of obtaining a low-dimensional latent vector according to the latent space matrix and the dance movement features through a multi-head self-attention mechanism includes: The latent vector is obtained according to the latent space matrix and the dance movement features through a multi-head self-attention mechanism and contrastive learning loss.

6. The dance generation method according to any one of claims 1 to 5, characterized in that: Generating a dance according to the music feature and the dance style signal comprises: Encoding the music feature to obtain the encoded music feature; Get the starting action feature; splicing the encoded music feature, the starting action feature and the dance style signal to obtain a spliced ​​vector; The concatenated vectors are decoded to obtain a dance action sequence corresponding to the dance style signal.

7. The dance generation method according to any one of claims 1 to 5, characterized in that: The music feature includes at least one of a Mel-cephalogram coefficient feature, a beat feature, a root mean square energy feature, a constant Q transform feature, and a fundamental frequency feature.

8. A dance generation device, characterized in that: include: A learning module, used for learning and obtaining a dance style signal according to input dance movements; An acquisition module, used for acquiring music features of input music; A generation module is used to generate a dance according to the music feature and the dance style signal.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the processor is used to implement the steps of the dance generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by at least one processor to enable the at least one processor to perform the steps of the dance generation method according to any one of claims 1 to 7.

Images