Dance generation method and system based on deep learning model

Abstract

The invention discloses a dance generation method and system based on a deep learning model, and belongs to the field of dance choreography, and the transmission method comprises the steps: collecting a dance motion video of a dancer, extracting skeleton data of each frame through a computer vision algorithm, and converting the skeleton data into a key point sequence; a convolutional neural network is used to extract spatial features of the dance movement video, a time sequence feature of the dance movement video is extracted through a graph convolutional network, the time sequence and rhythm of the movement are captured, and time coding is performed on each movement; constructing a multi-modal deep learning model, and training the model to generate a corresponding dance action sequence according to the input dance style and music rhythm; a style control unit is added, a dance style is converted into a style vector understood by a model through a style migration network, and music and dance actions are synchronously performed by using a music rhythm prediction model; and generating a new dance action through the deep learning model. According to the method, the dynamic change and the structural relationship of the dance movement are accurately captured, and the generated movement is smoother.

Description

Technical Field

[0001] The present invention belongs to the field of dance choreography, and more specifically, particularly relates to a dance generation method and system based on a deep learning model. Background Art

[0002] In daily life, whether it is watching a dance performance or learning to dance, it is a very common thing. It can be seen that dance is gradually entering the lives of ordinary people. The existing dances are rich in both quantity and variety. However, for some people, there is no dance among the existing dances that can arouse their interest, suit their own physical fitness, and meet their needs. They are very eager to have a tailor-made dance that meets their own needs. But currently, dance choreography still belongs to a relatively niche field. There are many things to consider in dance choreography, and the choreography difficulty is great. Therefore, even professional choreographers need to spend a lot of time and energy when choreographing dances.

[0003] High-quality and fully annotated dance movement datasets are relatively limited. Most publicly available datasets cannot cover diverse dance styles, complex movements, and long time series. Collecting dance videos and performing skeleton key point extraction and annotation require professional equipment and a large amount of manpower. Different dance styles may be severely unbalanced in the dataset. For example, there is more data on popular dances, while there is less data on traditional ethnic dances. Data deviation may lead to insufficient generation ability of the model for certain dance styles and cannot cover diverse user needs. Summary of the Invention

[0004] The purpose of this part is to outline some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Simplifications or omissions may be made in this part, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this part, the abstract, and the title, and such simplifications or omissions shall not be used to limit the scope of the present invention.

[0005] In view of the above or existing problems of the dance generation method based on a deep learning model, the present invention is proposed.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: An embodiment of the present invention provides a dance generation method based on a deep learning model, including: collecting a dancer's dance action video and performing data annotation, using a computer vision algorithm to extract the skeleton data of each frame, and converting it into a key point sequence; using a convolutional neural network to extract spatial features from the dance action video, and using a graph convolutional network to extract time series features from the dance action video, capturing the timing and rhythm of the actions, and performing time encoding on each action; constructing a multi-modal deep learning model, training the model to generate a corresponding dance action sequence according to the input dance style and music rhythm; adding a style control unit during the generation process of the dance action sequence, converting the dance style into a style vector understood by the model through a style transfer network, and synchronizing the music and dance actions using an RNN-based music rhythm prediction model; and generating new dance actions through the trained deep learning model.

[0007] As a preferred solution of the dance generation method based on the deep learning model of the present invention, wherein: the step of using a computer vision algorithm to extract the skeleton data of each frame and convert it into a key point sequence includes: The computer vision algorithm identifies the human body from the image, segments the area where each human body is located, and performs feature extraction through a convolutional neural network. Its expression is as follows: where x is the input image, W i is the convolutional kernel weight, b i is the bias, ∗ is the convolutional operation, f is the activation function, and yi is the feature map output by the convolutional layer; Detect the key points of the human body through OpenPose, extract the skeleton structure of the human body in each frame, and convert the key points extracted from each frame into a sequence in chronological order. Each key point sequence includes the coordinates of multiple joints, forming a skeleton sequence connected by multiple joints; In skeleton detection, represent the key points of the human body as heatmaps. The value of each pixel in the heatmap represents the possibility of the corresponding joint. For each key point k, the generated heatmap Hk is represented in the following way: where (x k , y k ) is the true position of joint k in the image, (x, y) is a pixel position in the heatmap, and σ k is the standard deviation of the Gaussian distribution.

[0008] As a preferred solution of the dance generation method based on the deep learning model of the present invention, wherein: the step of using a graph convolutional network to extract time series features from the dance action video includes: Fuse the feature information of each node in the graph with the feature information of its neighbor nodes to update the representation of the node. The formula for graph convolution is as follows: where, H (l) is the node feature matrix of the l-th layer, A is the adjacency matrix of the graph, W (l) is the weight matrix of the l-th layer, and σ is the activation function; Each key point in each frame of the video serves as a node of the graph. According to the spatial relationship between the joints, construct the adjacency matrix of the graph. Establish edge connections between each joint and its neighboring joints. Use the joint point features of each frame of the video as the input of graph convolution. Through the graph convolution network model, incorporate the information in the time dimension into the features of each node to form a time-dependent node representation. Through the multi-layer graph convolution network structure, capture the dynamic changes of human body movements.

[0009] As a preferred solution of the dance generation method based on the deep learning model described in the present invention, wherein: capturing the timing and rhythm of the actions, and performing time encoding on each action, including: When the rhythm of the dance action changes rapidly, pay more attention. Use the attention mechanism to assign a weight to each time step and learn the relative importance of different time periods. The formula for the self-attention mechanism is as follows: where, Q, K, and V are the query, key, and value matrices respectively, representing different time steps of the time series data.

[0010] As a preferred solution of the dance generation method based on the deep learning model described in the present invention, wherein: constructing a multi-modal deep learning model, and training the model to generate corresponding dance action sequences according to the input dance style and music rhythm, including: Design a multi-modal generation model. Convert the dance style into a vector representation through the embedding layer, process the music rhythm through the fully connected layer, and extract the timing features of the rhythm. Process the audio features through the CNN network to extract deep spectral information; Use LSTM to capture the time dynamics of the dance actions to ensure that the model can understand the continuity and rhythm between different actions. Use the generative adversarial network to optimize the quality of the generated dance actions. The generated dance actions are presented in the form of skeleton data or pose vectors.

[0011] As a preferred solution of the dance generation method based on the deep learning model described in the present invention, wherein: converting the dance style into a style vector that can be understood by the model through the style transfer network, and synchronizing the music and dance actions using the RNN-based music rhythm prediction model, including: By converting the dance style into a low-dimensional style vector, if the dance style is a text description, a pre-trained language model is used to encode the text into a style vector; if the dance style is a discrete label, One-hot encoding is used, and its embedding representation is learned through a neural network; the style adjustment module controls the details of the generated actions by adjusting the hidden layer of the generator. Use Fourier transform to extract rhythm information, process the time series data of music rhythm through LSTM, synchronize the rhythm curve with the timing information of dance action generation, and adjust the speed, duration or rhythm change of the action according to the rhythm information; in the timing generation network, the style vector and the rhythm features at each time step are passed to the LSTM unit, and the LSTM generates the skeleton data of the dance action according to the input.

[0012] As a preferred solution of the dance generation method based on the deep learning model described in the present invention, wherein: generating new dance actions through the trained deep learning model includes: Input the specified dance style, convert it into a style vector through the style control unit, input the music corresponding to the dance, extract rhythm features, extract features from the music, and use Fourier transform and Mel spectrogram analysis. Divide the music into small segments, each corresponding to a segment of dance action. Use the trained style embedding network to convert the input dance style into a style vector v style , encode the music signal into a rhythm sequence r = [r1, r2,..., r T , each r i includes features such as pitch and beat intensity, and provide the action at the initial moment to the model as the starting point of the generated sequence. Combine the style vector v style and the music rhythm encoding r to form the model input x t : where, a t-1 is the action generated in the previous time step. The action generation network receives the input x t frame by frame and outputs the action vector a t for each frame. The action generation network generates a new action sequence based on the current input and historical information, and the model gradually generates a complete action sequence A = [a1, a2,..., a T , each a t represents the skeleton data of one frame of action.

[0013] A dance generation system based on a deep learning model, comprising: a video acquisition module for collecting the dance action videos of dancers, performing data annotation, extracting the skeleton data of each frame using computer vision algorithms, and converting it into a key point sequence; A feature extraction module for extracting spatial features from the dance action videos using a convolutional neural network, extracting temporal sequence features from the dance action videos through a graph convolutional network, capturing the temporality and rhythm of the actions, and performing time encoding on each action; A model construction module for constructing a multi-modal deep learning model, and training the model to generate corresponding dance action sequences according to the input dance style and music rhythm; A style and rhythm control module for adding a style control unit during the generation process of the dance action sequence, converting the dance style into a style vector understood by the model through a style transfer network, and synchronizing the music and dance actions using an RNN-based music rhythm prediction model; A dance generation module for generating new dance actions through the trained deep learning model. A computing device, the computing device comprising: At least one processor, a memory, and an input / output unit; Wherein, the memory is used for storing computer programs, and the processor is used for calling the computer programs stored in the memory to execute the steps of the dance generation method based on the deep learning model.

[0014] A computer-readable storage medium, which includes instructions that, when running on a computer, cause the computer to execute the steps of the dance generation method based on the deep learning model.

[0015] The beneficial effects of the present invention are as follows: The present invention can accurately capture the dynamic changes and structural relationships of dance actions, and the generated dance actions are more fluent and conform to the laws of human kinematics. Combining with the style transfer network, the present invention can flexibly generate action sequences of different dance styles, making the generated results more natural and realistic. The cross-modal deep learning design enables the model to generate creative dances according to different music inputs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings without creative efforts based on these drawings.

[0017] Figure 1 It is a flowchart of a dance generation method based on a deep learning model provided by an embodiment of the present invention.

[0018] Figure 2 This is a schematic structural diagram of a dance generation system based on a deep learning model provided by an embodiment of the present invention.

[0019] Figure 3 Schematically shows a schematic structural diagram of a medium according to an embodiment of the present invention.

[0020] Figure 4 Schematically shows a schematic structural diagram of a computing device according to an embodiment of the present invention.

[0021] In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed implementation manners

[0022] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following detailed description of the specific implementation manners of the present invention will be given in conjunction with the accompanying drawings of the specification.

[0023] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0024] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that can be included in at least one implementation manner of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that excludes other embodiments.

[0025] Embodiment 1 The following references Figure 1 , Figure 1 This is a flowchart of a dance generation method based on a deep learning model provided by an embodiment of the present invention. It should be noted that the implementation manner of the present invention can be applied to any applicable scenario.

[0026] Figure 1 The process of the dance generation method based on a deep learning model provided by an embodiment of the present invention shown in S1: Collect the dance action videos of the dancer and perform data annotation, use computer vision algorithms to extract the skeleton data of each frame, and convert it into a key point sequence.

[0027] Preferably, the computer vision algorithm recognizes the human body from the image, segments the area where each human body is located, and performs feature extraction through a convolutional neural network. Its expression is as follows: Among them, x is the input image, and W i is the convolutional kernel weight, b i is the bias, ∗ is the convolution operation, f is the activation function, and yi is the feature map output by the convolutional layer; Detect the key points of the human body through OpenPose, extract the skeleton structure of the human body in each frame, and convert the key points extracted in each frame into a sequence in chronological order. Each key point sequence includes the coordinates of multiple joints, forming a skeleton sequence connected by multiple joints; In skeleton detection, represent the key points of the human body as heatmaps. The value of each pixel in the heatmap represents the probability of the corresponding joint. For each key point k, the generated heatmap Hk is represented as follows: where (x k , y k ) is the true position of joint k in the image, (x, y) is a pixel position in the heatmap, and σ k is the standard deviation of the Gaussian distribution.

[0028] Furthermore, each key point is represented by its two-dimensional coordinates (x, y), or the depth dimension z is added in the depth camera to represent the three-dimensional coordinates (x, y, z). For each frame, the skeleton of the human body can be represented by the coordinates of a set of key points: Collect the skeleton key points of each frame in chronological order of the video frames to form a time series: where T is the total number of video frames; Each key point sequence describes the trajectory of a specific joint: where J i represents the movement trajectory of the i-th joint in the entire video.

[0029] Furthermore, taking a certain key point of the human body (such as the neck) as a reference point, centralize the coordinates of all key points of the skeleton: The skeleton sequence is represented by a matrix. Each frame of the skeleton can be represented as an N×3 matrix: where N is the number of key points; The skeleton sequence of the entire video can be represented as a tensor T×N×3, where T is the number of frames, N is the number of key points, and 3 is the coordinate dimension of each key point.

[0030] S2: Extract spatial features from the dance movement video using a convolutional neural network, and extract temporal sequence features from the dance movement video through a graph convolutional network to capture the timing and rhythm of the movements and perform temporal encoding on each movement.

[0031] Preferably, fuse the feature information of each node in the graph with the feature information of its neighbor nodes to update the representation of the node. The formula for graph convolution is: where, H (l) is the node feature matrix of the l-th layer, A is the adjacency matrix of the graph, W (l) is the weight matrix of the l-th layer, and σ is the activation function; Each key point in each frame of the video serves as a node of the graph. According to the spatial relationship between the joints, construct the adjacency matrix of the graph, establish edge connections between each joint and its neighboring joints, use the joint point features of each frame of the video as the input of graph convolution, and incorporate the information in the time dimension into the features of each node through the graph convolutional network model to form a time-dependent node representation. Through the multi-layer graph convolutional network structure, capture the dynamic changes of human movement.

[0032] Preferably, when the rhythm of the dance movement changes rapidly, give more attention, use the attention mechanism, assign a weight to each time step, and learn the relative importance of different time periods. The formula for the self-attention mechanism is as follows: where, Q, K, and V are the query, key, and value matrices respectively, representing different time steps of the time series data.

[0033] Furthermore, the human skeleton key points in each frame of the video are represented as an undirected graph G=(V, E), where the node V is the key point of the human skeleton (such as the 18 joint points defined by the COCO model), and the edge E is the connection relationship between the key points; Use an adjacency matrix A to represent the connection relationship between the joint points: If the human skeleton contains N = 18 joint points, the size of the adjacency matrix A is 18×18; The initial feature of each node vi∈V is its three-dimensional coordinates (xi, yi, zi): If it is a multi-frame video, stack it in the time dimension to form a tensor F0∈R T×N×D where, T is the number of frames, N is the number of key points, and D is the feature dimension of each key point; To capture the time-dependence of motion, a time-based graph convolution operation is designed to increase the information in the time dimension and construct cross-frame connections: A t Combined the connection relationships within and between time frames.

[0034] S3: Construct a multi-modal deep learning model and train the model to generate corresponding dance movement sequences according to the input dance style and music rhythm.

[0035] Preferably, a multi-modal generation model is designed. The dance style is converted into a vector representation through an embedding layer, the music rhythm is processed through a fully connected layer, and the temporal features of the rhythm are extracted. The audio features are processed through a CNN network to extract deep spectral information; Use LSTM to capture the time dynamics of dance movements to ensure that the model can understand the continuity and rhythm between different movements. Use a generative adversarial network to optimize the quality of the generated dance movements, and the generated dance movements are presented in the form of skeleton data or pose vectors.

[0036] Furthermore, a multi-modal input, dance style embedding layer: Input the user-specified dance style, discretize the dance style into classification labels {s1, s2, …, sk}, and use the embedding layer Embedding(S) to convert the label into a vector v s ∈R d ; Music rhythm processing module: Input the beat features of the music, use a fully connected layer to extract the rhythm feature f r , use an RNN (LSTM / GRU)-based time series processing model to extract the temporal features h r , and output Hr = [hr1, hr2, …, hrT], representing the time series features of the music rhythm.

[0037] Furthermore, to capture the time dynamics and rhythm information of the dance movement sequence, input the preprocessed dance skeleton sequence X = {x1, x2, …, xT}, where x t ∈R d represents the skeleton joint coordinates of the t-th frame. Use a single-layer or multi-layer LSTM, where ht represents the hidden state at the t-th moment, and the last-layer hidden state H = {h1, h2, …, hT} represents the time dynamic features of the movement sequence.

[0038] S4: Add a style control unit during the generation process of the dance movement sequence, convert the dance style into a style vector understood by the model through a style transfer network, and synchronize the music and dance movements using an RNN-based music rhythm prediction model.

[0039] Preferably, by converting the dance style into a low-dimensional style vector, if the dance style is a text description, a pre-trained language model is used to encode the text into a style vector; if the dance style is a discrete label, one-hot encoding is used, and its embedding representation is learned through a neural network; the style adjustment module controls the details of the generated actions by adjusting the hidden layer of the generator. The Fourier transform is used to extract rhythm information, and the time series data of the music rhythm is processed by LSTM to synchronize the rhythm curve with the timing information of the generated dance movements, and the speed, duration or rhythm change of the movements is adjusted according to the rhythm information; in the temporal generation network, the style vector and the rhythm features at each time step are passed to the LSTM unit, and the LSTM generates the skeleton data of the dance movements according to the input.

[0040] Furthermore, the input music audio x(t) is decomposed into a time-frequency domain signal through short-time Fourier transform: where w(n) is the window function; The spectral energy E(f) is calculated, and the energy change in a specific frequency range (such as 0.5 - 4 Hz, corresponding to common music beats) is extracted to generate the rhythm feature M t , for the rhythm curve M t is normalized so that its range is within [0,1].

[0041] According to the intensity change of the rhythm feature M t , the time interval of the generated frames is dynamically adjusted: A time window is used to smooth the rhythm feature, capture sudden rhythm changes (such as crescendo, diminuendo), and smoothly adjust the movement amplitude.

[0042] The rhythm feature Mt of the input test audio and the dance style V are input style , the skeleton data S0 is initialized, and the skeleton sequence {St} is generated frame by frame through the LSTM model t=1 T .

[0043] S5: Generate new dance movements through the trained deep learning model.

[0044] Preferably, a specified dance style is input, converted into a style vector through the style control unit, the music corresponding to the dance is input, the rhythm feature is extracted, features are extracted from the music, the Fourier transform and Mel spectrum analysis are used to divide the music into small segments, and each segment corresponds to a dance movement; the trained style embedding network is used to convert the input dance style into the style vector v style, encode the music signal into a rhythm sequence r = [r1, r2,..., r T , where each r i includes features such as pitch and beat intensity, and provide the initial moment action to the model as the starting point of the generated sequence; Merge the style vector v style with the music rhythm encoding r to form the model input x t : where a t-1 is the action generated in the previous time step; The action generation network receives the input x frame by frame t , and outputs the action vector a for each frame t . The action generation network generates a new action sequence based on the current input and historical information, and the model gradually generates a complete action sequence A = [a1, a2,..., a T , where each a t represents the skeleton data of one frame of action.

[0045] Furthermore, for different dance styles C, an embedding layer E is established to convert C into a low-dimensional vector representation V style . The style vector V style is used to guide the motion features in the action generation model, including action amplitude, frequency, and space utilization. Calculate the Mel spectrogram through the Mel filter bank to obtain the Mel spectrogram S(t, m), which is used to capture the deep features of the audio. According to the peaks of the rhythm curve R t , divide the music into several segments, each segment with a duration L, and the corresponding rhythm feature R i and the spectrogram feature S i are used as the input for generating actions.

[0046] After introducing the method of the exemplary embodiment of the present invention, next, refer to Figure 2 to describe an exemplary embodiment of the present invention of a dance generation system based on a deep learning model. The system includes: A video acquisition module for acquiring the dance action video of the dancer and performing data annotation, using computer vision algorithms to extract the skeleton data of each frame and convert it into a key point sequence; A feature extraction module for using a convolutional neural network to extract spatial features from the dance action video, and using a graph convolutional network to extract time series features from the dance action video, capturing the timing and rhythm of the actions, and performing time encoding on each action; A model construction module for constructing a multi-modal deep learning model and training the model to generate a corresponding dance action sequence according to the input dance style and music rhythm; A style rhythm control module is used to add a style control unit during the generation process of a dance movement sequence. It converts the dance style into a style vector that can be understood by the model through a style transfer network, and synchronizes the music and dance movements using an RNN-based music rhythm prediction model. A dance generation module is used to generate new dance movements through a trained deep learning model.

[0047] After introducing the methods and systems of the exemplary embodiments of the present invention, next, refer to Figure 3 The computer-readable storage medium of the exemplary embodiments of the present invention will be described. Please refer to Figure 3 , which shows that the computer-readable storage medium is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will implement the steps recorded in the above method embodiments. For example, it collects the dance movement videos of dancers and performs data annotation, uses computer vision algorithms to extract the skeleton data of each frame, and converts it into a key point sequence; uses a convolutional neural network to extract spatial features from the dance movement videos, and uses a graph convolutional network to extract time series features from the dance movement videos to capture the timing and rhythm of the movements, and performs time encoding on each movement; constructs a multi-modal deep learning model, trains the model to generate corresponding dance movement sequences according to the input dance style and music rhythm; adds a style control unit during the generation process of the dance movement sequence, converts the dance style into a style vector that can be understood by the model through a style transfer network, and synchronizes the music and dance movements using an RNN-based music rhythm prediction model; generates new dance movements through a trained deep learning model; the specific implementation methods of each step will not be repeated here.

[0048] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other optical and magnetic storage media, which will not be elaborated one by one here.

[0049] After introducing the methods, systems, and media of the exemplary embodiments of the present invention, next, refer to Figure 4 The computing device for dance generation based on a deep learning model of the exemplary embodiments of the present invention.

[0050] Figure 4 The block diagram of an exemplary computing device 40 suitable for implementing the embodiments of the present invention is shown. The computing device 40 may be a computer system or a server. Figure 4The illustrated computing device 40 is only an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0051] As Figure 4 shown, the components of the computing device 40 may include, but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 that couples different system components (including the system memory 402 and the processing unit 401).

[0052] The computing device 40 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computing device 40, including volatile and non-volatile media, removable and non-removable media.

[0053] The system memory 402 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 4021 and / or cache memory 4022. The computing device 40 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a ROM 4023 may be used to read and write on a non-removable, non-volatile magnetic medium ( Figure 4 not shown, typically referred to as a "hard disk drive"). Although not shown in Figure 4 the figure, a disk drive for reading and writing on a removable non-volatile disk (such as a "floppy disk"), and an optical disk drive for reading and writing on a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to the bus 403 via one or more data media interfaces. The system memory 402 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0054] A program / utility 4025 having a set (at least one) of program modules 4024 may be stored, for example, in the system memory 402, and such program modules 4024 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data, and an implementation of a network environment may be included in each or some combination of these examples. The program modules 4024 generally perform the functions and / or methods described in the embodiments of the present invention.

[0055] The computing device 40 can also communicate with one or more external devices 404 (such as a keyboard, a pointing device, a display, etc.). This communication can be carried out through the input / output (I / O) interface 405. Moreover, the computing device 40 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through the network adapter 406. As Figure 4 shown, the network adapter 406 communicates with other modules (such as the processing unit 401, etc.) of the computing device 40 through the bus 403. It should be understood that although Figure 4 not shown in the figure, other hardware and / or software modules can be used in combination with the computing device 40.

[0056] The processing unit 401 executes various functional applications and data processing by running programs stored in the system memory 402. For example, it acquires the dance action video of the dancer and performs data annotation, uses computer vision algorithms to extract the skeleton data of each frame and converts it into a key point sequence; uses a convolutional neural network to extract spatial features from the dance action video, and uses a graph convolutional network to extract time series features from the dance action video to capture the timing and rhythm of the actions and perform time encoding on each action; constructs a multi-modal deep learning model, trains the model to generate a corresponding dance action sequence according to the input dance style and music rhythm; adds a style control unit during the generation of the dance action sequence, converts the dance style into a style vector understood by the model through a style transfer network, and synchronizes the music and dance actions using an RNN-based music rhythm prediction model; generates new dance actions through the trained deep learning model. The specific implementation methods of each step will not be repeated here. It should be noted that although several units / modules or sub-units / sub-modules of the synchronous escape wiring device based on multi-commodity flow are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to the embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0057] In the description of the present invention, it should be noted that the terms "first", "second", "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0058] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0059] In several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some communication interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0060] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0061] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0062] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0063] Finally, it should be noted that the above-described embodiments are only specific embodiments of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions described in the foregoing embodiments, or can easily conceive of changes, or make equivalent replacements for some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

[0064] In addition, although the operations of the method of the present invention are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

Claims

1. A dance generation method based on a deep learning model, characterized in that: include: Collect dancers’ dance action videos and annotate the data, use computer vision algorithms to extract skeleton data of each frame, and convert it into a sequence of key points; Use convolutional neural networks to extract spatial features from dance action videos, and use graph convolutional networks to extract time series features from dance action videos, capture the timing and rhythm of the movements, and perform time encoding on each movement. Build a multimodal deep learning model and train the model to generate corresponding dance movement sequences based on the input dance style and music rhythm; A style control unit is added during the generation of dance action sequences. The dance style is converted into a style vector that can be understood by the model through a style transfer network. The music rhythm prediction model based on RNN is used to synchronize the music and dance actions. Generate new dance moves through the trained deep learning model.

2. The dance generation method based on the deep learning model as claimed in claim 1, characterized in that: The method of using a computer vision algorithm to extract skeleton data of each frame and converting it into a key point sequence includes: The computer vision algorithm recognizes the human body from the image and segments the area where each human body is located. The convolutional neural network is used to extract features. The expression is as follows: Where x is the input image, W i is the convolution kernel weight, b i is the bias, ∗ is the convolution operation, f is the activation function, and yi is the feature map output by the convolution layer; OpenPose is used to detect the key points of the human body and extract the skeleton structure of the human body in each frame. The key points extracted from each frame are converted into a sequence in chronological order. Each key point sequence includes the coordinates of multiple joints, forming a skeleton sequence connected by multiple joints. In skeleton detection, the key points of the human body are represented as heat maps. The value of each pixel in the heat map represents the possibility of the corresponding joint. For each key point k, the generated heat map Hk is represented as follows: Among them, (x k ,y k ) is the true position of joint k in the image, (x,y) is a pixel position in the heat map, σ k is the standard deviation of the Gaussian distribution.

3. The dance generation method based on the deep learning model as claimed in claim 1, characterized in that: The method of extracting time series features from a dance action video by using a graph convolutional network includes: The feature information of each node in the graph is fused with the feature information of its neighboring nodes to update the representation of the node. The formula for graph convolution is: Among them, H (l) is the node feature matrix of the lth layer, A is the adjacency matrix of the graph, W (l) is the weight matrix of the lth layer, σ is the activation function; Each key point in each frame of video is taken as a node of the graph. According to the spatial relationship between the joints, the adjacency matrix of the graph is constructed. Edge connections are established between each joint and its adjacent joints. The joint point features of each frame of video are used as the input of the graph convolution. The information of the time dimension is integrated into the features of each node through the graph convolution network model to form a time-dependent node representation. The dynamic changes of human body movement are captured through the multi-layer graph convolution network structure.

4. The dance generation method based on the deep learning model as claimed in claim 1, characterized in that: The capturing of the timing and rhythm of the actions and time coding of each action include: When the dance moves change at a fast pace, more attention is paid to them. The attention mechanism is used to assign a weight to each time step and learn the relative importance of different time periods. The formula of the self-attention mechanism is as follows: Among them, Q, K, and V are query, key, and value matrices, respectively, representing different time steps of the time series data.

5. The dance generation method based on deep learning model as claimed in claim 1, characterized in that: The multimodal deep learning model is constructed, and the training model generates corresponding dance action sequences according to the input dance style and music rhythm, including: Design a multimodal generative model, convert the dance style into a vector representation through the embedding layer, process the music rhythm through the fully connected layer and extract the temporal features of the rhythm, process the audio features through the CNN network, and extract deep spectrum information; LSTM is used to capture the temporal dynamics of dance movements, ensuring that the model can understand the continuity and rhythm between different movements. A generative adversarial network is used to optimize the quality of the generated dance movements, which are presented in the form of skeleton data or pose vectors.

6. The dance generation method based on deep learning model as claimed in claim 1, characterized in that: The dance style is converted into a style vector understood by the model through a style transfer network, and the music and dance movements are synchronized using an RNN-based music rhythm prediction model, including: By converting the dance style into a low-dimensional style vector, if the dance style is a text description, the pre-trained language model is used to encode the text into a style vector; if the dance style is a discrete label, one-hot encoding is used and its embedding representation is learned through a neural network; the style adjustment module controls the details of the generated action by adjusting the hidden layer of the generator; Fourier transform is used to extract rhythm information, and LSTM is used to process the time series data of music rhythm. The rhythm curve is synchronized with the timing information generated by dance movements, and the speed, duration or rhythm changes of the movements are adjusted according to the rhythm information. In the timing generation network, the style vector is passed to the LSTM unit together with the rhythm features of each time step, and the LSTM generates the skeleton data of the dance movement based on the input.

7. The dance generation method based on deep learning model as claimed in claim 1, characterized in that: The method of generating new dance moves through the trained deep learning model includes: Input the specified dance style, convert it into a style vector through the style control unit, input the music corresponding to the dance, extract the rhythm features, extract features from the music, use Fourier transform and Mel spectrum analysis, Divide the music into small sections, each corresponding to a dance move; Use the trained style embedding network to convert the input dance style into a style vector v style , encode the music signal into a rhythm sequence r=[r1,r2,...,r T ], each r i Including features such as pitch and beat intensity, providing the initial moment action to the model as the starting point for generating the sequence; The style vector v style Combined with the music rhythm code r to form the model input x t : Among them, a t-1 is the action generated in the previous time step; The action generation network receives input x frame by frame t , output the action vector a of each frame t , the action generation network generates a new action sequence based on the current input and historical information, and the model gradually generates a complete action sequence A=[a1,a2,...,a T ], each a t Skeleton data representing one frame of motion.

8. A dance generation system based on a deep learning model, characterized in that: include: The video acquisition module is used to collect dancers’ dance movement videos and annotate the data. It uses computer vision algorithms to extract skeleton data of each frame and convert it into a sequence of key points. The feature extraction module is used to extract spatial features from dance action videos using convolutional neural networks, extract time series features from dance action videos through graph convolutional networks, capture the timing and rhythm of actions, and perform time encoding on each action; The model building module is used to build a multimodal deep learning model and train the model to generate corresponding dance movement sequences based on the input dance style and music rhythm; The style rhythm control module is used to add a style control unit to the dance action sequence generation process, convert the dance style into a style vector understood by the model through the style transfer network, and use the RNN-based music rhythm prediction model to synchronize the music and dance actions; The dance generation module is used to generate new dance moves through the trained deep learning model.

9. A computing device, comprising: at least one processor, memory, and input-output unit; The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the steps of the dance generation method based on a deep learning model as described in any one of claims 1 to 7.

10. A computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the steps of the dance generation method based on a deep learning model as described in any one of claims 1 to 7.

Images