Method for generating hand posture action sequence by music based on diffusion model

Abstract

The invention discloses a method for generating a hand posture action sequence by music based on a diffusion model. The method comprises the following steps: firstly, performing feature extraction on an audio through a pre-trained audio encoder to obtain a feature vector corresponding to the audio; learning position codes based on audio features, and obtaining a corresponding hand position through a hand position predictor; a time step code and a hand posture decoder are introduced, a diffusion model is utilized, and the relation between audio features and hand postures is learned in the noise adding and denoising process. According to the method, through the steps of audio feature coding, hand position decoding, posture embedding and the like, generation of a sequence from music to hand posture actions is achieved, hand actions which are synchronous with the music and have rich details can be generated, and the method is efficient and has good robustness.

Description

Technical Field

[0001] The present invention relates to a method for generating hand playing postures driven by piano audio in the field of generation, and in particular to a method for generating hand posture action sequences from music based on a diffusion model. Background Art

[0002] In the field of computer vision and human-computer interaction, hand gesture generation has always been an important research direction. Its applications cover many fields such as virtual reality, augmented reality, sign language translation, and human-computer interaction. Due to the high degree of freedom and complex movement patterns of the hands, generating accurate and natural hand movements has become a challenging task. Traditional motion capture-based methods usually rely on expensive hardware equipment and complex preliminary work, making them difficult to be widely used.

[0003] With the rapid development of deep learning, especially the emergence of generative adversarial networks (GANs) and variational autoencoders (VAEs), data-driven methods have made some progress in generating hand gestures. These methods learn the distribution of hand movements through a large amount of training data, so that they can generate relatively natural action sequences. However, these methods still face some problems. For example, the generated action sequences may lack temporal correlation with the input signal (such as music, speech), resulting in a lack of synchronization between the action and the audio. In addition, since the generation process of these models lacks modeling of long-range contextual dependencies, the generated action sequences often lack details and changes.

[0004] Recently, diffusion models, as a new generative model, have been proposed for image generation and have made breakthrough progress. Diffusion models gradually generate target data by defining a recursive process from simple noise distribution to complex data distribution. Its superior generation ability and robustness enable it to perform well in tasks such as audio generation and image restoration. However, its application in the field of action generation, especially in generating hand gesture action sequences based on music, has not been fully explored.

[0005] Generating hand gesture sequences based on music has important practical value. In the field of virtual reality and digital entertainment, it can improve the naturalness and interactivity of virtual avatars; in sign language translation, audio signals can be converted into sign language movements to facilitate hearing-impaired people to obtain information. Therefore, studying a method for generating hand gesture sequences from music based on a diffusion model can not only make full use of the generation ability of the diffusion model, but also achieve a high degree of synchronization between music and hand movements, which has important practical significance and application prospects. Summary of the invention

[0006] In order to generate natural hand gesture sequences from music using a diffusion model, and solve the problems of difficulty in synchronizing hand gesture generation with music and lack of details in the generated movements in the existing methods, the present invention proposes a method for generating hand gesture sequences from music based on a diffusion model. The method completes feature extraction and temporal correlation modeling of music signals through steps such as audio feature encoding, hand position decoding, and gesture embedding, and realizes the generation of hand gesture sequences from music. The generated hand gesture sequences are natural and smooth. The present invention is efficient, robust, and easy to implement, and can generate hand movements that are synchronized with music and have rich details.

[0007] The technical contents of the present invention are as follows:

[0008] 1. A method for generating hand gesture sequences from music based on diffusion model

[0009] S1: obtaining a piano playing video containing hand movements, separating an audio signal from the piano playing video, and extracting a hand position sequence and a hand posture sequence from the piano playing video to form a training data set;

[0010] S2: extracting audio features from the audio signals in the training data set to obtain audio features, and training the position generation module using the extracted audio features and the hand position sequence in the training data set to obtain a hand position predictor;

[0011] S3: After training the diffusion model according to the extracted audio features and the hand position sequence and hand posture sequence in the training data set, a trained diffusion model is obtained;

[0012] S4: After the actual audio signal is input into the hand position predictor, a predicted hand position sequence is obtained, and after the audio feature extraction is performed on the actual audio signal, the actual audio feature is obtained. Then, the predicted hand position sequence and the actual audio feature are input into the trained diffusion model, and the model outputs a predicted posture sequence.

[0013] In S2, a pre-trained audio feature encoder is used to extract audio features from the audio signal to obtain audio features.

[0014] The pre-trained audio feature encoder is wav2vec2.

[0015] In S2, after training the position generation module using the extracted audio features and the hand position sequence in the training data set, a hand position predictor is obtained, which specifically includes:

[0016] The position generation module includes a hand position decoder and a linear mapping layer. First, the hand position decoder is used to embed and map the extracted audio features into potential position features with richer temporal information to obtain potential features in the position space. The linear mapping layer is then used to perform dimension mapping on the potential features in the position space to obtain the position information of the left and right hands respectively. The hand position decoder is iteratively trained according to the loss between the position information of the left and right hands and the hand position sequence until the training is completed. The trained position generation module is obtained and recorded as the hand position predictor.

[0017] The S3 is specifically:

[0018] The diffusion model includes the noise addition process and the noise removal process;

[0019] For the noise adding process, during the training process, a time step t is randomly sampled from the range of [0, N], where N is a hyperparameter, t Gaussian noise matrices are sampled, and the t noise matrices are sequentially superimposed on the input hand posture sequence to obtain the hand posture sequence after t noise additions;

[0020] For the denoising process, the input of the denoising neural network consists of two parts: noise input and conditional input. For a time step t, the noise input is the hand posture sequence after t times of noise addition. The conditional input is composed of the hand position sequence and the audio features concatenated through the output of the posture decoder. The noise input and the conditional input are concatenated and used as the input of the denoising neural network. The hand posture sequence before the tth noise addition is calculated by predicting the noise matrix after the tth noise addition, and the training of the posture decoder and the denoising neural network is completed to obtain the trained diffusion model.

[0021] In S4, the predicted hand position sequence and the actual audio features are input into the trained diffusion model, and the model outputs a predicted posture sequence, specifically including:

[0022] The actual audio features are input into the posture decoder in the trained diffusion model to obtain the posture features. The posture features are concatenated with the predicted hand position sequence as the conditional input of the denoising neural network. The noise input of the denoising neural network is a directly sampled Gaussian noise matrix. The conditional input and the noise input are concatenated to form the input of the denoising neural network. The output of the neural network is a denoised hand posture sequence and serves as the noise input for the next denoising. The conditional input and the updated noise input are concatenated to form the input of the next denoising of the denoising neural network, and the hand movement sequence is obtained after removing the noise N times.

[0023] In S3, the loss function of the diffusion model includes the position loss and speed loss The specific formula is as follows:

[0024]

[0025] in, is the L1 loss between the predicted pose sequence and the pose sequence in the dataset, is the predicted pose sequence of the model, P is the true pose sequence in the dataset, and are respectively the pose sequence predicted by the model and the pose sequence predicted by the model one frame later, P n and P n-1 They are the pose sequence in the dataset and the pose sequence in the dataset lagged by one frame, respectively, indicating the change of the pose sequence in the time dimension, and ‖‖2 is the L2 loss.

[0026] 2. A computer device

[0027] The device comprises a memory and a processor, wherein the memory stores a computer program, and the steps of the method are implemented when the processor executes the computer program.

[0028] 3. A computer-readable storage medium

[0029] The storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0030] IV. A COMPUTER PROGRAM PRODUCT

[0031] The product comprises a computer program / instructions which, when executed by a processor, implement the steps of the method.

[0032] The beneficial effects of the present invention are:

[0033] By adopting the above technical solution, the present invention can efficiently generate natural hand gesture sequences from music. The diffusion model is used to generate hand movements that are highly synchronized with music and rich in details, while having high robustness and efficiency. Without relying on a large number of high-precision data sets, the method directly generates high-quality hand gesture sequences, which has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is an overall flow chart of a method for generating a hand gesture action sequence from music based on a diffusion model according to an embodiment of the present invention.

[0035] Figure 2 It is a visualization solution process and a display gradient propagation route of an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the above-mentioned objects, features and advantages of the present invention simpler and easier to understand, the present invention is described in detail below in conjunction with the drawings in the embodiments of the present invention, and the technical solutions in the embodiments of the present invention are clearly and completely described, but it does not constitute a limitation of the present invention.

[0037] The hardware platform of this embodiment uses an Intel i9-12900X CPU and an NVIDIA GeForce GTX3090Ti graphics card. The system program is written in Python and uses pytorch, numpy, and librosa libraries.

[0038] like Figure 1 and Figure 2 As shown, the embodiments of the present invention and their implementation process are as follows:

[0039] S1: Use a camera to capture and obtain a piano playing video containing hand movements. The sampling rate of the piano playing video is 16kHz, and the specific parameters of the hand position are the hand position P and the hand posture Θ. Separate the audio signal from the piano playing video, and extract the hand position sequence and posture sequence from the piano playing video, so that the training data set is composed of the audio signal, the hand position sequence and the posture sequence;

[0040] S2: extracting audio features from the audio signals in the training data set to obtain audio features, and training the position generation module using the extracted audio features and the hand position sequence in the training data set to obtain a hand position predictor;

[0041] In S2, a pre-trained audio feature encoder is used to extract audio features from the audio signal to obtain audio features.

[0042] The pre-trained audio feature encoder is wav2vec2. wav2vec2 takes the original audio as input, first performs preliminary audio feature extraction through a convolutional neural network, and then uses the Transformer encoder structure to aggregate primary audio features, utilize the long-term dependent characteristics, and utilize the contextual relationship of audio features. The two are combined to form an audio feature encoder. The formula is as follows:

[0043]

[0044] Among them, f a is the audio feature, Φ a It is wav2vec2, A is the input audio, N is the number of frames corresponding to the audio, and C is the dimension of the audio feature.

[0045] In S2, after training the position generation module using the extracted audio features and the hand position sequence in the training data set, a hand position predictor is obtained, which specifically includes:

[0046] The position generation module includes a hand position decoder and a linear mapping layer. First, the position is represented by the three-dimensional coordinates of both hands, with a total of six dimensions. Since it is difficult to obtain the hand position by mapping audio features, the hand position decoder is used to embed the extracted audio features into potential position features with richer time information to obtain potential features in the position space. The linear mapping layer is then used to perform dimension mapping on the potential features in the position space to obtain the position information of the left and right hands respectively. The formula is as follows:

[0047]

[0048] Among them, f p is the potential feature in the position space, Φ p () is the hand position decoder, P is the coordinates of the two hands, Φ linear () is a linear mapping layer that maps the potential vector in the position space to the position coordinates of both hands.

[0049] Then, the hand position decoder is iteratively trained according to the loss between the position information of the left and right hands and the hand position sequence until the training is completed, and the trained position generation module is obtained and recorded as the hand position predictor. In the next stage of training, its parameters are frozen as the hand position generation module, which together with the audio features and hand postures constitutes the input of the next stage.

[0050] S3: After training the diffusion model according to the extracted audio features and the hand position sequence and posture sequence in the training data set, a trained diffusion model is obtained;

[0051] S3 is specifically:

[0052] For the denoising process, first predefine a total number of denoising and denoising as a hyperparameter N. The number of denoising and denoising is referred to as the time step below. During the training process, a time step t is randomly sampled from the range of [0, N]. N is a hyperparameter, specifically the total number of denoising and denoising. The input of the denoising process is the hand posture sequence in the training data set. T Gaussian noise matrices are sampled. The size of the Gaussian noise matrix is ​​consistent with the size of the input hand posture sequence. The t noise matrices are sequentially superimposed on the input hand posture sequence to obtain the hand posture sequence after t times of denoising, which will be used as part of the input of the denoising neural network later.

[0053] For the denoising process, the diffusion model uses a neural network for denoising, that is, the hand posture sequence after noise is input, and the hand posture sequence before noise is predicted by the neural network. The specific process is as follows: the input of the denoising neural network consists of two parts: noise input and conditional input. For a time step t, the noise input is the hand posture sequence after t times of noise, and the conditional input is composed of the hand position sequence and the audio features spliced ​​by the output of the posture decoder. The posture decoder is specifically a decoder based on the Transformer model. After splicing the noise input and the conditional input, it is used as the input of the denoising neural network, where the conditional input is independent of the time step. The denoising neural network is a neural network model based on U-Net, which is composed of a convolutional neural network, combined with downsampling and upsampling. By predicting the noise matrix of the tth noise, the hand posture sequence before the tth noise is calculated, that is, the hand posture sequence after t-1 times of noise is used as the output of the denoising neural network, and the posture decoder and the denoising neural network are trained to obtain a trained diffusion model.

[0054] More specifically, the diffusion model can effectively find the distribution space of complex hand movements in the data set through the process of adding noise and denoising. During the training process, the initial posture sequence input is the true value input of the action sequence. In the forward diffusion process of the model, a Markov chain is established. In T time steps, the corresponding noise is generated and gradually added to the posture part. The noise addition formula is as follows:

[0055]

[0056] Among them, q(Θ t |x0) is the probability distribution of noise introduced at t time steps under the condition that the input is x0, x0 is the true hand posture sequence in the training data set, Θ t represents the corresponding noisy pose vector at time step t, represents a Gaussian distribution, is the noise coefficient, β s Represents the Gaussian distribution of the variance; I is the unit matrix, s is the time step, s is calculated incrementally starting from 1, and the maximum value is equal to the current sampling time step t.

[0057] Another Markov chain is introduced into the obtained posture noise vector as the inverse process of denoising, and the neural network is used to remove the noise. The number of denoising is the same as the number of denoising. Each denoising neural network will use the signal before denoising at the corresponding time step as the true value of the loss function to learn the mapping p(Θ t-1 |Θ t), after serialization denoising, the denoised output is finally obtained. In the denoising model, each layer of the neural network is a 4-layer U-Net structure, and the dimensions of each layer of the network in the structure are 256, 512, 1024, and 2048 respectively.

[0058] In order to reduce the impact of audio noise on the model, the present invention introduces a gesture decoder Φ g , embed the audio features into the feature space of implicit posture information, the formula is as follows:

[0059] f g =Φ g (f a )

[0060] Among them, f g is the feature vector in the posture feature space, Φ g Represents the pose decoder.

[0061] The output of the pose decoder f g It will be combined with the position output P of the hand position predictor as the conditional input of the diffusion model, which can generate corresponding conditional postures according to the position information and audio information during the posture generation process.

[0062] Therefore, the overall formula of the denoising part is as follows:

[0063]

[0064] in, is the result of N frames of hand posture, G() is the denoising model formed by cascading t U-Net networks, Θ t is the noise pose vector obtained by adding noise after t time steps.

[0065] In S3, the loss function of the diffusion model includes position loss and speed loss The position loss is used to calculate the error of the predicted hand position, and the speed loss introduced can consider the temporal stability, generate smoother movements, and reduce the jitter problem of the movement. The specific formula is as follows:

[0066]

[0067] in, is the L1 loss between the predicted pose sequence and the pose sequence in the dataset, is the predicted pose sequence of the model, P is the true pose sequence in the dataset, and are respectively the pose sequence predicted by the model and the pose sequence predicted by the model one frame later, P n and P n-1They are the pose sequence in the dataset and the pose sequence in the dataset lagged by one frame, respectively, indicating the change of the pose sequence in the time dimension, and ‖‖2 is the L2 loss.

[0068] S4: After the actual audio signal is input into the hand position predictor, a predicted hand position sequence is obtained, and after the audio feature extraction is performed on the actual audio signal, the actual audio feature is obtained. Then, the predicted hand position sequence and the actual audio feature are input into the trained diffusion model, and the model outputs a predicted posture sequence.

[0069] In S4, the predicted hand position sequence and the actual audio features are input into the trained diffusion model, and the model outputs the predicted posture sequence, including:

[0070] During the inference process, the actual audio features are input into the posture decoder in the trained diffusion model to obtain posture features. The posture features are concatenated with the predicted hand position sequence as the conditional input of the denoising neural network. The noise input of the denoising neural network is a directly sampled Gaussian noise matrix with the same size as the hand posture sequence to be predicted. The above-mentioned conditional input and noise input are concatenated to form the input of the denoising neural network. The output of the neural network is the hand posture sequence after one denoising and serves as the noise input for the next denoising. The conditional input and the updated noise input are concatenated to form the input of the next denoising of the denoising neural network. After N iterations, the hand action sequence obtained after removing the noise N times is the output of the diffusion model. The hand posture sequence is then visualized and rendered to obtain a video of the playing.

[0071] The embodiment of the present invention proposes a method for generating a hand gesture sequence from music based on a diffusion model, which can efficiently generate a hand gesture sequence synchronized with the music. A pre-trained audio feature encoder is first used to perform preliminary audio feature extraction through a convolutional neural network. The audio feature is embedded into the potential position feature space using a hand position decoder, and mapped to obtain the three-dimensional position information of the left and right hands respectively. A hand position predictor is obtained through training and used as a position module in the next stage of training. The hand position output is spliced ​​with the posture embedding feature of the audio feature, combined with time embedding, and finally obtained by denoising and neural network denoising. The output of each denoising neural network is used to calculate the loss function with the pre-noise signal of the corresponding time step, and the model parameters are updated. In the denoising model, each layer of the neural network adopts a 4-layer U-Net structure. This method is not only applicable to the generation of hand gestures from music, but can also be extended to other fields, such as individual portrait generation, virtual humans, metaverses, and game character animation.

[0072] An embodiment of the present invention provides a device for generating a hand gesture movement sequence from music based on a diffusion model, including one or more processors for implementing a method for generating a hand gesture movement sequence from music based on a diffusion model in the above embodiment.

[0073] The embodiment of the device for generating a sequence of hand gestures and movements from music based on a diffusion model proposed by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located to read the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, it is a hardware structure diagram of any device with data processing capabilities where the device for generating a sequence of hand gestures and movements from music based on a diffusion model of the present invention is located. In addition to the processor, memory, network interface, and non-volatile memory, the manufacturing of any device with data processing capabilities where the device in the embodiment is located is usually based on the actual function of the device with data processing capabilities, and other hardware may also be included, which will not be repeated.

[0074] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0075] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] An embodiment of the present invention further provides a readable storage medium having a program stored thereon. When the program is executed by a processor, a method for generating a hand gesture action sequence from music based on a diffusion model in the above embodiment is implemented.

[0077] The readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0078] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating hand gesture action sequences from music based on a diffusion model, characterized in that: The following steps are involved: S1: obtaining a piano playing video containing hand movements, separating an audio signal from the piano playing video, and extracting a hand position sequence and a hand posture sequence from the piano playing video to form a training data set; S2: extracting audio features from the audio signals in the training data set to obtain audio features, and training the position generation module using the extracted audio features and the hand position sequence in the training data set to obtain a hand position predictor; S3: After training the diffusion model according to the extracted audio features and the hand position sequence and hand posture sequence in the training data set, a trained diffusion model is obtained; S4: After the actual audio signal is input into the hand position predictor, a predicted hand position sequence is obtained, and after the audio feature extraction is performed on the actual audio signal, the actual audio feature is obtained. Then, the predicted hand position sequence and the actual audio feature are input into the trained diffusion model, and the model outputs a predicted posture sequence.

2. The method for generating hand gesture sequences from music based on a diffusion model according to claim 1, characterized in that: In S2, a pre-trained audio feature encoder is used to extract audio features from the audio signal to obtain audio features.

3. The method for generating hand gesture sequences from music based on a diffusion model according to claim 2, characterized in that: The pre-trained audio feature encoder is wav2vec2.

4. The method for generating hand gesture sequences from music based on a diffusion model according to claim 1, characterized in that: In S2, after training the position generation module using the extracted audio features and the hand position sequence in the training data set, a hand position predictor is obtained, which specifically includes: The position generation module includes a hand position decoder and a linear mapping layer. First, the hand position decoder is used to embed and map the extracted audio features into potential position features with richer temporal information to obtain potential features in the position space. The linear mapping layer is then used to perform dimension mapping on the potential features in the position space to obtain the position information of the left and right hands respectively. The hand position decoder is iteratively trained according to the loss between the position information of the left and right hands and the hand position sequence until the training is completed. The trained position generation module is obtained and recorded as the hand position predictor.

5. The method for generating hand gesture action sequences from music based on a diffusion model according to claim 1, characterized in that: The S3 is specifically: The diffusion model includes the noise addition process and the noise removal process; For the noise adding process, during the training process, a time step t is randomly sampled from the range of [0, N], where N is a hyperparameter, t Gaussian noise matrices are sampled, and the t noise matrices are sequentially superimposed on the input hand posture sequence to obtain the hand posture sequence after t noise additions; For the denoising process, the input of the denoising neural network consists of two parts: noise input and conditional input. For a time step t, the noise input is the hand posture sequence after t times of noise addition. The conditional input is composed of the hand position sequence and the audio features concatenated through the output of the posture decoder. The noise input and the conditional input are concatenated and used as the input of the denoising neural network. The hand posture sequence before the tth noise addition is calculated by predicting the noise matrix after the tth noise addition, and the training of the posture decoder and the denoising neural network is completed to obtain the trained diffusion model.

6. The method for generating hand gesture sequences from music based on a diffusion model according to claim 1, characterized in that: In S4, the predicted hand position sequence and the actual audio features are input into the trained diffusion model, and the model outputs a predicted posture sequence, specifically including: The actual audio features are input into the posture decoder in the trained diffusion model to obtain the posture features. The posture features are concatenated with the predicted hand position sequence as the conditional input of the denoising neural network. The noise input of the denoising neural network is a directly sampled Gaussian noise matrix. The conditional input and the noise input are concatenated to form the input of the denoising neural network. The output of the neural network is a denoised hand posture sequence and serves as the noise input for the next denoising. The conditional input and the updated noise input are concatenated to form the input of the next denoising of the denoising neural network, and the hand movement sequence is obtained after removing the noise N times.

7. The method for generating hand gesture sequences from music based on a diffusion model according to claim 1, characterized in that: In S3, the loss function of the diffusion model includes the position loss and speed loss The specific formula is as follows: in, is the L1 loss between the predicted pose sequence and the pose sequence in the dataset, is the predicted pose sequence of the model, P is the true pose sequence in the dataset, and are respectively the pose sequence predicted by the model and the pose sequence predicted by the model one frame later, P n and P n-1 They are the pose sequence in the dataset and the pose sequence in the dataset lagged by one frame, respectively, indicating the change of the pose sequence in the time dimension, and ‖‖2 is the L2 loss.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images