Cooperative voice gesture generation method and system based on deep learning

Abstract

The invention discloses a collaborative voice gesture generation method and system based on deep learning, and the method comprises the steps: obtaining an initial gesture sequence, and carrying out the forward diffusion of the initial gesture sequence, and obtaining a disturbance sequence; obtaining a voice feature, an initial attitude feature and a time embedding vector, and carrying out cross-modal fusion on the voice feature, the initial attitude feature and the time embedding vector to obtain a multi-modal feature; and iteratively denoising the disturbance sequence under the guidance of the multimode features by using an extended long-short-term memory network to obtain a new gesture sequence. According to the method, the extended long-short-term memory network and the extended model are combined and used, the correlation between different multi-modal features and gestures can be effectively captured, natural and expressive gestures can be deduced from condition information, and the generated collaborative gestures have higher quality and diversity.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a method and system for collaborative voice gesture generation based on deep learning. Background Art

[0002] Collaborative speech gestures refer to hand movements that occur naturally when speech is spoken. These gestures play an important role in enhancing communication, conveying additional information, and expressing emotions. Automatically generating natural and expressive collaborative speech gestures is of great significance in applications such as virtual character animation and human-computer interaction systems.

[0003] Existing collaborative speech gesture generation methods mainly include rule-based methods and data-driven methods. Rule-based methods rely on predefined gesture libraries and mapping rules, have poor flexibility, and are difficult to generate natural and diverse gestures. Data-driven methods usually use sequence generation models such as recurrent neural networks (RNN) or long short-term memory networks (LSTM) to model the relationship between speech and gestures. However, these methods often have the following problems: 1) The generated gestures lack diversity and are prone to mean regression, resulting in limited naturalness and expressiveness of gestures; 2) It is difficult to capture the long-term dependency between speech and gestures, resulting in insufficient coordination between the generated gestures and the speech content, affecting the synchronization and naturalness of the gestures; 3) The generation process lacks controllability, making it difficult to adjust the generation results according to demand, limiting the flexibility and adaptability of gesture generation; 4) Existing technologies fail to effectively integrate multimodal information such as speech, posture, and time, resulting in a lack of coordination and expressiveness in gesture generation.

[0004] Therefore, how to design a collaborative speech gesture generation method that can generate natural, diverse and highly coordinated speech content is a technical problem that needs to be urgently solved in this field. Summary of the invention

[0005] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a method and system for collaborative speech gesture generation based on deep learning is provided, which combines the use of extended long short-term memory networks and extended models to effectively capture the correlation between different multimodal features and gestures, and help infer natural and expressive gestures from conditional information, so that the generated collaborative gestures have higher quality and diversity.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A collaborative speech gesture generation method based on deep learning comprises the following steps:

[0008] Obtain an initial gesture sequence, perform forward diffusion on the initial gesture sequence, and obtain a disturbance sequence;

[0009] Acquire speech features, initial posture features and time embedding vectors, perform cross-modal fusion on the speech features, initial posture features and time embedding vectors, and obtain multi-modal features;

[0010] The extended long short-term memory network is used to iteratively denoise the perturbation sequence under the guidance of multimodal features to obtain a new gesture sequence.

[0011] Furthermore, when the initial gesture sequence is forward diffused, specifically, the forward diffusion process is applied to the initial gesture sequence, and Gaussian noise is gradually added. The expression is as follows:

[0012]

[0013] Among them, x0 represents the initial gesture sequence, α t represents the time-dependent scaling factor and ε represents the Gaussian noise sampled from a normal distribution.

[0014] Furthermore, the speech feature is specifically a frame-level speech feature extracted from the speech by a speech feature extractor.

[0015] Furthermore, the initial posture feature is specifically obtained by extracting posture features from the initial gesture sequence through a pre-trained skeleton sequence autoencoder.

[0016] Furthermore, the time embedding vector is specifically a time embedding vector obtained by processing the diffusion time step through cosine transform, and the diffusion time step includes the time step used for forward diffusion of the initial gesture sequence and iterative denoising of the disturbance sequence.

[0017] Furthermore, when cross-modal fusion of speech features, initial posture features and time embedding vectors is performed, it specifically includes:

[0018] The speech features, initial posture features and time embedding vector are fused into a feature vector, which is expressed as follows:

[0019] z=concat(a,x,t)

[0020] Among them, a represents the speech feature, x represents the initial posture feature, t represents the time embedding vector, and concat() represents the concatenation operation;

[0021] The feature vector is compressed through a linear layer, tanh activation, linear layer, and ReLU activation function, and then the compressed feature vector is decompressed through an MLP layer, and the decompressed feature vector is used as a multimodal feature.

[0022] Furthermore, when using the extended long short-term memory network to iteratively denoise the disturbance sequence under the guidance of multi-modal features, it specifically includes:

[0023] To remove noise from the disturbance sequence, the expression is as follows

[0024]

[0025] in, Indicates the state x at time t t and condition variables When x t-1 The probability distribution of x t represents the disturbance sequence at time t, Represents multimodal features, Represents x t-1 It follows a normal distribution, represents the mean of the normal distribution, β t I represents the covariance matrix of the normal distribution, where β t represents the scalar variance associated with the time step, I is the identity matrix, and θ represents the bias;

[0026] The disturbance sequence after noise removal and multimodal features are input into the extended long short-term memory network, so that the extended long short-term memory network updates the hidden state at each step. The expression is as follows:

[0027]

[0028] Among them, H t is the hidden state at step t, H t-1 is the hidden state of the previous step of step t;

[0029] Use the decoder to generate the denoised gesture sequence from the updated hidden state.

[0030] Furthermore, the step of designing a loss function is included, and the expression of the loss function is as follows:

[0031] L(θ)=L diffusuon (θ)+L fusion

[0032] Among them, L(θ) represents the loss function, L fusion is the loss function of cross-modal fusion, L diffusion (θ) is the loss function of the denoising process.

[0033] Furthermore, the loss function L of the cross-modal fusion is fusion And the loss function L of the denoising process diffusion The expression of (θ) is as follows:

[0034]

[0035] Among them, z represents the feature vector that fuses the speech feature, the initial posture feature and the time embedding vector, represents the multimodal features after compressing and decompressing the feature vector z, According to the perturbation sequence Multimodal features and the noise predicted at time step t, ∈ represents the actual noise, E q Represents expected value operation.

[0036] The present invention also proposes a deep learning-based collaborative voice gesture generation system, comprising a microprocessor and a computer-readable storage medium connected to each other, wherein the microprocessor is programmed or configured to execute any one of the deep learning-based collaborative voice gesture generation methods.

[0037] Compared with the prior art, the advantages of the present invention are:

[0038] 1. The present invention makes full use of the advantages of the diffusion model and the extended long short-term memory network to improve the quality and diversity of generated gestures, while enhancing the ability to model long-term dependencies.

[0039] 2. The present invention integrates the cross-modal feature fusion module into the gesture generation framework, effectively integrating voice, initial posture, and time information, and improving the coordination and expressiveness of gesture generation.

[0040] 3. The present invention adopts an iterative denoising generation strategy to overcome the mean regression problem in traditional methods and generate more natural and varied gesture sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.

[0042] Figure 2 Schematic diagram of the working principle of the diffusion model of an embodiment of the present invention.

[0043] Figure 3 Schematic diagram of the structure of a cross-modal feature fusion module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0045] Embodiment 1

[0046] This embodiment proposes a method for generating collaborative speech gestures based on deep learning to improve the naturalness, diversity and coordination of gesture generation with speech content. Figure 1 As shown, the following steps are included:

[0047] S1) Diffusion Process stage, including forward diffusion and cross-modal fusion, where:

[0048] The forward diffusion process specifically obtains the initial gesture sequence, performs forward diffusion on the initial gesture sequence, and obtains the perturbation sequence;

[0049] The cross-modal fusion process specifically involves obtaining speech features, initial posture features, and time embedding vectors, and cross-modally fusing the speech features, initial posture features, and time embedding vectors to obtain multi-modal features.

[0050] S2) Diffusion xLSTM-based gesture generation stage, including:

[0051] The extended long short-term memory network is used to iteratively denoise the perturbation sequence under the guidance of multimodal features to obtain a new gesture sequence.

[0052] This embodiment constructs a collaborative speech gesture generation model through the above steps. The gesture sequence is first processed by the diffusion process, which increases the diversity of the generated results and effectively integrates the audio, posture and time information. The xLSTM is combined with the diffusion model to ensure the effective modeling of the timing information and improve the quality and naturalness of the generated gestures.

[0053] The following is a detailed description of each step.

[0054] like Figure 2 As shown, the forward diffusion process of this embodiment starts with an initial gesture sequence x0. When the initial gesture sequence is forward diffused, the forward diffusion process is applied to the initial gesture sequence x0, and Gaussian noise is gradually added to obtain a disrupted gesture sequence, which is called a disturbance sequence x in this embodiment. t , the expression is as follows:

[0055]

[0056] Among them, x0 represents the initial gesture sequence, α t represents the time-dependent scaling factor and ε represents the Gaussian noise sampled from a normal distribution.

[0057] In this embodiment, the speech feature is specifically the frame-level speech feature extracted by the speech through the speech feature extractor. When acquiring the speech feature in the cross-modal fusion process, a segment of speech is specifically input into the speech feature extractor (AcousticFeatureExtractor). The speech feature extractor uses a speech feature extraction network based on a convolutional neural network to extract audio features, thereby obtaining the frame-level speech feature a.

[0058] In this embodiment, the initial posture feature is specifically to extract the posture feature of the initial gesture sequence through a pre-trained skeleton sequence autoencoder. When obtaining the initial posture feature in the cross-modal fusion process, the initial gesture sequence x0 is specifically input into the pre-trained skeleton sequence autoencoder, and the skeleton sequence autoencoder extracts the spatiotemporal features of the initial gesture to obtain the initial posture feature x.

[0059] In this embodiment, the time embedding vector is specifically a time embedding vector obtained by processing the diffusion time step through a cosine transform. The diffusion time step is obtained by simulating the process of data from an initial distribution to a target distribution, specifically including a forward diffusion process and a reverse generation process. In this embodiment, it includes the time step used for forward diffusion of the initial gesture sequence and iterative denoising of the disturbance sequence.

[0060] This embodiment designs a corresponding cross-modal fusion module for the cross-modal fusion process, and its structure is as follows: Figure 3 As shown in FIG. 1 , when cross-modal fusion of speech features, initial posture features and time embedding vectors is performed, the following is specifically included:

[0061] The speech features, initial posture features and time embedding vector are fused into the feature vector z, which is expressed as follows:

[0062] z=concat(a,x,t)

[0063] Among them, a represents the speech feature, x represents the initial posture feature, t represents the time embedding vector, and concat() represents the concatenation operation;

[0064] The feature vector z is compressed through a linear layer, tanh activation, linear layer, and ReLU activation function, and then the compressed feature vector is decompressed back to a high-dimensional feature through an MLP layer. The decompressed feature vector As a multimodal feature, in the decompression stage, the model can learn more effective feature representation by restoring low-dimensional features to high-dimensional features through a multi-layer perceptron (MLP), thereby achieving better performance and robustness in the subsequent modeling process.

[0065] The loss function in the cross-modal fusion process is achieved by minimizing the mean square error (MSE), that is:

[0066] In this embodiment, a network structure including an extended long short-term memory unit is designed for the gesture generation stage of the diffusion model based on xLSTM to enhance the modeling ability of long-term dependencies. The xLSTM network has multimodal features. Under the guidance of , iterative denoising xt is performed, such as Figure 2As shown in the figure, one sLSTM module and three mLSTM modules are integrated into the xLSTM network. These two modules work together to process multimodal features and noisy gestures xt. The sLSTM module adopts a scalar update mechanism to provide fine-grained memory unit control. This helps to enhance the robustness of the model when processing long sequences, especially in capturing subtle changes in audio features and posture features. The mLSTM module extends vector operations to matrix calculations to improve parallel processing capabilities. This enables the mLSTM module to effectively process time series data and capture long-term dependencies, thereby maintaining temporal coherence when generating gesture sequences.

[0067] Therefore, in this embodiment, when the extended long short-term memory network is used to iteratively denoise the disturbance sequence under the guidance of multi-modal features, the disturbance sequence x is specifically t and multimodal features Input the xLSTM model to generate new gesture sequences through an iterative denoising process Specifically include:

[0068] First, the perturbation sequence is de-noised by reverse denoising. In the reverse denoising stage, the model gradually removes the noise and learns the conditional distribution of the time step t∈(1, T), which is expressed as follows:

[0069]

[0070] in, Indicates the state x at time t t and condition variables When x t-1 The probability distribution of x t represents the disturbance sequence at time t, Represents multimodal features, Represents x t-1 It follows a normal distribution, represents the mean of the normal distribution, β t I represents the covariance matrix of the normal distribution, where β t represents the scalar variance β1…β associated with the time step T , I is the unit matrix, θ represents the bias;

[0071] Then the disturbance sequence after noise removal and the multimodal features are input into the extended long short-term memory network, so that the extended long short-term memory network uses the multimodal features At each step, the hidden state is updated as follows:

[0072]

[0073] Among them, H t is the hidden state at step t, H t-1is the hidden state of the previous step of step t, x t is the input feature corresponding to the t-th step, that is, the disturbance sequence after noise removal corresponding to the t-th step;

[0074] Finally, decoder D is used to extract the updated hidden state H t Generate denoised gesture sequence

[0075] Through the above steps, xLSTM combines the state information and multimodal features obtained from the denoising step while retaining the historical information about the time series. The xLSTM model gradually recovers and generates new gesture sequences through an iterative denoising process. In each iteration, the model first denoises and then updates the hidden state. The denoising process uses the current state and multimodal information to predict the state of the previous moment. The xLSTM network then updates its internal state by combining the newly obtained state, multimodal features, and historical hidden states. This alternating process allows the model to gradually remove noise and effectively integrate multimodal information, ultimately generating a sample that approximates real data. This process fully utilizes the advantages of xLSTM in modeling long-term dependencies while combining the generation capabilities of the diffusion model.

[0076] In this embodiment, the goal of the gesture generation phase of the xLSTM-based diffusion model is to minimize the prediction noise. The difference between the actual noise ε and the loss function of the denoising process is as follows:

[0077]

[0078] Among them, z represents the feature vector that fuses the speech feature, the initial posture feature and the time embedding vector, represents the multimodal features after compressing and decompressing the feature vector z, According to the perturbation sequence Multimodal features and the noise predicted at time step t, ∈ represents the actual noise, E q Represents expected value operation

[0079] In this embodiment, after the model is constructed through step S1 and step S2, a corresponding loss function is also designed for model training. The expression of the loss function is as follows:

[0080] L(θ)=L diffusion (θ)+L fusion

[0081] Among them, L(θ) represents the loss function, L fusion is the loss function of cross-modal fusion, L diffusion (θ) is the loss function of the denoising process.

[0082] During the model training process, the overall training goal is to minimize the loss function. It should be noted that the specific process of training the model and updating the network parameters according to the aforementioned diffusion process stage and the diffusion model gesture generation stage based on xLSTM with the goal of minimizing the loss function is well known to those skilled in the art. The method of this embodiment does not involve improvements in the relevant process, so the specific calculation process will not be repeated.

[0083] Embodiment 2

[0084] This embodiment proposes a deep learning-based collaborative voice gesture generation system, including a microprocessor and a computer-readable storage medium connected to each other, and the microprocessor is programmed or configured to execute the deep learning-based collaborative voice gesture generation method described in Example 1.

[0085] In summary, the present invention proposes a method and system for collaborative speech gesture generation based on deep learning. The implementation steps are divided into two stages. The first stage is the diffusion process. This stage also includes a cross-modal feature fusion module. The operation steps are as follows: The initial gesture sequence is converted into a damaged gesture x through the diffusion process. t ; Preprocess the input audio signal and extract audio features; obtain initial posture information; generate time embedding; process these conditional features (audio features, initial posture, time embedding) through the cross-modal feature fusion module to obtain the fused feature z. The second stage is the diffusion model gesture generation based on xLSTM (Diffusion xLSTM). The operation steps are as follows: t With the fused features The xLSTM model gradually recovers and generates new gesture sequences through an iterative denoising process. This process fully utilizes the advantages of xLSTM in modeling long-term dependencies while combining the generation capabilities of the diffusion model. Compared with the prior art, the present invention can generate more natural, diverse and highly synchronized collaborative gestures with the speech content, effectively alleviating the common problems of over-smoothing and loss of details in gesture generation.

[0086] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A collaborative speech gesture generation method based on deep learning, characterized in that: The following steps are involved: Obtain an initial gesture sequence, perform forward diffusion on the initial gesture sequence, and obtain a disturbance sequence; Acquire speech features, initial posture features and time embedding vectors, and perform cross-modal fusion on the speech features, initial posture features and time embedding vectors to obtain multi-modal features; The extended long short-term memory network is used to iteratively denoise the perturbation sequence under the guidance of multimodal features to obtain a new gesture sequence.

2. The method for generating collaborative speech gestures based on deep learning according to claim 1, characterized in that: When the initial gesture sequence is forward diffused, the forward diffusion process is applied to the initial gesture sequence, and Gaussian noise is gradually added. The expression is as follows: Among them, x0 represents the initial gesture sequence, α t represents the time-dependent scaling factor and ε represents the Gaussian noise sampled from a normal distribution.

3. The method for generating collaborative speech gestures based on deep learning according to claim 1, characterized in that: The speech features are specifically frame-level speech features extracted from the speech by a speech feature extractor.

4. The method for generating collaborative speech gestures based on deep learning according to claim 1, characterized in that: The initial posture feature is specifically extracted by passing the initial gesture sequence through a pre-trained skeleton sequence autoencoder.

5. The method for generating collaborative speech gestures based on deep learning according to claim 1, characterized in that: The time embedding vector is specifically a time embedding vector obtained by processing the diffusion time step through cosine transform, and the diffusion time step includes the time step used for forward diffusion of the initial gesture sequence and iterative denoising of the disturbance sequence.

6. The method for generating collaborative speech gestures based on deep learning according to claim 1, characterized in that: When cross-modal fusion of speech features, initial posture features and time embedding vectors is performed, it specifically includes: The speech features, initial posture features and time embedding vector are fused into a feature vector, which is expressed as follows: z=concat(a,x,t) Among them, a represents the speech feature, x represents the initial posture feature, t represents the time embedding vector, and concat() represents the concatenation operation; The feature vector is compressed through a linear layer, tanh activation, linear layer, and ReLU activation function, and then the compressed feature vector is decompressed through an MLP layer, and the decompressed feature vector is used as a multimodal feature.

7. The method for generating collaborative speech gestures based on deep learning according to claim 1, characterized in that: When using the extended long short-term memory network to iteratively denoise the perturbation sequence under the guidance of multimodal features, it specifically includes: To remove noise from the disturbance sequence, the expression is as follows in, Indicates the state x at time t t and condition variables When x t-1 The probability distribution of x t represents the disturbance sequence at time t, Represents multimodal features, Represents x t-1 It follows a normal distribution, represents the mean of the normal distribution, β t I represents the covariance matrix of the normal distribution, where β t represents the scalar variance associated with the time step, I is the identity matrix, and θ represents the bias; The disturbance sequence after noise removal and multimodal features are input into the extended long short-term memory network, so that the extended long short-term memory network updates the hidden state at each step. The expression is as follows: Among them, H t is the hidden state at step t, H t-1 is the hidden state of the previous step of step t; Use the decoder to generate the denoised gesture sequence from the updated hidden states.

8. The method for generating collaborative speech gestures based on deep learning according to claim 1, characterized in that: The method also includes the step of designing a loss function, wherein the expression of the loss function is as follows: L(θ)=L diffusion (θ)+L fusion Among them, L(θ) represents the loss function, L fusion is the loss function of cross-modal fusion, L diffusion (θ) is the loss function of the denoising process.

9. The method for generating collaborative speech gestures based on deep learning according to claim 8, characterized in that: The loss function L of the cross-modal fusion fusion And the loss function L of the denoising process diffusion The expression of (θ) is as follows: Among them, z represents the feature vector that fuses the speech feature, the initial posture feature and the time embedding vector, represents the multimodal features after compressing and decompressing the feature vector z, According to the perturbation sequence Multimodal features and the noise predicted at time step t, ∈ represents the actual noise, E q Represents expected value operation.

10. A collaborative speech gesture generation system based on deep learning, characterized in that: It comprises a microprocessor and a computer-readable storage medium connected to each other, wherein the microprocessor is programmed or configured to execute the deep learning-based collaborative voice gesture generation method according to any one of claims 1 to 9.

Images