A deep learning-based collaborative speech gesture generation method and system
By combining extended long short-term memory networks with a diffusion model, natural and diverse collaborative gestures are generated using speech, posture, and temporal features. This solves the problem of insufficient diversity and coordination in gesture generation in existing technologies, and achieves higher quality gesture generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUNAN UNIV
- Filing Date
- 2024-11-29
- Publication Date
- 2026-04-10
AI Technical Summary
Existing collaborative speech gesture generation methods suffer from problems such as a lack of diversity in generated gestures, difficulty in capturing the long-term dependency between speech and gestures, a lack of controllability in the generation process, and insufficient integration of multimodal information, resulting in insufficient naturalness and expressiveness of the generated gestures.
By combining Extended Long Short-Term Memory (LSTM) networks with a diffusion model, speech, gesture, and temporal features are obtained through forward diffusion and cross-modal fusion. The LSTM network is then used for iterative denoising to generate natural and diverse collaborative gestures.
It improves the quality and diversity of generated gestures, enhances the coordination and expressiveness of gestures and speech content, overcomes the mean regression problem in traditional methods, and generates more natural and varied gesture sequences.
Smart Images
Figure CN119577686B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of human-computer interaction, in particular to a collaborative speech gesture generation method and system based on deep learning. BACKGROUND
[0002] Collaborative speech gesture refers to the natural hand movements that accompany speech during conversation. These gestures play an important role in enhancing communication effectiveness, conveying additional information, and expressing emotions. In applications such as virtual character animation and human-computer interaction systems, automatically generating natural and expressive collaborative speech gestures is of great significance.
[0003] Existing collaborative speech gesture generation methods mainly include rule-based methods and data-driven methods. Rule-based methods rely on predefined gesture libraries and mapping rules, which have poor flexibility and are difficult to generate natural and diverse gestures. Data-driven methods usually use sequence generation models such as recurrent neural networks (RNN) or long short-term memory networks (LSTM) to model the relationship between speech and gestures. However, these methods often have the following problems: 1) The generated gestures lack diversity and are prone to mean regression, limiting the naturalness and expressiveness of the gestures; 2) It is difficult to capture long-term dependencies between speech and gestures, resulting in generated gestures that are not well coordinated with speech content, affecting the synchronization and naturalness of the gestures; 3) The generation process lacks controllability, making it difficult to adjust the generation results according to requirements, limiting the flexibility and adaptability of gesture generation; 4) Existing technologies fail to effectively integrate multi-modal information such as speech, posture, and time, resulting in a lack of coordination and expressiveness in gesture generation.
[0004] Therefore, how to design a collaborative speech gesture generation method that can generate natural, diverse, and highly coordinated gestures with speech content is a technical problem that needs to be solved in the field. SUMMARY
[0005] The technical problem to be solved by the present application is to provide a collaborative speech gesture generation method and system based on deep learning, which combines the use of extended long short-term memory networks and extended models to effectively capture the correlation between different multi-modal features and gestures, helping to infer natural and expressive gestures from conditional information, and making the generated collaborative gestures have higher quality and diversity.
[0006] To solve the above technical problems, the technical solution adopted by the present application is:
[0007] A collaborative speech gesture generation method based on deep learning, comprising the following steps:
[0008] Obtain an initial gesture sequence, and perform forward diffusion on the initial gesture sequence to obtain a perturbation sequence;
[0009] obtain a multi-modal feature by obtaining a speech feature, an initial gesture feature and a time embedding vector, and performing cross-modal fusion on the speech feature, the initial gesture feature and the time embedding vector;
[0010] using an extended long short-term memory network to iteratively denoise the perturbation sequence under the guidance of the multi-modal feature, to obtain a new gesture sequence.
[0011] Further, when performing forward diffusion on the initial gesture sequence, specifically, a forward diffusion process is applied to the initial gesture sequence, and Gaussian noise is gradually added, and the expression is as follows:
[0012]
[0013] wherein x0 represents the initial gesture sequence, a t represents a time-dependent scaling factor, and ε represents Gaussian noise sampled from a normal distribution.
[0014] Further, the speech feature is specifically a frame-level speech feature extracted by a speech feature extractor.
[0015] Further, the initial gesture feature is specifically a gesture feature extracted by a pre-trained skeleton sequence autoencoder from the initial gesture sequence.
[0016] Further, the time embedding vector is specifically a time embedding vector obtained by processing a diffusion time step through a cosine transformation, and the diffusion time step includes a time step used for forward diffusion on the initial gesture sequence and iterative denoising of the perturbation sequence.
[0017] Further, when performing cross-modal fusion on the speech feature, the initial gesture feature and the time embedding vector, specifically includes:
[0018] The speech feature, the initial gesture feature and the time embedding vector are fused into a feature vector, and the expression is as follows:
[0019] z=concat(a,x,t)
[0020] wherein a represents the speech feature, x represents the initial gesture feature, t represents the time embedding vector, and concat() represents a concatenation operation.
[0021] The feature vector is compressed through a linear layer, a tanh activation, a linear layer and a ReLU activation function, and then the compressed feature vector is decompressed through an MLP layer, and the decompressed feature vector is used as a multi-modal feature.
[0022] Further, when using an extended long short-term memory network to iteratively denoise the perturbation sequence under the guidance of the multi-modal feature, specifically includes:
[0023] The noise is removed from the disturbance sequence, and the expression is as follows
[0024]
[0025] wherein, represents the probability distribution of x t and the conditional variable at time t, x t-1 , x t represents the disturbance sequence at time t, represents the multimodal feature, represents the normal distribution to which x t-1 obeys, represents the mean of the normal distribution, β t I represents the covariance matrix of the normal distribution, wherein β t represents the scalar variance related to time step, I is a unit matrix, and θ represents a bias;
[0026] The noise-removed disturbance sequence and the multimodal feature are input into the extended long short-term memory network, so that the extended long short-term memory network updates the hidden state at each step, and the expression is as follows:
[0027]
[0028] wherein, H t is the hidden state at the t-th step, and H t-1 is the hidden state of the previous step at the t-th step;
[0029] The decoder is used to generate the noise-removed gesture sequence from the updated hidden state.
[0030] Further, the step of designing a loss function is further included, and the expression of the loss function is as follows:
[0031] L(θ)=L diffusuon (θ)+L fusion
[0032] wherein, L(θ) represents the loss function, L fusion is the loss function of cross-modal fusion, and L diffusion (θ) is the loss function of the denoising process.
[0033] Further, the expressions of the loss function L fusion of cross-modal fusion and the loss function L diffusion (θ) of the denoising process are as follows:
[0034]
[0035] wherein z represents a feature vector obtained by fusing the speech feature, the initial pose feature, and the time embedding vector, represents a multimodal feature after the feature vector z is compressed and decompressed, represents a multimodal feature after the feature vector z is compressed and decompressed, multimodal feature and the noise predicted at time step t, ∈ represents the actual noise, E q represents an expected value operation.
[0036] The application also provides a deep learning-based collaborative speech gesture generation system, comprising a microprocessor and a computer-readable storage medium connected to each other, wherein the microprocessor is programmed or configured to perform any of the deep learning-based collaborative speech gesture generation methods.
[0037] Compared with the prior art, the application has the following advantages:
[0038] 1. The application takes full advantage of the diffusion model and the extended long short-term memory network, improves the quality and diversity of generated gestures, and enhances the modeling capability of long-term dependencies.
[0039] 2. The application integrates the cross-modal feature fusion module into the gesture generation framework, effectively integrates speech, initial pose, and time information, and improves the collaboration and expressiveness of gesture generation.
[0040] 3. The application adopts an iterative denoising generation strategy, overcomes the mean regression problem in traditional methods, and generates more natural and varied gesture sequences. BRIEF DESCRIPTION OF DRAWINGS
[0041] Figure 1 is a method flowchart of an embodiment of the application.
[0042] Figure 2 is a working principle diagram of the diffusion model of an embodiment of the application.
[0043] Figure 3 is a structure diagram of the cross-modal feature fusion module of an embodiment of the application. DETAILED DESCRIPTION
[0044] The application will be further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the application is not limited thereby.
[0045] Embodiment One
[0046] The embodiment provides a deep learning-based collaborative speech gesture generation method to improve the naturalness, diversity, and coordination with speech content of gesture generation. As shown in Figure 1 the following steps are included:
[0047] S1) The diffusion process stage includes two processes: forward diffusion and cross-modal fusion.
[0048] The forward diffusion process specifically involves acquiring an initial gesture sequence, performing forward diffusion on the initial gesture sequence, and obtaining a perturbation sequence.
[0049] The cross-modal fusion process specifically involves acquiring speech features, initial pose features, and temporal embedding vectors, and then fusing these features across modalities to obtain multimodal features.
[0050] S2) The xLSTM-based diffusion model gesture generation stage includes:
[0051] Using an extended long short-term memory network, guided by multimodal features, iterative denoising of the perturbation sequence is performed to obtain a new gesture sequence.
[0052] This embodiment constructs a collaborative speech gesture generation model through the above steps. First, the gesture sequence is processed using a diffusion process, which increases the diversity of the generated results and effectively integrates audio, posture, and temporal information. Then, xLSTM is combined with the diffusion model, which not only ensures effective modeling of temporal information but also improves the quality and naturalness of the generated gestures.
[0053] The following is a detailed explanation of each step.
[0054] like Figure 2 As shown, the forward diffusion process in this embodiment begins with an initial gesture sequence x0. Specifically, the forward diffusion process is applied to the initial gesture sequence x0, gradually adding Gaussian noise to obtain a disrupted gesture sequence, referred to in this embodiment as the perturbation sequence x. t The expression is as follows:
[0055]
[0056] Where x0 represents the initial gesture sequence, α t ε represents the time-dependent scaling factor, and ε represents the Gaussian noise sampled from a normal distribution.
[0057] In this embodiment, the speech features are specifically frame-level speech features extracted by the speech feature extractor. When obtaining speech features during cross-modal fusion, a speech segment is input into the speech feature extractor. The speech feature extractor uses a speech feature extraction network based on a convolutional neural network to extract audio features, thereby obtaining frame-level speech feature a.
[0058] In this embodiment, the initial pose feature is specifically the pose feature extracted by the pre-trained skeleton sequence autoencoder from the initial gesture sequence. When obtaining the initial pose feature in the cross-modal fusion process, the initial gesture sequence x0 is input into the pre-trained skeleton sequence autoencoder, and the skeleton sequence autoencoder extracts the spatial and temporal features of the initial gesture to obtain the initial pose feature x.
[0059] In this embodiment, the time embedding vector is specifically the time embedding vector obtained by processing the diffusion time step through the cosine transformation. The diffusion time step is obtained by simulating the process of data from the initial distribution to the target distribution, which specifically includes a forward diffusion process and a reverse generation process. In this embodiment, it includes the time step used for forward diffusion of the initial gesture sequence and iterative denoising of the perturbation sequence.
[0060] This embodiment designs a corresponding cross-modal fusion module for the cross-modal fusion process, and the structure is as shown in Figure 3 When fusing the speech feature, the initial pose feature and the time embedding vector, the cross-modal fusion process specifically includes:
[0061] The speech feature, the initial pose feature and the time embedding vector are fused into a feature vector z, and the expression is as follows:
[0062] z = concat (a, x, t)
[0063] Wherein, a represents the speech feature, x represents the initial pose feature, t represents the time embedding vector, and concat() represents the concatenation operation.
[0064] The feature vector z is compressed through a linear layer, a tanh activation, a linear layer and a ReLU activation function, and then the compressed feature vector is decompressed back to a high-dimensional feature The decompressed feature vector is used as a multi-modal feature. In the decompression stage, the multi-layer perceptron (MLP) is used to recover from low-dimensional features to high-dimensional features. The model can learn more effective feature representations, thereby achieving better performance and robustness in the subsequent modeling process.
[0065] The loss function in the cross-modal fusion process is realized by minimizing the mean square error (MSE), that is:
[0066] In this embodiment, a network structure containing an extended long short-term memory unit is designed for the gesture generation stage of the diffusion model based on xLSTM, which enhances the modeling ability of long-term dependencies. The xLSTM network iteratively denoises xt under the guidance of the multi-modal feature . Figure 2As shown, the xLSTM network integrates one sLSTM module and three mLSTM modules. These two modules work together to process multimodal features and noisy gestures xt. The sLSTM module employs a scalar update mechanism, providing fine-grained memory cell control. This helps enhance the model's robustness when processing long sequences, especially in capturing subtle changes in audio and pose features. The mLSTM modules extend vector operations to matrix computation, improving parallel processing capabilities. This enables the mLSTM modules to efficiently process time-series data, capturing long-term dependencies and thus maintaining temporal coherence when generating gesture sequences.
[0067] Therefore, in this embodiment, when using an extended long short-term memory network to iteratively denoise the perturbation sequence under the guidance of multimodal features, specifically, the perturbation sequence x is... t and multimodal features Input an xLSTM model and generate new gesture sequences through an iterative denoising process. Specifically, it includes:
[0068] First, noise is removed from the perturbation sequence through reverse denoising. In the reverse denoising stage, the model gradually removes noise and learns the conditional distribution of the time step t∈(1,T), as expressed below.
[0069]
[0070] in, Indicates the state x at time t t and condition variables At that time, x t-1 The probability distribution of x t This represents the perturbation sequence at time t. Represents multimodal features, x represents t-1 It follows a normal distribution. β represents the mean of a normal distribution. t I represents the covariance matrix of the normal distribution, where β t The scalar variance β1……β represents the time step correlation. T I is the identity matrix, and θ represents the bias.
[0071] Then, the noise-removed perturbation sequence and multimodal features are input into the Extended Long Short-Term Memory (ESLM) network, enabling the ELM network to utilize the multimodal features. The hidden state is updated at each step using the following expression:
[0072]
[0073] Among them, H t H is the hidden state at step t. t-1is the hidden state of the previous step for the t-th step t is the input feature of the t-th step, i.e., the denoised perturbation sequence of the t-th step
[0074] Finally, the updated hidden state H t generates the denoised gesture sequence
[0075] Through the above steps, the xLSTM combines the state information obtained from the denoising step and the multi-modal features, while preserving the historical information about the time series. The xLSTM model gradually recovers and generates a new gesture sequence through an iterative denoising process. In each iteration, the model first denoises and then updates the hidden state. The denoising process uses the current state and multi-modal information to predict the state at the previous time. Then, the xLSTM network updates its internal state by comprehensively using the newly obtained state, multi-modal features, and historical hidden state. This alternating process allows the model to gradually remove noise and effectively integrate multi-modal information, ultimately generating a sample that approximates the real data. This process fully utilizes the advantages of xLSTM in modeling long-term dependencies, while combining the generation ability of diffusion models.
[0076] In this embodiment, the goal of the gesture generation stage of the xLSTM-based diffusion model is to minimize the difference between the predicted noise and the actual noise ε, and the loss function of the denoising process is expressed as follows:
[0077]
[0078] where z represents a feature vector that combines the speech features, initial pose features, and time embedding vectors, represents the multi-modal features after compressing and decompressing the feature vector z, represents the noise predicted according to the perturbation sequence the multi-modal features and the noise predicted at time step t, and ∈ represents the actual noise, E q represents the expected value operation
[0079] In this embodiment, after constructing the model through steps S1 and S2, a corresponding loss function is designed for the training of the model, and the expression of the loss function is as follows:
[0080] L(θ) = L diffusion (θ) + L fusion
[0081] where L(θ) represents the loss function, L fusion is the loss function of cross-modal fusion, L diffusion (θ) is the loss function of the denoising process.
[0082] During model training, the overall training objective is to minimize the loss function. It should be noted that the specific process of training the model and updating the network parameters according to the aforementioned diffusion process stages and the xLSTM-based diffusion model gesture generation stage, with the goal of minimizing the loss function, is well-known to those skilled in the art. The method in this embodiment does not involve improvements to these processes, and therefore the specific calculation process will not be elaborated further.
[0083] Example 2
[0084] This embodiment proposes a deep learning-based collaborative voice and gesture generation system, including interconnected microprocessors and computer-readable storage media, wherein the microprocessors are programmed or configured to execute the deep learning-based collaborative voice and gesture generation method described in Embodiment 1.
[0085] In summary, this invention proposes a deep learning-based collaborative speech gesture generation method and system. The implementation steps are divided into two stages. The first stage is a diffusion process, which also includes a cross-modal feature fusion module. The operation steps are as follows: the initial gesture sequence is transformed into a corrupted gesture x through the diffusion process. t The first stage involves preprocessing the input audio signal to extract audio features; obtaining initial pose information; generating temporal embeddings; and processing these conditional features (audio features, initial pose, and temporal embeddings) through a cross-modal feature fusion module to obtain the fused feature z. The second stage is the diffusion model gesture generation based on xLSTM (Diffusion xLSTM), with the following steps: The damaged gesture x obtained in the first stage... t Features after fusion The inputs are fed into the xLSTM model. The xLSTM model iteratively reconstructs and generates new gesture sequences through a denoising process. This process fully leverages the advantages of xLSTM in modeling long-term dependencies while combining the generative capabilities of a diffusion model. Compared to existing technologies, this invention can generate more natural, diverse, and highly synchronized collaborative gestures with the speech content, effectively alleviating the common problems of over-smoothing and loss of detail in gesture generation.
[0086] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1.A method for generating collaborative speech gesture based on deep learning, characterized in that, The method comprises the following steps: obtaining an initial gesture sequence, performing forward diffusion on the initial gesture sequence to obtain a perturbation sequence, and specifically applying a forward diffusion process to the initial gesture sequence and gradually adding Gaussian noise, the expression being as follows: wherein denotes an initial gesture sequence, denotes a time-dependent scaling factor, denotes a Gaussian noise sampled from a normal distribution; obtaining a speech feature, an initial posture feature and a time embedding vector, performing cross-modal fusion on the speech feature, the initial posture feature and the time embedding vector to obtain a multi-modal feature, and specifically comprising the following steps: fusing the speech feature, the initial posture feature and the time embedding vector into a feature vector, the expression being as follows: wherein, represents a speech feature, represents an initial pose feature, represents a temporal embedding vector, represents a concatenation operation; compressing the feature vector through a linear layer, a tanh activation, a linear layer and a ReLU activation function, then decompressing the compressed feature vector through an MLP layer, and taking the decompressed feature vector as the multi-modal feature; using an extended long short-term memory network to iteratively denoise the perturbation sequence under the guidance of the multi-modal feature to obtain a new gesture sequence, and specifically comprising the following steps: removing noise from the perturbation sequence, the expression being as follows wherein denotes the state at time t and the conditional variables at time t, the probability distribution of denotes the disturbance sequence at time t, denotes the multi-modal feature, denotes a normal distribution subject to denotes the mean of the normal distribution, denotes the covariance matrix of the normal distribution, wherein denotes the time step dependent scalar variance, is the identity matrix, denotes a bias; inputting the noise-removed perturbation sequence and the multi-modal feature into the extended long short-term memory network to update the hidden state at each step of the extended long short-term memory network, the expression being as follows: wherein, is the hidden state of the t-th step, is the hidden state of the previous step of the t-th step; generating a denoised gesture sequence from the updated hidden state using a decoder. 2.The deep learning-based collaborative speech gesture generation method of claim 1, wherein, The speech feature is specifically a frame-level speech feature extracted by a speech feature extractor from the speech. 3.The deep learning based collaborative speech gesture generation method of claim 1, wherein, The initial posture feature is specifically a posture feature extracted by a pre-trained skeleton sequence autoencoder from the initial gesture sequence. 4.The deep learning based collaborative speech gesture generation method of claim 1, wherein, The time embedding vector is specifically a time embedding vector obtained by processing a diffusion time step through a cosine transformation, the diffusion time step including a time step used for forward diffusion on the initial gesture sequence and iterative denoising of the perturbation sequence. 5.The deep learning based collaborative speech gesture generation method of claim 1, wherein, The method further comprises a step of designing a loss function, the expression of the loss function being as follows: wherein, represents a loss function, is a loss function for cross-modal fusion, is a loss function for the denoising process. 6.The deep learning based collaborative speech gesture generation method of claim 5, wherein, The loss function of the cross-modal fusion And the loss function of the denoising process The expression is as follows: wherein, represents a feature vector that fuses the speech feature, the initial pose feature, and the time embedding vector, represents a feature vector that fuses the speech feature and the initial pose feature, compresses the multi-modal feature after decompression, represents a predicted noise according to a perturbation sequence , a multi-modal feature , and a time step , represents an actual noise, represents an expected value operation. 7.A deep learning based collaborative speech gesture generation system, characterized by, The system comprises an interconnected microprocessor and a computer-readable storage medium, the microprocessor being programmed or configured to perform the deep learning-based collaborative speech gesture generation method of any one of claims 1-6.