A speech-driven gesture generation method based on semantic fusion and discrete coding
By constructing a semantically enhanced audio representation extraction module and a diffusion model generator, speech features are decoupled and weighted to generate gesture sequences that coordinate with speech rhythm, semantics, and style. This solves the problem of insufficient semantic expressiveness of gestures in existing technologies and achieves high-quality speech-driven gesture generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-03-02
- Publication Date
- 2026-05-29
Smart Images

Figure CN122111232A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of human-computer interaction and speech synthesis technology, specifically relating to a speech-driven gesture generation method based on semantic fusion and discrete coding. Background Technology
[0002] In the fields of virtual reality, social robotics, and human-computer interaction, endowing virtual agents with the ability to automatically generate voice-coordinated gestures is a key technology for improving the naturalness and immersion of interactions. Existing collaborative voice-gesture generation methods are typically based on multimodal data-driven approaches. They involve temporally aligning audio recordings and their corresponding transcribed text, then inputting these into a deep learning model to learn the mapping from speech to human posture. However, these methods largely focus on the temporal alignment of multimodal features. While they can ensure the synchronization of generated gestures with the rhythm of speech to some extent, their processing paradigm limits their ability to effectively separate and enhance information strongly correlated with the semantics of the text from the audio signal.
[0003] In practical applications, rhythmic information in speech signals often dominates, while semantic gestures, such as illustrative and metaphorical gestures, which carry specific meanings, are relatively sparse in the data. This causes existing time-alignment-based models to tend to fit rhythmic patterns during training, neglecting semantic features deeply bound to the language content. As a result, while the generated gestures may be rhythmically synchronized with the speech, their semantic expressiveness is weak, and their relevance to the language content is insufficient. They struggle to accurately convey abstract concepts and semantic emphasis through the visual channel, significantly limiting the expressive richness and realism of the generated gestures. Therefore, existing technologies suffer from the problem that multimodal feature processing methods cannot effectively decouple and enhance semantic information in speech, leading to the masking or loss of semantic information during generation, resulting in insufficient relevance to the language content and weak semantic expressiveness in the generated gestures. Summary of the Invention
[0004] To address the shortcomings of existing technologies, the present invention aims to provide a speech-driven gesture generation method based on semantic fusion and discrete coding. This method solves the problem in existing technologies where multimodal feature processing methods are difficult to effectively decouple and enhance semantic information in speech, resulting in semantic information being masked or lost during the generation process. Consequently, the generated gestures lack sufficient relevance to the language content and have weak semantic expressiveness.
[0005] The objective of this invention can be achieved through the following technical solutions: A speech-driven gesture generation method based on semantic fusion and discrete coding includes the following steps: The system acquires audio recordings, corresponding text transcriptions, and the actual gesture sequences and style identifiers of the speakers, and performs time-series alignment and segmentation to obtain data samples. Based on data samples, a semantically enhanced audio representation extraction module is constructed and trained. The text transcription content is used as the semantic anchor. Through contrastive learning, the speech and audio features are decoupled into semantically related features and rhythmic component features. The decoupled features are then adaptively weighted and fused with style identifiers to obtain semantically enhanced audio representations. A text-to-gesture module is built and trained based on data samples. Real gesture sequences are used as the reconstruction target, and the transcribed text content is used as the input to generate sparse gesture sequences associated with semantic content, which serve as semantic motion priors. A generator based on a diffusion model is built and trained based on data samples. Guided by the joint conditions formed by the fusion of semantically enhanced audio representation and semantic motion prior, a gesture sequence that is coordinated with the input speech in terms of rhythm, semantics and style is generated. During inference, the target speech audio, corresponding text transcription content, and speaker style identifier are obtained. Based on the trained semantically enhanced audio representation extraction module, text-to-gesture module, and generator, the final gesture sequence is generated.
[0006] Furthermore, the semantically enhanced audio representation extraction module includes a text encoder, a positive sample audio encoder, a negative sample audio encoder, and a style encoder; The transcribed text is converted into a sequence of word vectors using a pre-trained word embedding model, and then input into a text encoder to obtain text features. The speech audio is converted into a Mel spectrogram as the raw acoustic feature, which is then input into the positive sample audio encoder and the negative sample audio encoder respectively to obtain the positive sample audio feature and the negative sample audio feature. Define a semantic contrast loss function, using text features as positive sample anchors, to bring text features closer to positive sample audio features, while simultaneously increasing the distance between text features and negative sample audio features, as well as the distance between text features and other sample text features. By optimizing the semantic contrast loss function, the positive sample audio encoder learns to extract semantically relevant features aligned with the text semantics, while the negative sample audio encoder learns to extract rhythmic components that are unrelated to semantics. The style identifier is input into the style encoder to generate adaptive weights, and the semantically related features, rhythmic component features and original acoustic features are weighted and fused to obtain a semantically enhanced audio representation.
[0007] Furthermore, the semantic contrastive loss function The specific expression is as follows: In the formula, This represents the function for calculating cosine similarity. N Indicates the size of the training batch; and Representing the first in each batchn Positive and negative audio features of each sample; Indicates semantically relevant features; Indicates the characteristics of rhythmic components; Represents text features.
[0008] Furthermore, the semantically related features, rhythmic component features, and original acoustic features are weighted and fused, as shown in the following expression: In the formula, This represents semantically enhanced audio representation; , This represents a three-dimensional weight vector generated by the style encoder based on the style identifier, and all elements of the weight vector are non-negative. It represents the original acoustic characteristics.
[0009] Furthermore, the text-to-gesture module employs a vector quantization variational autoencoder structure, including a motion encoder. Motion decoder and text-to-code encoder ; During training, real gesture sequences are input into the motion encoder to obtain latent features. , Potential characteristics Discrete quantization code is obtained through vector quantization. The gesture sequence is then reconstructed by the motion decoder. , Simultaneously, the text will be transcribed into content. Input text to code encoder to obtain latent text representation ; Based on real gesture sequences, reconstructed gesture sequences Potential characteristics Discrete quantization code Textual latent representation Construct a comprehensive loss function; constrain the reconstruction of the gesture sequence by minimizing the comprehensive loss function. Consistency with real gesture sequences and enabling latent text representation With discrete quantization code Align in the potential space; During inference, only the transcribed text content is used as input. A sparse sequence of gestures associated with the semantic content is generated by a text-to-code encoder and a motion decoder, serving as a semantic motion prior.
[0010] Furthermore, the specific expression for the comprehensive loss function is as follows: In the formula, Indicates the overall loss; Represents a real gesture sequence; This indicates that the gradient operation is stopped.
[0011] Furthermore, the generator based on the diffusion model includes a denoising network that concatenates the semantically enhanced audio representation with the semantic motion prior in the feature dimension to form a joint conditional vector c; During training, noisy gesture sequences were used. The original clean gesture sequence is predicted by a denoising network using the diffusion time step t and the joint conditional vector c as inputs. A composite loss function including reconstruction loss, velocity loss and semantic contrast loss is constructed, and the denoising network is trained by optimizing the composite loss function. During inference, starting with random Gaussian noise, multi-step iterative denoising is performed guided by the joint condition vector c to generate the final gesture sequence.
[0012] Furthermore, the specific expression for the composite loss function is as follows: In the formula, Indicates compound loss; This represents the reconstruction loss based on Huber loss, ensuring overall consistency between the generated gesture and the real gesture; It is a speed loss that constrains the smoothness of gesture changes between adjacent frames; and It is a hyperparameter that balances the weights of various components; This represents the actual sequence of actions from the second frame to the last frame; This represents the actual sequence of actions from the first frame to the second-to-last frame; This represents the predicted action sequence from the second frame to the last frame; This represents the predicted action sequence from the first frame to the penultimate frame.
[0013] Furthermore, the denoising network adopts a stacked hybrid module architecture, where each module integrates a multi-head self-attention mechanism and a state space model, and injects the joint conditional vector and the embedding information of the diffusion time step through an adaptive instance normalization layer.
[0014] The beneficial effects of this invention are: This invention constructs a semantically enhanced audio representation extraction module, using text transcription content as semantic anchors. It employs a contrastive learning mechanism to actively decouple speech audio features into semantically relevant features and rhythmic component features, effectively solving the problem of semantic information being easily masked by rhythmic information in speech, thus achieving explicit enhancement and expression of semantic features. Simultaneously, by constructing an independent text-to-gesture module, it generates sparse gesture sequences associated with semantic content as explicit semantic motion priors, providing pure semantic guidance signals for subsequent gesture generation. Based on this, the semantically enhanced audio representation and semantic motion priors are fused to form joint conditions, which are input into a diffusion-based generator, allowing the generation process to be simultaneously guided by speech rhythm, deep semantics, and explicit semantic gesture priors. Therefore, this invention can generate natural gesture sequences that are not only rhythmically synchronized with speech but also precisely aligned with the meaning of language in terms of semantic content, and possess a personalized style, significantly improving the semantic expressiveness, realism, and naturalness of human-computer interaction of the generated gestures. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a schematic diagram of the overall process of the voice-driven gesture generation method of the present invention; Figure 2 This is a schematic diagram of the semantic enhancement audio representation extraction module of the present invention; Figure 3 This is a schematic diagram of the generator structure based on the diffusion model of the present invention; Figure 4 This is a schematic diagram of the Transformer-Mamba hybrid module structure in the denoising network of this invention; Figure 5 This is a schematic diagram of the text-to-gesture module structure of the present invention. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] like Figures 1 to 5 As shown, a speech-driven gesture generation method based on semantic fusion and discrete coding includes the following steps: The system acquires audio recordings, corresponding text transcriptions, and the actual gesture sequences and style identifiers of the speakers, and performs time-series alignment and segmentation to obtain data samples. Based on data samples, a semantically enhanced audio representation extraction module is constructed and trained. The text transcription content is used as the semantic anchor. Through contrastive learning, the speech and audio features are decoupled into semantically related features and rhythmic component features. The decoupled features are then adaptively weighted and fused with style identifiers to obtain semantically enhanced audio representations. A text-to-gesture module is built and trained based on data samples. Real gesture sequences are used as the reconstruction target, and the transcribed text content is used as the input to generate sparse gesture sequences associated with semantic content, which serve as semantic motion priors. A generator based on a diffusion model is built and trained based on data samples. Guided by the joint conditions formed by the fusion of semantically enhanced audio representation and semantic motion prior, a gesture sequence that is coordinated with the input speech in terms of rhythm, semantics and style is generated. During inference, the target speech audio, corresponding text transcription content, and speaker style identifier are obtained. Based on the trained semantically enhanced audio representation extraction module, text-to-gesture module, and generator, the final gesture sequence is generated.
[0019] This invention constructs a semantically enhanced audio representation extraction module, using text transcription content as semantic anchors. It employs a contrastive learning mechanism to actively decouple speech audio features into semantically relevant features and rhythmic component features, effectively solving the problem of semantic information being easily masked by rhythmic information in speech, thus achieving explicit enhancement and expression of semantic features. Simultaneously, by constructing an independent text-to-gesture module, it generates sparse gesture sequences associated with semantic content as explicit semantic motion priors, providing pure semantic guidance signals for subsequent gesture generation. Based on this, the semantically enhanced audio representation and semantic motion priors are fused to form joint conditions, which are input into a diffusion-based generator, allowing the generation process to be simultaneously guided by speech rhythm, deep semantics, and explicit semantic gesture priors. Therefore, this invention can generate natural gesture sequences that are not only rhythmically synchronized with speech but also precisely aligned with the meaning of language in terms of semantic content, and possess a personalized style, significantly improving the semantic expressiveness, realism, and naturalness of human-computer interaction of the generated gestures.
[0020] like Figure 2 As shown, the semantically enhanced audio representation extraction module includes a text encoder, a positive sample audio encoder, a negative sample audio encoder, and a style encoder; The transcribed text is converted into a sequence of word vectors using a pre-trained word embedding model, and then input into a text encoder to obtain text features. The speech audio is converted into a Mel spectrogram as the raw acoustic feature, which is then input into the positive sample audio encoder and the negative sample audio encoder respectively to obtain the positive sample audio feature and the negative sample audio feature. Define a semantic contrast loss function, using text features as positive sample anchors, to bring text features closer to positive sample audio features, while simultaneously increasing the distance between text features and negative sample audio features, as well as the distance between text features and other sample text features. By optimizing the semantic contrast loss function, the positive sample audio encoder learns to extract semantically relevant features aligned with the text semantics, while the negative sample audio encoder learns to extract rhythmic components that are unrelated to semantics. The style identifier is input into the style encoder to generate adaptive weights, and the semantically related features, rhythmic component features and original acoustic features are weighted and fused to obtain a semantically enhanced audio representation.
[0021] Semantic contrastive loss function The specific expression is as follows: In the formula, This represents the function for calculating cosine similarity. N Indicates the size of the training batch; and Representing the first in each batch n Positive and negative audio features of each sample; Indicates semantically relevant features; Indicates the characteristics of rhythmic components; Represents text features.
[0022] The semantically related features, rhythmic component features, and original acoustic features are weighted and fused, as shown in the following expression: In the formula, This represents semantically enhanced audio representation; , This represents a three-dimensional weight vector generated by the style encoder based on the style identifier, and all elements of the weight vector are non-negative. Indicates the original acoustic characteristics; In the formula, This indicates a style tag.
[0023] The text-to-gesture module employs a vector quantization variational autoencoder structure, including a motion encoder. Motion decoder and text-to-code encoder ; During training, real gesture sequences are input into the motion encoder to obtain latent features. , Potential characteristics Discrete quantization code is obtained through vector quantization. The gesture sequence is then reconstructed by the motion decoder. , Simultaneously, the text will be transcribed into content. Input text to code encoder to obtain latent text representation ; Based on real gesture sequences, reconstructed gesture sequences Potential characteristics Discrete quantization code Textual latent representation Construct a comprehensive loss function; constrain the reconstruction of the gesture sequence by minimizing the comprehensive loss function. Consistency with real gesture sequences and enabling latent text representation With discrete quantization code Alignment in the latent space; the training process of the text-to-gesture module (t2g module) is as follows: Figure 5 As shown; During inference, only the transcribed text content is used as input. A sparse sequence of gestures associated with the semantic content is generated by a text-to-code encoder and a motion decoder, serving as a semantic motion prior.
[0024] The specific expression for the comprehensive loss function is as follows: In the formula, Indicates the overall loss; Represents a real gesture sequence; This indicates that the gradient operation is stopped.
[0025] The generator based on the diffusion model includes a denoising network that concatenates semantically enhanced audio representations and semantic motion priors along the feature dimension to form a joint conditional vector c. During training, noisy gesture sequences were used. The original clean gesture sequence is predicted by a denoising network using the diffusion time step t and the joint conditional vector c as inputs. A composite loss function including reconstruction loss, velocity loss and semantic contrast loss is constructed, and the denoising network is trained by optimizing the composite loss function. During inference, starting with random Gaussian noise, multi-step iterative denoising is performed guided by the joint condition vector c to generate the final gesture sequence.
[0026] The specific expression for the composite loss function is as follows: In the formula, Indicates compound loss; This represents the reconstruction loss based on Huber loss, ensuring overall consistency between the generated gesture and the real gesture; It is a speed loss that constrains the smoothness of gesture changes between adjacent frames; and It is a hyperparameter that balances the weights of various components; This represents the actual sequence of actions from the second frame to the last frame; This represents the actual sequence of actions from the first frame to the second-to-last frame; This represents the predicted action sequence from the second frame to the last frame; This represents the predicted action sequence from the first frame to the penultimate frame.
[0027] The denoising network adopts a stacked hybrid module architecture, where each module integrates a multi-head self-attention mechanism and a state space model, and injects joint conditional vectors and embedding information of diffusion time steps through an adaptive instance normalization layer; The denoising network in this invention It adopts an innovative backbone architecture, such as Figure 4 As shown, it consists of multiple cascaded "Transformer-Mamba hybrid blocks". The specific structure of each hybrid block is shown in the attached figure. Figure 4 First, layer normalization is performed. Then, a multi-head self-attention module captures global dependencies. After further normalization, the input is fed into a Mamba module (selective state-space model) to efficiently model long-term dependencies. Finally, the joint conditional vector is processed through an adaptive instance normalization (AdaIN) layer. and diffusion time step This allows for fine-grained control by embedding information into features. This hybrid design balances global interaction with sequence processing efficiency.
[0028] Once the model is trained, it can be applied to real-world scenarios for gesture generation. During the inference phase, the user only needs to provide the target audio text and a specified style identifier. The system then executes two branches in parallel: one branch processes the audio to obtain a semantically enhanced audio representation. Another branch processes text transcription, generating semantic motion priors. Next, and Integration as a condition This data is then input into a pre-trained diffusion model generator. The generator draws from a randomly sampled Gaussian noise... Initially, a multi-step iterative denoising process is performed, with each step subject to conditions. And precise guidance from style identifiers. Finally, the synthesized gesture sequence is output. This sequence is not only highly synchronized with the prosodic rhythm of the input speech in time, but its gestures also closely match the semantic content of the speech and naturally reflect the specified personalized gesture style, thus achieving high-quality generation of semantic and rhythmic coordination.
[0029] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0030] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.
Claims
1. A speech-driven gesture generation method based on semantic fusion and discrete coding, characterized in that, Includes the following steps: The system acquires audio recordings, corresponding text transcriptions, and the actual gesture sequences and style identifiers of the speakers, and performs temporal alignment and segmentation to obtain data samples. Based on data samples, a semantically enhanced audio representation extraction module is constructed and trained. The text transcription content is used as the semantic anchor. Through contrastive learning, the speech and audio features are decoupled into semantically related features and rhythmic component features. The decoupled features are then adaptively weighted and fused with style identifiers to obtain semantically enhanced audio representations. A text-to-gesture module is built and trained based on data samples. Real gesture sequences are used as the reconstruction target, and the transcribed text content is used as the input to generate sparse gesture sequences associated with semantic content, which serve as semantic motion priors. A generator based on a diffusion model is built and trained based on data samples. Guided by the joint conditions formed by the fusion of semantically enhanced audio representation and semantic motion prior, a gesture sequence that is coordinated with the input speech in terms of rhythm, semantics and style is generated. During inference, the target speech audio, corresponding text transcription content, and speaker style identifier are obtained. Based on the trained semantically enhanced audio representation extraction module, text-to-gesture module, and generator, the final gesture sequence is generated.
2. The speech-driven gesture generation method based on semantic fusion and discrete coding according to claim 1, characterized in that, The semantically enhanced audio representation extraction module includes a text encoder, a positive sample audio encoder, a negative sample audio encoder, and a style encoder; The transcribed text is converted into a sequence of word vectors using a pre-trained word embedding model, and then input into a text encoder to obtain text features. The speech audio is converted into a Mel spectrogram as the raw acoustic feature, which is then input into the positive sample audio encoder and the negative sample audio encoder respectively to obtain the positive sample audio feature and the negative sample audio feature. Define a semantic contrast loss function, using text features as positive sample anchors, to bring text features closer to positive sample audio features, while simultaneously increasing the distance between text features and negative sample audio features, as well as the distance between text features and other sample text features. By optimizing the semantic contrast loss function, the positive sample audio encoder learns to extract semantically relevant features aligned with the text semantics, while the negative sample audio encoder learns to extract rhythmic components that are unrelated to semantics. The style identifier is input into the style encoder to generate adaptive weights, and the semantically related features, rhythmic component features and original acoustic features are weighted and fused to obtain a semantically enhanced audio representation.
3. The speech-driven gesture generation method based on semantic fusion and discrete coding according to claim 2, characterized in that, Semantic contrastive loss function The specific expression is as follows: In the formula, This represents the function for calculating cosine similarity. N Indicates the size of the training batch; and Representing the first in each batch n Positive and negative audio features of each sample; Indicates semantically relevant features; Indicates the characteristics of rhythmic components; Represents text features.
4. The speech-driven gesture generation method based on semantic fusion and discrete coding according to claim 2, characterized in that, The semantically related features, rhythmic component features, and original acoustic features are weighted and fused, as shown in the following expression: In the formula, This represents semantically enhanced audio representation; , This represents a three-dimensional weight vector generated by the style encoder based on the style identifier, and all elements of the weight vector are non-negative. It represents the original acoustic characteristics.
5. The speech-driven gesture generation method based on semantic fusion and discrete coding according to claim 1, characterized in that, The text-to-gesture module employs a vector quantization variational autoencoder structure, including a motion encoder. Motion decoder and text-to-code encoder ; During training, real gesture sequences are input into the motion encoder to obtain latent features. , Potential characteristics Discrete quantization code is obtained through vector quantization. The gesture sequence is then reconstructed by the motion decoder. , Simultaneously, the text will be transcribed into content. Input text to code encoder to obtain latent text representation ; Based on real gesture sequences, reconstructed gesture sequences Potential characteristics Discrete quantization code Textual latent representation Construct a comprehensive loss function; constrain the reconstruction of the gesture sequence by minimizing the comprehensive loss function. Consistency with real gesture sequences and enabling latent text representation With discrete quantization code Align in the potential space; During inference, only the transcribed text content is used as input. A sparse sequence of gestures associated with the semantic content is generated by a text-to-code encoder and a motion decoder, serving as a semantic motion prior.
6. The speech-driven gesture generation method based on semantic fusion and discrete coding according to claim 5, characterized in that, The specific expression for the comprehensive loss function is as follows: In the formula, Indicates overall loss; Represents a real gesture sequence; This indicates that the gradient operation is stopped.
7. The speech-driven gesture generation method based on semantic fusion and discrete coding according to claim 1, characterized in that, The generator based on the diffusion model includes a denoising network that concatenates semantically enhanced audio representations and semantic motion priors along the feature dimension to form a joint conditional vector c. During training, noisy gesture sequences were used. The original clean gesture sequence is predicted by a denoising network using the diffusion time step t and the joint conditional vector c as inputs. A composite loss function including reconstruction loss, velocity loss and semantic contrast loss is constructed, and the denoising network is trained by optimizing the composite loss function. During inference, starting with random Gaussian noise, multi-step iterative denoising is performed guided by the joint condition vector c to generate the final gesture sequence.
8. The speech-driven gesture generation method based on semantic fusion and discrete coding according to claim 7, characterized in that, The specific expression for the composite loss function is as follows: In the formula, Indicates compound loss; This represents the reconstruction loss based on Huber loss, ensuring overall consistency between the generated gesture and the real gesture; It is a speed loss that constrains the smoothness of gesture changes between adjacent frames; and It is a hyperparameter that balances the weights of various components; This represents the actual sequence of actions from the second frame to the last frame; This represents the actual sequence of actions from the first frame to the second-to-last frame; This represents the predicted action sequence from the second frame to the last frame; This represents the predicted action sequence from the first frame to the penultimate frame.
9. The speech-driven gesture generation method based on semantic fusion and discrete coding according to claim 7, characterized in that, The denoising network adopts a stacked hybrid module architecture, where each module integrates a multi-head self-attention mechanism and a state space model, and injects the joint conditional vector and the embedding information of the diffusion time step through an adaptive instance normalization layer.