Gesture generation method and system based on diffusion model with attention mechanism, readable storage medium

Abstract

The application relates to the fields of computer technology and artificial intelligence, and discloses a diffusion model gesture generation method and system with an attention mechanism and a readable storage medium, which comprises the following specific steps: a conditional embedding model comprising an encoder and an attention-based fusion network is constructed; multi-modal data is input into the conditional embedding model; comprehensive features of the multi-modal data are obtained through the encoder; the comprehensive features are fused and a gesture generation framework is constructed through the attention-based fusion network; a denoising model based on a latent diffusion model is constructed, wherein the denoising model comprises a self-attention network; the gesture generation framework is input into the denoising model; conditional information is extracted in the denoising model and an initial latent gesture sequence is generated; and the initial latent gesture sequence is subjected to forward diffusion and reverse diffusion according to the conditional information, so that a final gesture sequence is generated. The application solves the problem that the diversity and fidelity of features in the prior art are limited, and has the characteristics of high generation efficiency and quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer technology and artificial intelligence, and more particularly, to a gesture generation method, system and readable storage medium with attention mechanism of diffusion model. BACKGROUND

[0002] In the field of human-computer interaction, it is still a challenging task to correlate speech and gesture with each other. Early researchers such as Cassell, Kipp and Kopp proposed to rely on rule-based methods, using linguistic rules to convert speech into pre-defined gesture segments. These methods generate interpretable and controllable results, but require a lot of time and effort to create gesture datasets and design rules. To reduce manual work, researchers explored data-driven methods. Using data-driven methods, researchers began to learn mapping rules directly from data using statistical models. Early data-driven methods use statistical models to learn mapping rules and use them in combination with pre-defined gesture units. However, relying on pre-defined units limits the diversity and flexibility of generated gestures. With the rise of deep neural networks, researchers began to train complex end-to-end models directly from raw speech-gesture data for co-articulatory gesture generation. These models include deterministic models such as multi-layer perceptron (MLP), convolutional neural network (CNN) and recurrent neural network (RNN), as well as generative models such as flow-based models, variational autoencoder (VAE) and discrete VAE (VQ-VAE). However, the solutions proposed now focus on explicitly incorporating rhythm and semantic information, improving training supervision strategies to enhance content matching. In addition, hybrid systems, such as combining deep features and motion graphs, have been proposed to take advantage of the strengths of different methods.

[0003] Recently, there is prior art that models long-term temporal dependencies through a denoising model, effectively capturing cross-modal audio-to-gesture associations and maintaining temporal consistency, but they do not generate realistic and natural gestures.

[0004] However, the prior art has the problem of limited diversity and fidelity of features, so how to invent a high-diversity and high-fidelity gesture generation method is a technical problem that needs to be solved in the technical field. SUMMARY

[0005] In order to solve the problem of limited diversity and fidelity of features in the prior art, the present application provides a gesture generation method, system and readable storage medium with attention mechanism of diffusion model, which has the characteristics of high generation efficiency and quality.

[0006] To achieve the above-mentioned purposes of the present application, the technical solutions adopted are as follows:

[0007] The gesture generation method of the diffusion model with attention mechanism comprises the following specific steps:

[0008] A conditional embedding model comprising an encoder and an attention-based fusion network is constructed.

[0009] Multi-modal data is obtained and input into the conditional embedding model; in the conditional embedding model, the comprehensive features of the multi-modal data are obtained through the encoder; the comprehensive features are fused and a gesture generation framework is constructed through the attention-based fusion network.

[0010] A denoising model based on a latent diffusion model is constructed, wherein the denoising model comprises a self-attention network.

[0011] A seed gesture dataset is initialized; the seed gesture dataset and the gesture generation framework are input into the denoising model.

[0012] In the denoising model, the seed gesture dataset and the gesture generation framework are input into the self-attention network to extract conditional information and generate an initial latent gesture sequence; then, according to the conditional information, the initial latent gesture sequence is subjected to forward diffusion and reverse diffusion to generate a final gesture sequence.

[0013] Preferably, the encoder comprises an audio encoder, a text encoder, a speaker recognition encoder, and an emotion encoder.

[0014] Further, the audio encoder is composed of a 12-layer time convolutional network (TCN) with skip connection; the text encoder is composed of an 8-layer time convolutional network (TCN) with skip connection; the speaker recognition encoder is composed of an embedding layer (EID); the emotion encoder is composed of a 4-layer time convolutional network (TCN) and an embedding layer (Eemo); and the attention-based fusion network is composed of an attention-aware Transformer.

[0015] Further, in the conditional embedding model, the comprehensive features of the multi-modal data are obtained through the encoder, and the specific steps are as follows:

[0016] The audio in the multi-modal data is edited; the edited audio A is input into the audio encoder, and the audio encoder converts A into a down-sampled latent code feature wherein z i ∈R 128 , and C is the dimension of the latent space.

[0017] The text in the multi-modal data is converted into a word embedding set of early layers, wherein t i ∈R 300; input the word embedding set into the text encoder to fine-tune the word set, and obtain the final latent feature z of the text by merging data from a plurality of frames for each frame i text ∈R 128 ;

[0018] input the speaker information ID in the multi-modal data into the speaker recognition encoder to obtain the latent speaker feature Z ID =E ID (ID), wherein EID is an embedding layer;

[0019] input the emotion information in the multi-modal data into the emotion encoder to obtain the latent emotion feature Z emo =E emo (emo i-f ,…,emo i+f ), wherein wherein f represents the receptive field of the emotion encoder, and emo i-f ,…,emo i+f represent 2f+1 emotion vectors around the i-th gesture.

[0020] Further, the comprehensive feature is fused and a gesture generation framework is constructed through an attention-based fusion network, and the specific steps are as follows:

[0021] connect Z audio ,Z text ,Z ID , and Z emo to obtain an intermediate feature vector

[0022] project to obtain the final representation vector

[0023] through the sine absolute position encoding technology of the attention-based fusion network, the position encoding is integrated into to obtain a gesture generation framework integrated with audio, text, speaker ID, and emotion features.

[0024] Further, the denoising model based on the latent diffusion model is specific to:

[0025] x0′=∈ θ (x t ,t,z att )

[0026] wherein x0′ is the final gesture sequence, z att represents the extracted conditional information, ∈ θ is the denoising model and the measured ∈ θ ; the denoising model is used to minimize the predicted noise ∈ θMean square error loss between the real noise ∈.

[0027] Further, the seed pose dataset is combined with the gesture generation framework to input the self-attention network to extract the conditional information and generate an initial latent gesture sequence, and the specific steps are as follows:

[0028] The seed pose dataset P s = p1,…,p prompt is connected with the gesture generation framework to obtain an initial latent gesture sequence; wherein, p prompt is the length of the pose;

[0029] The initial latent gesture sequence is input into the self-attention network to extract the conditional information z att .

[0030] Further, according to the conditional information, the initial latent gesture sequence is forward diffused and backward diffused to generate a final gesture sequence, and the specific steps are as follows:

[0031] Based on the conditional information, the latent gesture sequence x0 of the seed pose dataset is diffused for t rounds to obtain t ∈ 0, 1,..., T, T is the maximum diffusion step, and Gaussian noise is added in the diffusion process to approximate the normal distribution

[0032]

[0033] Wherein, α t Based on the set variance plan, I is the set unit matrix; through the diffusion steps of t rounds, the gesture generation framework is forward diffused;

[0034] Based on the conditional information, x T is predicted and the noise in x is removed one by one until x0' after denoising is obtained, and x0' after denoising is taken as the generated final gesture sequence, and the negative diffusion is completed.

[0035] The diffusion model gesture generation system with attention mechanism includes a cascaded model construction module, a data acquisition module, a gesture generation framework construction module, a conditional information extraction module and a diffusion reasoning module.

[0036] The model construction module is used to construct a conditional embedding model including an encoder and an attention-based fusion network, and to construct a denoising model based on a latent diffusion model, and the denoising model includes a self-attention network.

[0037] The data acquisition module is used to acquire multi-modal data and initialize a seed pose dataset.

[0038] The gesture generation framework construction module is used for inputting multi-modal data into a conditional embedding model comprising an encoder and an attention-based fusion network; in the conditional embedding model, comprehensive features of the multi-modal data are obtained through the encoder; and the comprehensive features are fused and the gesture generation framework is constructed through the attention-based fusion network.

[0039] The diffusion reasoning module is used for initializing a seed gesture dataset; inputting the seed gesture dataset and the gesture generation framework into a denoising model; in the denoising model, the seed gesture dataset is combined with the gesture generation framework to input a self-attention network to extract conditional information and generate an initial latent gesture sequence, and then according to the conditional information, the initial latent gesture sequence is subjected to forward diffusion and reverse diffusion to generate a final gesture sequence.

[0040] A computer-readable storage medium comprises a diffusion model gesture generation program with an attention mechanism, which, when executed by a processor, implements the steps of the diffusion model gesture generation method with an attention mechanism.

[0041] The beneficial effects of the present application are as follows:

[0042] The present application constructs a conditional embedding model comprising an encoder and an attention-based fusion network, and a denoising model based on a latent diffusion model; the present application extracts multi-modal features through the conditional embedding model of the attention-based fusion network and constructs a gesture generation framework, and inputs the gesture generation framework into the denoising model to generate a final gesture sequence. The present application solves the problem of limited diversity and fidelity of features in the prior art, and has the characteristics of high generation efficiency and quality. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a flowchart of the diffusion model gesture generation method with an attention mechanism of the present application.

[0044] Figure 2 is a detailed framework diagram of the diffusion model gesture generation method with an attention mechanism of the present application in Example 2.

[0045] Figure 3 is a comparison diagram of visualized experimental results of the diffusion model gesture generation method with an attention mechanism of the present application on a BEAT dataset in Example 2.

[0046] Figure 4 is a comparison diagram of experimental results of the diffusion model gesture generation method with an attention mechanism of the present application under different modalities in Example 2. DETAILED DESCRIPTION

[0047] The present application will be described in detail below in conjunction with the drawings and specific embodiments.

[0048] Embodiment 1

[0049] As shown in the figure, the gesture generation method of the diffusion model with attention mechanism comprises the following specific steps: Figure 1

[0050] A conditional embedding model comprising an encoder and an attention-based fusion network is constructed;

[0051] Multi-modal data is obtained and input into the conditional embedding model; in the conditional embedding model, the comprehensive features of the multi-modal data are obtained through the encoder; the comprehensive features are fused and a gesture generation framework is constructed through the attention-based fusion network;

[0052] A denoising model based on a latent diffusion model is constructed, wherein the denoising model comprises a self-attention network;

[0053] A seed gesture dataset is initialized; the seed gesture dataset and the gesture generation framework are input into the denoising model;

[0054] In the denoising model, the seed gesture dataset is combined with the gesture generation framework to input into the self-attention network to extract conditional information and generate an initial latent gesture sequence; then, according to the conditional information, the initial latent gesture sequence is subjected to forward diffusion and reverse diffusion to generate a final gesture sequence.

[0055] As shown in the figure, in one specific embodiment, the encoder comprises an audio encoder, a text encoder, a speaker recognition encoder, and an emotion encoder. Figure 2 In one specific embodiment, the audio encoder is composed of a 12-layer time convolution network (TCN) with a skip connection; the text encoder is composed of an 8-layer time convolution network (TCN) with a skip connection; the speaker recognition encoder is composed of an embedding layer (EID); the emotion encoder is composed of a 4-layer time convolution network (TCN) and an embedding layer (Eemo); and the attention-based fusion network is composed of an attention-aware Transformer.

[0056] In one specific embodiment, in the conditional embedding model, the comprehensive features of the multi-modal data are obtained through the encoder, and the specific steps are as follows:

[0057] The audio in the multi-modal data is edited; the edited audio A is input into the audio encoder, and the audio encoder converts A into a down-sampled latent code feature

[0058] wherein z i ∈R 128 , C is the dimension of the latent space;

[0059] ​​Clipping the audio in the multi-modal data, specifically, representing the audio in the multi-modal data with raw audio waveforms and down-sampling it to 16kHz, treating the down-sampled audio as running at 15 frames per second, for each frame, the audio signal is represented by where a i ∈R 1067 ;

[0060] Converting the text in the multi-modal data into a set of word embeddings , where t i ∈R 300 ; inputting the set of word embeddings into a text encoder to fine-tune the set of words, for each frame i, combining data from a number of frames to obtain the final latent feature of the text z text ∈R 128 ;

[0061] In this embodiment, the text in the multi-modal data is converted into a set of word embeddings, specifically, using pre-trained parameter weights from fastText to convert the text in the multi-modal data into a set of word embeddings , where t i ∈R 300 ;

[0062] In this embodiment, for each frame i, the TCN combines data from 2f = 34 frames to create the final latent feature of the text, denoted as z text ∈R 128 ; the TCN takes as input a sequence of embeddings t i-f ,…,t i+f and applies a sequence of one-dimensional convolutions with different kernel sizes to capture local and global relationships between words. The TCN takes as input a sequence of embeddings t i-f ,…,t i+f and applies a sequence of one-dimensional convolutions with different kernel sizes to capture local and global relationships between words;

[0063] Inputting the speaker information ID in the multi-modal data into a speaker recognition encoder to obtain a latent speaker feature Z ID = E ID (ID), where EID is an embedding layer;

[0064] Inputting the sentiment information in the multi-modal data into a sentiment encoder to obtain a latent sentiment feature Z emo = E emo (emo i-f ,…,emo i+f ), where Z emoi ∈R 8 , where f denotes the receptive field of the sentiment encoder, emo i-f,…,emo i+f Let f(i) represent the 2f+1 emotion vectors surrounding the i-th gesture.

[0065] In one specific embodiment, a gesture generation framework is constructed by fusing comprehensive features and using an attention-based fusion network. The specific steps are as follows:

[0066] Z audio Z text Z ID , and Z emo Connect the features to obtain the intermediate feature vector.

[0067] Will Projecting onto the vector yields the final representation vector.

[0068] Considering the stable structure of the skeleton, we incorporate position encoding into the input sequence before passing it to the Transformer encoder;

[0069] By employing sinusoidal absolute position encoding based on an attention-based fusion network, this encoding method enables the attention-aware Transformer to effectively utilize and integrate joint position information; integrating position encoding into... In this process, a gesture generation framework integrating audio, text, speaker ID, and emotional features was obtained. These components together constitute a powerful gesture generation framework that integrates audio, text, speaker ID, and emotion.

[0070] In one specific embodiment, the denoising model based on the latent diffusion model is as follows:

[0071] x0′=∈ θ (x t ,t,z att )

[0072] Where x0′ is the final gesture sequence, z att Represents the extracted conditional information, ∈ θ For the denoising model and the measured ∈ θ The denoising model described above is used to minimize the predicted noise ∈ θ The mean squared error loss between the actual noise ∈.

[0073] In one specific embodiment, the seed pose dataset is combined with the gesture generation framework and input into the attention network to extract conditional information and generate an initial potential gesture sequence. The specific steps are as follows:

[0074] The seed pose dataset P is obtained through a sampling algorithm. s =p1,…,p promptConnecting with the gesture generation framework yields an initial potential gesture sequence; where p prompt It is the length of the posture;

[0075] The initial potential gesture sequence is input into the self-attention network to extract conditional information z. att .

[0076] In one specific embodiment, based on conditional information, the initial potential gesture sequence is subjected to forward and backward diffusion to generate the final gesture sequence. The specific steps are as follows:

[0077] Based on conditional information, the latent gesture sequence x0 in the seed pose dataset is diffused for t rounds to obtain t∈0,1,...,T, where T is the maximum number of diffusion steps. Gaussian noise is gradually added during the diffusion process to approximate a normal distribution.

[0078]

[0079] Where, α t Based on the set variance plan, I is the set identity matrix; the gesture generation framework is forward diffused through t rounds of diffusion steps;

[0080] Based on conditional information, from x T Start predicting and removing one by one The noise in the image is removed until the denoised x0′ is obtained. The denoised x0′ is then used as the final gesture sequence to complete the negative diffusion.

[0081] In this embodiment, to address the limitations of existing methods in terms of diversity and fidelity, as well as the mismatch between generated gestures and their semantic and emotional context, this study introduces a diffusion model gesture generation method with an attention mechanism. This method employs a complex denoising diffusion probability module to progressively transform a simple probability distribution into a more complex one, thereby generating a series of natural and diverse gestures. Furthermore, a pre-trained fastText model is used for text feature extraction, combined with an attention mechanism, to ensure that the generated gestures are consistent with the accompanying speech in terms of semantic content and emotional nuances.

[0082] In this embodiment, a series of rigorous objective experiments were conducted to verify the effectiveness of the proposed method. The results show that the diffusion model gesture generation method with an attention mechanism performs exceptionally well in generating natural and diverse gestures, accurately and coherently conveying the intended information, surpassing the benchmarks established by traditional methods. Specifically:

[0083] This invention has the following effects and advantages, which will have a positive impact on society, the economy, and technology:

[0084] High-quality gesture generation: The denoising model of this invention can recover high-quality original gestures from noisy latent sequences. Experiments were conducted on the large-scale gesture dataset BEAT using the diffusion model gesture generation method with an attention mechanism of this invention. Visualization of the experiments is attached. Figure 3 As shown, the denoising model in this invention is closer to the real value than existing models such as Speech2Gestrure, MultiContext, and CaMN, demonstrating the effectiveness of the denoising model in gesture generation tasks. On average, the denoising model maintains the naturalness and fluency of gestures while removing noise.

[0085] Multimodal fusion: This invention fuses audio, text, speaker ID, and emotional information to achieve multimodal gesture generation. We conducted comparative experiments on the fusion effects of different modalities, and the results are attached. Figure 4 As shown in the figure, the FGD and SRGR evaluation criteria show that the method of the present invention can better guarantee the diversity and realism of gesture generation compared to LSTM.

[0086] Efficient reasoning process: The reasoning process of this invention utilizes the DDPM sampling algorithm, which can quickly synthesize high-quality gesture sequences.

[0087] Application prospects: This invention can be applied to fields such as virtual characters, robots, and virtual assistants to improve the human-computer interaction experience.

[0088] Economic benefits: It improves the realism of virtual characters, which helps increase user engagement and satisfaction, thereby promoting the sales of related products.

[0089] In summary, the technical solution of this invention is innovative and practical in the field of gesture generation, and is expected to have a positive impact on society, economy and technology.

[0090] Example 2

[0091] A diffusion model gesture generation system with attention mechanism includes a cascaded model building module, a data acquisition module, a gesture generation framework building module, a conditional information extraction module, and a diffusion inference module;

[0092] The model building module is used to build a conditional embedding model including an encoder and an attention-based fusion network, and to build a denoising model based on a latent diffusion model, wherein the denoising model includes a self-attention network.

[0093] The data acquisition module is used to acquire multimodal data and initialize the seed pose dataset;

[0094] The gesture generation framework construction module is configured to input the multi-modal data into a conditional embedding model comprising an encoder and an attention-based fusion network; in the conditional embedding model, comprehensive features of the multi-modal data are obtained by the encoder; and the comprehensive features are fused and the gesture generation framework is constructed by the attention-based fusion network.

[0095] The diffusion reasoning module is configured to initialize a seed gesture dataset; input the seed gesture dataset and the gesture generation framework into a denoising model; in the denoising model, the seed gesture dataset is combined with the gesture generation framework to input into a self-attention network to extract conditional information and generate an initial latent gesture sequence, and then the initial latent gesture sequence is subjected to forward diffusion and reverse diffusion according to the conditional information to generate a final gesture sequence.

[0096] Embodiment 3

[0097] A computer-readable storage medium comprising a diffusion model gesture generation program with an attention mechanism, which, when executed by a processor, implements the steps of the diffusion model gesture generation method with an attention mechanism.

[0098] In this embodiment, the software environment of the diffusion model gesture generation program with an attention mechanism included in the computer-readable storage medium is a Python programming environment (Anaconda or Miniconda); the required Python libraries are installed, such as PyTorc, numpy, scipy, etc.; the audio processing library used is, for example, librosa; and the deep learning framework used for model training and inference.

[0099] The process of collecting the dataset is as follows: collect the gesture dataset, including audio, text and gesture motion annotations, and the present method uses the BEAT dataset.

[0100] The model training and evaluation method is as follows: use the collected dataset to train the audio encoder, text encoder, speaker recognition and emotion encoder; implement the denoising model and optimize the parameters to minimize the noise; and use cross-validation or other evaluation methods to evaluate the model performance.

[0101] The inference process is as follows: in actual application, the trained model is deployed into a generation system. The sampling algorithm of DDPM is used to synthesize the denoised gesture sequence.

[0102] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not intended to limit the embodiments of the present application. Any modification, equivalent replacement and improvement within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A diffusion model gesture generation method with attention mechanism, characterized in that: Comprise the following specific steps: A conditional embedding model comprising an encoder and an attention-based fusion network is constructed; Multimodal data including audio, text, speaker ID and emotion information is obtained, and the multimodal data is input into the conditional embedding model; in the conditional embedding model, the comprehensive features of the multimodal data are obtained through the encoder; The comprehensive features are fused and a gesture generation framework is constructed through the attention-based fusion network; A denoising model based on a latent diffusion model is constructed, and the denoising model comprises a self-attention network; A seed gesture dataset is initialized; the seed gesture dataset and the gesture generation framework are input into the denoising model; In the denoising model, the seed pose data set is combined with the gesture generation framework to input into a self-attention network to extract conditional information and generate an initial latent gesture sequence, and according to the conditional information, the initial latent gesture sequence is subjected to forward diffusion and backward diffusion to generate a final gesture sequence ; The denoising model based on the latent diffusion model comprises: wherein, denotes the extracted condition information; the denoising model is used to minimize the mean square error loss between the predicted noise and the real noise; The seed gesture dataset is combined with the gesture generation framework to extract conditional information and generate an initial latent gesture sequence in the self-attention network, and the specific steps are: seed pose dataset by a sampling algorithm obtaining an initial latent gesture sequence in connection with the gesture generation framework; Extracting conditional information from an initial potential gesture sequence into a self-attention network ; According to the conditional information, the initial latent gesture sequence is forward diffused and backward diffused to generate a final gesture sequence, and the specific steps are: Based on the conditional information, potential gesture sequences of the seed pose dataset Carrying out t rounds of diffusion to obtain , ∈[ , ], is the maximum diffusion step number, and Gaussian noise is added step by step in the diffusion process to approximate the normal distribution : wherein, for a variance plan based on the setting, is a set unit matrix; the gesture generation framework is forward diffused through a diffusion step of t rounds; Based on the conditional information, from Start predicting and removing noise in one by one until the denoised , the denoised as the final generated gesture sequence, complete the negative diffusion.

2. The gesture generation method of claim 1, wherein the diffusion model with attention mechanism is trained by: The encoder comprises an audio encoder, a text encoder, a speaker recognition encoder and an emotion encoder.

3. The gesture generation method of claim 2, wherein the diffusion model with attention mechanism is trained by: The audio encoder is composed of a 12-layer time convolution network (TCN) with skip connections; the text encoder is composed of an 8-layer time convolution network (TCN) with skip connections; the speaker recognition encoder is composed of an embedding layer ; the emotion encoder is composed of a 4-layer time convolution network (TCN) and an embedding layer ; and the attention-based fusion network is composed of an attention-aware Transformer.

4. The gesture generation method of claim 3, wherein the diffusion model with attention mechanism is trained by: In the conditional embedding model, the comprehensive features of the multimodal data are obtained through the encoder, and the specific steps are: clipping the audio in the multi-modal data; inputting the clipped audio A into an audio encoder to convert A into a down-sampled latent code feature wherein C is the dimension of the latent space; Converting text in multi-modal data into a set of word embeddings of early layers, wherein ; inputting the set of word embeddings into a text encoder to fine-tune the set of word embeddings, for each frame i, merging data from several frames to obtain final latent features of text ; Inputting the speaker ID information in the multi-modal data into a speaker recognition encoder to obtain latent speaker ID features wherein is an embedding layer; Inputting the sentiment information in the multi-modal data into a sentiment encoder to obtain latent sentiment features , wherein , wherein f represents a receptive field of the sentiment encoder, represents 2f+1 sentiment vectors around the i-th gesture.

5. The gesture generation method of claim 4, wherein the diffusion model with attention mechanism is trained by: The comprehensive features are fused and a gesture generation framework is constructed through the attention-based fusion network, and the specific steps are: Connect to get the intermediate feature vector Connect to get the intermediate feature vector ; Will Projecting onto a higher-dimensional space yields the final representation vector. ; By the sine absolute position encoding technology based on the attention fusion network, the position encoding is integrated into The gesture generation framework integrated with audio, text, speaker ID and emotion features is obtained.

6. A diffusion model gesture generation system with attention mechanism, characterized by: For realizing the method as claimed in any one of claims 1-5, comprising a cascaded model construction module, a data acquisition module, a gesture generation framework construction module, a conditional information extraction module and a diffusion reasoning module; The model construction module is used to construct a conditional embedding model comprising an encoder and an attention-based fusion network, and to construct a denoising model based on a latent diffusion model, wherein the denoising model comprises a self-attention network; The data acquisition module is used to acquire multimodal data and initialize a seed gesture dataset; The gesture generation framework construction module is used to input the multimodal data into the conditional embedding model comprising an encoder and an attention-based fusion network; in the conditional embedding model, the comprehensive features of the multimodal data are obtained through the encoder; The comprehensive features are fused and a gesture generation framework is constructed through the attention-based fusion network; The diffusion reasoning module is used to initialize a seed gesture dataset; the seed gesture dataset and the gesture generation framework are input into the denoising model; in the denoising model, the seed gesture dataset is combined with the gesture generation framework to extract conditional information and generate an initial latent gesture sequence in the self-attention network; and then according to the conditional information, the initial latent gesture sequence is forward diffused and backward diffused to generate a final gesture sequence.

7. A computer readable storage medium characterized in that, The computer-readable storage medium comprises a diffusion model gesture generation program with an attention mechanism, and when the diffusion model gesture generation program with the attention mechanism is executed by a processor, the steps of the diffusion model gesture generation method with the attention mechanism as claimed in any one of claims 1-5 are realized.

Images