Generative adversarial network collaborative gesture synthesis method and device based on dual-path Transformer

Abstract

The present invention discloses a generative adversarial network collaborative gesture synthesis method and device based on dual-path Transformer, and relates to the field of human-computer interaction technology. It includes: obtaining text data, audio data and speaker identity data, inputting them into a constructed Transformer-based cross-modal fusion network model, and obtaining gesture synthesis results; wherein, the Transformer-based cross-modal fusion network model includes: a front-end single-modal encoder module, a cross-modal fusion module and a gesture translation module. The present invention introduces an innovative multimodal fusion mechanism to achieve end-to-end collaborative voice gesture generation. The fusion mechanism is carried out through the synergy of parallel directional cross-modal Transformers and interactive cascaded two-dimensional attention modules, which fills the shortcomings of a single modality and helps the model focus on the context and voice information related to gestures. It realizes the generation of three-dimensional gestures without pre-definition, rather than simply predicting gesture categories, and achieves the best effect in implementation.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a generative adversarial network collaborative gesture synthesis method and device based on a dual-path Transformer. Background Art

[0002] In recent decades, as the world has increasingly used digital and virtual platforms for daily communication and interaction, HRI (Human-Robot Interaction) has been used to enhance traditional face-to-face interactions in various applications, such as online learning, interviews, video games, and virtual reality. These applications understand multimodal inputs, including verbal and non-verbal information, such as spoken language, voice, facial expressions, and body movements. As an important part of non-verbal communication, gestures refer to gestures made with hands, arms, head postures, and body postures. If virtual agents can use human-like gestures to accompany their language, gestures will play an important role in communication.

[0003] The existing patent (CN116524074A) discloses a method, device, equipment and storage medium for generating digital human gestures, and specifically discloses that the embodiment of the present invention provides a method, device, equipment and storage medium for generating digital human gestures, the method comprising: obtaining a target audio file for generating digital human gestures; determining the action sequence corresponding to the target audio file based on a script generation model; and controlling the generated representative gestures and rhythmic gestures to be synthesized into digital human gestures corresponding to the target audio file based on the action sequence and the gesture generation model. The method provided by the present invention effectively controls the synthesis of digital human gestures under synchronous speech through the action sequence corresponding to the target audio file determined by the script generation model, decouples and models the gestures to obtain a representative gesture generation model and a rhythmic gesture generation model, and combines the representative gestures and rhythmic gestures obtained by the gesture generation model to generate more natural and rich gestures, making the effect of digital human gestures more realistic.

[0004] The existing patent (CN113674184A) discloses a method, device, equipment and storage medium for generating body gestures of a virtual speaker, and specifically discloses the data processing technology field involved in the embodiments of this application, specifically a method, device, equipment and storage medium for generating body gestures of a virtual speaker, which aims to increase the naturalness of the synthesis result of the body gestures of the virtual speaker and improve the information transmission effect of the virtual image. The method comprises: inputting the speech audio, speech text and initialization gesture sequence into the self-attention body gesture generation network; extracting features of the speech audio, speech text and initialization gesture sequence through the self-attention body gesture generation network to obtain audio features, text features and initialization gesture sequence features; fusing the audio features, text features and initialization gesture sequence features through the self-attention body gesture generation network to obtain a synthesized gesture sequence.

[0005] Collaborative speech and gestures have a significant impact on conveying information. For social agents, generating realistic and fluent gestures is crucial to achieve natural interactions with humans, which is a challenging task that depends on many influencing factors (such as speech audio, content, and interacting objects).

[0006] It is theorized that speech and gesture are highly correlated. Therefore, in recent decades, a large number of learning-based methods have focused on generating human gestures that co-occur with speech content.

[0007] The rule-based method requires a lot of manpower to define various rules for speech-gesture mapping, which consumes a lot of manpower and time (Wagner P, Malisz Z, Kopp S. Gesture and speech in interaction: Anoverview[J]. Speech Communication, 2014, 57: 209-232.). Therefore, due to the limited set of designed gestures, the rule-based system cannot fully capture the rich gestures of humans (Kucherenko T, Hasegawa D, Henter GE, et al. Analyzing input and output representations for speech-driven gesture generation[C] / / Proceedings of the 19th ACM InternationalConference on Intelligent Virtual Agents. 2019: 97-104.). In addition, when new speech or text appears, manual settings are required to define new gestures. Despite this, it is still the mainstream technology for commercial robots due to its simple operation. For example, NAO and Pepper robots (Robotics, S.: pepper and NAO robotseducation) have a set of gestures specified and rules designed to link phonetic words with gestures.

[0008] In order to overcome the dependence of rule-based methods on human labor, data-driven methods have been proposed to find rules based on human demonstrations through machine learning techniques. Thanks to the recent progress in deep neural networks and massive data, complex problems are being solved in an end-to-end manner without breaking them down into smaller sub-problems (Yoon, Y.,Ko, W.-R., Jang, M., Lee, J., Kim, J., Lee, G.: Robots learn social skills:End-to-end learning of co-speech gesture generation for humanoid robots. In:International Conference on Robotics and Automation, pp. 4303–4309 (2019).IEEE). Although they rely on complex models to make up for the shortcomings of humans in designing rules, the generated gestures are more realistic than rule-based gestures, so they have higher attention and are also the focus of this technology.

[0009] In addition, various modalities can be used as input to drive gesture generation models. Many methods also use text records to generate collaborative speech gestures. In the work of Yoon et al., RNN (Recurrent Neural Network) is used to learn text-gesture mapping. Despite the success of gesture translation, these text-based methods ignore the relationship between gestures and speech acoustic information such as intonation, prelude, and loudness. Only a few works consider using audio and text to drive gesture generation models, such as Chiu et al. (Chiu, C.-C., Morency, L.-P., Marsella, S.: Predicting co-verbal gestures: a deep and temporal modeling approach. In: International Conference on Intelligent Virtual Agents, pp.152–166 (2015). Springer), but it predicts predefined gesture categories instead of generating arbitrary gestures; the closest to the technology of the present invention is another work by Yoon et al. (Yoon, Y., Cha, B., Lee, J.-H., Jang, M., Lee, J., Kim, J., Lee, G.: Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics 39(6), 1–16 (2020)), which uses both audio and text as input. However, the encoded unimodal features are simply concatenated for gesture generation, and the relationship between the modalities is not taken into account. Summary of the invention

[0010] In order to solve the technical problem that in the prior art, the encoded single-modal features are simply connected in series, while the relationship between modalities is not taken into account during the gesture generation process, the embodiment of the present invention provides a generative adversarial network collaborative gesture synthesis method and device based on a dual-path Transformer. The technical solution is as follows:

[0011] On the one hand, a generative adversarial network collaborative gesture synthesis method based on a dual-path Transformer is provided, the method is implemented by a gesture synthesis device, and the method includes:

[0012] S1. Obtain input data for gesture synthesis.

[0013] The input data includes text data, audio data and speaker identity data.

[0014] S2. Input the input data into the constructed Transformer-based cross-modal fusion network model.

[0015] Among them, the Transformer-based cross-modal fusion network model includes: a front-end single-modal encoder module, a cross-modal fusion module, and a gesture translation module.

[0016] S3. Obtain the gesture synthesis result based on the input data and the Transformer-based cross-modal fusion network model.

[0017] Optionally, the construction process of the Transformer-based cross-modal fusion network model in S2 includes:

[0018] S21. Obtain sample data.

[0019] The sample data includes an audio input sequence, a text input sequence, and a speaker ID.

[0020] S22. Input the sample data into the front-end single-modal encoder module to obtain audio feature representation, text feature representation, and speaker identity feature representation.

[0021] S23, input the audio feature representation and the text feature representation into a cross-modal fusion module to generate an audio-text cross feature representation, and obtain a multimodal feature representation based on the audio-text cross feature representation and the speaker identity feature representation.

[0022] S24. Input the multimodal feature representation into the gesture translation module to obtain a three-dimensional gesture synthesis result.

[0023] Optionally, the front-end single-mode encoder includes: an audio encoder, a text encoder, and a speaker ID encoder.

[0024] In S22, the sample data is input into the front-end single-mode encoder module to obtain audio feature representation, text feature representation and speaker identity feature representation, including:

[0025] S221, input the audio input sequence into the audio encoder, and encode the audio input sequence using a cascaded one-dimensional convolutional layer to obtain an audio feature representation, as shown in the following formula (1):

[0026] (1)

[0027] In the formula, represents the audio feature representation, Represents the audio codec, , Indicates the length of the audio input sequence, represents the field of real numbers, Indicates the duration of the audio feature. represents the audio feature dimension, Represents an audio input sequence.

[0028] S222. Input the text input sequence into the text encoder, use a one-hot vector to represent the index of each word in the text input sequence, and obtain a text feature representation.

[0029] S223: Input the speaker ID into the speaker ID encoder, encode the speaker ID using a one-hot vector, and obtain a speaker identity feature representation.

[0030] Optionally, the cross-modal fusion module includes: a directional cross-modal attention module and a two-dimensional attention module.

[0031] In S23, the audio feature representation and the text feature representation are input into the cross-modal fusion module to generate an audio-text cross feature representation, including:

[0032] S231. Input the audio feature representation and the text feature representation into a directional cross-modal attention module to obtain an audio-to-text feature representation and a text-to-audio feature representation, and then obtain a cross-feature representation.

[0033] S232. Input the cross-feature representation into a two-dimensional attention module, perform self-attention along the feature dimension and self-attention along the time dimension, and interact to obtain an audio-text cross-feature representation.

[0034] Optionally, the feature representation of audio to text in S231 is as shown in the following formula (2):

[0035] (2)

[0036] In the formula, Represents the feature representation of audio to text, represents the cross-attention mechanism from audio to text, Represents the input text features, represents the input audio features, , represents the learnable weights, , represents the learnable weights, represents the transpose of a matrix, express The characteristic dimension of The Value matrix represents the value of the feature representation. represents the learnable weights.

[0037] Optionally, the self-attention along the feature dimension in S232 is as shown in the following formula (3):

[0038] (3)

[0039] In the formula, represents the self-attention along the feature dimension, express The time of the self-attention module through the feature dimension, represents the feature representation of the concatenated feature dimensions, , Indicates splicing, Represents the feature representation of text to audio, represents a full connection operation, The query matrix represents the feature representation along the feature dimension, The key matrix representing the feature representation along the feature dimension, represents the length of the time series, The Value matrix represents the value of the feature representation along the feature dimension, represents the dimension of the matrix, Indicates the length of the gesture;

[0040] Self-attention along the time dimension is shown in equation (4):

[0041] (4)

[0042] In the formula, represents the self-attention along the time dimension, Represents the feature representation of the time dimension Time after splicing, The query matrix representing the feature representation along the time dimension, The key matrix representing the feature representation along the time dimension, A matrix of values ​​representing the feature representation along the time dimension.

[0043] Optionally, the construction process of the Transformer-based cross-modal fusion network model in S2 further includes:

[0044] The mean pose difference (MPD) is used to evaluate the Transformer-based cross-modal fusion network model.

[0045] On the other hand, a dual-path Transformer-based generative adversarial network collaborative gesture synthesis device is provided, and the device is applied to a dual-path Transformer-based generative adversarial network collaborative gesture synthesis method, and the device includes:

[0046] The acquisition module is used to acquire input data for gesture synthesis.

[0047] The input data includes text data, audio data and speaker identity data.

[0048] The input module is used to input the input data into the constructed Transformer-based cross-modal fusion network model.

[0049] Among them, the Transformer-based cross-modal fusion network model includes: a front-end single-modal encoder module, a cross-modal fusion module, and a gesture translation module.

[0050] The output module is used to obtain the gesture synthesis result based on the input data and the Transformer-based cross-modal fusion network model.

[0051] Optionally, the input module is further configured to:

[0052] S21. Obtain sample data.

[0053] The sample data includes an audio input sequence, a text input sequence, and a speaker ID.

[0054] S22. Input the sample data into the front-end single-modal encoder module to obtain audio feature representation, text feature representation, and speaker identity feature representation.

[0055] S23, input the audio feature representation and the text feature representation into a cross-modal fusion module to generate an audio-text cross feature representation, and obtain a multimodal feature representation based on the audio-text cross feature representation and the speaker identity feature representation.

[0056] S24. Input the multimodal feature representation into the gesture translation module to obtain a three-dimensional gesture synthesis result.

[0057] Optionally, the front-end single-mode encoder includes: an audio encoder, a text encoder, and a speaker ID encoder.

[0058] Input module, further used for:

[0059] S221, input the audio input sequence into the audio encoder, and encode the audio input sequence using a cascaded one-dimensional convolutional layer to obtain an audio feature representation, as shown in the following formula (1):

[0060] (1)

[0061] In the formula, represents the audio feature representation, Represents the audio codec, , Indicates the length of the audio input sequence, represents the field of real numbers, Indicates the duration of the audio feature. represents the audio feature dimension, Represents an audio input sequence.

[0062] S222. Input the text input sequence into the text encoder, use a one-hot vector to represent the index of each word in the text input sequence, and obtain a text feature representation.

[0063] S223: Input the speaker ID into the speaker ID encoder, encode the speaker ID using a one-hot vector, and obtain a speaker identity feature representation.

[0064] Optionally, the cross-modal fusion module includes: a directional cross-modal attention module and a two-dimensional attention module.

[0065] Input module, further used for:

[0066] S231. Input the audio feature representation and the text feature representation into a directional cross-modal attention module to obtain an audio-to-text feature representation and a text-to-audio feature representation, and then obtain a cross-feature representation.

[0067] S232. Input the cross-feature representation into a two-dimensional attention module, perform self-attention along the feature dimension and self-attention along the time dimension, and interact to obtain an audio-text cross-feature representation.

[0068] Optionally, the feature representation of audio to text is as shown in the following formula (2):

[0069] (2)

[0070] In the formula, Represents the feature representation of audio to text, represents the cross-attention mechanism from audio to text, Represents the input text features, represents the input audio features, , represents the learnable weights, , represents the learnable weights, represents the transpose of a matrix, express The characteristic dimension of The Value matrix represents the value of the feature representation. represents the learnable weights.

[0071] Optionally, self-attention along the feature dimension is as shown in Equation (3):

[0072] (3)

[0073] In the formula, represents the self-attention along the feature dimension, express The time of the self-attention module through the feature dimension, The feature representation representing the concatenated feature dimension, , Indicates splicing, Represents the feature representation of text to audio, represents a full connection operation, The query matrix represents the feature representation along the feature dimension, The key matrix representing the feature representation along the feature dimension, represents the length of the time series, The Value matrix represents the value of the feature representation along the feature dimension, represents the dimension of the matrix, Indicates the length of the gesture;

[0074] Self-attention along the time dimension is shown in equation (4):

[0075] (4)

[0076] In the formula, represents the self-attention along the time dimension, Represents the feature representation of the time dimension Time after splicing, The query matrix representing the feature representation along the time dimension, The key matrix representing the feature representation along the time dimension, A matrix of values ​​representing the feature representation along the time dimension.

[0077] Optionally, the input module is further configured to:

[0078] The mean pose difference (MPD) is used to evaluate the Transformer-based cross-modal fusion network model.

[0079] On the other hand, a gesture synthesis device is provided, which includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned generative adversarial network collaborative gesture synthesis methods based on dual-path Transformer is implemented.

[0080] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned dual-path Transformer-based generative adversarial network collaborative gesture synthesis methods.

[0081] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0082] In the embodiment of the present invention, as an important part of non-verbal communication, gestures refer to gestures made with hands, arms, head postures and body postures. If virtual agents can use human-like gestures to match their language, then gestures will play an important role in communication. Since gestures may change with the change of language, there is no need for predefined gesture categories, but arbitrary gestures can be generated. Therefore, it can be deduced that the advantages of the present invention are:

[0083] (1) Generating 3D gestures: The present invention differs from conventional methods in that it is able to generate 3D gestures rather than simply predicting gesture categories. And the generated gestures are not limited to pre-determined categories, but are freely generated. This improves the realism and expressiveness of the generated gestures, making them closer to the natural way humans communicate.

[0084] (2) The key to achieving natural interaction: This method achieves natural interaction with humans by generating realistic and smooth gestures. This is crucial for social agents to communicate effectively and naturally with users in various applications.

[0085] (3) Solving the cross-modal fusion problem: By introducing a novel fusion mechanism, especially the parallel directional cross-modal Transformer and the interactive cascaded two-dimensional attention module, this method successfully solves the fusion problem of cross-modal information (speech and gesture).

[0086] (4) Selective fusion of gesture-related clues: Through mechanism design, this method achieves the selective fusion of gesture-related clues, which can improve the accuracy and adaptability of generated gestures, making the generated gestures more in line with the context and content.

[0087] (5) Evaluation method without 3D pose annotation: We propose new metrics for evaluating gesture diversity and speech-gesture correspondence without the need for cumbersome 3D pose annotations. This simplifies the evaluation process and makes the method easier to apply and generalize. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0089] Figure 1 It is a flow chart of a generative adversarial network collaborative gesture synthesis method based on dual-path Transformer provided by an embodiment of the present invention;

[0090] Figure 2 It is a framework diagram of a generative adversarial network collaborative gesture synthesis method based on a dual-path Transformer provided by an embodiment of the present invention;

[0091] Figure 3 is a framework diagram of an audio, text and speaker identity encoder provided by an embodiment of the present invention;

[0092] Figure 4 is a directional cross-modal attention framework diagram provided by an embodiment of the present invention;

[0093] Figure 5 It is a two-dimensional attention module framework provided by an embodiment of the present invention Block diagram;

[0094] Figure 6 It is a two-dimensional attention module framework provided by an embodiment of the present invention Block diagram;

[0095] Figure 7 It is a block diagram of a generative adversarial network collaborative gesture synthesis device based on a dual-path Transformer provided by an embodiment of the present invention;

[0096] Figure 8 It is a structural schematic diagram of a gesture synthesis device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0097] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0098] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0099] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0100] In the embodiments of the present invention, sometimes the subscripts such as W 1 It may be written in non-subscript form such as W1. When the difference is not emphasized, the meaning is the same.

[0101] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0102] The embodiment of the present invention provides a generative adversarial network collaborative gesture synthesis method based on dual-path Transformer, which can be implemented by a gesture synthesis device, which can be a terminal or a server. Figure 1 The flowchart of the generative adversarial network collaborative gesture synthesis method based on dual-path Transformer is shown. The processing flow of the method may include the following steps:

[0103] S1. Obtain input data for gesture synthesis.

[0104] The input data may include text data, audio data, speaker identity data, etc.

[0105] S2. Input the input data into the constructed Transformer-based cross-modal fusion network model.

[0106] Among them, the Transformer-based cross-modal fusion network model may include: a front-end single-modal encoder module, a cross-modal fusion module, and a gesture translation module.

[0107] Transformer: A neural network architecture based on the self-attention mechanism, widely used in natural language processing and other sequence learning tasks, which realizes parallel processing of sequence data through the attention mechanism.

[0108] Optionally, the construction process of the Transformer-based cross-modal fusion network model in S2 may include the following steps S21-S24:

[0109] S21. Obtain sample data.

[0110] The sample data may include an audio input sequence, a text input sequence, and a speaker ID.

[0111] In a feasible implementation manner, the present invention generates the required three-dimensional upper body gestures ( ). Set the input to involve three modes: text ( )、Audio( ) and speaker identity ( ).

[0112] Furthermore, the present invention will and represent audio and text input sequences respectively ( and represents the original audio and text sequence length), The goal of this invention is to predict the simultaneous three-dimensional gestures ( represents the gesture length), train an end-to-end model, and its formula is:

[0113] (1)

[0114] in, It is a model A gesture is usually represented by different human joints. coordinate: in is the number of human joints considered. , which is equivalent to ten joints in three-dimensional coordinates (spine, neck, nose, head, left and right shoulders, elbows, and wrists).

[0115] like Figure 2 As shown in Figure 1, the overall architecture of the TCMF (Transformer-based Cross-Modal) network consists of three main stages, including S22-S24:

[0116] S22. Input the sample data into the front-end single-modal encoder module to obtain audio feature representation, text feature representation, and speaker identity feature representation.

[0117] Alternatively, if Figure 3 As shown in the figure, the front-end uni-modal encoders, namely the audio encoder, the text encoder and the speaker ID encoder, are used to independently extract the feature representation of each modality. Given an input gesture sequence, the present invention first encodes each modality into frame-level features.

[0118] The present invention is used represents the encoded unimodal features extracted from the same utterance where In the present invention, and Represent the sequence duration and feature dimension respectively.

[0119] The above step S22 may include the following steps S221-S223:

[0120] S221. Input the audio input sequence into the audio encoder, and encode the audio input sequence using a cascaded one-dimensional convolutional layer to obtain an audio feature representation.

[0121] In one possible implementation, the audio representation encodes the original audio wave using cascaded one-dimensional convolutional layers to obtain an equal amount of encoded audio vectors and output gesture frames (e.g. Figure 3 Middle (a)):

[0122] (2)

[0123] In the formula, represents the audio feature representation, Represents the audio codec, , Indicates the length of the audio input sequence, represents the field of real numbers, Indicates the duration of the audio feature. represents the audio feature dimension, Represents an audio input sequence.

[0124] S222. Input the text input sequence into the text encoder, use a one-hot vector to represent the index of each word in the text input sequence, and obtain a text feature representation.

[0125] In one feasible implementation, text representation: a text is a series of words, and each word is represented by a one-hot vector to represent the index of the word dictionary.

[0126] To align the time with other modalities, padding markers (0 index) are inserted in the known utterance time. Taking the spoken text “This is an amazing story” as an example, the corresponding five text markers are placed at the word start frame, while the other frames use padding markers. FastText is chosen to provide pre-trained text embeddings because its subword-level encoding strategy can more accurately represent unseen words. Then, a temporal convolutional network (see Appendix) with convolutional operations is used. Figure 3 (c) to process sequential text information. The formula is as follows:

[0127] (3)

[0128] in, Represents a text encoder.

[0129] S223: Input the speaker ID into the speaker ID encoder, encode the speaker ID using a one-hot vector, and obtain a speaker identity feature representation.

[0130] In one feasible implementation, speaker identity representation: considering that different people have different gestures under the same speech input, the speaker identity information is encoded by a one-hot vector to reflect the individual gesture characteristics. Figure 3 As shown in (b), the one-hot vector first enters the embedding layer, then enters three fully connected (FC) layers, and finally the outputs of the two FC layers are added together. The formula is as follows:

[0131] (4)

[0132] In the formula, Indicates the speaker identity feature representation, represents the speaker ID encoder, Indicates the length of the audio input sequence, Indicates the duration of speaker identity feature, It is important to note that there is only one speaker in a gesture segment, so exist There is no change in dimensions.

[0133] All three modalities are resampled to the same frame rate as the hand gestures (15 frames per second) through the above encoder, where In addition, the length of audio and text features is the same, i.e. .

[0134] S23, input the audio feature representation and the text feature representation into a cross-modal fusion module to generate an audio-text cross feature representation, and obtain a multimodal feature representation based on the audio-text cross feature representation and the speaker identity feature representation.

[0135] Optionally, a cross-modal fusion stage (i.e., cross-modal Transformer and 2D attention) fuses independently encoded features to generate more representative multi-modal features for gesture generation.

[0136] In a feasible implementation, for the cross-modal Transformer: directly concatenating independently encoded unimodal features cannot fully utilize the task-related cross-modal information. The present invention proposes a method of using a directional cross-modal Transformer to realize potential audio-text to assist in the generation of end-to-end gestures.

[0137] A general Transformer block consists of a cascaded encoder and decoder connected by paired encoder-decoder attention blocks. In this paper, the core is the directional cross-modal attention (e.g. Figure 4 ), it can learn the interactions between audio to text and text to audio throughout the segment.

[0138] The above step S23 may include the following steps S231-S232:

[0139] S231. Input the audio feature representation and the text feature representation into a directional cross-modal attention module to obtain an audio-to-text feature representation and a text-to-audio feature representation, and then obtain a cross-feature representation.

[0140] In one possible implementation, given the audio and text features and After that, the present invention will and They are respectively represented as the input of the Transformer encoder-decoder attention module. Query is defined as , define Keys as , define Values ​​as . , and are learnable weights, where and Respectively represent and as well as The latent adaptation from audio to text is achieved through cross-modal attention, as Figure 4 As shown, the formula is as follows:

[0141] (5)

[0142] In the formula, Represents the feature representation of audio to text, represents the cross-attention mechanism from audio to text, Represents the input text features, represents the input audio features, , represents the learnable weights, , represents the learnable weights, represents the transpose of a matrix, express The characteristic dimension of The Value matrix represents the value of the feature representation. represents the learnable weights.

[0143] for , the calculation method is the same as the above formula, but the positions of E and A are swapped. and Represented as the final output of two parallel cross-modal Transformers. Then, they are concatenated to form the final audio-text cross representation:

[0144] (6)

[0145] in Indicates connection, . We do not use causal masks in the cross-modal Transformer so that the module can learn contextual information from the entire segment.

[0146] S232. Input the cross-feature representation into a two-dimensional attention module, perform self-attention along the feature dimension and self-attention along the time dimension, and interact to obtain an audio-text cross-feature representation.

[0147] In a feasible implementation, for the 2D attention module: Based on preliminary experiments, the present invention finds that the cross-modal Transformer alone is not sufficient to learn a good representation for this deformable gesture generation task. Therefore, the present invention proposes a 2D attention module to simulate the temporal and feature dynamics of audio-text representations.

[0148] Specifically, the present invention learns the representation Self-attention is applied along the time dimension and feature dimension respectively, and they interact with each other. The two-dimensional attention module consists of stacked interactive and Module composition, details such as Figure 5 , Figure 6 For example, in piece( Figure 6 )middle, It is entered along the feature dimension, Query 、Key and Value is the learning weight through the fully connected layer , and The calculated ( is the embedding length). and Compute the self-attention matrix to focus on the value Finally, the result goes through a fully connected layer to make it equal to the input dimension:

[0149] (7)

[0150] In the formula, represents the self-attention along the feature dimension, express The time of the self-attention module through the feature dimension, where , It represents the feature representation of the concatenated feature dimension. In the following text, the subscript containing C represents the concatenated variable. , Indicates connection splicing, Represents the feature representation of text to audio, represents a full connection operation, The query matrix representing the feature representation, represents the key matrix of feature representation, represents the length of the time series, The Value matrix represents the value of the feature representation. represents the dimension of the matrix, Indicates the gesture length.

[0151] for The calculation process is the same, except Perform self-attention along the time dimension:

[0152] (8)

[0153] In the formula, represents the self-attention along the time dimension, Represents the feature representation of the time dimension Time after splicing, The query matrix representing the feature representation along the time dimension, The key matrix representing the feature representation along the time dimension, A matrix of values ​​representing the feature representation along the time dimension.

[0154] Furthermore, the transposed join in , and then add a fully connected layer to achieve the The result of the second two-dimensional attention fusion (temporal feature). The final output after the two-dimensional attention module is:

[0155] (9)

[0156] in, Represents pairwise addition.

[0157] The entire 2D attention module consists of multiple cascaded time / feature interaction modules, which gradually update the multimodal features. The above steps are repeated The final encoding is expressed as .

[0158] S24. Input the multimodal feature representation into the gesture translation module to obtain a three-dimensional gesture synthesis result.

[0159] In a feasible implementation, the gesture translation stage includes a pose generator and a pose discriminator.

[0160] For gesture generation: an adversarial scheme is adopted in end-to-end training to generate more realistic human gestures. For each training segment, the generator: GRU (bi-directional Gated Recurrent Unit): uses the encoded multimodal latent representation As input, a gesture is generated. The discriminator is used as a binary classifier to distinguish between real gestures and generated gestures. It uses a fully connected layer to aggregate the outputs of multi-layer bidirectional GRU to make a binary decision of real or fake gestures.

[0161] Specific training parameters of the present invention: The present invention uses the Adam optimizer, the learning rate is 0.0005, and the weight decay is 100% for each epoch. and . It is trained for 100 epochs with a batch size of 128. The first 10 epochs are used for warm-up and no adversarial training is used. Independently encoded audio, text, and speaker identity features ( Figure 3 ) are 32, 32, and 16 respectively (corresponding to Figure 2 and Figure 3 In , and ). The present invention selects To demonstrate the advantages of the cross-modal Transformer. During training, the present invention generates T = 30 (15 frames / second, 2 seconds) poses, while during synthesis, the generation length depends on the length of the utterance. For the cross-modal Transformer, the number of encoders and decoders is set to 1 and 2, respectively. Query, Key, and Value are mapped to different embedding spaces with the same dimensions as the unimodal features ( ). For two-dimensional attention, the present invention uses Cascading The block interactively fuses cross-modal features along the time axis and feature axis, where the size of the feed-forward layer is the same as the input feature. For both the directional cross-modal Transformer and the two-dimensional attention, the present invention uses multi-head attention with head=8.

[0162] Optionally, the construction process of the Transformer-based cross-modal fusion network model in S2 further includes:

[0163] The mean pose difference (MPD) is used to evaluate the Transformer-based cross-modal fusion network model.

[0164] In one possible implementation, the present invention uses an objective evaluation metric. Although subjective user studies are very important, objective evaluation metrics are still needed for fair comparisons. Therefore, the present invention uses FGD (Fréchet Gesture Distance). It measures the quality of motion by considering the latent feature distribution between real and generated samples rather than the average posture error. Therefore, it is more perceptually appropriate than other measurements and has been validated by the consistency of human judgment.

[0165] The main limitation of FGD is that it requires 3D pose annotations (which are not always available) to train the feature extractor and evaluate the method. To address this issue, we propose a new metric, Mean Pose Difference (MPD), which computes the average pose difference from the initial pose and is expressed as follows:

[0166] (10)

[0167] in, represents the expected value within the range of t, represents the predicted value at time t, Represents the predicted value at the initial moment.

[0168] In addition to posture diversity, good generation results also require that gestures and speech are highly correlated (and aligned). to measure the gesture diversity of different speech segments. Since speech energy contains information related to gestures (for example, when a speaker speaks with higher energy, he will make more gestures to attract the audience’s attention), and It is obtained based on the MPD formula on frames with energy below / above the median threshold (the present invention does not use voice activity labels to avoid no available annotations). Therefore, The larger the value, the better the result.

[0169] We use the TED gesture dataset instead of other public datasets such as AVATAR and Personality Dyads Corpus because it has rich speech-gesture mapping data and a variety of speech content and speakers. In addition, it is one of the few datasets that has both audio and text input and 3D gesture annotations.

[0170] S3. Obtain the gesture synthesis result based on the input data and the Transformer-based cross-modal fusion network model.

[0171] For example, the present invention is first compared with the main method in FGD, see Table 1:

[0172] Table 1

[0173]

[0174] The results show that TCMF (the present invention) performs best in FGD, with a minimum value of 2.851. TCMF (the present invention) can generate different gestures for different speech contents. In addition, the generation results also depend on the rhythm and the presence or absence of speech. For example, when pronouncing the keyword "force", the character's two arms swing up and down. After a short pause, when reading "ourselves", the character raises his right arm with a large amplitude. After the pronunciation is finished, the character puts down his arms and finally returns to a neutral posture.

[0175] Table 2 lists the MPD comparison results. The results show that the MPD value generated by TCMF (the present invention) is the highest and closest to the true label. Therefore, the gesture variation level generated by TCMF (the present invention) is similar to that of humans. (measures the correspondence between audio and gesture), TCMF (the present invention) obtained the highest score.

[0176] Table 2

[0177]

[0178] The present invention conducts ablation experiments and analyzes the TCMF (present invention) proposed by the present invention in detail. The present invention eliminates each component separately, and the results are listed in Table 3. As can be seen from the table, after using the directional cross-modal Transformer, the FGD drops from 4.334 (baseline) to 3.548. Therefore, the potential feature adaptation from one modality to another (i.e., A→E and E→A) contributes. When only the two-dimensional attention module is used, the FGD drops to 3.553. Therefore, the model network also benefits from cascading and interactive temporal feature fusion. Finally, when the two modules are combined for end-to-end training, the present invention can obtain the best FGD of 2.851.

[0179] Table 3

[0180]

[0181] The present invention solves the cross-modal fusion problem through a novel fusion mechanism, thereby achieving collaborative speech-gesture generation based on end-to-end learning. In particular, the present invention utilizes parallel directional cross-modal Transformers and interactive cascaded 2D attention modules to achieve selective fusion of gesture-related cues. In addition, the present invention proposes new metrics to evaluate gesture diversity and speech-gesture correspondence without the requirement of 3D pose annotation. Experiments on a public dataset show that the proposed method can successfully generate diverse human-like poses, and its performance is better than other competitive state-of-the-art methods.

[0182] In the embodiment of the present invention, as an important part of non-verbal communication, gestures refer to gestures made with hands, arms, head postures and body postures. If virtual agents can use human-like gestures to match their language, then gestures will play an important role in communication. Since gestures may change with the change of language, there is no need for predefined gesture categories, but arbitrary gestures can be generated. Therefore, it can be deduced that the advantages of the present invention are:

[0183] (1) Generating 3D gestures: The present invention differs from conventional methods in that it is able to generate 3D gestures rather than simply predicting gesture categories. And the generated gestures are not limited to pre-determined categories, but are freely generated. This improves the realism and expressiveness of the generated gestures, making them closer to the natural way humans communicate.

[0184] (2) The key to achieving natural interaction: This method achieves natural interaction with humans by generating realistic and smooth gestures. This is crucial for social agents to communicate effectively and naturally with users in various applications.

[0185] (3) Solving the cross-modal fusion problem: By introducing a novel fusion mechanism, especially the parallel directional cross-modal Transformer and the interactive cascaded two-dimensional attention module, this method successfully solves the fusion problem of cross-modal information (speech and gesture).

[0186] (4) Selective fusion of gesture-related clues: Through mechanism design, this method achieves the selective fusion of gesture-related clues, which can improve the accuracy and adaptability of generated gestures, making the generated gestures more in line with the context and content.

[0187] (5) Evaluation method without 3D pose annotation: We propose new metrics for evaluating gesture diversity and speech-gesture correspondence without the need for cumbersome 3D pose annotations. This simplifies the evaluation process and makes the method easier to apply and generalize.

[0188] Figure 7 The block diagram of a device for synthesizing gestures using a generative adversarial network based on a dual-path Transformer is shown according to an exemplary embodiment. The device is used in a method for synthesizing gestures using a generative adversarial network based on a dual-path Transformer. Figure 7 The device includes an acquisition module 310, an input module 320 and an output module 330. Among them:

[0189] The acquisition module 310 is used to acquire input data to be synthesized into gestures.

[0190] The input data includes text data, audio data and speaker identity data.

[0191] The input module 320 is used to input the input data into the constructed Transformer-based cross-modal fusion network model.

[0192] Among them, the Transformer-based cross-modal fusion network model includes: a front-end single-modal encoder module, a cross-modal fusion module, and a gesture translation module.

[0193] The output module 330 is used to obtain a gesture synthesis result according to the input data and the Transformer-based cross-modal fusion network model.

[0194] In the embodiment of the present invention, as an important part of non-verbal communication, gestures refer to gestures made with hands, arms, head postures and body postures. If virtual agents can use human-like gestures to match their language, then gestures will play an important role in communication. Since gestures may change with the change of language, there is no need for predefined gesture categories, but arbitrary gestures can be generated. Therefore, it can be deduced that the advantages of the present invention are:

[0195] (1) Generating 3D gestures: The present invention differs from conventional methods in that it is able to generate 3D gestures rather than simply predicting gesture categories. And the generated gestures are not limited to pre-determined categories, but are freely generated. This improves the realism and expressiveness of the generated gestures, making them closer to the natural way humans communicate.

[0196] (2) The key to achieving natural interaction: This method achieves natural interaction with humans by generating realistic and smooth gestures. This is crucial for social agents to communicate effectively and naturally with users in various applications.

[0197] (3) Solving the cross-modal fusion problem: By introducing a novel fusion mechanism, especially the parallel directional cross-modal Transformer and the interactive cascaded two-dimensional attention module, this method successfully solves the fusion problem of cross-modal information (speech and gesture).

[0198] (4) Selective fusion of gesture-related clues: Through mechanism design, this method achieves the selective fusion of gesture-related clues, which can improve the accuracy and adaptability of generated gestures, making the generated gestures more in line with the context and content.

[0199] (5) Evaluation method without 3D pose annotation: We propose new metrics for evaluating gesture diversity and speech-gesture correspondence without the need for cumbersome 3D pose annotations. This simplifies the evaluation process and makes the method easier to apply and generalize.

[0200] Figure 8 is a structural diagram of a gesture synthesis device provided by an embodiment of the present invention, such as Figure 8 As shown, the gesture synthesis device may include the above Figure 7 The shown dual-path Transformer-based generative adversarial network collaborative gesture synthesis device. Optionally, the gesture synthesis device 410 may include a first processor 2001.

[0201] Optionally, the gesture synthesis device 410 may further include a memory 2002 and a transceiver 2003 .

[0202] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0203] Combine the following Figure 8 The components of the gesture synthesis device 410 are described in detail:

[0204] The first processor 2001 is the control center of the gesture synthesis device 410, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0205] Optionally, the first processor 2001 may perform various functions of the gesture synthesis device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0206] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 are shown in FIG.

[0207] In a specific implementation, as an embodiment, the gesture synthesis device 410 may also include multiple processors, such as Figure 8 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0208] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0209] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently and access the first processor 2001 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0210] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0211] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 8 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0212] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0213] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A generative adversarial network collaborative gesture synthesis method based on dual-path Transformer, characterized in that: The method comprises: S1, obtaining input data for gesture synthesis; Wherein, the input data includes: text data, audio data and speaker identity data; S2, inputting the input data into the constructed Transformer-based cross-modal fusion network model; The Transformer-based cross-modal fusion network model includes: a front-end single-modal encoder module, a cross-modal fusion module, and a gesture translation module; S3, obtaining a gesture synthesis result according to the input data and a Transformer-based cross-modal fusion network model; The construction process of the Transformer-based cross-modal fusion network model in S2 includes: S21, obtaining sample data; The sample data includes an audio input sequence, a text input sequence, and a speaker ID; S22, inputting the sample data into the front-end single-mode encoder module to obtain audio feature representation, text feature representation and speaker identity feature representation; S23, inputting the audio feature representation and the text feature representation into the cross-modal fusion module to generate an audio-text cross feature representation, and obtaining a multimodal feature representation according to the audio-text cross feature representation and the speaker identity feature representation; S24, inputting the multimodal feature representation into a gesture translation module to obtain a three-dimensional gesture synthesis result; The cross-modal fusion module includes: a directional cross-modal attention module and a two-dimensional attention module; The step of inputting the audio feature representation and the text feature representation into the cross-modal fusion module to generate an audio-text cross feature representation in S23 includes: S231, inputting the audio feature representation and the text feature representation into the directional cross-modal attention module to obtain an audio-to-text feature representation and a text-to-audio feature representation, and then obtaining a cross feature representation; S232. Input the cross-feature representation into the two-dimensional attention module, perform self-attention along the feature dimension and self-attention along the time dimension, and interact to obtain an audio-text cross-feature representation.

2. The method for collaborative gesture synthesis based on a dual-path Transformer generative adversarial network according to claim 1 is characterized in that: The front-end single-mode encoder includes: an audio encoder, a text encoder and a speaker ID encoder; The step of inputting the sample data into the front-end single-mode encoder module in S22 to obtain an audio feature representation, a text feature representation, and a speaker identity feature representation includes: S221, input the audio input sequence into the audio encoder, and encode the audio input sequence using a cascaded one-dimensional convolutional layer to obtain an audio feature representation, as shown in the following formula (1): Where, X A represents the audio feature representation, f A Represents the audio codec, T a represents the length of the audio input sequence, R represents the real number domain, T A Indicates the duration of the audio feature, d A represents the audio feature dimension, and A represents the audio input sequence; S222, inputting the text input sequence into the text encoder, using a one-hot vector to represent the index of each word in the text input sequence, and obtaining a text feature representation; S223 , input the speaker identity ID into the speaker ID encoder, encode the speaker identity ID using a one-hot vector, and obtain a speaker identity feature representation.

3. The method for collaborative gesture synthesis based on a dual-path Transformer generative adversarial network according to claim 1, characterized in that: The feature representation of audio to text in S231 is shown in the following formula (2): In the formula, Feature representation for audio to text, CM A→E represents the cross-attention mechanism from audio to text, Represents the input text features, represents the input audio features, represents the learnable weights, represents the learnable weight, T represents the transpose of the matrix, d k K A The characteristic dimension, V A The Value matrix represents the value of the feature representation. represents the learnable weights.

4. The method for collaborative gesture synthesis based on a dual-path Transformer generative adversarial network according to claim 3 is characterized in that: The self-attention along the feature dimension in S232 is shown in the following formula (3): In the formula, represents the self-attention along the feature dimension, and n represents X C The time of the self-attention module through the feature dimension, X C，f The feature representation representing the concatenated feature dimension, Indicates splicing, X E→A represents the feature representation of text to audio, FC represents the fully connected operation, Q C，f The query matrix K represents the feature representation along the feature dimension. C，f The key matrix representing the feature representation along the feature dimension, T C represents the length of the time series, V C，f The Value matrix representing the feature representation along the feature dimension, d c represents the dimension of the matrix, T′ represents the gesture length; The self-attention along the time dimension is shown in the following formula (4): In the formula, represents the self-attention along the time dimension, X represents the feature representation of the time dimension Time after splicing, and Q C，t The query matrix K represents the feature representation along the time dimension. C，t The key matrix representing the feature representation along the time dimension, V C，t A matrix of values ​​representing the feature representation along the time dimension.

5. The method for collaborative gesture synthesis based on a dual-path Transformer-based generative adversarial network according to claim 1, characterized in that: The construction process of the Transformer-based cross-modal fusion network model in S2 also includes: The mean pose difference (MPD) is used to evaluate the Transformer-based cross-modal fusion network model.

6. A generative adversarial network collaborative gesture synthesis device based on dual-path Transformer, the generative adversarial network collaborative gesture synthesis device based on dual-path Transformer is used to implement the generative adversarial network collaborative gesture synthesis method based on dual-path Transformer as claimed in any one of claims 1 to 5, characterized in that: The device comprises: An acquisition module, used to acquire input data to be synthesized for gestures; Wherein, the input data includes: text data, audio data and speaker identity data; An input module, used to input the input data into the constructed Transformer-based cross-modal fusion network model; The Transformer-based cross-modal fusion network model includes: a front-end single-modal encoder module, a cross-modal fusion module, and a gesture translation module; An output module, used to obtain a gesture synthesis result according to the input data and a Transformer-based cross-modal fusion network model; The construction process of the Transformer-based cross-modal fusion network model includes: S21, obtaining sample data; The sample data includes an audio input sequence, a text input sequence, and a speaker ID; S22, inputting the sample data into the front-end single-mode encoder module to obtain audio feature representation, text feature representation and speaker identity feature representation; S23, inputting the audio feature representation and the text feature representation into the cross-modal fusion module to generate an audio-text cross feature representation, and obtaining a multimodal feature representation according to the audio-text cross feature representation and the speaker identity feature representation; S24, inputting the multimodal feature representation into a gesture translation module to obtain a three-dimensional gesture synthesis result; The cross-modal fusion module includes: a directional cross-modal attention module and a two-dimensional attention module; The step of inputting the audio feature representation and the text feature representation into the cross-modal fusion module to generate an audio-text cross feature representation in S23 includes: S231, inputting the audio feature representation and the text feature representation into the directional cross-modal attention module to obtain an audio-to-text feature representation and a text-to-audio feature representation, and then obtaining a cross feature representation; S232. Input the cross-feature representation into the two-dimensional attention module, perform self-attention along the feature dimension and self-attention along the time dimension, and interact to obtain an audio-text cross-feature representation.

7. A gesture synthesis device, characterized in that: The gesture synthesis device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 5.

Images