Virtual speaker body gesture generation method, device, equipment and storage medium
By using a self-attentional gesture generation network to extract and fuse features from spoken audio, text, and initial gesture sequences, this technology addresses the problem of insufficient modeling of long-term dependencies and complex temporal patterns in virtual speaker gesture synthesis in existing technologies, thereby improving the naturalness and expressiveness of virtual speaker gestures.
Patent Information
- Application Number
- CN202110814936.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-19
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-07-19
AI Technical Summary
In existing technologies, recurrent neural networks and convolutional neural networks cannot effectively model long-term dependencies and complex temporal patterns in the synthesis of virtual speaker gestures, resulting in low naturalness and insufficient expressiveness of the generated gestures, which differ significantly from the gestures of real people.
A self-attention gesture generation network is adopted. By extracting and fusing features from spoken audio, spoken text and initial gesture sequences, a deep self-attention network is used to model the long-term dependencies and complex temporal patterns of multimodal inputs, generating virtual speaker gestures with high expressiveness and semantic relevance.
It improves the naturalness of virtual speaker gestures and the effectiveness of information transmission, enhancing the expressiveness of virtual avatars.
Smart Images

Figure CN113674184B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a method, device, equipment and storage medium for generating body gestures of a virtual speaker. Background Art
[0002] Virtual speaker synthesis technology is a hot research topic in the field of artificial intelligence. It mainly drives virtual speakers to perform actions through audio and has broad application prospects. The synthesis of speaker body gestures is a relatively important task in virtual speaker synthesis technology. How to synthesize expressive body gestures that match the speech content and have emotions for the speaker to improve the expression effect of the virtual speaker is a research focus. In the existing technology, the speech audio and speech text are usually input into a recurrent neural network (RNN) or a convolutional neural network (CNN), and a sequence of key points of the virtual character's joints is output. This key node sequence can be used to drive the virtual character to make corresponding gestures.
[0003] In the prior art, recurrent neural networks (RNNs) or convolutional neural networks (CNNs) are used as the basic structure of the algorithm model. However, due to the limitations of their own structural characteristics, these two neural networks mainly model the short-term patterns of the speaker's body gestures, and are unable to model the long-term dependencies and more complex temporal patterns that are widely present in body gesture sequences. As a result, the generated body gestures are limited in naturalness and lack expressiveness, and are quite different from real people's gestures, resulting in low naturalness of the virtual speaker. Summary of the invention
[0004] The embodiments of the present application provide a method, device, equipment and storage medium for generating body gestures of a virtual speaker, aiming to increase the naturalness of the synthesis results of the body gestures of the virtual speaker and improve the information communication effect of the virtual image.
[0005] A first aspect of an embodiment of the present application provides a method for generating a virtual speaker's body gestures, the method comprising:
[0006] Input the speech audio, speech text and initial gesture sequence into the self-attention body gesture generation network;
[0007] The self-attention body gesture generation network is used to extract features from the speech audio, speech text, and initialization gesture sequence to obtain audio features, text features, and initialization gesture sequence features;
[0008] The audio features, text features and initialized gesture sequence features are fused through the self-attention body gesture generation network to obtain a synthetic gesture sequence.
[0009] Optionally, the training step of the self-attention body gesture generation network includes:
[0010] Collect videos of speakers;
[0011] Processing the speaker video to obtain speech audio, speech text, and an initialization gesture sequence in the speaker video;
[0012] The speech audio, speech text and initialization gesture sequence are used as training sets, and the speaker video is used as a control group and input into the self-attention body gesture generation network to train the network, thereby obtaining a trained self-attention body gesture generation network.
[0013] Optionally, the step of extracting features from the speech audio by using the self-attention body gesture generation network includes:
[0014] Calculate the spoken audio to obtain a Mel spectrum of the audio;
[0015] The Mel spectrum is input into an audio encoder to obtain the audio feature.
[0016] Optionally, the step of extracting features from the spoken text by the self-attention body gesture generation network includes:
[0017] Using a pre-trained language model to process the spoken text to obtain semantic features;
[0018] Performing time-series alignment on the semantic feature and the audio feature to obtain an aligned semantic feature;
[0019] The aligned semantic features are input into a semantic encoder to obtain the text features.
[0020] Optionally, the step of extracting features from the initialization gesture sequence by the self-attention body gesture generation network includes:
[0021] The initialization gesture sequence is input into an initialization gesture sequence encoder to obtain the initialization gesture sequence feature.
[0022] Optionally, the audio features, text features and initialization gesture sequence features are fused by the self-attention body gesture generation network to obtain a synthetic gesture sequence, including:
[0023] Adding the audio feature and the text feature to obtain a summed feature;
[0024] The added features and the initialization gesture sequence features are input into a decoder to obtain the synthesized gesture sequence.
[0025] A second aspect of an embodiment of the present application provides a device for generating body gestures of a virtual speaker, the device comprising:
[0026] A data input module, used to input speech audio, speech text and initialization gesture sequence into the self-attention body gesture generation network;
[0027] A feature extraction module, used to extract features from the speech audio, speech text and initialization gesture sequence through the self-attention body gesture generation network to obtain audio features, text features and initialization gesture sequence features;
[0028] The gesture synthesis module is used to fuse the audio features, text features and initialization gesture sequence features through the self-attention body gesture generation network to obtain a synthesized gesture sequence.
[0029] Optionally, the device further comprises:
[0030] A video collection module, used to collect speaker videos;
[0031] A video processing module, used to process the speaker video to obtain the speaking audio, speaking text and initialization gesture sequence in the speaker video;
[0032] The model training module is used to use the speech audio, speech text and initialization gesture sequence as a training set, and the speaker video as a control group to input into the self-attention body gesture generation network to train the network, so as to obtain a trained self-attention body gesture generation network.
[0033] Optionally, the feature extraction module includes:
[0034] The audio calculation submodule is used to calculate the spoken audio to obtain the Mel spectrum of the audio segment;
[0035] The audio feature extraction submodules are configured to input the Mel spectrum into an audio encoder to obtain the audio features.
[0036] Optionally, the feature extraction module further includes:
[0037] A semantic feature extraction submodule, used to process the spoken text using a pre-trained language model to obtain semantic features;
[0038] The semantic feature and its submodule are used to perform time sequence alignment between the semantic feature and the audio feature to obtain an aligned semantic feature;
[0039] The text feature extraction submodule is used to input the aligned semantic features into a semantic encoder to obtain the text features.
[0040] Optionally, the feature extraction module further includes:
[0041] The initialization gesture sequence feature extraction submodule is used to input the initialization gesture sequence into the initialization gesture sequence encoder to obtain the initialization gesture sequence feature.
[0042] Optionally, the gesture synthesis module includes:
[0043] A feature summing submodule, used for summing the audio feature and the text feature to obtain a summed feature;
[0044] The gesture synthesis submodule is used to input the added features and the initialization gesture sequence features into a decoder to obtain the synthesized gesture sequence.
[0045] A third aspect of an embodiment of the present application provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the method described in the first aspect of the present application are implemented.
[0046] A fourth aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect of the present application are implemented.
[0047] The method for generating virtual speaker body gestures provided in the present application is adopted to input speech audio, speech text and initialization gesture sequence into the self-attention body gesture generation network; the self-attention body gesture generation network is used to extract features of the speech audio, speech text and initialization gesture sequence to obtain audio features, text features and initialization gesture sequence features; the self-attention body gesture generation network is used to fuse the audio features, text features and initialization gesture sequence features to obtain a synthetic gesture sequence. In the present application, speech audio, speech text and initialization gesture features are input into the self-attention body gesture generation network for processing, and a deep self-attention network is used to process multimodal input. The deep self-attention network effectively models the long-term dependencies and complex temporal patterns of the input content, and generates virtual speaker body gestures with high expressiveness and semantic relevance. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0049] Figure 1 is a flow chart of a method for generating body gestures of a virtual speaker proposed in an embodiment of the present application;
[0050] Figure 2 This is a schematic diagram of the basic modules of the self-attention body gesture generation network proposed in one embodiment of the present application;
[0051] Figure 3 It is a flowchart of a method for generating body gestures of a virtual speaker proposed in an embodiment of the present application;
[0052] Figure 4 Schematic diagram of a virtual speaker body gesture generation device proposed in an embodiment of the present application. DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0054] refer to Figure 1 , Figure 1 : is a flow chart of a method for generating a virtual speaker's body gestures according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:
[0055] S11: Input the speech audio, speech text and initialization gesture sequence into the self-attention body gesture generation network.
[0056] In this embodiment, the speech audio is an arbitrary audio containing a single speaker, the speech text is the speech content in the speech audio, the initialization gesture sequence is the initial coordinates of the key points on the virtual speaker's limbs, and the self-attention limb gesture generation network is a deep neural network that uses a self-attention mechanism, which is used to generate a gesture sequence based on the speech audio, the speech text and the initialization gesture sequence.
[0057] For example, the speech audio can be a video of a person giving a speech, the speech text is the text content of the person's speech in the speech audio, and the initialization gesture sequence is a pre-set initial sequence of key points on the virtual speaker's body, for example, it can be set to initial body movements such as clasping fists and clapping, and it is best to present the emotions expressed in the speech content in the speech audio.
[0058] In this embodiment, the self-attention mechanism performs well in modeling the long-term dependencies and complex temporal relationships that are widely present in body gesture sequences. The deep neural network using the self-attention mechanism can process multimodal data and synthesize the speaker's body gestures with high expressiveness and semantic relevance.
[0059] S12: Extract features of the speech audio, speech text and initialization gesture sequence through the self-attention body gesture generation network to obtain audio features, text features and initialization gesture sequence features.
[0060] In this embodiment, the step of extracting features from the speech audio by using the self-attention body gesture generation network includes:
[0061] S12-1-1: Calculate the speech audio to obtain the Mel spectrum of the audio.
[0062] In this embodiment, after the speech audio is input into the self-attention body gesture generation network, the network will automatically calculate the Mel spectrum of the speech audio to obtain the Mel spectrum of the audio segment.
[0063] S12-1-2: Input the Mel spectrum into an audio encoder to obtain the audio features.
[0064] In this embodiment, the Mel spectrum is input into an audio encoder, and the audio encoder encodes the Mel spectrum to obtain audio features.
[0065] The step of extracting features from the spoken text by the self-attention body gesture generation network comprises:
[0066] S12-2-1: Use a pre-trained language model to process the spoken text to obtain semantic features.
[0067] In this embodiment, a pre-trained language model is set in the self-attention body gesture generation network. The pre-trained language model performs semantic recognition on the spoken text to obtain an initial semantic feature sequence, namely, a semantic feature.
[0068] For example, the language model can use a commonly used language recognition model, and there is no limitation here. The pre-trained language model is integrated in the self-attention body gesture generation network. There is no need to process the text in advance to extract semantic features. The text can be directly input into the network for processing.
[0069] S12-2-2: Perform time-series alignment on the initial text features and the audio features to obtain aligned semantic features.
[0070] In this embodiment, the semantic features need to be time-sequence aligned with the audio features, that is, the spoken content in the audio is matched one by one with the words in the text, and they are arranged in time sequence to obtain aligned semantic features.
[0071] S12-2-3: Input the aligned semantic features into a semantic encoder to obtain the semantic features.
[0072] In this embodiment, the aligned semantic features are input into a semantic encoder, and the aligned semantic features are encoded by the semantic encoder to obtain text features.
[0073] In this embodiment, the step of extracting features from the initialization gesture sequence through the self-attention body gesture generation network includes:
[0074] S12-3-1: Input the initialization gesture sequence into the initialization gesture sequence encoder to obtain the initialization gesture sequence features.
[0075] In this embodiment, the initialization gesture sequence is the coordinates of key points on the limbs, and the specific gestures can be determined by these coordinates, such as the coordinates of the elbow joint, wrist joint, and shoulder joint. The initialization gesture sequence is input into the initialization gesture sequence encoder to obtain the initialization gesture sequence feature. The initialization gesture sequence can be set according to the content of the input speech audio and speech text.
[0076] S13: The audio features, text features and initialized gesture sequence features are fused through the self-attention body gesture generation network to obtain a synthetic gesture sequence.
[0077] In this embodiment, the synthesized gesture sequence is the coordinates of the key nodes on the limbs of the virtual speaker synthesized in each frame of the image. The synthesized gesture sequence is obtained based on the audio features, text features and the initialization gesture sequence features. The specific steps of fusing the audio features, text features and the initialization gesture sequence features are as follows:
[0078] S13: Add the audio feature and the text feature to obtain a summed feature.
[0079] In this embodiment, after the audio features and the text features are added together, a feature vector can be obtained, and the feature vector is the added feature.
[0080] S14: Input the added features and the initialization gesture sequence features into a decoder to obtain the synthesized gesture sequence.
[0081] In this embodiment, the summed features and the initialization gesture sequence features are input into the decoder, and the decoder combines the summed features and the initialization gesture sequence to obtain a synthetic gesture sequence. The synthetic gesture sequence is used to determine the body gestures of the virtual speaker.
[0082] For example, a piece of audio of storytelling, the corresponding text and the corresponding initialization gesture sequence are input into the self-attention body gesture generation network. The self-attention body gesture generation network will generate a synthetic gesture sequence. The synthetic gesture sequence corresponds to the audio and text of the storytelling and can express the corresponding emotion. For example, there is a sentence in the storytelling, "To know what will happen next, please listen to the next episode", and the corresponding synthetic gesture sequence can be the right hand raised high and then put down, which corresponds to the action of the storyteller hitting the wake-up stick.
[0083] In this embodiment, the Mel spectrum of the spoken audio is extracted, and then the audio encoder is used to obtain the audio features; for the spoken text, the pre-trained language model is first used to perform semantic recognition on the spoken text to obtain the semantic features, and then the semantic features are aligned with the audio features in time sequence to obtain the aligned semantic features, and the aligned semantic features are encoded by the semantic encoder to obtain the text features; for the initialization gesture sequence, the initialization gesture sequence encoder is used to encode the initialization gesture sequence features; when performing feature fusion, the audio features and the text features are first added to obtain a summed vector, and the vector and the initialization gesture features are input into the decoder to obtain a synthesized gesture sequence. A pre-trained language model is set in the body gesture generation network, which can more conveniently and effectively extract text features, and then generate body gestures of a virtual speaker with high semantic relevance. At the same time, the body gesture generation network is constructed based on the self-attention mechanism. When multimodal data is input into the network, more attention can be paid to the key parts of the data, which is conducive to synthesizing virtual speaker body gestures with more high expressiveness and semantic relevance, thereby increasing the naturalness of the virtual speaker body gesture synthesis result and improving the information communication effect of the virtual image.
[0084] In another embodiment of the present application, the training step of the self-attention body gesture generation network includes:
[0085] S21: Collect speaker videos.
[0086] In this embodiment, the self-attention body gesture generation network needs to be trained. First, speaker videos are collected. The speaker in the speaker video is a single person, and the speaker needs to have changes in body gestures when speaking, which facilitates the learning and training of the neural network.
[0087] For example, the speaker video may be a speech video, a recitation video, etc.
[0088] S22: Process the speaker video to obtain the speaking audio, speaking text, and initialization gesture sequence in the speaker video.
[0089] In this embodiment, the audio in the speaker's video is first extracted. The audio can be extracted using existing audio extraction tools without limitation. Then, the text of the speaker's speech is extracted. The speaker's text can be extracted using existing speech recognition tools without limitation.
[0090] The initialization gesture sequence is obtained by capturing the first few frames of the speaker's speech in the video, determining the positions of key points on the speaker's body according to the speaker's body positions in the captured images, and obtaining the position coordinates of the key points on the speaker's body.
[0091] S23: The speech audio, speech text and initialization gesture sequence are used as a training set, and the speaker video is used as a control group and input into the self-attention body gesture generation network to train the network, thereby obtaining a trained self-attention body gesture generation network.
[0092] In this embodiment, after the speech audio, speech text and initialization gesture sequence are input into the self-attention body gesture generation network, the network obtains a synthetic gesture sequence based on the input content. The network obtains a loss value by comparing the synthetic gesture sequence with the gesture sequence of the speaker in the original speaker video, and then adjusts the parameters of the network. When the parameters are adjusted to the optimal level, a trained self-attention body gesture generation network is obtained.
[0093] In another embodiment of the present application, the present application will be further described in conjunction with the basic modules of the body gesture generation network:
[0094] In this embodiment, Figure 2 As shown, Figure 2 : is a schematic diagram of a basic module of a self-attention body gesture generation network proposed in an embodiment of the present application, such as Figure 2 As described above, the basic modules of the network model include an encoder and a decoder, wherein the encoder includes four units, each of which includes a multi-head attention layer and a one-dimensional convolution layer, and both the multi-head attention layer and the one-dimensional convolution layer include summation and regularization operations. The decoder also includes 4 units, each of which includes two multi-head attention layers and a one-dimensional convolution layer, wherein the input of the second attention layer includes the decoder internal data stream and the input of the initialized gesture encoder. The structures of the audio encoder, semantic encoder, and initialized gesture sequence encoder are the same, but as shown in the figure, Figure 2As shown in the figure, the output of the initialized gesture sequence encoder is connected to the second multi-head attention layer of the decoder. For the audio encoder and semantic encoder, their outputs are first summed, and the summed result is used as the input of the decoder and input into the first multi-head attention layer. Multiple self-attention layers are connected to the multi-head attention layer, and the input feature vector is processed through the self-attention mechanism.
[0095] For the audio encoder and semantic encoder, after the feature vector is input into the encoder, it enters the multi-head attention layer. The feature vector processed by the multi-head attention layer is summed and regularized with the original feature vector. The obtained vector is input into the one-dimensional convolution layer. After being processed by the one-dimensional convolution layer, it is summed and regularized again with the vector after the first summation and regularization. The processed feature vector is output.
[0096] For the initialization gesture sequence encoder, the processing steps of the feature vector are the same as those of the audio encoder and semantic encoder, which will not be described here, but the output is connected to the second multi-head attention layer of the decoder.
[0097] In the decoder, the sum of the audio features and the text features is input into the encoder and first enters the first multi-head attention layer. After being processed by the multi-head attention layer, the original feature vector and the processed vector are summed and regularized, and the processed vector is output. The processed vector is then input into the second multi-head attention layer together with the initialized gesture sequence feature vector. After being processed by the second multi-head attention layer, a fused vector is obtained. The fused vector and the vector output by the first summation and regularization layer are summed and regularized again to obtain a vector after the second summation and regularization. The vector after the second summation and regularization is input into the one-dimensional convolution layer. After being processed by the one-dimensional convolution layer, the vector processed by the one-dimensional convolution layer and the vector after the second summation and regularization are summed and regularized again. After calculation, a synthetic gesture sequence is output.
[0098] In this embodiment, the multi-head attention layer in the module can assign weights to the sequences in the input data. The part with a larger weight is the more important part of the sequence, and the relationship between the data is captured very well, which is more conducive to modeling long-term dependencies and complex temporal relationships in the sequence.
[0099] In another embodiment of the present application, the present application is further described in conjunction with a flow chart of a method for generating a virtual speaker's body gestures.
[0100] like Figure 3 As shown, Figure 3 This is a flowchart of a method for generating body gestures of a virtual speaker proposed in an embodiment of the present application.
[0101] like Figure 3 As stated, Figure 3 In the process, the speech audio, speech text, and initial gesture sequence are input into the self-attention body gesture generation network.
[0102] For spoken audio, the network extracts the Mel spectrum of the audio and performs position encoding on the Mel spectrum. Position encoding is used to encode the temporal relationship of the input feature vector, that is, to encode the position sequence of the vector. The audio encoder outputs the audio feature vector, i.e., the audio feature.
[0103] For spoken text, the pre-trained language model is used to perform semantic recognition on the text to obtain a semantic feature vector, i.e., a semantic feature. Before semantic recognition, the text sequence must also be positionally encoded. The semantic feature vector is then aligned with the audio feature vector, and the aligned semantic feature vector, i.e., the aligned semantic feature, is positionally encoded and input into the semantic encoder to obtain a text vector, i.e., a text feature.
[0104] For the initialization gesture sequence, start with Figure 3 As can be seen in FIG, the initialization gesture sequence is the coordinates of the key points of the initial body movements of the virtual speaker. After position encoding of the initialization gesture sequence, the initialization gesture sequence encoder is input to obtain the initialization gesture sequence feature vector, that is, the initialization gesture sequence feature.
[0105] When fusing feature vectors, the audio feature vector and the text feature vector are first added together as the input of a decoder. Then, the initialization gesture sequence feature vector input and the added feature vector are input into the decoder together to obtain a synthesized gesture sequence, and then the subsequent body movements of the virtual speaker are synthesized.
[0106] Based on the same inventive concept, an embodiment of the present application provides a virtual speaker body gesture generation device. Figure 4 , Figure 4 FIG. 4 is a schematic diagram of a virtual speaker body gesture generation device 400 proposed in an embodiment of the present application. Figure 4 As shown, the device comprises:
[0107] A data input module 401 is used to input speech audio, speech text and initialization gesture sequence into the self-attention body gesture generation network;
[0108] A feature extraction module 402 is used to extract features from the speech audio, speech text and initialization gesture sequence through the self-attention body gesture generation network to obtain audio features, text features and initialization gesture sequence features;
[0109] The gesture synthesis module 403 is used to fuse the audio features, text features and initialization gesture sequence features through the self-attention body gesture generation network to obtain a synthesized gesture sequence.
[0110] Optionally, the device further comprises:
[0111] A video collection module, used to collect speaker videos;
[0112] A video processing module, used to process the speaker video to obtain the speaking audio, speaking text and initialization gesture sequence in the speaker video;
[0113] The model training module is used to use the speech audio, speech text and initialization gesture sequence as a training set, and the speaker video as a control group to input into the self-attention body gesture generation network to train the network, so as to obtain a trained self-attention body gesture generation network.
[0114] Optionally, the feature extraction module includes:
[0115] The audio calculation submodule is used to calculate the spoken audio to obtain the Mel spectrum of the audio segment;
[0116] The audio feature extraction submodules are configured to input the Mel spectrum into an audio encoder to obtain the audio features.
[0117] Optionally, the feature extraction module further includes:
[0118] A semantic feature extraction submodule, used to process the spoken text using a pre-trained language model to obtain semantic features;
[0119] The semantic feature and its submodule are used to perform time sequence alignment between the semantic feature and the audio feature to obtain an aligned semantic feature;
[0120] The text feature extraction submodule is used to input the aligned semantic features into a semantic encoder to obtain the text features.
[0121] Optionally, the feature extraction module further includes:
[0122] The initialization gesture sequence feature extraction submodule is used to input the initialization gesture sequence into the initialization gesture sequence encoder to obtain the initialization gesture sequence feature.
[0123] Optionally, the gesture synthesis module includes:
[0124] A feature summing submodule, used for summing the audio feature and the text feature to obtain a summed feature;
[0125] The gesture synthesis submodule is used to input the added features and the initialization gesture sequence features into a decoder to obtain the synthesized gesture sequence.
[0126] Based on the same inventive concept, another embodiment of the present application provides a readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the method for generating a virtual speaker's body gestures as described in any of the above embodiments of the present application are implemented.
[0127] Based on the same inventive concept, another embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes, the steps in the method for generating virtual speaker body gestures described in any of the above embodiments of the present application are implemented.
[0128] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0129] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0131] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0132] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0134] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0135] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0136] The above is a detailed introduction to the virtual speaker body gesture generation method, device, equipment and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for a person skilled in the art, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for generating a virtual speaker's body gestures, the method comprising: Input the speech audio, speech text and the initialization gesture sequence into the self-attention body gesture generation network, the self-attention body gesture generation network includes an encoder and a decoder, the encoder includes four units, each of the units includes a multi-head attention layer and a one-dimensional convolution layer, the multi-head attention layer and the convolution layer both include summation and regularization operations, the decoder includes four units, each of the units includes two multi-head attention layers and one one-dimensional convolution layer, the input of the second multi-head attention layer includes the decoder internal data stream and the input of the initialization gesture sequence encoder; The self-attention body gesture generation network is used to extract features from the speech audio, speech text, and initialization gesture sequence to obtain audio features, text features, and initialization gesture sequence features; The audio features, text features and initialized gesture sequence features are fused through the self-attention body gesture generation network to obtain a synthetic gesture sequence.
2. The method according to claim 1, characterized in that The training steps of the self-attention body gesture generation network include: Collect video of speakers; Processing the speaker video to obtain speech audio, speech text, and an initialization gesture sequence in the speaker video; The speech audio, speech text and initialization gesture sequence are used as training sets, and the speaker video is used as a control group and input into the self-attention body gesture generation network to train the network, thereby obtaining a trained self-attention body gesture generation network.
3. The method according to claim 1, characterized in that The step of extracting features from the speech audio by the self-attention body gesture generation network comprises: Calculate the spoken audio to obtain a Mel spectrum of the audio; The Mel spectrum is input into an audio encoder to obtain the audio feature.
4. The method according to claim 1, characterized in that: The step of extracting features from the spoken text by the self-attention body gesture generation network comprises: Using a pre-trained language model to process the spoken text to obtain semantic features; Performing time-series alignment on the semantic feature and the audio feature to obtain an aligned semantic feature; The aligned semantic features are input into a semantic encoder to obtain the text features.
5. The method according to claim 1, characterized in that The step of extracting features of the initialization gesture sequence by the self-attention body gesture generation network comprises: The initialization gesture sequence is input into an initialization gesture sequence encoder to obtain the initialization gesture sequence feature.
6. The method according to claim 1, characterized in that The audio features, text features and initialization gesture sequence features are fused through the self-attention body gesture generation network to obtain a synthetic gesture sequence, including: Adding the audio feature and the text feature to obtain a summed feature; The added features and the initialization gesture sequence features are input into a decoder to obtain the synthesized gesture sequence.
7. A virtual speaker body gesture generation device, characterized in that: The device comprises: A data input module, used for inputting speech audio, speech text and initialization gesture sequence into a self-attention body gesture generation network, wherein the self-attention body gesture generation network comprises an encoder and a decoder, wherein the encoder comprises four units, each of which comprises a multi-head attention layer and a one-dimensional convolution layer, wherein the multi-head attention layer and the convolution layer both comprise summation and regularization operations, and the decoder comprises four units, each of which comprises two multi-head attention layers and one-dimensional convolution layer, wherein the input of the second multi-head attention layer comprises the internal data stream of the decoder and the input of the initialization gesture sequence encoder; A feature extraction module, used to extract features from the speech audio, speech text and initialization gesture sequence through the self-attention body gesture generation network to obtain audio features, text features and initialization gesture sequence features; The gesture synthesis module is used to fuse the audio features, text features and initialization gesture sequence features through the self-attention body gesture generation network to obtain a synthesized gesture sequence.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 6 are implemented.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
