Gesture generation methods and electronic devices

Abstract

This application discloses a gesture generation method and an electronic device. The method includes: responding to received speech information, first text information, and first generation style information of a gesture to be generated; encoding the first text information and the first generation style information to obtain second text information and second generation style information; predicting noise based on the speech information, second text information, and second generation style information; denoising the speech information, second text information, and second generation style information based on the predicted noise to obtain a gesture encoding sequence; and inputting the gesture encoding sequence into a pre-trained gesture generation model to generate the corresponding gesture. This application uses a gesture generation model to generate corresponding gestures from the received speech information, first text information, and first generation style information of the gesture to be generated, enabling gestures to be generated from multiple communication methods and improving generation accuracy.

Description

Technical Field

[0001] This application relates to the field of computer graphics technology, specifically to a gesture generation method and an electronic device. Background Technology

[0002] Gestures refer to the continuous, stylized hand movements humans make while speaking. Appropriate gestures can enhance the vividness of conversation and visualize its content. In the field of computer graphics, motion capture technology is used to record common hand gestures to drive biomimetic movements in digital characters, making their movements more lifelike. However, implementing such solutions is often quite challenging. Therefore, automatically generating high-quality gestures from input speech and text using computers has become a hot technological trend.

[0003] The current mainstream technical solution utilizes neural network technology to learn patterns from large speech-gesture datasets, thereby enabling the direct prediction of corresponding gestures based on input speech and text. However, this approach lacks sufficient controllability, making it difficult for users to freely control the style of generated gestures according to their personal intentions. Although numerous works have been dedicated to speech-text-driven stylized gesture generation for digital humans, accurately understanding user intentions and efficiently achieving stylized control over gesture generation remains an important and unsolved problem. Summary of the Invention

[0004] To address the above issues, this application provides a gesture generation method and an electronic device. By using a pre-trained gesture generation model to generate corresponding gestures from the received speech information, first text information, and first generation style information of the gesture to be generated, the corresponding gesture can be generated accurately according to the user's intention, and the gesture can be generated efficiently according to the style required by the user.

[0005] In a first aspect, embodiments of this application provide a gesture generation method, comprising: responding to received speech information, first text information, and first generation style information of a gesture to be generated; encoding the first text information and the first generation style information to obtain second text information and second generation style information; predicting noise based on the speech information, the second text information, and the second generation style information; denoising the speech information, the second text information, and the second generation style information based on the predicted noise to obtain a gesture encoding sequence to be generated; and inputting the gesture encoding sequence to be generated into a pre-trained gesture generation model to generate the corresponding gesture.

[0006] An embodiment of the second aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method described in the first aspect above.

[0007] An embodiment of the third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0008] An embodiment of the fourth aspect of this application provides a computer program product including a computer program that is executed by a processor to implement the method described in the first aspect above.

[0009] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0010] In this embodiment, in response to received voice information, first text information, and first generation style information of the gesture to be generated, the first text information and the first generation style information are encoded to obtain second text information and second generation style information. Noise is predicted based on the voice information, second text information, and second generation style information. By encoding and predicting noise from three dimensions—voice information, first text information, and first generation style information—the intent of the user to generate a gesture can be understood and analyzed from multiple dimensions, effectively improving the intelligence of the generated gesture.

[0011] Furthermore, the speech information, second text information, and second generation style information are denoised based on the predicted noise to obtain the gesture action encoding sequence to be generated. This sequence is then input into a pre-trained gesture action generation model to generate the corresponding gesture action. Noise significantly affects the gesture generation results. Denoising the speech information, second text information, and second generation style information better preserves the details within the information, reduces the interference of noise data, improves the accuracy of the generated gesture action encoding sequence, and reduces errors. In addition, pre-training the gesture action generation model ensures fast and accurate generation of the corresponding gesture actions.

[0012] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0013] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0014] Figure 1 A flowchart of a gesture generation method provided in an embodiment of this application is shown;

[0015] Figure 2 The diagram shows the training structure of the gesture action text feature extraction model provided in an embodiment of this application;

[0016] Figure 3 This invention provides a schematic diagram of the structure of a denoising network that predicts noise based on speech information, second text information, and second generation style information.

[0017] Figure 4 This paper shows a schematic diagram of the structure of a gesture generation device provided in an embodiment of this application;

[0018] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0020] First, the terms used in the embodiments of this application will be explained:

[0021] Gestures: refer to the continuous, stylized hand movements that humans make while speaking.

[0022] Noise refers to data containing errors or anomalies (deviations from expected values), which interfere with data analysis.

[0023] In real life, appropriate gestures can enhance the vividness of conversations and make the content more concrete. In the field of computer graphics, to make the movements of digital characters more lifelike, motion capture technology is used to record common hand gestures to drive the digital characters in biomimetic movements. First, neural network technology is used to learn patterns from a large amount of speech-gesture datasets, thereby enabling the direct prediction of corresponding gestures based on input speech and text. However, this neural network-based method lacks sufficient controllability and is difficult to accurately convey user intentions, making it difficult for users to freely control the style of the generated gestures according to their personal intentions.

[0024] Based on this, embodiments of this application provide a gesture generation method and an electronic device. This application performs noise prediction on speech information, text information, and generation style information, and then performs denoising processing on the predicted noise to generate a gesture encoding sequence. This denoising process can remove erroneous or abnormal data in the speech information, text information, and generation style information, reducing interference and thus improving the accuracy of understanding the user's conveyed intent. Finally, a pre-trained gesture generation model generates the corresponding gesture based on the gesture encoding sequence, improving the generation speed and diversity of generation styles. The following detailed description of the embodiments of this application is provided in conjunction with the accompanying drawings.

[0025] See Figure 1 The flowchart shown illustrates a method for generating gesture actions, which specifically includes the following steps:

[0026] Step 101: In response to the received voice information, first text information and first generation style information of the gesture action to be generated.

[0027] In this embodiment, the received voice information for the gesture action to be generated can be MP3, WAV, WMA, MP2, Flac, MIDI, RA, APE, AAC, CDA, MOV, etc.; the first text information can be TXT, WORD, EXCEL, PPT, XML, etc.; the first generation style information can be one or more file formats among text format, video format and three-dimensional motion format. Those skilled in the art can set it according to actual needs, and this application does not make specific limitations.

[0028] In some embodiments, a display page and an input button can be provided for users to directly input the voice information of the gesture to be generated, so that users can directly input the voice information of the gesture to be generated through the input button to generate voice information; an upload button can also be provided for users to upload the voice information of the gesture to be generated, so that users can select a pre-recorded voice file of the gesture to be generated by clicking the upload button to upload and generate voice information.

[0029] Furthermore, the display page can also provide an input box for directly inputting the first text information of the gesture action to be generated, so that users can directly input the first text information of the gesture action to be generated into the input box to generate the first text information; it can also provide an upload button for uploading the first text information of the gesture action to be generated, so that users can select the pre-generated first text file of the gesture action to be generated by clicking the upload button to upload and generate the first text information.

[0030] Furthermore, the display page can also provide a button or drop-down box for selecting the file format. When the file format is selected as text, the same implementation method as described above for generating the first text information is provided, with an input box for the first generation style information of the gesture action to be generated, so that the user can directly input the first generation style information of the gesture action to be generated into the input box to generate the first generation style information. Alternatively, an upload button can be provided to upload the generation style information of the gesture action to be generated, so that the user can select the pre-generated first generation style text file of the gesture action to be generated by clicking the upload button to upload and generate the first generation style information. Alternatively, when the file format is selected as video or 3D motion format, the implementation principle is the same as that of selecting the text format, that is, an input button or upload button is provided for data input, which will not be repeated here.

[0031] Those skilled in the art can make specific settings for the above embodiments according to actual conditions, and this application does not impose any specific limitations.

[0032] Step 102: Encode the first text information and the first generation style information to obtain the second text information and the second generation style information.

[0033] After obtaining the first text information and the first generation style information of the gesture action to be generated through step 101, the first text information and the first generation style information are encoded to obtain the second text information and the second generation style information.

[0034] In some embodiments, when encoding the first text information, a pre-trained gesture action text feature extraction model can be used to encode the first text information to obtain the second text information; and when encoding the first generated style information, it can be first determined whether the format of the first generated style information is one or more of text format, video format, or 3D motion format. When the format of the first generated style information is text format, a text encoder based on a contrast-based language-image pre-trained CLIP model can be used to encode the first generated style information to obtain the second generated style information; when the format of the first generated style information is video format, a video encoder based on a CLIP model can be used to encode the first generated style information to obtain the second generated style information; when the format of the first generated style information is 3D motion format, a motion encoder based on a contrast-based language-image-motion pre-trained MotionCLIP model can be used to encode the first generated style information to obtain the second generated style information.

[0035] Specifically, before encoding the first text information using a pre-trained gesture action text feature extraction model to obtain the second text information, the gesture action text feature extraction model can be trained first, such as... Figure 2 As shown, the gesture action text feature extraction model can include a gesture encoder, a text encoder, and a joint vector space. The pre-prepared third gesture action information to be trained is input into the gesture encoder for encoding to obtain a first gesture feature. Then, the first gesture feature is subjected to max pooling to obtain a second gesture feature. The corresponding text information to be trained is input into the text encoder for encoding to obtain a third text feature. The third text feature is subjected to max pooling to obtain a fourth text feature. The second gesture feature and the fourth text feature are mapped to the joint vector space and temporally matched. The successfully matched second gesture feature and fourth text feature are used as the first training pair, and other training pairs besides the first training pair are used as the second training pair. Finally, the network is trained using the first and second training pairs, and a second training value is determined. Based on this second training value, when a second preset convergence condition is met, the gesture action text feature extraction model is obtained.

[0036] After obtaining the gesture text feature extraction model, the first text information of the gesture to be generated can be input into the gesture text feature extraction model for encoding processing to obtain the second text information.

[0037] It should be noted that both the gesture encoder and the text encoder in the gesture action text feature extraction model process data on a sentence-by-sentence basis.

[0038] Max pooling refers to the process of obtaining the data with the largest sequence value among each data feature in the processed data. The first gesture feature is max pooled to obtain the second gesture feature, i.e., the feature with the largest sequence value among each feature of the first gesture feature is used as the second gesture feature; similarly, the third text feature is max pooled to obtain the fourth text feature, i.e., the feature with the largest sequence value among each feature of the third text feature is used as the fourth text feature.

[0039] Through the max pooling process described above, the most semantically relevant information in each feature sequence of the first gesture feature and the third text feature can be aggregated respectively. This ensures the temporal correspondence between gestures and text during the training of the gesture action text feature extraction model, effectively improving the accuracy of feature extraction by the gesture action text feature extraction model and laying a solid foundation for subsequent gesture generation.

[0040] The second training value can refer to the training loss value during network training, or it can refer to the number of training iterations during network training. When the second training value represents the training loss value, a second preset convergence condition is met when the training loss value is less than or equal to a preset second training loss value; when the second training value represents the number of training iterations, a second preset convergence condition is met when the number of training iterations is greater than a preset second number. Those skilled in the art can set this value according to actual circumstances, and this application does not impose specific limitations.

[0041] By setting convergence conditions to determine whether the gesture action text feature extraction model has been trained, the accuracy of model training and the accuracy of model output can be significantly improved, so as to achieve the generation effect actually needed by users.

[0042] Furthermore, before encoding the first generated style information using the CLIP model video encoder to obtain the second generated style information, the CLIP model video encoder can be trained first. Specifically, the pre-selected 3D motion data of the hand gesture to be trained is first input into the 3D engine for rendering to obtain the corresponding 2D video data to be trained. The 3D motion data of the hand gesture to be trained is then input into the motion encoder in the MotionCLIP model for encoding to obtain the 3D motion features of the hand gesture to be trained. The 2D video data to be trained is then input into the CLIP model-based video encoder to be trained for encoding to obtain the 2D video features of the hand gesture to be trained. The network is trained using the 3D motion features of the hand gesture to be trained and the 2D video features of the hand gesture to be trained, and a third training value is determined. Based on the third training value, when the third preset convergence condition is met, the CLIP model-based video encoder is obtained.

[0043] It's important to note that a 3D engine is a graphics tool that encapsulates graphics algorithms and can be used for editing, generating, and rendering 3D scenes and objects. Common 3D engines include the Unreal Engine (UE) series, the Unity series, and so on.

[0044] The third training value can refer to the training loss value during network training, or it can refer to the number of training iterations during network training. When the third training value represents the training loss value, a third preset convergence condition is met when the training loss value is less than or equal to a preset third training loss value. When the third training value represents the number of training iterations, a third preset convergence condition is met when the number of training iterations is greater than a preset third number. Those skilled in the art can set these values ​​according to actual circumstances, and this application does not impose specific limitations.

[0045] Since the text encoder of the CLIP model and the motion encoder of the MotionCLIP model are encoders commonly used by those skilled in the art, the operating principle of encoding the first generated style information by the text encoder of the CLIP model and encoding the first generated style information by the motion encoder of the MotionCLIP model in this embodiment will not be described in detail here.

[0046] In other embodiments, during the training of the gesture text feature extraction model, the network parameters of the gesture encoder and the momentum version of the text encoder of the gesture text feature extraction model can be updated by exponential moving average, so that when errors occur during the training of the gesture text feature extraction model, the network parameters of the gesture encoder and the momentum version of the text encoder of the gesture text feature extraction model can be corrected by updating the network parameters of the gesture encoder and the momentum version of the text encoder of the gesture text feature extraction model.

[0047] It should be noted that the momentum versions of the gesture encoder and text encoder refer to backup versions that are consistent with the training versions. These backup versions are used to continue training when errors occur during the training of the gesture action text feature extraction model, ensuring successful training and saving training resources.

[0048] An exponential moving average is a moving average that uses exponentially decreasing weights. The weighting influence of each value decreases exponentially over time, with more recent data having a greater weighting influence, but older data is also given a certain weighting value. The advantage of using an exponential moving average is that it does not need to store the actual values ​​of all previous times, thus reducing memory usage and effectively correcting for biases.

[0049] In other embodiments, when an error occurs during network training using the first and second training pairs, the network parameters of the momentum versions of the gesture encoder and the text encoder of the gesture action text feature extraction model are used to re-match the second gesture features with the fourth text features in a temporal sequence. The successfully re-matched second gesture features and the fourth text features are used as the new first training pair, and other training pairs other than the new first training pair are used as the new second training pair. Then, the network training continues using the new first training pair and the new second training pair.

[0050] Step 103: Predict noise based on speech information, second text information, and second generation style information.

[0051] After obtaining the second text information and the second generation style information of the gesture action to be generated through step 102, noise is predicted based on the speech information received in step 101 and the second text information and the second generation style information.

[0052] In some embodiments, the speech information, the second text information, and the second generated style information can be input into a pre-trained denoising diffusion stochastic model, and the noise can be predicted by the denoising network in the denoising diffusion stochastic model.

[0053] Specifically, such as Figure 3 As shown, the denoising network can adopt a Transformer network architecture, including a causal attention layer, a semantic awareness attention layer, an adaptive instance normalization layer, and a sub-network. Then, the causal attention layer concatenates the speech rhythm features and intensity features in the speech information with the preset noisy gesture code to obtain new speech information features, which are then transmitted to the adaptive instance normalization layer.

[0054] It should be noted that the preset noise-added gesture code is a code set by those skilled in the art according to actual needs, or it can be a code obtained by those skilled in the art after adjusting the set code according to actual needs. This embodiment does not make any specific limitations.

[0055] Furthermore, the first text features in the second text information are fused with the preset intermediate features through the semantic awareness attention layer to obtain new intermediate features. The preset intermediate features are then replaced with the new intermediate features and transmitted to the adaptive instance normalization layer.

[0056] It should be noted that the preset intermediate features are features set by those skilled in the art according to actual needs, or features obtained by those skilled in the art after adjusting the set features according to actual needs. This embodiment does not make specific limitations.

[0057] In some implementations, after the first text feature in the second text information is obtained through the semantic awareness attention layer, the first text feature is subjected to max pooling to obtain the second text feature. The product of the first text feature and the second text feature is used as the semantic saliency of the second text information. The first text feature and the semantic saliency are then fused with a preset intermediate feature to obtain a new intermediate feature.

[0058] By calculating the semantic saliency of the second text information, the semantic awareness attention layer's ability to recognize semantics can be significantly improved. By fusing it with preset intermediate features, the accuracy of the denoising network in predicting noise can be improved.

[0059] In other implementations, a cross-attention mechanism can be used to map the first text features to a first matrix and a second matrix, and a preset intermediate feature to a third matrix. Based on the first matrix, the second matrix, the third matrix, and the semantic saliency, the features are fused using the following formula to obtain new intermediate features:

[0060]

[0061] Among them, I c T represents a new intermediate feature. p Let K represent the coefficients of the matrix transpose operation, V represent the first matrix, Q represent the second matrix, and s represent the third matrix. t denoted by semantic saliency, and C represents a constant.

[0062] Furthermore, the first generated style features in the second generated style information are mapped to numerical parameters through an adaptive instance normalization layer. The mean and variance of the new speech information features and the new intermediate features transmitted to the adaptive instance normalization layer are modified according to the numerical parameters. Finally, the sub-network is used to perform fusion calculation on the modified new speech information features and the new intermediate features, and the calculation result is used as the predicted noise.

[0063] In some implementations, a multilayer perceptron network can be used to map the first generated style features in the second generated style information to numerical parameters, and modify the mean and variance of the new speech information features and new intermediate features transmitted to the adaptive instance normalization layer according to the numerical parameters.

[0064] Step 104: Denoise the speech information, the second text information, and the second generation style information based on the predicted noise to obtain the gesture action encoding sequence to be generated.

[0065] After predicting noise through step 103 above, the speech information, the second text information, and the second generation style information are denoised to obtain the gesture action encoding sequence to be generated.

[0066] In some embodiments, the hand gesture encoding sequence to be generated can be calculated based on the predicted noise using the following formula:

[0067]

[0068] Among them, Z n-1 E represents the sequence of gesture codes to be generated. n Let x represent the predicted noise, which is randomly distributed according to a standard normal distribution, and n represent the number of diffusion steps. n , and σ n All of these represent constant coefficients.

[0069] Step 105: Input the encoded sequence of the gesture action to be generated into the pre-trained gesture action generation model to generate the corresponding gesture action.

[0070] The hand gesture encoding sequence to be generated, calculated through step 104 above, is input into the pre-trained hand gesture generation model to generate the corresponding hand gesture.

[0071] In some embodiments, the gesture generation model can be trained before the encoded sequence of gesture actions to be generated is input into the pre-trained gesture generation model.

[0072] Specifically, the gesture generation model includes an encoder and a decoder. The first training gesture information is input into the encoder for downsampling to obtain a first implicit coding sequence. According to a preset discrete codebook, each hidden vector in the first implicit coding sequence is mapped to obtain a second implicit coding sequence. Then, the second implicit coding sequence is decoded by the decoder to obtain the second training gesture information. Finally, a first training value is determined based on the second training gesture information and the first training gesture information. When the first preset convergence condition is met based on the first training value, the gesture generation model is obtained.

[0073] It should be noted that downsampling refers to sampling the first gesture action information to be trained a preset number of times at preset time intervals. The preset time interval and preset number of times can be set by those skilled in the art according to actual needs, or can be obtained by those skilled in the art after adjusting the set time interval and number of times according to actual needs. This embodiment does not make specific limitations.

[0074] The first training value can refer to the training loss value during network training, or it can refer to the number of training iterations during network training. When the first training value represents the training loss value, a first preset convergence condition is met when the training loss value is less than or equal to a preset first training loss value; when the first training value represents the number of training iterations, a first preset convergence condition is met when the number of training iterations is greater than a preset first iteration number. Those skilled in the art can set these values ​​according to actual conditions, and this embodiment does not impose specific limitations.

[0075] This application provides a gesture generation method and an electronic device. The method predicts noise from speech information, text information, and generation style information, and generates a gesture encoding sequence after denoising the predicted noise. This denoising process can remove erroneous or abnormal data in the speech information, text information, and generation style information, thereby improving the accuracy of understanding the user's intention. Finally, a pre-trained gesture generation model generates the corresponding gesture based on the gesture encoding sequence, thereby improving the generation speed and the diversity of generation styles.

[0076] See Figure 4 This application also provides a gesture generation device for performing the gesture generation method described in the above embodiments. The device includes:

[0077] Response module 11 is used to respond to the received voice information, first text information and first generation style information of the gesture action to be generated;

[0078] Encoding module 12 is used to encode the first text information and the first generation style information to obtain second text information and second generation style information;

[0079] Prediction module 13 is used to predict noise based on the speech information, the second text information, and the second generation style information;

[0080] The denoising module 14 is used to denoise the speech information, the second text information and the second generation style information according to the predicted noise to obtain the gesture action encoding sequence to be generated;

[0081] The gesture generation module 15 is used to input the encoded sequence of the gesture to be generated into the pre-trained gesture generation model to generate the corresponding gesture.

[0082] The gesture generation device provided in this application embodiment and the gesture generation method provided in the above embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0083] This application also provides an electronic device corresponding to the gesture generation method provided in the foregoing embodiments. Please refer to... Figure 5 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 2 As shown, the electronic device 30 may include: a processor 300, a memory 301, a bus 302, and a communication interface 303. The processor 300, the communication interface 303, and the memory 301 are connected through the bus 302. The memory 301 stores a computer program that can run on the processor 300. When the processor 300 runs the computer program, it executes the gesture generation method provided in any of the foregoing embodiments of this application.

[0084] The memory 301 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one physical port 303 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0085] Bus 302 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 301 is used to store programs. After receiving an execution instruction, the processor 300 executes the program. The camera pose estimation method disclosed in any of the foregoing embodiments of this application can be applied to the processor 300, or implemented by the processor 300.

[0086] The processor 300 may be an integrated circuit with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 300 or by instructions in software form. The processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 301. The processor 300 reads the information in memory 301 and, in conjunction with its hardware, completes the steps of the above method.

[0087] The electronic device provided in this application embodiment and the gesture action generation method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0088] This application also provides a computer-readable storage medium corresponding to the gesture generation method provided in the foregoing embodiments, which stores a computer program (i.e., a program product) thereon. When the computer program is run by a processor, it executes the gesture generation method provided in any of the foregoing embodiments.

[0089] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0090] This application also provides a computer program product corresponding to the gesture generation method provided in the foregoing embodiments, including a computer program that is executed by a processor to implement the gesture generation method provided in the above embodiments.

[0091] The computer-readable storage medium and computer program product provided in the above embodiments of this application are based on the same inventive concept as the gesture action generation method provided in the embodiments of this application, and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0092] It should be noted that:

[0093] The algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used in conjunction with the teachings herein. The required structure for constructing such devices is apparent from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0094] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0095] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0096] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0097] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0098] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the virtual machine creation apparatus according to embodiments of this application. This application can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0099] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0100] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A gesture motion generation method characterized by, The method comprises the following steps: in response to the received voice information of the to-be-generated gesture action, first text information and first generation style information; encoding processing is performed on the first text information and the first generation style information to obtain second text information and second generation style information; predicting noise according to the voice information, the second text information and the second generation style information; performing denoising processing on the voice information, the second text information and the second generation style information according to the predicted noise to obtain a to-be-generated gesture action encoding sequence; inputting the to-be-generated gesture action encoding sequence into a pre-trained gesture action generation model to generate a corresponding gesture action; wherein the step of predicting noise according to the voice information, the second text information and the second generation style information comprises: inputting the voice information, the second text information and the second generation style information into a pre-trained denoising diffusion random model, and predicting the noise through a denoising network in the denoising diffusion random model, wherein the denoising network comprises a causal attention layer, a semantic perception attention layer, an adaptive instance normalization layer and a subnetwork; wherein the step of predicting the noise through the denoising network in the denoising diffusion random model comprises: splicing the voice rhythm features and the voice intensity features in the voice information with a preset noisy gesture encoding through the causal attention layer to obtain new voice information features, which are transmitted to the adaptive instance normalization layer; fusing the first text features in the second text information with a preset intermediate feature through the semantic perception attention layer to obtain new intermediate features; and replacing the preset intermediate feature with the new intermediate features, which are transmitted to the adaptive instance normalization layer; mapping the first generation style features in the second generation style information into numerical parameters through the adaptive instance normalization layer; and modifying the mean and variance of the new voice information features and the new intermediate features transmitted to the adaptive instance normalization layer according to the numerical parameters; performing fusion calculation on the modified new voice information features and the new intermediate features by using the subnetwork, and taking the calculation result as the predicted noise.

2. The gesture motion generation method of claim 1, wherein, The gesture action generation model comprises an encoder and a decoder, and before the to-be-generated gesture action encoding sequence is input into the pre-trained gesture action generation model, the method further comprises the following steps: inputting first to-be-trained gesture action information into the encoder for down-sampling to obtain a first implicit encoding sequence; mapping each implicit vector in the first implicit encoding sequence according to a preset discrete codebook to obtain a second implicit encoding sequence; decoding the second implicit encoding sequence through the decoder to obtain second to-be-trained gesture action information; determining a first training value according to the second to-be-trained gesture action information and the first to-be-trained gesture action information; when it is determined that the first preset convergence condition is reached based on the first training value, obtaining the gesture action generation model.

3. The gesture motion generation method of claim 1, wherein, The second text information and the second generation style information are obtained by encoding the first text information and the first generation style information, and the encoding processing of the first text information and the first generation style information comprises: The second text information and the second generation style information are obtained by encoding the first text information and the first generation style information, and the encoding processing of the first text information and the first generation style information comprises: wherein the denotes the gesture action encoding sequence to be generated, the denotes the predicted noise, the denotes a random noise of a standard normal distribution, the n denotes a number of diffusion steps, the , and all denote constant coefficients.

4. The gesture motion generation method of claim 1, characterized by, The first text feature and the semantic saliency are fused with the preset intermediate feature to obtain a new intermediate feature. The second text feature obtained by maximum pooling processing of the first text feature is obtained. The product of the first text feature and the second text feature is taken as the semantic saliency of the second text information. The first text feature and the semantic saliency are fused with the preset intermediate feature to obtain a new intermediate feature.

5. The gesture motion generation method of claim 4, wherein, The first text feature and the semantic saliency are fused with the preset intermediate feature to obtain a new intermediate feature, which comprises: The first text feature is mapped into a first matrix and a second matrix by adopting a cross-attention mechanism, and the preset intermediate feature is mapped into a third matrix; The first matrix, the second matrix, the third matrix and the semantic saliency are fused by the following formula to obtain a new intermediate feature: Wherein the represents a new intermediate feature, represents a matrix transpose operation coefficient, K represents the first matrix, V represents the second matrix, and Q represents the third matrix, represents the semantic saliency, and C represents a constant.

6. The gesture motion generation method of claim 1, wherein, The format of the first generation style information at least comprises one of a text format, a video format and a three-dimensional action format, and the encoding processing of the first text information and the first generation style information to obtain the second text information and the second generation style information comprises: The first text information is encoded by adopting a pre-trained gesture action text feature extraction model to obtain the second text information; It is judged whether the format of the first generation style information is a text format, a video format or a three-dimensional action format; When the format of the first generation style information is a text format, a text encoder of a contrast-based language-image pre-training CLIP model is adopted to encode the first generation style information to obtain the second generation style information; When the format of the first generation style information is a video format, a video encoder of the CLIP model is adopted to encode the first generation style information to obtain the second generation style information; When the format of the first generation style information is a three-dimensional action format, an action encoder of a contrast-based language-image-action pre-training MotionCLIP model is adopted to encode the first generation style information to obtain the second generation style information.

7. The gesture motion generation method of claim 6, wherein, The gesture action text feature extraction model comprises a gesture encoder, a text encoder and a joint vector space, and before the first text information is encoded by adopting the pre-trained gesture action text feature extraction model to obtain the second text information, it further comprises: A third to-be-trained gesture action information is input into the gesture encoder for encoding to obtain a first gesture feature, and a second gesture feature is obtained by maximum pooling processing of the first gesture feature; The third to-be-trained gesture action information corresponding to-be-trained text information is input into the text encoder for encoding to obtain third text features, and fourth text features are obtained after maximum pooling processing of the third text features; The second gesture features and the fourth text features are mapped into the joint vector space and matched in time sequence, the second gesture features and the fourth text features that are successfully matched are taken as a first training pair, and other training pairs other than the first training pair are taken as a second training pair; The network is trained by using the first training pair and the second training pair, and a second training value is determined; When it is determined that a second preset convergence condition is reached based on the second training value, the gesture action text feature extraction model is obtained; And / or, Before the video encoder adopting the CLIP model encodes and processes the first generated style information to obtain the second generated style information, the method further includes: The to-be-trained gesture three-dimensional action data is input into a three-dimensional engine for rendering to obtain to-be-trained two-dimensional video data corresponding to the to-be-trained gesture three-dimensional action data; The to-be-trained gesture three-dimensional action data is input into a motion encoder in the MotionCLIP model for encoding to obtain to-be-trained gesture three-dimensional action gesture features; The to-be-trained two-dimensional video data is input into a to-be-trained video encoder based on a CLIP model for encoding to obtain to-be-trained two-dimensional video features; The network is trained by using the to-be-trained gesture three-dimensional action gesture features and the to-be-trained two-dimensional video features, and a third training value is determined; When it is determined that a third preset convergence condition is reached based on the third training value, the video encoder based on the CLIP model is obtained.

8. The gesture motion generation method of claim 7, wherein, The method further includes: The network parameters of the momentum versions of the gesture encoder and the text encoder are updated respectively by an exponential moving average method; And / or, When an error occurs in training the network by using the first training pair and the second training pair, the network parameters of the momentum versions of the gesture encoder and the text encoder are used to re-match the second gesture features and the fourth text features in time sequence, the second gesture features and the fourth text features that are successfully re-matched are taken as a new first training pair, and other training pairs other than the new first training pair are taken as a new second training pair.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method in any one of claims 1-8. The processor executes the computer program to implement the method in any one of claims 1-8.

Images