A method and apparatus for generating motion of a three-dimensional digital human, and an electronic device

Abstract

The application relates to the technical field of three-dimensional digital people, and discloses a three-dimensional digital person action generation method and device and electronic equipment. In the application, two mutually independent facial mouth shape feature coding units and body action feature coding units are adopted to code the related feature information of the three-dimensional digital person, and a bidirectional coding feature interaction unit is used to interact the feature information of the mutually independent facial mouth shape feature coding units and body action feature coding units, so that the generated facial mouth shape and body action can be ensured to be sufficiently coordinated, the mutually independent coding modes are used, the two coding units can more deeply learn the respective unique feature distribution, and the finally generated facial mouth shape is accurate and the body posture is expressive.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional digital human technology, specifically to a method, apparatus, and electronic device for generating motion in a three-dimensional digital human. Background Technology

[0002] A 3D digital human, also known as a digital virtual human body, is a three-dimensional structural model generated by integrating real human body tomographic data and using computer 3D reconstruction technology.

[0003] In related technologies, three-dimensional digital human network models are generally formed using separate modeling, unidirectional modeling, or unified modeling methods to generate the movements of the three-dimensional digital human. Separate modeling uses two independent network models that do not interact with each other, which can easily lead to inconsistencies between facial expressions and body movements in the generated 3D digital human. Unidirectional modeling cannot achieve bidirectional dynamic interaction between facial expressions and body movements. Unified modeling, in practice, forcibly merges the facial and body, which have different features, causing the model to sacrifice its ability to capture the fine movements of each individual face and body, resulting in unclear movements and a loss of the uniqueness and expressiveness of each feature. Therefore, the movements of the 3D digital humans generated by these methods are neither accurate nor coordinated. Summary of the Invention

[0004] This invention provides a method, apparatus, and electronic device for generating motion in a 3D digital human, in order to solve the problem that network models in related technologies are not accurate or coordinated enough when generating motion in a 3D digital human.

[0005] In a first aspect, the present invention provides a method for generating the motion of a three-dimensional digital human, the method comprising:

[0006] Acquire facial lip features, body movement features, and audio output features of the 3D digital human output; Facial lip features, body movement features, and audio output features are input into a 3D digital human model for prediction, resulting in a predicted target action of the 3D digital human. The 3D digital human model includes a generation module and a fine-tuning module. The generation module includes: A facial lip-shape feature encoding unit is used to encode a first audio feature associated with facial lip-shape features, wherein the first audio feature is derived from a portion of the audio output features; A body motion feature encoding unit is used to encode a second audio feature associated with the body motion feature; the second audio feature is also derived from a portion of the audio output feature. A bidirectional coding feature interaction unit is used to bidirectionally interact the first audio feature and the second audio feature to obtain the audio feature interaction result; The full-body coordination feature decoding unit is used to decode the third audio feature associated with the audio feature interaction result and output the full-body coordination movement of the three-dimensional digital human; the third audio feature also comes from a part of the audio output feature. The fine-tuning module includes: The personalized body movement adaptation unit is used to fine-tune the target movement prediction results based on the personalized body movement characteristics of the 3D digital human.

[0007] In some specific implementations, the 3D digital human outputs facial lip features, body movement features, and audio output features according to preset identity roles and preset emotional states; The facial lip feature encoding unit and the body motion feature encoding unit are two independent transformer encoders.

[0008] The bidirectional coding feature interaction unit performs bidirectional interaction between the first audio feature and the second audio feature based on the cross-attention mechanism to obtain the audio feature interaction result. The cross-attention mechanism is executed through the following process: When the first audio feature and the second audio feature flow through the preset intermediate layer of the bidirectional coding feature interaction unit during forward propagation, a bidirectional interaction action is triggered.

[0009] In some specific implementations, if the first audio feature is used as the querying party, then the second audio feature is used as the queryed party. When the current feature vector of the querying party is linearly projected to generate a query vector, the current feature vector of the queryed party is linearly projected to generate a key vector and a value vector, respectively.

[0010] In some specific implementations, during the query process, the similarity between the query vector and the key vector is calculated, and the similarity is normalized to obtain the attention weight. Then, the attention weight is used to perform a weighted summation of the value vector to extract the context features containing the dynamic information of the queried party.

[0011] In some specific implementations, the contextual features obtained from the query are fused with the input features corresponding to the queryer in the result processing, thereby obtaining an audio feature interaction result that incorporates bidirectional information.

[0012] In some specific implementations, the personalized body movement adaptation unit fine-tunes the target movement prediction results by using an adaptive calculation dynamic scaling gating mechanism based on the personalized body movement characteristics of the 3D digital human.

[0013] In some specific implementations, the dynamic scaling gating mechanism for adaptive computation is executed in the following manner: The input vector of personalized body movement features of the 3D digital human is fed into the linear projection layer, where matrix multiplication is performed to map it into a scalar for dimensionality reduction. Then, the scalar is processed through the activation layer to filter out negative values ​​and retain positive activation strength, thereby obtaining non-negative dynamic weight coefficients. The dynamic weight coefficient is then multiplied element-wise with the adaptation feature processed by the bottleneck layer to dynamically adjust the magnitude of the adaptation feature. The adjusted result is then added to and fused with the output of the backbone network in the generation module.

[0014] Secondly, the present invention provides a motion generation device for a three-dimensional digital human, the device comprising: The acquisition module is used to acquire facial lip features, body movement features, and audio output features of the 3D digital human output. The prediction module is used to input facial lip features, body movement features, and audio output features into the 3D digital human model for prediction, and obtain the target movement prediction result of the 3D digital human; the 3D digital human model includes: a generation module and a fine-tuning module; The generation module includes: A facial lip-shape feature encoding unit is used to encode a first audio feature associated with facial lip-shape features, wherein the first audio feature is derived from a portion of the audio output features; A body motion feature encoding unit is used to encode a second audio feature associated with the body motion feature; the second audio feature is also derived from a portion of the audio output feature. A bidirectional coding feature interaction unit is used to bidirectionally interact the first audio feature and the second audio feature to obtain the audio feature interaction result; The whole-body coordination feature decoding unit is used to encode the third audio feature associated with the audio feature interaction result; the third audio feature also comes from a part of the audio output feature. The fine-tuning module includes: The personalized body movement adaptation unit is used to fine-tune the target movement prediction results based on the personalized body movement characteristics of the 3D digital human.

[0015] Thirdly, the present invention provides an electronic device, comprising: The memory and processor are interconnected and communicate with each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the motion generation method of the three-dimensional digital human in the first aspect or any embodiment of the first aspect.

[0016] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions, the computer instructions being used to cause a computer to execute the motion generation method for a three-dimensional digital human in the first aspect or any embodiment of the first aspect.

[0017] The technical solution of this invention has the following advantages: This invention discloses a method, apparatus, and electronic device for generating motion in a three-dimensional digital human. Because the three-dimensional digital human model uses two independent facial lip-shape feature encoding units and body motion feature encoding units to encode the relevant feature information of the three-dimensional digital human, and through a bidirectional encoding feature interaction unit, the feature information of the independent facial lip-shape feature encoding units and body motion feature encoding units can be exchanged, which helps to ensure that the generated facial lip-shape and body motion are sufficiently coordinated. By using an independent encoding method, the two encoding units can learn their unique feature distributions more deeply, ensuring that the final generated facial lip-shape is accurate and the body posture is expressive. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a method for generating motion in a three-dimensional digital human according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a three-dimensional digital human model according to an embodiment of the present invention; Figure 3 This is a structural block diagram of a motion generation device for a three-dimensional digital human according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0022] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0023] A 3D digital human, also known as a digital virtual human body, is a three-dimensional structural model generated by integrating real human body tomographic data and using computer 3D reconstruction technology.

[0024] In related technologies, three-dimensional digital human network models are generally formed using separate modeling, unidirectional modeling, or unified modeling methods. Separate modeling uses two independent network models that do not interact with each other, which can easily lead to inconsistencies between facial expressions and body movements in the generated 3D digital human. Unidirectional modeling cannot achieve bidirectional dynamic interaction between facial expressions and body movements. Unified modeling, in practice, forcibly merges the face and body, which have different features, causing the model to sacrifice its ability to capture the fine movements of each individual face and body, resulting in unclear movements and a loss of the uniqueness and expressiveness of each feature.

[0025] In view of this, this embodiment provides a method for generating motion of a three-dimensional digital human to solve the technical problems existing in the above-mentioned solutions.

[0026] This application provides a method for generating motion in a three-dimensional digital human, such as... Figure 1 As shown, it can be used with computer devices such as mobile phones, tablets, desktop computers, laptops, and servers. Figure 1 This is a flowchart of a three-dimensional digital human motion generation method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain the facial lip features, body movement features, and audio output features of the 3D digital human output.

[0027] Step S102: Input facial lip features, body movement features, and audio output features into the 3D digital human model for prediction to obtain the target movement prediction result of the 3D digital human; wherein, the 3D digital human model includes: a generation module and a fine-tuning module.

[0028] Among them, such as Figure 2 As shown, this is a schematic diagram of the structure of a three-dimensional digital human model. The three-dimensional digital human model 2 includes: a generation module 21 and a fine-tuning module 22.

[0029] The generation module 21 includes: a facial lip shape feature encoding unit 211, a bidirectional encoding feature interaction unit 212, and an audio environment feature encoding unit 213.

[0030] The facial lip-shape feature encoding unit is used to encode a first audio feature associated with the facial lip-shape feature, wherein the first audio feature is derived from a portion of the audio output feature; the body motion feature encoding unit is used to encode a second audio feature associated with the body motion feature; the second audio feature is also derived from a portion of the audio output feature.

[0031] Specifically, the facial lip-sync feature encoding unit is mainly used to encode a first audio feature associated with facial lip-sync features. This first audio feature can be an audio feature strongly correlated with facial lip-sync pronunciation, such as phonemes and rhythm. The body motion feature encoding unit is mainly used to encode a second audio feature associated with body motion features. This second audio feature can be a second audio feature associated with body motion features, including but not limited to posture and gesture-related audio features, such as semantics and rhythm.

[0032] The bidirectional coding feature interaction unit is used to perform bidirectional interaction between the first audio feature and the second audio feature to obtain the audio feature interaction result.

[0033] Specifically, the bidirectional coding feature interaction unit is mainly used for information exchange between the first audio feature and the second audio feature.

[0034] The full-body coordination feature decoding unit is used to decode the third audio feature associated with the audio feature interaction result and output the full-body coordination movement of the three-dimensional digital human; the third audio feature also comes from a part of the audio output feature.

[0035] exist Figure 2 In the middle, the fine-tuning module 22 includes: a personalized body movement adaptation unit 221, which is used to fine-tune the target movement prediction result based on the personalized body movement characteristics of the three-dimensional digital human.

[0036] Specifically, the whole-body coordination feature decoding unit can be a transformer decoder. In other words, in order to ensure that the generated overall movement is highly aligned with the speech signal, the whole-body coordination feature decoding unit will also simultaneously receive the audio output features of the three-dimensional digital human and decode these features in combination with the audio feature interaction results.

[0037] exist Figure 2The diagram shows that after the bidirectional interaction between the facial lip-sync feature encoding unit and the body motion feature encoding unit, the facial lip-sync features and body motion features are added and fused together, and then fed as the main input into a unified full-body coordination feature decoding unit, namely the transformer decoder. At this stage, to ensure high alignment between the generated overall motion and the speech signal, the full-body coordination feature decoding unit also simultaneously receives the audio output features from the 3D digital human. Specifically, the audio output features extracted from the 3D digital human, such as audio semantic features, audio rhythm features, and audio content features, are added together and fused into the full-body coordination feature decoding unit as conditional features through a cross-attention mechanism. Subsequently, this full-body coordination feature decoding unit is responsible for learning the joint probability distribution of facial and body features under the constraints of the audio conditions, and finally, through a single output head, directly predicts the target motion prediction result of the 3D digital human, which includes facial lip-sync features and body motion features.

[0038] Specifically, to enhance the accuracy of feature extraction for each branch, this embodiment employs a multi-task supervision strategy. At specific locations, two auxiliary prediction heads are connected to the final output layers of the facial lip-shape feature encoding unit and the body motion feature encoding unit, respectively. The entire 3D digital human model includes a composite loss function of auxiliary loss and main loss. The computer device first predicts the intermediate coefficients of the face and body using the auxiliary heads, calculating the differences between them and the true labels; simultaneously, the whole-body coordination feature decoding unit outputs the difference between the overall motion coefficients and the actual overall motion. Finally, this application obtains the total loss value through weighted summation and uses the backpropagation algorithm to update the network parameters of each prediction head in the facial lip-shape feature encoding unit, the body motion feature encoding unit, and the whole-body coordination feature decoding unit based on this total loss value, thereby achieving simultaneous optimization of local features and overall coordination.

[0039] In some specific implementations, the 3D digital human outputs facial lip-sync features, body motion features, and audio output features according to a preset identity role and preset emotional state. The facial lip-sync feature encoding unit and the body motion feature encoding unit are two independent transformer encoders. The bidirectional encoding feature interaction unit performs bidirectional interaction between the first audio feature and the second audio feature based on a cross-attention mechanism to obtain the audio feature interaction result; The cross-attention mechanism is executed through the following process: When the first audio feature and the second audio feature flow through the preset intermediate layer of the bidirectional coding feature interaction unit during forward propagation, a bidirectional interaction action is triggered.

[0040] Specifically, the preset identity role can represent the 3D digital human playing the role of an artificial intelligence assistant, and the preset emotional state can represent the 3D digital human maintaining an objective, neutral, and stable emotional state.

[0041] Facial lip-sync features are primarily used to encode the first audio feature associated with facial lip-sync features. This first audio feature can be an audio feature strongly correlated with facial lip-sync pronunciation, such as phonemes and rhythms. Body movement features include, but are not limited to, posture and gesture-related audio features, such as semantics and rhythms. Audio output features include audio semantic features, audio rhythm features, and audio content features output by the 3D digital human.

[0042] This application outputs facial lip features, body movement features, and audio output features according to preset identity roles and preset emotional states, which is beneficial for accurately predicting the target movements of three-dimensional digital humans.

[0043] This application uses this separate and dedicated encoding method, which allows the facial lip feature encoding unit and the body motion feature encoding unit to learn more deeply the unique motion distribution of their respective fields, ensuring that the final generated facial expressions, lip shapes and body postures are expressive.

[0044] In some specific implementations, if the first audio feature is used as the querying party, then the second audio feature is used as the queryed party. When the current feature vector of the querying party is linearly projected to generate a query vector, the current feature vector of the queryed party is linearly projected to generate a key vector and a value vector, respectively.

[0045] In some specific implementations, during the query process, the similarity between the query vector and the key vector is calculated, and the similarity is normalized to obtain the attention weight. Then, the attention weight is used to perform a weighted summation of the value vector to extract the context features containing the dynamic information of the queried party.

[0046] In some specific implementations, the contextual features obtained from the query are fused with the input features corresponding to the queryer in the result processing, thereby obtaining an audio feature interaction result that incorporates bidirectional information.

[0047] Specifically, in terms of the triggering timing of the three-dimensional digital human model, when the feature data of the two branches, audio lip-sync features and body movement features, flow through the preset intermediate layer of the encoder during the forward propagation process, usually the third layer of the encoder, the bidirectional interactive action is triggered.

[0048] Regarding the definition of query elements, if the first audio feature is taken as the querying element, its current feature vector is linearly projected to generate a query vector; while the second audio feature is taken as the querying element's feature vector and linearly projected to generate a key vector and a value vector, respectively. The reason why the two can query each other is that they are aligned in the time dimension and are mapped to a shared latent feature space through projection, thus enabling a feature of one modality to be used as an index to retrieve relevant information of another modality.

[0049] During the query process, the similarity between the query vector and the key vector is calculated, and the similarity is normalized to obtain the attention weight. Then, the value vector is weighted and summed using this weight to extract the context features containing the dynamic information of the second branch.

[0050] In result processing, the contextual features obtained from the query are fused with the original input features of the first branch layer. This fusion process includes residual connections, layer normalization, and multilayer perceptron transformation to obtain updated features that incorporate bidirectional information. This result is then used as the input for the next layer. This process is bidirectional: facial lip-shape features flow to body motion features, and vice versa. This bidirectional interaction mechanism simulates the mutual influence of head, face, and body postures in the real world, ensuring that the features possess prior knowledge of overall coordination during the encoding stage.

[0051] In some specific implementations, the personalized body movement adaptation unit fine-tunes the target movement prediction results by using an adaptive calculation dynamic scaling gating mechanism based on the personalized body movement characteristics of the 3D digital human.

[0052] In some specific implementations, the motion generation method for 3D digital humans uses an adaptive calculation-based dynamic scaling gating mechanism, which is executed in the following manner: The input vector of personalized body movement features of the 3D digital human is fed into the linear projection layer, where matrix multiplication is performed to map it into a scalar for dimensionality reduction. Then, the scalar is processed through the activation layer to filter out negative values ​​and retain positive activation strength, thereby obtaining non-negative dynamic weight coefficients. The dynamic weight coefficient is then multiplied element-wise with the adaptation feature processed by the bottleneck layer to dynamically adjust the magnitude of the adaptation feature. The adjusted result is then added to and fused with the output of the backbone network in the generation module.

[0053] Specifically, the personalized body movement adaptation unit decouples general movement generation capabilities from the personalized style of the 3D digital human. First, the personalized body movement adaptation unit described above is fully pre-trained on a dataset with a fixed identity and neutral emotion, aiming to enable the 3D digital human model to learn a high-quality, general movement expression capability.

[0054] When a new identity or emotion needs to be adapted, all parameters of the backbone network in the pre-trained 3D digital human model are frozen, and only the lightweight adaptation module, which is inserted into the encoder layer in a bypass parallel manner, is fine-tuned. In terms of internal processing logic, this module first receives the external conditional vector (identity or emotion) to modulate the features, and then uses a bottleneck structure of dimensionality reduction-nonlinear activation-dimensionality increase to transform the features, which greatly reduces the number of parameters (accounting for only about 10% of the backbone network).

[0055] Specifically, in the output fusion stage, this embodiment introduces a dynamic scaling gating mechanism based on adaptive calculation of input features. In the model architecture design, this mechanism is implemented through an independent linear projection layer. The specific calculation principle is as follows: First, the input feature vector of the current layer is passed to this linear projection layer, and matrix multiplication is performed to map it to a scalar for dimensionality reduction. Then, this scalar is processed through an activation layer to filter out negative values ​​and retain positive activation strengths, thereby obtaining non-negative dynamic weight coefficients. Finally, the system multiplies these dynamic weight coefficients element-wise with the adapted features processed by the bottleneck layer, achieving dynamic adjustment of the adapted feature amplitude. The adjusted result is then added to and fused with the output of the original backbone network. This mechanism can automatically calculate and determine the integration ratio of personalized features based on the specific context of the action at the current moment, thereby achieving precise personalized adjustments while retaining general action knowledge.

[0056] Therefore, the motion generation method for 3D digital humans in this application uses two independent facial lip-shape feature encoding units and body motion feature encoding units to encode the relevant feature information of the 3D digital human model. Furthermore, through a bidirectional encoding feature interaction unit, the feature information of the independent facial lip-shape feature encoding units and body motion feature encoding units is exchanged. This helps to ensure that the generated facial lip-shape and body motion are sufficiently coordinated. By using an independent encoding method, the two encoding units can learn their unique feature distributions more deeply, ensuring that the final generated facial lip-shape is accurate and the body posture is expressive.

[0057] This embodiment also provides a motion generation device for a three-dimensional digital human, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0058] This embodiment provides a motion generation device for a three-dimensional digital human, such as... Figure 3 As shown, it includes: The acquisition module 301 is used to acquire the facial lip features, body movement features, and audio output features of the 3D digital human output; The prediction module 302 is used to input facial lip features, body movement features, and audio output features into the 3D digital human model for prediction, and obtain the target movement prediction result of the 3D digital human; wherein, the 3D digital human model includes: a generation module and a fine-tuning module; The generation module includes: A facial lip-shape feature encoding unit is used to encode a first audio feature associated with facial lip-shape features, wherein the first audio feature is derived from a portion of the audio output features; A body motion feature encoding unit is used to encode a second audio feature associated with the body motion feature; the second audio feature is also derived from a portion of the audio output feature. A bidirectional coding feature interaction unit is used to bidirectionally interact the first audio feature and the second audio feature to obtain the audio feature interaction result; The full-body coordination feature decoding unit is used to decode the third audio feature associated with the audio feature interaction result and output the full-body coordination movement of the three-dimensional digital human; the third audio feature also comes from a part of the audio output feature. The fine-tuning module includes: The personalized body movement adaptation unit is used to fine-tune the target movement prediction results based on the personalized body movement characteristics of the 3D digital human.

[0059] In some optional implementations, the 3D digital human outputs facial lip features, body movement features, and audio output features according to a preset identity role and preset emotional state; The facial lip feature encoding unit and the body motion feature encoding unit are two independent transformer encoders.

[0060] The bidirectional coding feature interaction unit performs bidirectional interaction between the first audio feature and the second audio feature based on the cross-attention mechanism to obtain the audio feature interaction result. The cross-attention mechanism is executed through the following process: When the first audio feature and the second audio feature flow through the preset intermediate layer of the bidirectional coding feature interaction unit during forward propagation, a bidirectional interaction action is triggered.

[0061] In some optional implementations, if the first audio feature is used as the querying party, then the second audio feature is used as the queryed party. When the current feature vector of the querying party is linearly projected to generate a query vector, the current feature vector of the queryed party is linearly projected to generate a key vector and a value vector, respectively.

[0062] In some alternative implementations, during the query process, the similarity between the query vector and the key vector is calculated, and the similarity is normalized to obtain the attention weight. The attention weight is then used to perform a weighted summation of the value vectors to extract contextual features containing dynamic information of the queried party.

[0063] In some optional implementations, in the result processing, the context features obtained from the query are fused with the input features corresponding to the queryer, thereby obtaining an audio feature interaction result that incorporates bidirectional information.

[0064] In some optional implementations, the personalized body movement adaptation unit fine-tunes the target movement prediction results using an adaptive calculation dynamic scaling gating mechanism based on the personalized body movement characteristics of the 3D digital human.

[0065] In some alternative implementations, the dynamic scaling gating mechanism for adaptive computation is performed as follows: The input vector of personalized body movement features of the 3D digital human is fed into the linear projection layer, where matrix multiplication is performed to map it into a scalar for dimensionality reduction. Then, the scalar is processed through the activation layer to filter out negative values ​​and retain positive activation strength, thereby obtaining non-negative dynamic weight coefficients. The dynamic weight coefficient is then multiplied element-wise with the adaptation feature processed by the bottleneck layer to dynamically adjust the magnitude of the adaptation feature. The adjusted result is then added to and fused with the output of the backbone network in the generation module.

[0066] The motion generation device for three-dimensional digital humans provided in this embodiment of the invention can execute the motion generation method for three-dimensional digital humans provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method. Further functional descriptions of the various modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.

[0067] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0068] The following is a detailed reference. Figure 4 , Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0069] The following is a detailed reference. Figure 4 This diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 401, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 402 or a program loaded from memory 408 into random access random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of the electronic device. The processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0070] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0071] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 409, or installed from a memory 408, or installed from a ROM 402. When the computer program is executed by the processor 401, it performs the functions defined in the three-dimensional digital human motion generation method of the embodiments of the present invention.

[0072] Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0073] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the motion generation method for the three-dimensional digital human shown in the above embodiments is implemented.

[0074] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0075] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for generating motion in a three-dimensional digital human, characterized in that, The method includes: Acquire facial lip features, body movement features, and audio output features of the 3D digital human output; The facial lip features, body movement features, and audio output features are input into the 3D digital human model for prediction to obtain the target movement prediction result of the 3D digital human; wherein, the 3D digital human model includes: a generation module and a fine-tuning module; The generation module includes: A facial lip-shape feature encoding unit is used to encode a first audio feature associated with the facial lip-shape feature, wherein the first audio feature is derived from a portion of the audio output feature; A body motion feature encoding unit is used to encode a second audio feature associated with the body motion feature; the second audio feature is also derived from a portion of the audio output feature. A bidirectional coding feature interaction unit is used to bidirectionally interact the first audio feature and the second audio feature to obtain an audio feature interaction result; A full-body coordination feature decoding unit is used to decode a third audio feature associated with the interaction result of the audio feature and output the full-body coordination movement of the three-dimensional digital human; the third audio feature is also derived from a portion of the audio output feature. The fine-tuning module includes: A personalized body movement adaptation unit is used to fine-tune the target movement prediction result based on the personalized body movement characteristics of the three-dimensional digital human.

2. The method for generating motion in a three-dimensional digital human according to claim 1, characterized in that, The three-dimensional digital human outputs facial lip features, body movement features, and audio output features according to a preset identity role and preset emotional state; The facial lip shape feature encoding unit and the body motion feature encoding unit are two independent transformer encoders; The bidirectional coding feature interaction unit performs bidirectional interaction between the first audio feature and the second audio feature based on the cross-attention mechanism to obtain the audio feature interaction result. The cross-attention mechanism is executed through the following process: When the first audio feature and the second audio feature pass through the preset intermediate layer of the bidirectional coding feature interaction unit during forward propagation, a bidirectional interaction action is triggered.

3. The method for generating motion in a three-dimensional digital human according to claim 2, characterized in that, If the first audio feature is used as the querying party, then the second audio feature is used as the queryed party. When the current feature vector of the querying party is linearly projected to generate a query vector, the current feature vector of the queryed party is linearly projected to generate a key vector and a value vector, respectively.

4. The method for generating motion in a three-dimensional digital human according to claim 3, characterized in that, During the query process, the similarity between the query vector and the key vector is calculated, and the similarity is normalized to obtain the attention weight. Then, the attention weight is used to perform a weighted summation on the value vector to extract the context features containing the dynamic information of the queried party.

5. The method for generating motion in a three-dimensional digital human according to claim 4, characterized in that, In terms of result processing, the context features obtained from the query are fused with the input features corresponding to the querying party to obtain the audio feature interaction result that incorporates bidirectional information.

6. The method for generating motion in a three-dimensional digital human according to claim 1, characterized in that, The personalized body movement adaptation unit fine-tunes the target movement prediction results based on the personalized body movement characteristics of the three-dimensional digital human using an adaptive calculation dynamic scaling gating mechanism.

7. The method for generating motion in a three-dimensional digital human according to claim 6, characterized in that, The dynamic scaling gating mechanism for adaptive computation is implemented as follows: The input vector of personalized body movement features of the 3D digital human is fed into the linear projection layer, where matrix multiplication is performed to map it into a scalar for dimensionality reduction. Then, the scalar is processed through the activation layer to filter out negative values ​​and retain positive activation strength, thereby obtaining non-negative dynamic weight coefficients. The dynamic weight coefficient is then multiplied element-wise with the adaptation feature processed by the bottleneck layer to dynamically adjust the magnitude of the adaptation feature. The adjusted result is then added to and fused with the output of the backbone network in the generation module.

8. A motion generation device for a three-dimensional digital human, characterized in that, The device includes: The acquisition module is used to acquire facial lip features, body movement features, and audio output features of the 3D digital human output. The prediction module is used to input the facial lip features, body movement features, and audio output features into the 3D digital human model for prediction, and obtain the target movement prediction result of the 3D digital human; wherein, the 3D digital human model includes: a generation module and a fine-tuning module; The generation module includes: A facial lip-shape feature encoding unit is used to encode a first audio feature associated with the facial lip-shape feature, wherein the first audio feature is derived from a portion of the audio output feature; A body motion feature encoding unit is used to encode a second audio feature associated with the body motion feature; the second audio feature is also derived from a portion of the audio output feature. A bidirectional coding feature interaction unit is used to bidirectionally interact the first audio feature and the second audio feature to obtain an audio feature interaction result; A full-body coordination feature decoding unit is used to decode a third audio feature associated with the interaction result of the audio feature and output the full-body coordination movement of the three-dimensional digital human; the third audio feature is also derived from a portion of the audio output feature. The fine-tuning module includes: A personalized body movement adaptation unit is used to fine-tune the target movement prediction result based on the personalized body movement characteristics of the three-dimensional digital human.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the motion generation method for a three-dimensional digital human as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the motion generation method for a three-dimensional digital human as described in any one of claims 1 to 7.

Images