Digital human action generation method and device, storage medium and computer program product

Abstract

The application provides a digital human action generation method and device, a storage medium and a computer program product. The method comprises: obtaining multi-modal text information, and determining a context vector based on the multi-modal text information; inputting the context vector into a preset model to output a first frame latent action vector and a first weight matrix; for the t-th frame latent action vector, inputting a (t-1)-th frame latent action vector into the preset model to output a second weight matrix, and generating the t-th frame latent action vector based on the second weight matrix and all historical weight matrices generated before the second weight matrix, t being a positive integer greater than 1, and the historical weight matrices at least including the first weight matrix; determining a latent action vector sequence based on the first frame latent action vector and the t-th frame latent action vector; and determining an action vector of each frame based on the latent action vector sequence to control a digital human to perform corresponding actions according to the action vector.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, device, storage medium, and computer program product for generating digital human motion. Background Technology

[0002] In recent years, fields such as metaverse and digital humans have seen rapid development. The realism and expressiveness of digital humans largely depend on the naturalness, diversity, and interactivity of their actions. Therefore, efficient and natural digital human motion generation technology has become a core technological bottleneck and research focus in this field.

[0003] However, traditional motion generation methods mainly rely on manual creation and procedural rules. The model outputs motion instructions for the digital human to execute based on the input motion information. The motions generated in this way are relatively simple. Moreover, most related technologies only generate motions within 10 seconds. When the motion generation time is longer, the motion tends to be static, which does not meet the actual needs. Summary of the Invention

[0004] In view of this, embodiments of this application provide a digital human motion generation method, device, storage medium, and computer program product, which can generate diverse digital human motions that are not limited by duration and meet actual needs.

[0005] The technical solution of this application is implemented as follows: In a first aspect, this application provides a method for generating digital human actions, the method comprising: Acquire multimodal text information and determine the context vector based on the multimodal text information; Input the context vector into the preset model, and output the first frame potential action vector and the first weight matrix; For the potential action vector of frame t, the potential action vector of frame (t-1) is obtained and input into the preset model, and a second weight matrix is ​​output. Based on the second weight matrix and all historical weight matrices generated before the second weight matrix, the potential action vector of frame t is generated, where t is a positive integer greater than 1, and the historical weight matrix includes at least the first weight matrix. Based on the potential motion vector of the first frame and the potential motion vector of the t-th frame, a potential motion vector sequence is determined; and based on the potential motion vector sequence, a motion vector for each frame is determined to control the digital human to perform corresponding actions according to the motion vector.

[0006] Secondly, this application provides a digital human motion generation device, the digital human motion generation device comprising: The acquisition unit is used to acquire multimodal text information; A determining unit is used to determine a context vector based on the multimodal text information; The output unit is used to input the context vector into the preset model and output the first frame potential action vector and the first weight matrix; The output unit is also used to obtain the potential action vector of frame t-1 for the potential action vector of frame t-1, input it into the preset model, output the second weight matrix, and generate the potential action vector of frame t based on the second weight matrix and all historical weight matrices generated before the second weight matrix, where t is a positive integer greater than 1, and the historical weight matrix includes at least the first weight matrix. The determining unit is further configured to determine a potential action vector sequence based on the potential action vector of the first frame and the potential action vector of the t-th frame; and to determine the action vector of each frame based on the potential action vector sequence, so as to control the digital human to perform corresponding actions according to the action vector.

[0007] Thirdly, this application provides a digital human motion generation device, which includes a processor and a memory; the processor executes the running program stored in the memory to implement the above method.

[0008] Fourthly, this application provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described digital human motion generation method.

[0009] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described digital human motion generation method.

[0010] This application provides a method, device, storage medium, and computer program product for generating digital human actions. The method includes: acquiring multimodal text information and determining a context vector based on the multimodal text information; inputting the context vector into a preset model and outputting a first frame potential action vector and a first weight matrix; for the t-th frame potential action vector, acquiring the (t-1)-th frame potential action vector and inputting it into the preset model, outputting a second weight matrix, and generating the t-th frame potential action vector based on the second weight matrix and all historical weight matrices generated before the second weight matrix, where t is a positive integer greater than 1, and the historical weight matrix includes at least the first weight matrix; determining a potential action vector sequence based on the first frame potential action vector and the t-th frame potential action vector; and determining the action vector for each frame based on the potential action vector sequence to control the digital human to perform corresponding actions according to the action vectors. By adopting the above implementation scheme, when generating digital human actions, multimodal text information is used as the data to determine the input preset model context vector. The context vector is determined based on the text information of multiple modalities. The digital human actions generated by the context vector that integrates text information of multiple modalities have diversity. When generating digital human actions based on the context vector, the motion latent vector of the previous frame is referenced to generate a second weight matrix, and the motion latent vector of the current frame is further combined with the historical weight matrix to generate the motion latent vector of the current frame. In this way, without setting a duration, the digital human motion latent vector of the current frame can be continuously output according to the historical motion latent vector to obtain the final motion vector of each frame. In this way, without limiting the action generation duration, action output can be continuously performed to meet the needs of digital humans in real-world environments. Attached Figure Description

[0011] Figure 1 This application provides a schematic flowchart of a digital human motion generation method. Figure 2 This is a schematic diagram of the overall process of a digital human motion generation method provided in an embodiment of this application; Figure 3 A schematic diagram of the composition structure of a digital human motion generation device provided in this application embodiment. Figure 1 ; Figure 4 A schematic diagram of the composition structure of a digital human motion generation device provided in this application embodiment. Figure 2 . Detailed Implementation

[0012] To gain a more detailed understanding of the features and technical content of the embodiments of this application, the technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments. The accompanying drawings are for reference only and are not intended to limit the embodiments of this application.

[0013] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0014] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first / second / third" used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein.

[0015] In recent years, fields such as metaverse and digital humans have accelerated their development. The realism and expressiveness of digital humans largely depend on the naturalness, diversity, and interactivity of their actions. Therefore, efficient and natural digital human motion generation technology has become a core technological bottleneck and research focus in this field. Traditional motion generation methods mainly rely on manual production and programmed rules, but their production costs are high, the cycle is long, and they struggle to meet the demands of real-time interaction and large-scale content production. On the other hand, for idle actions in digital human applications, the industry generally adopts a solution combining state machines with preset animation loops, that is, combining and playing a limited number of pre-made animation clips (such as breathing, shaking, looking around, etc.). While this method is efficient, it cannot eliminate the mechanical repetition of actions. After responding to user input, the digital human returns to a mechanical state, making the digital human lack a true sense of "life." Besides smooth and natural movements, users also want each digital human to have a unique style with corresponding emotions and identities, making it more suitable for various scenarios and closer to real people.

[0016] With the development of Artificial Intelligence Generated Content (AIGC) technology in recent years, data-driven AI-generated digital human motion methods have become a hot topic. Among related technologies, the following are AI-based digital human motion generation methods: Method 1: This method uses a large language model to drive digital human actions. The core of this method is to solve the problem of stiff and slow response of traditional digital human actions. The large language model is used to parse action-related tags from dialogues, and smoothing algorithms are used to process these tags to ensure smooth and natural transitions between actions. A digital human rule base is established and continuously updated through user feedback, thereby generating emotional, coherent and highly adaptive dynamic performances of digital humans.

[0017] Method 2: Employing a multilingual 3D digital human interaction method using a Generative Pre-trained Transformer (GPT) allows the digital human to respond to user input in a streaming manner. The core of this method's action generation process lies in generating the digital human's speaking actions, without involving the generation of full-body movements of the digital human.

[0018] Method 3: Employ a high-granularity digital human action generation method based on user intent perception. The core of this method is that in each frame of the digital human's response, the system makes decisions through a value-policy neural network to achieve frame-by-frame fine-grained control and dynamically select the most suitable action frame.

[0019] While the above AI-based technical solutions have made significant progress in digital human motion generation, they still have the following drawbacks: 1. The above technical solutions mainly address the handling of user input responses and do not address the generation of idle actions between user inputs. The common approach in related technologies remains to integrate these actions with preset idle animations. Therefore, the key to making non-player characters (NPCs) in applications like Metaverse truly intelligent NPCs lies in enabling digital humans to proactively and creatively generate meaningful and non-repetitive idle actions based on their own settings and historical states during "idle" states without external commands, rather than relying on preset animations.

[0020] 2. The problem of multimodal information fusion: When generating digital human actions, in many digital human application scenarios, there are usually not only multiple modalities such as actions and emotions, but also multiple identities of digital humans in the same scene. Therefore, a technology that can fuse multimodal information is needed.

[0021] 3. When generating long-duration 3D human motion, most related technical solutions generate motion within 10 seconds. When the generated motion time is long, the motion tends to stagnate, does not conform to physical laws, or has a lot of noise.

[0022] Based on this, this application addresses the issues of failing to actively and creatively generate meaningful and non-repetitive idle actions, multimodal information fusion, and the long generation time of 3D human actions in related technologies. It proposes a text-driven method for generating long-duration 3D human actions based on multimodal fusion, mainly embodied in the following parts: Given the diverse application scenarios of digital humans, this application proposes a multimodal fusion context fusion module that can parse the digital human's emotion tags and identity tags from the user's input. These emotion tags and identity tags are then used as conditions to guide the generation of digital human actions, enabling the generation of actions that are more suitable for the usage scenario under different emotions and identities of the same action command.

[0023] To address the duration limitations of digital human motion generation, this application proposes an autoregressive motion generation module. This module generates motions based on multimodal text information and references historical latent motion vector sequences, thus solving the duration limitation problem in motion generation in related technologies. Furthermore, a diffusion model is incorporated into this module for generating motion latent vectors into motion vectors. When generating long-duration motions, this module can also control the diversity of motions, preventing long-duration motions from becoming static.

[0024] Regarding the generation of idle actions, this application also proposes to generate digital human actions based on an autoregressive action generation module and multimodal text information. This method does not rely on preset animations, but dynamically generates idle actions of the digital human that conform to the current usage scenario, making the digital human actions more realistic.

[0025] The detailed technical solutions of this application embodiment are as follows: This application provides a method for generating digital human actions, such as... Figure 1 As shown, the method may include: S101. Obtain multimodal text information and determine the context vector based on the multimodal text information.

[0026] In this embodiment of the application, the multimodal text information includes digital human action instruction text information, digital human style and emotion setting text information, and digital human identity text information.

[0027] In this embodiment, multimodal text information can be obtained by receiving multimodal text information input. Specifically, the user inputs digital human action instructions, digital human style and emotion settings, and digital human identity in text form. For example, the user inputs text information such as "You are now a cheerful and lively host. Wave and greet everyone in a happy tone."

[0028] It should be noted that multimodal text information is not limited to the three types of text information mentioned above. In particular, the selection can be made according to the actual situation, and no specific limitation is made in the embodiments of this application.

[0029] In one embodiment of this application, if one or more dimensions of the multimodal text information are missing after obtaining the multimodal text information, the complete multimodal text information can be obtained in the following way. Specifically, if one or more dimensions of the multimodal text information are missing, the text information corresponding to the missing dimensions is obtained through random sampling; or, based on the text information of at least one existing dimension in the multimodal text information, the text information corresponding to the missing dimensions is sampled and obtained.

[0030] In this application embodiment, for scenarios where one or more multidimensional text information is missing in multimodal text information, if one or more of these inputs (such as input digital human action instructions, digital human style emotion settings, and digital human identity text information) are missing during the initial application, the action generation device (or system) will randomly sample a mean state as the multimodal input based on the distribution learned from the training data.

[0031] For example, if the digital human identity text information is missing in the multimodal text information, the mean state corresponding to the digital human identity text information can be randomly sampled from the distribution learned from the training data of the digital human identity text information as the digital human identity text information in the multimodal text information.

[0032] It should be noted that the above applies to situations where at least one or all of the multimodal text information is missing.

[0033] It should be noted that "at least one" can be described as "one or more", and "one or more" can also be described as "at least one".

[0034] In the embodiments of this application, when there is at least one input in the multimodal text information, that is, when there is at least one dimension of text information (such as the text information of digital human style emotion setting) in the multimodal text information, the other modal (or dimension) text information (such as digital human identity text information) can be obtained by sampling the model input by the multimodal text information with the existing input (such as the text information of digital human style emotion setting) as a condition.

[0035] It should be noted that this applies to multimodal text information where at least one modality exists, but not to cases where all modalities are missing.

[0036] In this embodiment of the application, after obtaining the multimodal text information, the context vector is further determined by the multimodal context fusion module.

[0037] In this embodiment, the core objective of the multimodal context fusion module is to identify action instructions (e.g., "wave to everyone"), style and emotion (e.g., "cheerful and lively"), and digital human identity tags (e.g., "host") from user-input multimodal text information (e.g., "You are now a cheerful and lively host. Wave to everyone in a happy tone."). This multimodal input information is then transformed and fused into a unified, high-dimensional, information-dense context vector (i.e., a context vector). This vector serves as the conditional input for the subsequent autoregressive generation module, guiding it to generate actions of a specific style, content, and emotional tone. The multimodal context fusion module mainly consists of three parts: a structured semantic parsing unit, a multimodal encoder, and a fusion unit.

[0038] It should be noted that the multimodal context fusion module and autoregressive generation module mentioned above are components of a digital human motion generation architecture.

[0039] In this embodiment of the application, the context vector is determined based on multimodal text information, which can be achieved in the following way: parsing multimodal text information to obtain text information corresponding to multiple dimensions; performing vector transformation on the text information corresponding to each dimension to obtain multiple feature vectors corresponding to multiple text information, wherein one text information corresponds to one feature vector; and determining the context vector corresponding to the multimodal information based on the multiple feature vectors.

[0040] In this embodiment, multimodal text information is parsed by a structured semantic parsing unit to obtain text information corresponding to multiple dimensions. The structured semantic parsing unit is implemented by a pre-trained language model based on task fine-tuning. The training text data includes annotations for each word, which are labeled as A (action), E (emotional style), P (identity label), or O (other). The training objective is for the pre-trained language model to classify all lexical units of the input multimodal text information into these four categories, ultimately outputting three independent text segments: action text information. Mood and style text information Identity tag text information In this process, other types of text information obtained from classification are concatenated with action-type text information as auxiliary information.

[0041] In this embodiment, the acquired multimodal text information is input into a pre-trained language model to parse the multimodal text information, and finally outputs text information of multiple types (i.e., dimensions) from the multimodal text information, such as action text information. Mood and style text information Identity tag text information It can also output auxiliary information.

[0042] In this embodiment, the extracted structured text fragments of various types are mapped into feature vectors using a multimodal encoder. The multimodal encoder is an encoder trained for each modality (action, emotion, identity, etc.), and its input is the extracted structured text fragments (action text information). Mood and style text information Identity tag text information (or may output auxiliary information), the multimodal encoder will process the above action text information. Mood and style text information Identity tag text information The vector transformation process is performed on the auxiliary information, and each is mapped to a feature vector.

[0043] In this embodiment, the multimodal encoder used consists of a pre-trained language model and a mapping network, which is an encoder in a variational autoencoder (VAE).

[0044] In this embodiment, a multimodal encoder is used to perform vector transformation on the text information corresponding to each dimension to obtain multiple feature vectors corresponding to multiple text information. Specifically, this can be achieved as follows: perform word segmentation on the text information corresponding to each dimension to obtain multiple word units corresponding to each text information; calculate the embedding vectors corresponding to the multiple word units to obtain the word unit vector sequence corresponding to each text information; and determine multiple feature vectors corresponding to multiple text information based on the word unit vector sequence corresponding to each text information.

[0045] In this embodiment, each dimension of text information corresponds to an encoder, and text information processing is performed using the encoder corresponding to each text information. First, the pre-trained language model in the multimodal encoder is used to process the instruction text (i.e., action text information). Mood and style text information Identity tag text information Word segmentation is performed on text information (including auxiliary information, etc.) to divide it into word units, resulting in multiple word unit information corresponding to each text information. For example, by segmenting action text information into word units, multiple word unit information corresponding to the action text information can be obtained.

[0046] In this embodiment, the embedding vector (or lexical embedding vector) corresponding to each lexical information in the multiple lexical information is calculated, and a sequence of lexical vectors representing each text information is obtained based on the lexical embedding vector corresponding to each lexical information.

[0047] In this embodiment, the pre-trained language model, after the above processing, outputs a sequence of word vectors representing each piece of text information. Then, a mapping network in a multimodal encoder is used for further processing. Specifically, the word vector sequence is used as the input to the mapping network, and finally, a feature vector representing the modality information is output. The mapping network consists of Transformers.

[0048] In this embodiment of the application, the action text information is... Mood and style text information Identity tag text information The inputs to the Transformer mapping network include text, auxiliary information, etc., and the output of the Transformer mapping network is a feature vector corresponding to each text.

[0049] Based on this, in the specific implementation of the above process, the motion command encoder uses... As input, the final output is an action feature vector. The emotion state encoder is slightly different; the emotion state encoder uses... As input, the mapping network output of the emotion state encoder includes two results: one is a classification output of a predefined standard emotion classification. For example, "happy" or "sad," another result is the emotion feature vector. This allows for both coarse-grained emotion classification and features that preserve descriptive details. The identity label encoder uses... As input, the final output is an identity feature vector. If auxiliary information exists, feature vectors corresponding to that auxiliary information are also generated.

[0050] In this embodiment of the application, the feature vectors corresponding to the outputs representing the three modalities, i.e. the three dimensions of text information, are concatenated by the fusion unit in the multimodal context fusion module and fused into a context condition vector C, which is the context vector corresponding to the multimodal text information.

[0051] It should be noted that if auxiliary information exists, the feature vector corresponding to that auxiliary information also needs to be fused.

[0052] It should be noted that the context condition vector C in the embodiments of this application may be a fusion of... , , , The feature vector corresponding to the auxiliary information can be understood as fusing and splicing all the information output by the multimodal context fusion module into a context condition vector C, or it can be determined specifically according to the actual situation.

[0053] S102. Input the context vector into the preset model and output the potential action vector of the first frame and the first weight matrix.

[0054] In this embodiment, the autoregressive motion generation module is used to generate motion vectors for the digital human based on the context condition vector C. The autoregressive motion generation module is implemented using a Transformer network (i.e., a pre-defined model). Unlike previous autoregressive motion generation models, the autoregressive motion generation module in this embodiment does not directly predict the motion vectors, but instead first predicts a latent motion vector (i.e., a potential motion vector). .

[0055] In this embodiment, the Transformer model predicts the potential action vector of the first frame from the context condition vector C. In each subsequent prediction step, the Transformer network processes the entire predicted historical sequence through an attention mechanism, so that each position (or each frame) can be aware of the contextual condition C and the predicted historical potential action vector.

[0056] In this embodiment, the current context condition vector C and a start identifier [START] representing the start of an action are first used as inputs to the Transformer model to predict and output the potential action vector of the first frame. Meanwhile, in calculating the potential motion vector of the first frame... At that time, the first weight matrix based on the context condition vector C is calculated through the attention mechanism in the Transformer model, i.e. , , Weight matrix.

[0057] In this embodiment of the application, after calculation , , Afterwards, , The weight matrix is ​​stored in a cache.

[0058] S103. For the potential action vector of frame t, obtain the potential action vector of frame t-1 and input it into the preset model, output the second weight matrix, and generate the potential action vector of frame t based on the second weight matrix and all historical weight matrices generated before the second weight matrix; t is a positive integer greater than 1, and the historical weight matrix includes at least the first weight matrix.

[0059] In this embodiment of the application, after calculating the potential action vector of the first frame... Subsequently, in the calculation of the potential action vectors for frame 2 and subsequent frames (i.e., frame t), the attention mechanism in the Transformer model can be used based on the output of the previous frame. calculate , , The weight matrix will be newly generated after each calculation. and The weight matrix is ​​appended to the end of the buffer, further based on the output of the previous frame. The calculated weight matrix is ​​compared with all historical weight matrices to calculate the potential action vector for the current frame (i.e., frame t), i.e., using... Attention is calculated on all historical key and value values ​​in the cache to predict the action latent vector for frame t.

[0060] It should be noted that if calculating the latent action vector of the second frame, the historical weight matrix is ​​the same as the first weight matrix; if calculating the latent action vector of the third frame, the historical weight matrix includes both the first weight matrix and the weight matrix obtained when calculating the second frame. , , The weight matrix, and so on.

[0061] For example, the potential action vector in frame 1 is ,based on The first weight matrix is ​​calculated. , , ,Will and After being stored in the cache, the calculation for the second frame is performed based on the first frame. Specifically, the attention mechanism in the Transformer model is used based on the output of the first frame. calculate , , The weight matrix will be the newly generated and The weight matrix is ​​appended to the end of the cache, and then used... and storing history in the cache and Attention is calculated using the weight matrix to predict the latent action vector in the second frame. .

[0062] When calculating the latent action vector in the third frame, the calculation continues based on the second frame. Specifically, the attention mechanism in the Transformer model is used based on the output of the second frame. calculate , , The weight matrix will be the newly generated and The weight matrix is ​​appended to the end of the cache, and then used... and stored in the cache and as well as and Attention is calculated using the weight matrix to predict the latent action vector in frame 3. .

[0063] And so on, the prediction is obtained. The action of responding to user input text ends when the autoregressive model predicts that the next frame is .

[0064] It should be noted that this calculation method enables the Transformer model to generate long-term sequences of human movements.

[0065] It should be noted that during the training of this Transformer model, the actions in the training data will end with an " " marker. Therefore, when the Transformer model in the autoregressive motion generation module predicts that the next frame is " ", it means that the action of the corresponding user-input text has ended.

[0066] S104. Based on the potential motion vector of the first frame and the potential motion vector of the t-th frame, determine the potential motion vector sequence; and based on the potential motion vector sequence, determine the motion vector of each frame so as to control the digital human to perform the corresponding actions according to the motion vector.

[0067] In this embodiment of the application, the generated first frame potential action vector is... and the potential action vector of frame t , , The combination of these elements forms a predicted sequence of potential action vectors (or action latent vector sequence).

[0068] In the embodiments of this application, after obtaining the potential action vector sequence, for each potential action vector in the potential action vector sequence, the mapping of each potential action vector to an action vector is completed based on the diffusion model.

[0069] It should be noted that in traditional autoregressive models, a fully connected MLP layer is typically used to map the latent action vector to the output action vector. This can lead to a lack of diversity in the output and a tendency towards averaged actions over long periods. In this embodiment, a small conditional diffusion model is used as the output module for mapping latent action vectors to action vectors.

[0070] In this embodiment of the application, the motion vector of each frame is determined based on the potential motion vector sequence, which can be implemented in the following way: sampling to obtain noise vector and zero vector; for each potential motion vector in the potential motion vector sequence, a first motion vector is obtained based on the potential motion vector and noise vector, and a second motion vector is obtained based on the noise vector and zero vector; the motion vector of each frame is determined based on the first motion vector, the second motion vector and a preset first hyperparameter.

[0071] In this embodiment of the application, a potential action vector sequence is predicted. For each h in the potential action vector sequence, h is used as a condition in a small conditional diffusion model. Specifically, a pure noise vector is sampled from a standard normal distribution. And set a zero vector. As input to a small diffusion model, h (such as...) in the sequence of potential action vectors... As a condition for the small-scale diffusion model, the small-scale diffusion model will undergo s-step iterative denoising, and finally output the first action vector corresponding to h. .Will As input to the small-scale diffusion model, the zero vector is used as a condition for the model. The small-scale diffusion model undergoes s-step iterative denoising and finally outputs the second action vector corresponding to the zero vector. .

[0072] It should be noted that the small-scale diffusion model can be any type of diffusion model. Specifically, it can be selected according to the actual situation, and no specific limitation is made in the embodiments of this application.

[0073] In this embodiment of the application, based on the first action vector Second action vector Using the preset first hyperparameter, the motion vector corresponding to the potential motion vector of each frame is calculated, which can be obtained by the following formula (1): (1) Where w is the first hyperparameter, which can be set to a default value, and z represents the calculated action vector.

[0074] In this embodiment of the application, after calculating the motion vector of each frame using the above formula (1), multiple consecutive motion vectors that guide the digital human's actions can be obtained, and the obtained motion vectors are used to guide the digital human to perform actions corresponding to the motion vectors.

[0075] It should be noted that if no zero vector is set, the implementation method is to sample a pure noise vector from a standard normal distribution. ,Will As input to a small diffusion model, h (such as...) in the action latent vector sequence... As a condition for the small-scale diffusion model, the small-scale diffusion model will undergo s-step iterative denoising, and the final output will be the same as h (e.g., The corresponding action vector z.

[0076] In this embodiment of the application, after the generation of each frame motion vector corresponding to the multimodal text information is completed, that is, after the digital human completes the response to the multimodal text information input by the user, the digital human is in an idle state. In previous applications, pre-made idle animations are usually played or mixed, which makes the digital human feel mechanical.

[0077] Based on this, this application proposes a method of continuously and automatically generating actions using an autoregressive action generation module, which retains the perception of the current state of the digital human and makes the actions performed by the digital human in idle time more "human" by generating rather than relying on a pre-made animation library.

[0078] In this embodiment, when training the autoregressive action generation model (such as the Transformer model), the training dataset includes some meaningless idle animations, such as a digital human standing, strolling, or sitting for extended periods. The corresponding user input text is set as a zero vector; that is, the action text information in the multimodal text information is set to a zero vector, but emotional and identity text information is also present, such as a host strolling casually or pacing rapidly. The action vector generation method described in the previous embodiment can be used to train the autoregressive action generation model. A well-trained model can generate relevant actions of the digital human during idle periods.

[0079] In this embodiment of the application, during actual inference applications, when the autoregressive action generation model predicts the end marker , the digital human will be in an idle state, and the context vector C needs to be recalculated. Specifically, during the calculation, since the action vector is generated based on the user-input multimodal text information, an emotion feature vector is generated based on the emotion text information within the multimodal text information, and an identity feature vector is generated based on the identity text information, the previous emotion feature vector and identity feature vector can be directly reused here. That is, when recalculating the context vector C here… and If the input remains unchanged, and there is no input of action text information, then the action feature vector can be directly obtained from the VAE in the instruction action encoder. .

[0080] In the embodiments of this application, the obtained , and The vectors are spliced ​​together and merged to form a new context vector C.

[0081] In this embodiment, the attention mechanism is reused to calculate the weight matrix for the new context vector C, replacing the attention weight matrix cached in the autoregressive action generation module. At the same time, only the weight matrix of the most recent 2-second action (approximately 50 frames at a frame rate of 25) is retained in the weight matrix cache. The algorithm described in the previous embodiment is then used to continue predicting the action of the next frame, continuously generating the idle actions of the digital human.

[0082] Specifically, based on the new context vector, a new sequence of potential action vectors is predicted. For example, the new context condition vector C and a start flag [START] representing the start of an action are used as inputs to the Transformer model to predict the output of the first frame's potential action vectors. Meanwhile, in calculating the potential motion vector of the first frame... At that time, the first weight matrix based on the new context condition vector C is calculated through the attention mechanism in the Transformer model, i.e. , , Weight matrix.

[0083] In this embodiment of the application, after calculation , , Afterwards, , The weight matrix is ​​stored in a cache.

[0084] In this embodiment, the calculation for the second frame is performed based on the first frame. Specifically, the attention mechanism in the Transformer model is used based on the output of the first frame. calculate , , The weight matrix will be the newly generated and The weight matrix is ​​appended to the end of the cache, and then used... and storing history in the cache and Attention is calculated using the weight matrix to predict the latent action vector in the second frame. And so on, the prediction is... Finally, a new sequence of potential action vectors is obtained. .

[0085] In this embodiment of the application, after obtaining a new sequence of potential action vectors, a new action vector is determined based on the new sequence of potential action vectors. Specifically, this can be achieved by sampling to obtain a noise vector and a zero vector; for each new potential action vector in the new sequence of potential action vectors, a third action vector is obtained based on the noise vector and the new potential action vector, and a fourth action vector is obtained based on the noise vector and the zero vector; a new action vector is determined based on the third action vector, the fourth action vector, and a preset second hyperparameter, wherein the second hyperparameter is greater than the first hyperparameter.

[0086] In this embodiment of the application, a new potential action vector sequence is predicted. For each of the new potential action vector sequences ,Will It is used as a condition in a small-scale conditional diffusion model. Specifically, it involves sampling a pure noise vector from a standard normal distribution. And set a zero vector. As input to a small-scale diffusion model, the new sequence of potential action vectors... (like As a condition for the small-scale diffusion model, the small-scale diffusion model will undergo s-step iterative denoising, and the final output will be the same as... The corresponding third action vector .Will As input to the small-scale diffusion model, the zero vector is used as a condition for the model. The small-scale diffusion model undergoes s-step iterative denoising and finally outputs the fourth action vector corresponding to the zero vector. .

[0087] In this embodiment, to prevent prolonged idle periods from becoming static, a higher Classifier-Free Guidance (CFG) hyperparameter w is set when using the aforementioned small-scale diffusion module. The CFG hyperparameter plays a role in regulating diversity in the small-scale diffusion model, as shown in formula (1) above. For the generation of conditions (i.e., new potential action vectors) As a condition), This is unconditional generation (i.e., the condition h is a zero vector). When w takes a large value, the small diffusion model tends to reduce the constraints of the conditions and is allowed to generate more diverse and free actions.

[0088] In this embodiment, by adjusting the hyperparameter w, and based on the larger value of the adjusted hyperparameter w, combined with... and Generate new motion vectors for each of the multiple frames of the digital human in an idle state. Thus, through the new action vector Instruct the digital human to perform corresponding actions in the idle state.

[0089] In this embodiment, when the digital human system receives the next user input multimodal text information, it can repeat the process of generating the action vector z in the aforementioned embodiment, and generate three-dimensional human motion based on the action-related instructions in the user input text information.

[0090] It should be noted that if emotional text information or identity text information (identity tags) is missing in the structured semantic analysis, random sampling is not required. The previous emotional feature vector and identity feature vector can be used instead of randomly sampling to obtain the corresponding state as in the previous embodiment.

[0091] It is understood that the digital human motion generation method provided in this application uses multimodal text information as data to determine the input preset model context vector when generating digital human motion. The context vector is determined based on text information from multiple modalities. The digital human motion generated by the context vector that integrates text information from multiple modalities has diversity. Furthermore, when generating digital human motion based on the context vector, a second weight matrix is ​​generated by referring to the motion latent vector of the previous frame, and the motion latent vector of the current frame is generated by combining the historical weight matrix. This process continues, and without setting a duration, the digital human motion latent vector of the current frame can be continuously output based on the historical motion latent vector to obtain the final motion vector of each frame. In this way, without limiting the motion generation duration, motion output can be continuously performed to meet the needs of the digital human in the actual environment.

[0092] Based on the above embodiments, this application also provides a schematic diagram of the overall process of a digital human motion generation method, such as... Figure 2 As shown, the specific steps include: S1. Obtain multimodal text information; and parse the multimodal text information to obtain text information corresponding to multiple dimensions respectively.

[0093] S2. Perform word segmentation on the text information corresponding to each dimension to obtain multiple word elements for each text information.

[0094] S3. Calculate the embedding vectors corresponding to multiple word information to obtain the word vector sequence corresponding to each text information.

[0095] S4. Based on the word vector sequence corresponding to each text information, determine multiple feature vectors corresponding to multiple text information, wherein one text information corresponds to one feature vector.

[0096] S5. Based on multiple feature vectors, determine the context vector corresponding to the multimodal information.

[0097] S6. Input the context vector into the preset model and output the potential action vector and the first weight matrix of the first frame.

[0098] S7. For the potential action vector of frame t, obtain the potential action vector of frame t-1 and input it into the preset model, output the second weight matrix, and generate the potential action vector of frame t based on the second weight matrix and all historical weight matrices generated before the second weight matrix; t is a positive integer greater than 1, and the historical weight matrix includes at least the first weight matrix.

[0099] S8. Based on the potential action vectors of the first frame and the potential action vectors of the t-th frame, determine the potential action vector sequence.

[0100] S9. Sample to obtain the noise vector and zero vector.

[0101] S10. For each potential action vector in the potential action vector sequence, obtain the first action vector based on the potential action vector and the noise vector, and obtain the second action vector based on the noise vector and the zero vector.

[0102] S11. Determine the motion vector for each frame based on the first motion vector, the second motion vector, and the preset first hyperparameter.

[0103] After executing S11 above, the following action flow for generating the digital human's idle state can continue to be executed, such as S12-S14: S12. Obtain the emotional feature vector corresponding to the emotional text information and the identity feature vector corresponding to the identity text information, and randomly sample to obtain the action feature vector.

[0104] S13. Determine a new context vector based on the emotion feature vector, identity feature vector, and action feature vector.

[0105] S14. Based on the new context vector, predict a new sequence of potential action vectors; and based on the new sequence of potential action vectors, determine a new action vector to control the digital human to perform corresponding actions according to the new action vector.

[0106] It should be noted that the process of predicting a new sequence of potential action vectors based on the new context vectors, and determining new action vectors based on the new sequence of potential action vectors, is the same as the process of generating the aforementioned action vectors.

[0107] This application proposes a method for generating long-duration 3D human motion based on multimodal fusion and text-driven processing. The framework of this method consists of a multimodal context fusion module and an autoregressive diffusion motion generation module, each of which is composed of multiple units. This method enables the generation of digital human motion driven by multimodal text input. The method proposed in this application utilizes an autoregressive network and a designed sampling strategy to achieve 3D human motion generation with no time limit. By combining multimodal data such as text information, emotion information, and digital human set tags, diverse free movements of the digital human are generated. When the user's input is temporarily idle, the digital human can generate standby actions from zero vector input through the above multimodal information and historical information, rather than the preset idle animation in traditional technical solutions, making the digital human more human-like.

[0108] Based on the above embodiments, another embodiment of this application provides a digital human motion generation device 1, such as... Figure 3 As shown, the digital human motion generation device 1 includes: Acquisition unit 10 is used to acquire multimodal text information.

[0109] The determination unit 11 is used to determine the context vector based on multimodal text information.

[0110] Output unit 12 is used to input the context vector into the preset model and output the first frame potential action vector and the first weight matrix.

[0111] The output unit 12 is also used to obtain the potential action vector of the (t-1)th frame into the preset model for the potential action vector of the tth frame, output the second weight matrix, and generate the potential action vector of the tth frame based on the second weight matrix and all historical weight matrices generated before the second weight matrix, where t is a positive integer greater than 1, and the historical weight matrix includes at least the first weight matrix.

[0112] The determining unit 11 is also used to determine a sequence of potential action vectors based on the potential action vectors of the first frame and the potential action vectors of the t-th frame; and to determine the action vectors of each frame based on the sequence of potential action vectors, so as to control the digital human to perform corresponding actions according to the action vectors.

[0113] In one embodiment, the acquisition unit 10 is also used to sample and acquire noise vectors and zero vectors.

[0114] The determining unit 11 is also used to obtain a first action vector based on the potential action vector and the noise vector for each potential action vector in the potential action vector sequence, and to obtain a second action vector based on the noise vector and the zero vector.

[0115] The determining unit 11 is also used to determine the motion vector of each frame based on the first motion vector, the second motion vector and the preset first hyperparameter.

[0116] In one embodiment, the multimodal text information includes at least emotional text information and identity text information.

[0117] The acquisition unit 10 is also used to acquire the emotional feature vector corresponding to the emotional text information and the identity feature vector corresponding to the identity text information, and to randomly sample to obtain the action feature vector.

[0118] The determining unit 11 is also used to determine a new context vector based on the emotion feature vector, identity feature vector, and action feature vector.

[0119] The determining unit 11 is also used to predict a new sequence of potential action vectors based on the new context vector; and to determine a new action vector based on the new sequence of potential action vectors, so as to control the digital human to perform corresponding actions according to the new action vector.

[0120] In one embodiment, the acquisition unit 10 is also used to sample and acquire noise vectors and zero vectors.

[0121] The determining unit 11 is also used to obtain a third action vector based on the noise vector and the new potential action vector for each new potential action vector in the new potential action vector sequence, and to obtain a fourth action vector based on the noise vector and the zero vector.

[0122] The determining unit 11 is also used to determine a new action vector based on the third action vector, the fourth action vector and the preset second hyperparameter, wherein the second hyperparameter is greater than the first hyperparameter.

[0123] In one embodiment, the digital human motion generation device 1 may further include: a parsing unit and a conversion unit.

[0124] The parsing unit is used to parse multimodal text information to obtain text information corresponding to multiple dimensions.

[0125] The transformation unit is used to perform vector transformation on the text information corresponding to each dimension to obtain multiple feature vectors corresponding to multiple text information, wherein one text information corresponds to one feature vector.

[0126] The determining unit 11 is also used to determine the context vector corresponding to the multimodal information based on multiple feature vectors.

[0127] In one embodiment, the digital human motion generation device 1 may further include: a processing unit and a computing unit.

[0128] The processing unit is used to perform word segmentation on the text information corresponding to each dimension, and obtain multiple word element information corresponding to each text information.

[0129] The computation unit is used to calculate the embedding vectors corresponding to multiple lexical information, thereby obtaining the lexical vector sequence corresponding to each text information.

[0130] The determining unit 11 is also used to determine multiple feature vectors corresponding to multiple text information based on the word vector sequence corresponding to each text information.

[0131] In one embodiment, the acquisition unit 10 is further configured to acquire the text information corresponding to the missing one or more dimensions by random sampling when one or more dimensions of text information are missing in the multimodal text information; or, based on the text information of at least one dimension that already exists in the multimodal text information, sample and acquire the text information corresponding to the missing one or more dimensions in the multimodal text information.

[0132] This application provides a digital human motion generation device that acquires multimodal text information and determines a context vector based on the multimodal text information; inputs the context vector into a preset model and outputs a first frame potential motion vector and a first weight matrix; for the t-th frame potential motion vector, acquires the (t-1)-th frame potential motion vector and inputs it into the preset model, outputting a second weight matrix, and generates the t-th frame potential motion vector based on the second weight matrix and all historical weight matrices generated before the second weight matrix, where t is a positive integer greater than 1, and the historical weight matrix includes at least the first weight matrix; determines a potential motion vector sequence based on the first frame potential motion vector and the t-th frame potential motion vector; and determines the motion vector for each frame based on the potential motion vector sequence to control the digital human to perform corresponding actions according to the motion vectors. Therefore, the device proposed in this application... A digital human motion generation device uses multimodal text information as data to determine the input preset model context vector during digital human motion generation. The context vector is determined based on text information from multiple modalities, and the digital human motion generated by the context vector, which integrates text information from multiple modalities, is diverse. Furthermore, when generating digital human motion based on the context vector, a second weight matrix is ​​generated by referring to the motion latent vector of the previous frame, and the motion latent vector of the current frame is generated by combining the historical weight matrix. This process continues, and without setting a duration, the device can continuously output the digital human motion latent vector of the current frame based on the historical motion latent vector to obtain the final motion vector for each frame. In this way, without limiting the motion generation duration, motion output can be continuously performed to meet the needs of the digital human in a real-world environment.

[0133] Figure 4 This is a schematic diagram of the composition structure of a digital human motion generation device 1 provided in an embodiment of this application. In practical applications, based on the same disclosed concept of the above embodiments, such as... Figure 4As shown, the digital human motion generation device 1 of this application embodiment includes a processor 13, a memory 14 and a communication bus 15.

[0134] In specific embodiments, the aforementioned acquisition unit 10, determination unit 11, output unit 12, parsing unit, conversion unit, processing unit, and calculation unit can be implemented by a processor 13 located on the digital human motion generation device 1. The processor 13 can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above processor functions can also be other types; this application does not impose specific limitations.

[0135] In this embodiment, the communication bus 15 is used to establish a connection between the processor 13 and the memory 14; when the processor 13 executes the running program stored in the memory 14, it implements the following digital human motion generation method: Acquire multimodal text information and determine context vectors based on the multimodal text information; input the context vectors into a preset model and output the first frame potential action vector and the first weight matrix; for the t-th frame potential action vector, acquire the (t-1)-th frame potential action vector and input it into the preset model, output the second weight matrix, and generate the t-th frame potential action vector based on the second weight matrix and all historical weight matrices generated before the second weight matrix, where t is a positive integer greater than 1, and the historical weight matrix includes at least the first weight matrix; determine the potential action vector sequence based on the first frame potential action vector and the t-th frame potential action vector; and determine the action vector for each frame based on the potential action vector sequence to control the digital human to perform corresponding actions according to the action vectors.

[0136] In one embodiment, the processor 13 is further configured to sample and acquire noise vectors and zero vectors; for each potential action vector in the potential action vector sequence, obtain a first action vector based on the potential action vector and noise vector, and obtain a second action vector based on the noise vector and zero vector; and determine the action vector of each frame based on the first action vector, the second action vector, and a preset first hyperparameter.

[0137] In one embodiment, the multimodal text information includes at least emotional text information and identity text information.

[0138] The processor 13 described above is also used to obtain the emotional feature vector corresponding to the emotional text information and the identity feature vector corresponding to the identity text information, and randomly sample to obtain the action feature vector; determine a new context vector based on the emotional feature vector, the identity feature vector and the action feature vector; predict a new potential action vector sequence based on the new context vector; and determine a new action vector based on the new potential action vector sequence, so as to control the digital human to perform corresponding actions according to the new action vector.

[0139] In one embodiment, the processor 13 is further configured to sample and acquire noise vectors and zero vectors; for each new potential action vector in the new potential action vector sequence, obtain a third action vector based on the noise vector and the new potential action vector, and obtain a fourth action vector based on the noise vector and the zero vector; and determine a new action vector based on the third action vector, the fourth action vector, and a preset second hyperparameter, wherein the second hyperparameter is greater than the first hyperparameter.

[0140] In one embodiment, the processor 13 is further configured to parse multimodal text information to obtain text information corresponding to multiple dimensions; perform vector transformation on the text information corresponding to each dimension to obtain multiple feature vectors corresponding to multiple text information, wherein one text information corresponds to one feature vector; and determine the context vector corresponding to the multimodal information based on the multiple feature vectors.

[0141] In one embodiment, the processor 13 is further configured to perform word segmentation on the text information corresponding to each dimension to obtain multiple word elements corresponding to each text information; calculate the embedding vectors corresponding to the multiple word elements to obtain a word element vector sequence corresponding to each text information; and determine multiple feature vectors corresponding to the multiple text information based on the word element vector sequence corresponding to each text information.

[0142] In one embodiment, the processor 13 is further configured to, when there is a lack of text information in one or more dimensions of the multimodal text information, obtain the text information corresponding to the missing one or more dimensions by random sampling; or, based on the text information of at least one dimension that already exists in the multimodal text information, sample and obtain the text information corresponding to the missing one or more dimensions of the multimodal text information.

[0143] Based on the above embodiments, this application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors and applied in a digital human motion generation device. The computer program implements the digital human motion generation method as described above.

[0144] Based on the above embodiments, this application provides a computer program product, including a computer program that can be executed by one or more processors and applied in a digital human motion generation device. The computer program implements the digital human motion generation method as described above.

[0145] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the embodiments of this application.

[0147] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for generating digital human motion, characterized in that, The method includes: Acquire multimodal text information and determine the context vector based on the multimodal text information; Input the context vector into the preset model, and output the first frame potential action vector and the first weight matrix; For the potential action vector of frame t, the potential action vector of frame (t-1) is obtained and input into the preset model, and a second weight matrix is ​​output. Based on the second weight matrix and all historical weight matrices generated before the second weight matrix, the potential action vector of frame t is generated; t is a positive integer greater than 1, and the historical weight matrix includes at least the first weight matrix. Based on the potential motion vector of the first frame and the potential motion vector of the t-th frame, a potential motion vector sequence is determined; and based on the potential motion vector sequence, a motion vector for each frame is determined to control the digital human to perform corresponding actions according to the motion vector.

2. The method according to claim 1, characterized in that, The process of determining the motion vector for each frame based on the potential motion vector sequence includes: Sampling is used to obtain noise vectors and zero vectors; For each potential action vector in the potential action vector sequence, a first action vector is obtained based on the potential action vector and the noise vector, and a second action vector is obtained based on the noise vector and the zero vector; The motion vector for each frame is determined based on the first motion vector, the second motion vector, and the preset first hyperparameter.

3. The method according to claim 1, characterized in that, The multimodal text information includes at least emotional text information and identity text information. After determining the action vector for each frame based on the latent action vector sequence, the method further includes: Obtain the emotional feature vector corresponding to the emotional text information and the identity feature vector corresponding to the identity text information, and randomly sample to obtain the action feature vector; Based on the emotion feature vector, the identity feature vector, and the action feature vector, a new context vector is determined; Based on the new context vector, a new sequence of potential action vectors is predicted; and based on the new sequence of potential action vectors, a new action vector is determined to control the digital human to perform corresponding actions according to the new action vector.

4. The method according to claim 3, characterized in that, The process of determining a new action vector based on the new potential action vector sequence includes: Sampling is used to obtain noise vectors and zero vectors; For each new potential action vector in the new potential action vector sequence, a third action vector is obtained based on the noise vector and the new potential action vector, and a fourth action vector is obtained based on the noise vector and the zero vector; Based on the third action vector, the fourth action vector, and the preset second hyperparameter, the new action vector is determined, wherein the second hyperparameter is greater than the first hyperparameter.

5. The method according to claim 1, characterized in that, Determining the context vector based on the multimodal text information includes: The multimodal text information is parsed to obtain text information corresponding to multiple dimensions. Vector transformation is performed on the text information corresponding to each dimension to obtain multiple feature vectors corresponding to multiple text information, where one text information corresponds to one feature vector; Based on the multiple feature vectors, the context vector corresponding to the multimodal information is determined.

6. The method according to claim 5, characterized in that, The process of performing vector transformation on the text information corresponding to each dimension to obtain multiple feature vectors corresponding to multiple text information includes: The text information corresponding to each dimension is segmented to obtain multiple word elements for each text information. Calculate the embedding vectors corresponding to the multiple word information to obtain the word vector sequence corresponding to each text information; Based on the word vector sequence corresponding to each text information, multiple feature vectors corresponding to the multiple text information are determined.

7. The method according to claim 1, characterized in that, After acquiring the multimodal text information, the method further includes: In the case where one or more dimensions of text information are missing in the multimodal text information, the text information corresponding to the missing one or more dimensions is obtained by random sampling. Alternatively, based on the text information of at least one dimension that already exists in the multimodal text information, sample and obtain the text information corresponding to one or more missing dimensions in the multimodal text information.

8. A digital human motion generation device, characterized in that, The action generation device includes a processor and a memory; when the processor executes the running program stored in the memory, it implements the method as described in any one of claims 1 to 7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images