Action generation method, electronic device, storage medium, and computer program product

Abstract

The application discloses an action generation method, an electronic device, a storage medium and a computer program product, wherein the method comprises: receiving first description information, the first description information being used for describing a first action sequence of a digital person; generating a first action of the digital person based on the first description information through a first model; searching for an action matched with the first action and / or an action matched with the first description information in a first action library to obtain a second action of the digital person; and obtaining a second action sequence of the digital person based on the first action and the second action. Through the application, the diversity of the generated digital person action can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an action generation method, electronic device, storage medium, and computer program product. Background Technology

[0002] With breakthroughs in artificial intelligence and real-time rendering technology in recent years, digital human movements are gradually shifting from passively driven and playing pre-made animations according to rules to intelligent generation. Emerging fields such as metaverse and embodied intelligence are placing increasingly higher demands on the quality and intelligence of digital human movements. When users interact with digital humans or digital humans interact with scenes, they are required to present accurate, reasonable, more diverse, and human-like movements. To generate digital human movements, text can be input into relevant models to produce digital human movements; however, this approach is too simplistic, resulting in insufficient diversity of generated digital human movements. Summary of the Invention

[0003] To address the related technical issues, embodiments of this application provide an action generation method, an electronic device, a storage medium, and a computer program product.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an action generation method, the method comprising: Receive first description information, which is used to describe a first action sequence of the digital human; Based on the first description information, the first action of the digital human is generated through the first model; The second action of the digital human is obtained by retrieving the action that matches the first action and / or the action that matches the first description information from the first action library. The second action sequence of the digital human is obtained based on the first action and the second action.

[0005] In the above scheme, the first action sequence includes multiple first frames, each of the multiple first frames including parameters related to the pose of the digital human, and after receiving the first description information, the method further includes: The first description information is converted into one or more second description information, and each of the one or more second description information is used to describe one of the multiple first frames; The step of generating the first action of the digital human based on the first description information and through the first model includes: The first model is used to process the first description information and one or more second description information to generate the first action.

[0006] In the above scheme, the first model is a diffusion model, which includes a first attention layer, a second attention layer, a third attention layer, and an output layer. The processing of the first descriptive information and one or more pieces of second descriptive information using the first model includes: The first action sequence is subjected to self-attention operation through the first attention layer to obtain the first operation result; The second attention layer performs cross-attention operation on the encoding result of the first description information and the first operation result to obtain the second operation result; The third attention layer performs a cross-attention operation on the encoding results of one or more second descriptive information items and the second operation result to obtain the third operation result. The first action is obtained by converting the third operation result through the output layer.

[0007] In the above scheme, the first calculation result is positively correlated with the first similarity matrix, and the first similarity matrix represents the degree of correlation between different frames in the first action sequence; and / or, The second operation result is positively correlated with the second similarity matrix, which represents the correlation between each word in the first descriptive information and each frame in the first action sequence; and / or, The third operation result is positively correlated with the third similarity matrix, which represents the degree of association between each word in the second description information and different first parts, where the first part is the part of the digital human in the first action sequence.

[0008] In the above scheme, the step of retrieving actions matching the first action and / or actions matching the first description information from the first action library to obtain the second action of the digital human includes: The second model retrieves the first action and the first description information from the first database using the second model to obtain a first action set matching the first action and a second action set matching the first description information. The second model is used to encode the first action and the first description information into the same latent vector space. The second action is obtained based on the first action set and the second action set.

[0009] In the above scheme, obtaining the second action based on the first action set and the second action set includes: Determine whether there is an intersection between the first action set and the second action set; If there is an overlap in actions, the target action in the overlap is taken as the second action, and the target action is characterized as an action whose similarity to the first description information and the first action satisfies the corresponding condition; if there is no overlap in actions, one or more of the following are taken as the second action: the action in the first action set that has the highest matching degree with the first action; the action in the second action set that has the highest matching degree with the first description information.

[0010] In the above scheme, after obtaining the second action sequence of the digital human based on the first action and the second action, the method further includes: Send the second action sequence; Upon receiving a first instruction, the first weight corresponding to the first word in the second attention layer is adjusted according to the first instruction, wherein the first instruction is used to indicate that the second action sequence does not match the first word in the first description information; The action sequence of the digital human is regenerated based on the adjusted first weight.

[0011] This application also provides an action generation device, including: The first receiving unit is configured to receive first description information, which describes a first action sequence of the digital human. The first generation unit is configured to generate the first action of the digital human based on the first description information and through the first model; The first retrieval unit is used to retrieve actions that match the first action and / or actions that match the first description information from the first action library to obtain the second action of the digital human. The first processing unit obtains a second action sequence of the digital human based on the first action and the second action.

[0012] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor. The processor is used to execute the steps of any of the above-mentioned technical solutions when running the computer program.

[0013] This application also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0015] In this embodiment, first description information is received, which describes a first action sequence of a digital human. Based on the first description information, a first action of the digital human is generated through a first model. Actions matching the first action and / or actions matching the first description information are retrieved from a first action library to obtain a second action of the digital human. A second action sequence of the digital human is obtained based on the first action and the second action. It can be seen that this embodiment combines two methods for obtaining digital human actions: one is model generation, and the other is retrieval. Compared to a single technical path, this embodiment integrates both retrieval and model generation, increasing the diversity of technical paths for digital human action production, thereby increasing the diversity of generated digital human actions. Furthermore, in the retrieval stage, in addition to retrieval from description information to action, this embodiment also provides action-to-action retrieval, increasing the diversity of retrieval methods, thereby increasing the diversity of the final generated digital human actions. Attached Figure Description

[0016] Figure 1 A flowchart illustrating an action generation method provided in an embodiment of this application; Figure 2 A flowchart illustrating an application embodiment provided in this application; Figure 3 A flowchart illustrating another application embodiment provided in this application; Figure 4 This is a schematic diagram of the structure of an action generation device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0018] It should be understood that the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, the term "one or more" in this document is an exemplary expression and can be replaced with any possible expression, such as one or more, at least one, or at least one item, etc.

[0019] It should also be understood that the term "instruction" mentioned in the embodiments of this application can be a direct instruction, an indirect instruction, or an indication of a relationship. For example, A instructing B can mean that A directly instructs B, such as B being able to obtain information through A; it can also mean that A indirectly instructs B, such as A instructing C, so B can obtain information through C; or it can mean that there is a relationship between A and B.

[0020] It should also be understood that the term "correspondence" mentioned in the embodiments of this application may indicate a direct or indirect correspondence between the two, or an association between the two, or a relationship of instruction and being instructed, configuration and being configured, etc.

[0021] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0022] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0023] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and they all fall within the protection scope of the embodiments of this application.

[0024] 3D digital human motion generation technology is a product of the deep integration of artificial intelligence, computer graphics, and multimodal interaction technology. Previously, it primarily relied on motion capture, manual creation, and animation state machines. With recent breakthroughs in artificial intelligence and real-time rendering technology, digital human motion is gradually shifting from passively driven, rule-based playback of pre-made animations to intelligent generation, enabling direct driving of 3D human motion sequences through natural interaction methods such as voice and text. Emerging fields such as metaverse and embodied intelligence are placing increasingly higher demands on the quality and intelligence of digital human motion. When users interact with digital humans or with scenes, they are required to present accurate, reasonable, and more diverse, anthropomorphic movements, rather than monotonous pre-made animations. This application's embodiments can use text to drive 3D human motion. Users input text describing the digital human's actions, and this text drives the digital human to perform corresponding motion sequences.

[0025] The relevant technologies are mainly divided into two categories: one is based on generative or autoregressive models to achieve end-to-end generation of human actions from text, and the other is to use retrieval technologies and rules to generate actions based on existing action libraries.

[0026] The core idea of ​​generative or autoregressive model-based methods is to first encode and decode human movements using some method, such as a variational autoencoder. Then, an autoregressive model or a first-order model is used with text as input to generate human movement codes, which are then decoded to obtain the human movements. The following section uses techniques one, two, and three as examples to introduce generative or autoregressive model-based methods: The core of Technique 1 is to break down the human body into multiple parts, such as the head, hands, and legs, assign different weights, and build independent action coding libraries. It also divides the text into corresponding descriptive paragraphs, strengthens the association between text and local actions during the training of the neural network, uses vector quantization autoencoders to encode actions, and uses graph convolutional networks to simulate the topological relationships between different parts of the human body. Finally, it uses an attention mechanism and an autoregressive model to learn the mapping between two modalities, predicts the corresponding action code from the input text vector, and thus generates the action.

[0027] The fine-grained paraphrasing technique used in Technique 2 is similar to that in Technique 1. The difference between Technique 1 and Technique 2 is that Technique 2 uses a first model to generate subsequent text features into three-dimensional human motion.

[0028] The core of technology three is to use the first model as the generation model to improve the diversity of generated human movements, and to encode the human body as a whole.

[0029] The technical approach based on action retrieval uses various retrieval techniques to explore the relationship between text and human actions. It generates a variety of human actions through action fusion and splicing, physical calculation, and other methods. The following section introduces the technical approach based on action retrieval, using techniques four and five as examples.

[0030] Technique four uses rule matching for action retrieval. This method first establishes a text-action database, with corresponding text annotations for each action. When a user inputs text, the database is matched with the user's posture and scene using text keywords. The rationality of the user's actions is judged and adjusted according to predefined rules, and human actions that conform to both text and physical constraints are calculated based on physical constraints.

[0031] Technique five uses deep learning for cross-modal retrieval of text to action. It decomposes text and action sequences into features at multiple granularities, such as word level and instance level, and models cross-modal alignment through graph convolutional networks to achieve accurate alignment between text and motion sequences, thus solving the problem of retrieving complex actions.

[0032] The two existing technical solutions described above either generate new human motions or retrieve existing motions from a motion library. The solution using only model generation may produce motions that do not conform to human dynamics or motion logic, while the solution using only retrieval generates and fine-tunes motions based solely on an existing motion library, resulting in a lack of diversity. Therefore, the above technical solutions fail to meet the demands of various applications for high-quality, diverse, and intelligent human animation.

[0033] To address at least one of the aforementioned technical problems, this application proposes an action generation method. In this method, first descriptive information is received, which describes a first action sequence of a digital human. Based on the first descriptive information, a first action of the digital human is generated using a first model. Actions matching the first action and / or actions matching the first descriptive information are retrieved from a first action library to obtain a second action of the digital human. A second action sequence of the digital human is obtained based on the first action and the second action. It is evident that this application combines two methods for obtaining digital human actions: one through model generation and the other through retrieval. Compared to a single technical path, this application integrates both retrieval and model generation, increasing the diversity of technical paths for digital human action production, thereby increasing the diversity of generated digital human actions. Furthermore, in the retrieval stage, in addition to retrieval from descriptive information to action, this application also provides action-to-action retrieval, increasing the diversity of retrieval methods and thus increasing the diversity of the final generated digital human actions.

[0034] This application provides an action generation method. Please refer to [link / reference]. Figure 1 The method includes: Step 101: Receive first description information, which is used to describe the first action sequence of the digital human; It should be noted that this application does not limit the type of digital human. A digital human can be any possible type, such as a perceptual digital human, a virtual anchor, a virtual idol, an animated character, a game character, an intelligent character assistant, a virtual teacher, a virtual musical instrument player, an augmented reality (AR) digital human, a virtual reality (VR) digital human, a digital human in the metaverse, a digital human generated by a digital twin, etc.

[0035] It should be noted that the embodiments of this application are not limited to the field of digital human motion generation. For example, the embodiments of this application can also be applied to generating the motion of digital animals, generating the motion of virtual characters, and generating the motion of actual people or animals. Taking the application of this application to generate the motion of digital animals as an example, in this scenario, the first descriptive information is used to describe the motion sequence of the digital animal.

[0036] It should be noted that in the embodiments of this application, the action sequence (such as the first action sequence, the second action sequence, etc.) is a set of frames arranged in chronological order. Each frame in this set of frames describes the body posture of the digital human at a certain moment. Each frame in this set of frames can be represented as the posture information of the digital human at a time sampling point. For example, the frames in the action sequence can be represented as the positions of various parts of the digital human, or the frames in the action sequence can be represented as the rotation angle of the digital human's limbs, etc.

[0037] It should be noted that the first descriptive information describes the action sequence as the object, that is, it describes the action sequence at the granularity. Therefore, the first descriptive information can also be called sequence description text, sequence description information, or sequence-level description information, etc.

[0038] For example, in a typical application example of the embodiments of this application, the first description information is in the form of text. However, this application does not limit the modality of the first description information, and the first description information can also be in any possible form, such as audio or images.

[0039] For example, the first descriptive information can be information input by the user, such as text input by the user.

[0040] For example, a first instruction is sent to a first device, the first instruction being used to prompt the user to input descriptive information of an action sequence, the receiving of the first descriptive information including: receiving the first descriptive information input by the user.

[0041] Step 102: Based on the first description information, generate the first action of the digital human using the first model; It is understandable that "generating the first action of the digital human through the first model" does not mean that the image frame of the digital human has been generated. "Generating the first action of the digital human through the first model" can be understood as generating the action information of the digital human through the first model.

[0042] For example, the first action can be represented as an action sequence or an action vector, etc.

[0043] For example, in the embodiments of this application, the first model is a neural network model.

[0044] To improve the accuracy of the generated first action, embodiments of this application convert the first description information into frame-level description information with finer granularity. For specific examples, please refer to the following practical applications: In practical applications, the first action sequence includes multiple first frames, each of which includes parameters related to the pose of the digital human. After receiving the first description information, the method further includes: The first description information is converted into one or more second description information, and each of the one or more second description information is used to describe one of the multiple first frames; The step of generating the first action of the digital human based on the first description information and through the first model includes: The first model is used to process the first description information and one or more second description information to generate the first action.

[0045] It is understood that the embodiments of this application convert the description information based on the action sequence (i.e., the first description information) into fine-grained description information for each frame (i.e., the second description information), and then simultaneously input the sequence description text and the frame description text as conditions into the diffusion model, and use the diffusion model to generate actions. By combining the fine-grained description information, the accuracy and precision of the final generated digital human's actions can be improved.

[0046] It should be noted that the second descriptive information is frame-level information, so it can also be called frame description text, frame-level description text, or frame description information, etc. The first descriptive information is sequence-level description information. The granularity of the two is different. For example, if the first action sequence as a whole indicates the action of squatting, then the first descriptive information can be squatting. The first frame in the first action sequence represents the angle of knee bending during squatting. For example, if multiple first frames contain information of 30° and 60° respectively, then the second descriptive information corresponding to the multiple first frames are: knee bending 30° and knee bending 60° respectively.

[0047] For example, the first frame can be a key frame in the first action sequence. For instance, when extracting key frames from the first action sequence, key frames can be calculated based on changes in motion energy. For example, the difference in the amount of rotation change of each joint between adjacent frames can be calculated. When the difference in the amount of rotation change of the joint exceeds the corresponding threshold, the adjacent frame is regarded as a key frame.

[0048] For example, converting the first description information into one or more second description information includes: expanding the first description information into one or more second description information through a large language model.

[0049] The following describes how the above-mentioned large language model is trained in the embodiments of this application: For example, the large language model is trained using sequence description text and frame description text. Sequence description text is represented as text describing action sequences, belonging to sequence-level action description text. Frame description text is represented as text describing frames in an action sequence, belonging to frame-level action description text. Sequence description text has a coarser granularity, while frame description text has a finer granularity.

[0050] To train a large language model, this application embodiment requires constructing training datasets with both coarse and fine granularity. Specifically, sequence description text can be obtained through manual annotation, but frame description text is difficult to annotate frame by frame manually. Therefore, this application embodiment requires designing rules to convert the actions of various parts of the human body into text description units. Examples are provided for this purpose: For example, the open-source PoseScript rule library can be used to match and obtain frame description text. First, keyframes are extracted from the action sequence used for training. Keyframes are calculated based on changes in motion energy, specifically the difference in rotational changes of each joint between adjacent frames; frames exceeding a certain threshold are considered keyframes. For each keyframe's human pose, PoseScript defines how to convert 3D pose parameters (such as angular or positional relationships between joints) into a descriptive language. For instance, a knee angle <90 degrees can be converted to "squat," an elbow angle >150 degrees to "arms extended," and a hand distance greater than shoulder width to "arms outstretched," etc. Therefore, it is necessary to calculate the relative positions between human joints and the rotation angle of each bone point to obtain 3D pose parameters. These 3D pose parameters can be obtained directly from the action sequence or calculated based on it. The PoseScript rule base matching rules are filled in with the preset phrase "{body part 1} {state)}, {body part 2} {state}...", where the order of body parts description is consistent with the order of human body data key points, and the state in the preset phrase can be a description language obtained by matching 3D pose parameters in the PoseScript rule base. Based on the state in the preset phrase, the frame description text can be obtained.

[0051] The above method can construct text data at two granularities: paired sequences and frames. These paired sequence descriptions and frame descriptions can be used to fine-tune a large language model. During fine-tuning, low-rank adaptation (Lora) can be used, which involves freezing the pre-trained model parameters and injecting trainable parameters into the large language model through low-rank matrix factorization. After fine-tuning, the large language model can, referring to the given coarse-grained and fine-grained text examples, convert the user-input sequence description text into frame-level description text that describes the keyframes of the action sequence.

[0052] The above describes how to train a large language model and how to convert the first descriptive information into second descriptive information using the large language model. After obtaining these two levels of information, the first and second descriptive information, they can be input into the first model to generate the first action. The following describes the training process of the first model in this application embodiment and how to generate the first action using the first model: For example, the first model can be configured in the action generation module, which is trained using a dataset. In this embodiment, a diffusion model can be used as the first model. The goal of the diffusion model's training phase is to learn the conditional generation process from noise to motion sequences. In this example, the condition refers to action description text, including sequence description text and frame description text. This embodiment uses the CLIP pre-trained language model to encode the sequence description text and frame description text. Both the sequence description text and frame description text use token embedding encoding; that is, the sequence description text and frame description text are first segmented into words, and then an embedding vector is calculated for each token. Let the encoding of the sequence-level description text be denoted as... , These are the codes corresponding to each word element, as shown in the actual calculations below. Rewritten as a The matrix is ​​denoted by , where t is the number of terms in the sequence description text, and the encoding of the frame-level description text is denoted by . ,in, It is the word embedding sequence of each frame. , The number of terms in the frame description text. Assuming frames a and b in the action sequence are identified as keyframes, then... [a:b-1] are all used (i.e., the word embedding sequence of frame a) padding, in the actual calculations below Rewritten as a The matrix is ​​denoted by m, where m is the number of frames.

[0053] For example, the action generation module uses a U-Net network structure. The action generation module includes upsampling and downsampling parts. To learn the temporal relationships between frames in the action sequence, the precise correspondence between text and frames, and the correspondence between the human pose described in the text and the pose description in the action sequence, the action generation module designs the network layers in the sampling blocks based on an attention model (i.e., the first model). The three core network layers of the first model are the first attention layer, the second attention layer, and the third attention layer. The attention layers of the first model are described below: In practical applications, the first model is a diffusion model, which includes a first attention layer, a second attention layer, a third attention layer, and an output layer. The processing of the first descriptive information and one or more of the second descriptive information using the first model includes: The first action sequence is subjected to self-attention operation through the first attention layer to obtain the first operation result; The second attention layer performs cross-attention operation on the encoding result of the first description information and the first operation result to obtain the second operation result; The third attention layer performs a cross-attention operation on the encoding results of one or more second descriptive information items and the second operation result to obtain the third operation result. The first action is obtained by converting the third operation result through the output layer.

[0054] For example, the first attention layer is also called the self-attention layer, which is used to model the temporal correlation between frames in a human motion sequence (such as the first action sequence of a digital human).

[0055] For example, the second attention layer is also called the sequence-level cross-attention layer, while the first attention layer is used to establish the correspondence between text words (such as lexical units in the first descriptive information) and motion frames (such as frames in the first action sequence).

[0056] For example, the third attention layer is also called the frame-level cross-attention layer. The function of the third attention layer is to establish the correspondence between text words and motion frames at the frame level. For example, it can be used to establish the correspondence between the lexical units in the second description information and the parts of the digital human in each frame of the first action sequence. For example, if the lexical unit in the second description information is 30°, then the correspondence between 30° and the parts of the digital human such as the knee, elbow or head in the action sequence can be established.

[0057] For example, the computation of the third attention layer is performed between frames. For instance, when processing the second description information, the computation of the third attention layer can be performed between multiple first frames of the first action sequence.

[0058] In practical applications, the first calculation result is positively correlated with the first similarity matrix, which characterizes the correlation between different frames in the first action sequence; and / or, The second operation result is positively correlated with the second similarity matrix, which represents the correlation between each word in the first descriptive information and each frame in the first action sequence; and / or, The third operation result is positively correlated with the third similarity matrix, which represents the degree of association between each word in the second description information and different first parts, where the first part is the part of the digital human in the first action sequence.

[0059] It is understandable that the first model built through the above practical applications can learn the temporal relationship between frames in the action sequence, the precise correspondence between sequence-level descriptive text and frames, and the correspondence between the human posture described by the frame-level descriptive text and the posture description in the action sequence.

[0060] The first attention layer, the second attention layer, and the third attention layer in the embodiments of this application are described below with specific examples.

[0061] For example, let the first action sequence be denoted as ,in, Let m be the dimension of the motion vector, which can be a three-dimensional human motion vector. The motion vector can be represented by the coordinates of joints or rotations, etc. m is the number of frames in the first motion sequence. The first attention layer performs the following calculations: (1) (2) Where Q, K, and V are all calculated from the input first action sequence X through a fully connected layer, and the weights are... It is obtained through model training, where d is the length of the action embedding vector, which can be obtained by processing the first action sequence X. This is the output of the first attention layer, i.e., the result of the first operation. That is, the first similarity matrix. The value of the element reflects the correlation of motion between frames; frames with similar action segments should have a high degree of similarity.

[0062] The calculation for the second attention layer is as follows: (3) (4) It should be noted that the difference between the second attention layer and the first attention layer lies here. and Encoding from sequence level text 9j (i.e., the text encoding of the first descriptive information) is calculated, and From the output of the first attention layer mentioned above The calculated result, after processing by the second attention layer, is obtained by weighting with cross-attention weights. The similarity matrix of the second attention layer The second similarity matrix reflects the correlation between each word in the sequence-level text (such as the second descriptive information) and each frame in the action sequence (such as the first action sequence).

[0063] In the third attention layer, for [i] can be adopted [i] Perform the following calculation, where a mask can be used to ensure that the action in frame i corresponds to the description information in frame i:

[0064]

[0065]

[0066]

[0067] in, and From frame-level text encoding (For example, the encoding of the second descriptive information) is calculated, and From the output of the second attention layer mentioned above The calculated value, after processing by this attention layer, is obtained by weighting with frame-level cross-attention weights. The similarity matrix of the third attention layer. (i.e., third-order similarity evidence) reflects the correlation between words in the frame-level text and different parts of the action sequence in each frame. The final output... Given an m*d matrix, the output layer will transform it back to a matrix of length m*d. Action vector .

[0068] For example, during the training phase of the first model, it is trained by adding noise to the action sequence and model prediction noise, so that the first model can ultimately learn from the noise input. Obtain an action that matches the text. Thus, the action generation module in this embodiment of the application possesses action generation capabilities.

[0069] Step 103: Retrieve actions that match the first action and / or actions that match the first description information from the first action library to obtain the second action of the digital human; For example, the first action library can be a preset action library, which may include multiple action-related parameters, such as the rotation angle of a joint, the position of a limb, etc.

[0070] For example, "action matching the first action" is characterized as an action that is similar to the first action. For instance, a first similarity score can be calculated for each action in the first action library. The first similarity score is used to measure the similarity between the corresponding action and the first action. Actions with a first similarity score higher than the corresponding threshold are considered as actions matching the first action. Alternatively, the top K actions with the first similarity score are considered as actions matching the first action.

[0071] For example, an "action that matches the first description information" is characterized as an action that is similar to the first description information. For instance, a second similarity score can be calculated for each action in the first action library. The second similarity score is used to measure the similarity between the corresponding action and the first description information. Actions with a second similarity score higher than the corresponding threshold are considered as actions that match the first description information. Alternatively, the top K actions with the second similarity scores are considered as actions that match the first description information.

[0072] For example, the second action can be represented as an action sequence or an action vector, etc.

[0073] In practical applications, the step of retrieving actions from the first action library that match the first action and / or actions that match the first description information to obtain the second action of the digital human includes: The second model retrieves the first action and the first description information from the first database using the second model to obtain a first action set matching the first action and a second action set matching the first description information. The second model is used to encode the first action and the first description information into the same latent vector space. The second action is obtained based on the first action set and the second action set.

[0074] Understandably, the core of the second model is to encode text and action sequences into the same latent vector space, so that the latent space can serve as a measure of the similarity between actions and text.

[0075] For example, the second model described above can be a cross-modal retrieval model. The second model focuses on the overall semantics of the action, so it treats the action sequence as a whole for retrieval. During retrieval, it uses sequence-level descriptive text and the generated first action.

[0076] For example, the second model consists of a text encoder and an action sequence encoder. The text encoder uses a Contrastive Language-Image Pre-training (CLIP) pre-trained language model, taking the average of the word-level embedding vectors to encode the first descriptive information into a vector. The action encoder uses a Transformer model to encode the first action (which can be viewed as an action sequence) into a vector. , and The dimensions are the same. During training, the distance between the matched action vector and the text vector (i.e., positive sample pairs) is made as close as possible, while the distance between negative sample pairs is made as far apart as possible. Since both text and action are encoded in the same latent vector space, the second model can use text (such as the first descriptive information) as input to retrieve actions in the first action library, or it can use action sequences as input to retrieve semantically similar actions in the first action library.

[0077] For example, the second description information and the first action are input into the second model to retrieve action sequences from the first action library, respectively obtaining the top k actions most similar to the first action and the top k actions most similar to the first description information. The top k actions most similar to the first action form a first action set, and the top k actions most similar to the first description information form a second action set. For example, the similarity to the first action can be measured by a first similarity score, and the similarity to the first description information can be measured by a second similarity score. Therefore, the top k actions with the highest corresponding first similarity scores form the first action set, and the top k actions with the highest corresponding second similarity scores form the second action set.

[0078] In practical applications, obtaining the second action based on the first action set and the second action set includes: Determine whether there is an intersection between the first action set and the second action set; If there is an overlap in actions, the target action in the overlap is taken as the second action, and the target action is characterized as an action whose similarity to the first description information and the first action satisfies the corresponding condition; if there is no overlap in actions, one or more of the following are taken as the second action: the action in the first action set that has the highest matching degree with the first action; the action in the second action set that has the highest matching degree with the first description information.

[0079] Understandably, if there is an intersection between the first action set and the second action set, the action with the highest similarity score in the intersection is returned; if there is no intersection between the first action set and the second action set, the action sequence with the highest similarity between the two is returned.

[0080] For example, the "corresponding condition" mentioned above can be that the third similarity score corresponding to the target action in the action intersection is the highest, and the third similarity score is related to one or more of the following: the first similarity score of the corresponding action; the second similarity score of the corresponding action.

[0081] For example, the third similarity score can be obtained by weighted summation of the first and second similarity scores. With a weight of 0.4 for the first similarity score and 0.6 for the second similarity score, assuming the action intersection includes action 1 and action 2, and action 1 has a first similarity score of 5 with the first action and a second similarity score of 6 with the first descriptive information, then the third similarity score for action 1 is 0.4 × 5 + 0.6 × 6 = 5.6. Similarly, action 2 has a first similarity score of 4 with the first action and a second similarity score of 7 with the first descriptive information, so the third similarity score for action 2 is 0.4 × 4 + 0.6 × 7 = 5.8. Therefore, action 2 is the target action that satisfies the corresponding conditions, and action 2 is designated as the second action.

[0082] For example, one or more second actions retrieved from the first action library can be denoted as .

[0083] Step 104: Obtain the second action sequence of the digital human based on the first action and the second action.

[0084] For example, obtaining the second action sequence of the digital human based on the first action and the second action includes: fusing the first action and the second action by linear interpolation to obtain the second action sequence of the digital human.

[0085] For example, actions generated by the first model The action retrieved from the first action library as the first action. As the second action, linear interpolation is used for fusion, with the initial weight set to 0.5. Users can adjust the weight between 0 and 1 to select a satisfactory fusion effect. The purpose of this step is to use actions from the first action library that conform to logic and human dynamics (i.e., the second action) to constrain the generated action (i.e., the first action) in order to correct unreasonable parts of the generated action.

[0086] In practical applications, after obtaining the second action sequence of the digital human based on the first action and the second action, the method further includes: Send the second action sequence; Upon receiving a first instruction, the first weight corresponding to the first word in the second attention layer is adjusted according to the first instruction, wherein the first instruction is used to indicate that the second action sequence does not match the first word in the first description information; The action sequence of the digital human is regenerated based on the adjusted first weight.

[0087] Understandably, if the second action sequence obtained after the above steps does not meet the user's action requirements, the user can point out the mismatch between the second action sequence and the user's text by specifying a word (i.e., the first word element). For example, the user's text can be the text entered by the user, such as the first descriptive information entered by the user. Suppose the user's text contains "jump", and "jump" is located in the i-th position of the user's text, but the "jump" action does not appear in the generated second action sequence, then the user can point out that the second action sequence does not match the i-th word in the user's text.

[0088] For example, the first weight can be one of the weights in formula (3) above. .

[0089] For example, adjusting the first weight corresponding to the first word in the second attention layer according to the first instruction includes: increasing the first weight corresponding to the first word in the second attention layer according to the first instruction.

[0090] For example, based on user feedback, the similarity matrix in the first model of the action generation module can be adjusted. The second similarity matrix is ​​adjusted by increasing the first weight of word i in the second attention layer, which means adjusting its corresponding column. The generation process is repeated to regenerate a new action sequence until the user is satisfied.

[0091] It is understood that regenerating the digital human's action sequence based on the adjusted first weights means: adjusting the first weights of the first word units in the second attention layer of the first model, and then re-executing steps 102, 103, and related examples. That is, generating the digital human's actions based on the adjusted first model, re-searching in the first action library, and generating a new action sequence for the digital human based on the generated actions and the actions retrieved from the first action library.

[0092] The present application will be further described in detail below with reference to application examples.

[0093] This application proposes a retrieval-generation fusion framework to generate 3D human motion from text. The overall process involves the user inputting a motion sequence description text. The motion generation module rewrites the text using a large language model, expanding the motion sequence description text into fine-grained description text for each frame. Then, both the sequence description text and the frame description text are simultaneously input into a diffusion model to generate the motion. The generated motion and the sequence description text are then retrieved using a cross-modal retrieval module, and the retrieval results are returned for the user to view. After viewing the motion sequence, the user provides feedback, and the motion editing module adjusts the weight of each word in the text based on the user's feedback to regenerate the 3D human motion. The framework based on this application consists of three modules: a motion generation module, a motion library retrieval module, and a motion editing module.

[0094] The following description, in conjunction with specific steps, illustrates each module within the framework upon which the embodiments of this application are based. For details, please refer to [link / reference needed]. Figure 2 : The following steps 201-203 describe the action generation module: Step 201: Construct training datasets with both coarse and fine granularity; The training datasets, which are coarse-grained and fine-grained, include sequence description text and frame description text. Sequence description text is used to describe action sequences, while frame description text is used to describe frames in action sequences.

[0095] Step 202: Fine-tune the large language model using text data of various granularities, such as paired sequence description text and frame description text constructed in Step 201; Step 203: Train the action generation module using the dataset; For example, embodiments of this application use a diffusion model as the action generation model. The goal during the training phase is to learn the conditional generation process from noise to motion sequences through the diffusion model. The following steps 204-205 describe the action retrieval module: Step 204: Train the first model; For example, the first model can be a cross-modal retrieval model.

[0096] Step 205: Generate an action based on the first description information input by the user; For example, the user inputs first descriptive information, which is expanded into second descriptive information by a large language model. The first and second descriptive information are then input into the action generation module to obtain the generated action sequence. Then, the first description information and the generated action sequence are input into the second model to retrieve actions from the first action library, respectively, to obtain the top k actions that are closest to the first description information and the action sequence that is closest to the generated action sequence. The top k most similar actions are returned. If the two actions have an intersection, the action with the highest third similarity score in the intersection is returned. If the two actions have no intersection, the action sequence with the highest similarity between the two actions is returned. For example, action sequence This is an example of the first action mentioned above.

[0097] For example, the action retrieved from the first action library (i.e., the action that is finally returned) is denoted as , This is an example of the second action mentioned above.

[0098] Step 206: Use linear interpolation to fuse the two actions to obtain the second action sequence.

[0099] For example, linear interpolation fusion can be used. as well as The second action sequence is obtained.

[0100] Steps 207-208 will describe the action editing module.

[0101] Step 207: Receive user feedback; Step 208: Based on the user feedback in Step 207, adjust the similarity matrix in the three-layer attention model of the action generation module. Make adjustments and regenerate a new sequence of actions; The modules in the embodiments of this application have been described above. The action production process in the embodiments of this application is described below with reference to specific application examples. Please refer to [link / reference]. Figure 3 , Figure 3 This is an exemplary flowchart of an embodiment of this application. Figure 3 Includes the following steps: S301: Receive the first description information input by the user.

[0102] For example, the first descriptive information input by the user can be in the form of text, which can be sequence-level descriptive text used to describe the first sequence of actions of the digital human.

[0103] S302: Translate the first descriptive information into the second descriptive information using a large model; For example, the large model described above could be a large language model.

[0104] For example, the second descriptive information may be descriptive text of body parts in a keyframe.

[0105] S303: Based on the first and second descriptive information, the first action is generated by the first model through an attention mechanism; For example, the first model can be a diffusion model, which can include multiple attention layers to produce the first action through an attention mechanism.

[0106] S304a: Retrieve actions from the first action library that are similar to the first action; For example, an action similar to the first action can be an action whose corresponding semantics are similar to those of the first action. For instance, if action 3 in the first action library and the semantics corresponding to the first action are both "squat down", then action 3 can be considered as having a semantics similar to the first action.

[0107] S304b: Retrieve actions from the first action library that match the first description information; For example, actions similar to the first action can form a first action set, and actions similar to the first description information can form a second action set. The second action can be selected from the first action set and the second action set. For example, the second action can be the one with the highest third similarity score in the intersection of the actions of the first action set and the second action set.

[0108] S305: The generated first action and the retrieved second action are merged into a candidate action; For example, the candidate action can be represented as a second action sequence.

[0109] S306: Output candidate actions; For example, candidate actions can be sent to the user.

[0110] S307: User selection, adjustments are made based on user feedback until the user is satisfied.

[0111] For example, users can choose to delete some actions in the second action sequence or adjust the weights of certain words in the first model. After receiving this feedback from the user, the first model or the second action sequence can be adjusted until the user's needs are met.

[0112] As can be seen from the above, this application proposes a human motion generation framework that integrates retrieval and generation, which can respond more accurately to user-inputted text and is more natural and diverse. This application addresses the problems in related technologies, such as the single technical path leading to insufficient diversity of generated motions, weak controllability of text input on generated motions, and lack of conformity to human dynamics. By combining generation and retrieval technologies, utilizing a pre-set motion database to provide reasonable initialization for the generation model, and incorporating physical features based on motion calculation during the retrieval stage, the recall rate of the retrieval and the controllability of the final generated motions are improved.

[0113] Regarding the paraphrasing of fine-grained text, the aforementioned techniques one and two both paraphrase sequence-level descriptive text into descriptive text for each body part. However, the paraphrased text remains sequence-level descriptive text. In contrast, the fine-grained text in this application, in addition to describing body parts, also extracts keyframes from the sequence and describes these keyframes, achieving fine-grainedness not only in the action dimension but also in the time dimension. Simultaneously, the sequence descriptive text and frame descriptive text are used as conditional inputs to a diffusion model, which generates actions.

[0114] In terms of technical framework, compared with a single technical path, the embodiments of this application integrate two technical paths: retrieval and generation. It has both the diversity of generation and the reasonable constraints of the pre-set database for the generation model. In the retrieval stage, in addition to multimodal retrieval from text to action, action-to-action retrieval is also added. This framework solves the problems of insufficient realism of generated actions and insufficient diversity of retrieved actions.

[0115] Text-to-3D human motion generation technology is a core technology in multiple fields such as metaverse, digital humans, game development, virtual reality, and content creation. With the rapid development of the industry, the demand for high-quality, efficient, and highly controllable human motion generation methods is also increasing. The technical solution proposed in this application can effectively improve the realism, diversity, controllability, and rationality of generated motions. The framework design combining retrieval and generation in this application can ensure the rationality and diversity of motions. Furthermore, this application considers not only the granularity of the human body dimension but also the granularity of the time dimension in its multi-granular text integration, that is, it takes into account frame-level descriptive information, providing a comprehensive consideration and effectively improving the controllability of generated motions.

[0116] Based on the embodiments described above, this application also provides an action generation device, see [link to previous document]. Figure 4 The device includes: The first receiving unit 401 is used to receive first description information, which describes a first action sequence of the digital human. The first generation unit 402 is used to generate the first action of the digital human based on the first description information and through the first model. The first retrieval unit 403 is used to retrieve actions that match the first action and / or actions that match the first description information from the first action library to obtain the second action of the digital human. The first processing unit 404 obtains a second action sequence of the digital human based on the first action and the second action.

[0117] In one embodiment, the first action sequence includes a plurality of first frames, each of the plurality of first frames including parameters related to the pose of the digital human, and after receiving the first description information, the device further includes a first conversion unit, the first conversion unit being configured to: The first description information is converted into one or more second description information, and each of the one or more second description information is used to describe one of the multiple first frames; The first generation unit 402 generates the first action of the digital human based on the first description information and through the first model, including: The first model is used to process the first description information and one or more second description information to generate the first action.

[0118] In one embodiment, the first model is a diffusion model, which includes a first attention layer, a second attention layer, a third attention layer, and an output layer. The first generation unit 402 processes the first description information and one or more second description information through the first model, including: The first action sequence is subjected to self-attention operation through the first attention layer to obtain the first operation result; The second attention layer performs cross-attention operation on the encoding result of the first description information and the first operation result to obtain the second operation result; The third attention layer performs a cross-attention operation on the encoding results of one or more second descriptive information items and the second operation result to obtain the third operation result. The first action is obtained by converting the third operation result through the output layer.

[0119] In one embodiment, the first calculation result is positively correlated with a first similarity matrix, the first similarity matrix representing the degree of correlation between different frames in the first action sequence; and / or, The second operation result is positively correlated with the second similarity matrix, which represents the correlation between each word in the first descriptive information and each frame in the first action sequence; and / or, The third operation result is positively correlated with the third similarity matrix, which represents the degree of association between each word in the second description information and different first parts, where the first part is the part of the digital human in the first action sequence.

[0120] In one embodiment, the first retrieval unit 403 retrieves actions matching the first action and / or actions matching the first description information from a first action library to obtain the second action of the digital human, including: The second model retrieves the first action and the first description information from the first database using the second model to obtain a first action set matching the first action and a second action set matching the first description information. The second model is used to encode the first action and the first description information into the same latent vector space. The second action is obtained based on the first action set and the second action set.

[0121] In one embodiment, the first retrieval unit 403 obtains the second action based on the first action set and the second action set, including: Determine whether there is an intersection between the first action set and the second action set; If there is an overlap in actions, the target action in the overlap is taken as the second action, and the target action is characterized as an action whose similarity to the first description information and the first action satisfies the corresponding condition; if there is no overlap in actions, one or more of the following are taken as the second action: the action in the first action set that has the highest matching degree with the first action; the action in the second action set that has the highest matching degree with the first description information.

[0122] In one embodiment, the apparatus further includes a first transmitting unit, a first adjusting unit, and a second generating unit. After obtaining the second action sequence of the digital human based on the first action and the second action, the first transmitting unit is used to transmit the second action sequence. Upon receiving a first instruction, the first adjustment unit is configured to adjust the first weight corresponding to the first word in the second attention layer according to the first instruction, wherein the first instruction is configured to indicate that the second action sequence does not match the first word in the first description information; The first generation unit is used to regenerate the action sequence of the digital human based on the adjusted first weight.

[0123] In practical applications, the first generation unit 402, the first retrieval unit 403, the first processing unit 404, the first conversion unit, the first adjustment unit, and the second generation unit can be implemented by the processor in the action generation device. The first receiving unit 401, the first sending unit, the third receiving unit, etc., can be implemented by the processor in the action generation device in conjunction with the communication interface.

[0124] It should be noted that the action generation device provided in the above embodiments is only illustrated by the division of the above program modules when performing information processing. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the action generation device and the action generation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0125] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, this application also provides an electronic device, see [link to relevant documentation]. Figure 5 The electronic device includes: The first communication interface 1 is capable of exchanging information with other devices; The first processor 2 is connected to the first communication interface 1 to enable information interaction with other devices. When running a computer program, it executes the methods provided by one or more technical solutions in the above embodiments. The computer program is stored in the first memory 3.

[0126] Specifically, the first communication interface 1 is used to receive first description information, which describes a first action sequence of the digital human. The first processor 2 is configured to generate a first action of the digital human based on the first description information and a first model; The first processor 2 is further configured to retrieve actions that match the first action and / or actions that match the first description information from the first action library to obtain the second action of the digital human; The first processor 2 is further configured to obtain a second action sequence of the digital human based on the first action and the second action.

[0127] In one embodiment, the first action sequence includes a plurality of first frames, each of the plurality of first frames including parameters related to the pose of the digital human. After the first communication interface 1 receives the first description information, the first processor 2 is further configured to: The first description information is converted into one or more second description information, and each of the one or more second description information is used to describe one of the multiple first frames; The first processor 2 generates the first action of the digital human based on the first description information and through the first model, including: The first model is used to process the first description information and one or more second description information to generate the first action.

[0128] In one embodiment, the first model is a diffusion model, which includes a first attention layer, a second attention layer, a third attention layer, and an output layer. The first processor 2 processes the first description information and one or more second description information through the first model, including: The first action sequence is subjected to self-attention operation through the first attention layer to obtain the first operation result; The second attention layer performs cross-attention operation on the encoding result of the first description information and the first operation result to obtain the second operation result; The third attention layer performs a cross-attention operation on the encoding results of one or more second descriptive information items and the second operation result to obtain the third operation result. The first action is obtained by converting the third operation result through the output layer.

[0129] In one embodiment, the first calculation result is positively correlated with a first similarity matrix, the first similarity matrix representing the degree of correlation between different frames in the first action sequence; and / or, The second operation result is positively correlated with the second similarity matrix, which represents the correlation between each word in the first descriptive information and each frame in the first action sequence; and / or, The third operation result is positively correlated with the third similarity matrix, which represents the degree of association between each word in the second description information and different first parts, where the first part is the part of the digital human in the first action sequence.

[0130] In one embodiment, the first processor 2 retrieves actions matching the first action and / or actions matching the first description information from a first action library to obtain a second action of the digital human, including: The second model retrieves the first action and the first description information from the first database using the second model to obtain a first action set matching the first action and a second action set matching the first description information. The second model is used to encode the first action and the first description information into the same latent vector space. The second action is obtained based on the first action set and the second action set.

[0131] In one embodiment, the first processor 2 obtains the second action based on the first action set and the second action set, including: Determine whether there is an intersection between the first action set and the second action set; If there is an overlap in actions, the target action in the overlap is taken as the second action, and the target action is characterized as an action whose similarity to the first description information and the first action satisfies the corresponding condition; if there is no overlap in actions, one or more of the following are taken as the second action: the action in the first action set that has the highest matching degree with the first action; the action in the second action set that has the highest matching degree with the first description information.

[0132] In one embodiment, after the first processor 2 obtains the second action sequence of the digital human based on the first action and the second action, the first communication interface 1 is used to send the second action sequence; Upon receiving the first instruction, the first processor 2 is further configured to adjust the first weight corresponding to the first word in the second attention layer according to the first instruction, wherein the first instruction is configured to indicate that the second action sequence does not match the first word in the first description information; The first processor 2 is also configured to regenerate the action sequence of the digital human based on the adjusted first weight.

[0133] It should be noted that the specific processing procedure of the first communication interface 1 can be understood by referring to the above method.

[0134] Of course, in practical applications, the various components in an electronic device are coupled together through bus system 4. It can be understood that bus system 4 is used to achieve communication and connection between these components. In addition to the data bus, bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general will label all buses as Bus System 4.

[0135] The first memory 3 in this embodiment is used to store various types of data to support operation in the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0136] The methods disclosed in the embodiments of this application can be applied to the first processor 2, or implemented by the first processor 2. The first processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in the first processor 2. The first processor 2 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The first processor 2 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the first memory 3. The first processor 2 reads the information in the first memory 3 and completes the steps of the aforementioned method in combination with its hardware.

[0137] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned methods.

[0138] It is understood that the first memory 3 in this application embodiment can be a volatile memory pool or a non-volatile memory pool, or both. The non-volatile memory pool can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory pool can be a disk storage pool or a magnetic tape storage pool. The volatile memory pool can be a random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The storage pools described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of storage pools.

[0139] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a first memory 3 storing a computer program, which can be executed by a first processor 2 to complete the steps described in the aforementioned method.

[0140] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a first processor 2 to perform the steps described in the foregoing method.

[0141] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. An action generation method, characterized in that, The action generation method includes: Receive first description information, which is used to describe a first action sequence of the digital human; Based on the first description information, the first action of the digital human is generated through the first model; The second action of the digital human is obtained by retrieving the action that matches the first action and / or the action that matches the first description information from the first action library. The second action sequence of the digital human is obtained based on the first action and the second action.

2. The method according to claim 1, characterized in that, The first action sequence includes multiple first frames, each of the multiple first frames including parameters related to the pose of the digital human. After receiving the first description information, the method further includes: The first description information is converted into one or more second description information, and each of the one or more second description information is used to describe one of the multiple first frames; The step of generating the first action of the digital human based on the first description information and through the first model includes: The first model is used to process the first description information and one or more second description information to generate the first action.

3. The method according to claim 2, characterized in that, The first model is a diffusion model, which includes a first attention layer, a second attention layer, a third attention layer, and an output layer. The processing of the first descriptive information and one or more pieces of second descriptive information using the first model includes: The first action sequence is subjected to self-attention operation through the first attention layer to obtain the first operation result; The second attention layer performs cross-attention operation on the encoding result of the first description information and the first operation result to obtain the second operation result; The third attention layer performs a cross-attention operation on the encoding results of one or more second descriptive information items and the second operation result to obtain the third operation result. The first action is obtained by converting the third operation result through the output layer.

4. The method according to claim 3, characterized in that, The first calculation result is positively correlated with the first similarity matrix, which represents the degree of correlation between different frames in the first action sequence; And / or, The second operation result is positively correlated with the second similarity matrix, which represents the degree of association between each word in the first description information and each frame in the first action sequence. And / or, The third operation result is positively correlated with the third similarity matrix, which represents the degree of association between each word in the second description information and different first parts, where the first part is the part of the digital human in the first action sequence.

5. The method according to claim 1, characterized in that, The step of retrieving actions matching the first action from the first action library and / or actions matching the first description information to obtain the second action of the digital human includes: The second model retrieves the first action and the first description information from the first database using the second model to obtain a first action set matching the first action and a second action set matching the first description information. The second model is used to encode the first action and the first description information into the same latent vector space. The second action is obtained based on the first action set and the second action set.

6. The method according to claim 5, characterized in that, The process of obtaining the second action based on the first action set and the second action set includes: Determine whether there is an intersection between the first action set and the second action set; If there is an overlap in actions, the target action in the overlap is taken as the second action, and the target action is characterized as an action whose similarity to the first description information and the first action satisfies the corresponding condition; if there is no overlap in actions, one or more of the following are taken as the second action: the action in the first action set that has the highest matching degree with the first action; the action in the second action set that has the highest matching degree with the first description information.

7. The method according to claim 3, characterized in that, After obtaining the second action sequence of the digital human based on the first action and the second action, the method further includes: Send the second action sequence; Upon receiving a first instruction, the first weight corresponding to the first word in the second attention layer is adjusted according to the first instruction, wherein the first instruction is used to indicate that the second action sequence does not match the first word in the first description information; The action sequence of the digital human is regenerated based on the adjusted first weight.

8. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images