Action video generation method, related device and medium

Abstract

The disclosed embodiment discloses an action video generation method, related device and medium, which obtains the target audio associated with the target virtual character, encodes the target audio into target audio features; obtains the target guidance content associated with the target virtual character, inputs the target guidance content and the preset prompt text into the preset large language model for text prediction, generates at least one semantic tag for describing the target virtual character, and encodes the semantic tag into a semantic tag feature; uses a diffusion model to denoise the preset noise features according to the target audio features and the semantic tag features to obtain a target action sequence; generates a target action video of the target virtual character based on the target action sequence. The disclosed method can generate action data that better fits the characteristics of the virtual character itself, improves the precision and accuracy of the generated action data, and can be widely used in scenes such as virtual character modeling and virtual character driving.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to an action video generation method, related devices and media. Background Art

[0002] Virtual character modeling is widely used in games, video creation and other scenarios. In these scenarios, virtual characters are often required to make corresponding actions based on the audio emitted to improve interactivity, and action modeling for virtual characters is relatively important. With the development of technology, artificial intelligence technology can be used in related technologies to model the actions of virtual characters. However, when using artificial intelligence technology to model the actions of virtual characters, the action data of the virtual characters is often generated only based on voice. Due to the limited amount of information carried by voice, the generated action data of the virtual characters is insufficient in precision and accuracy, which reduces the quality of the generated action video. Summary of the invention

[0003] The present disclosure provides a method for generating an action video, a related device and a medium, which can improve the accuracy and precision of the generated action data, thereby improving the quality of the generated action video.

[0004] According to one aspect of the present disclosure, there is provided a method for generating an action video, comprising:

[0005] Acquire target audio associated with a target virtual character, and encode the target audio into a target audio feature;

[0006] Acquire target guidance content associated with the target virtual character, input the target guidance content and preset prompt text into a preset large language model for text prediction, generate at least one semantic tag for describing the target virtual character, encode at least one semantic tag into a semantic tag feature, and the preset prompt text is used to prompt the large language model to generate the semantic tag based on the target guidance content;

[0007] De-noising the preset noise features according to the target audio features and the semantic label features using a diffusion model to obtain a target action sequence, wherein the target action sequence includes action data of each action frame of the target virtual character in the target action video to be generated;

[0008] The target action video of the target virtual character is generated based on the target action sequence.

[0009] According to one aspect of the present disclosure, there is provided an action video generating device, comprising:

[0010] An audio acquisition module, used to acquire target audio associated with a target virtual character, and encode the target audio into a target audio feature;

[0011] A semantic tag processing module, used to obtain target guidance content associated with the target virtual character, input the target guidance content and preset prompt text into a preset large language model for text prediction, generate at least one semantic tag for describing the target virtual character, encode the at least one semantic tag into a semantic tag feature, and the preset prompt text is used to prompt the large language model to generate the semantic tag based on the target guidance content;

[0012] A diffusion module, configured to denoise a preset noise feature according to the target audio feature and the semantic tag feature using a diffusion model to obtain a target action sequence, wherein the target action sequence includes action data of each action frame of the target virtual character in the target action video to be generated;

[0013] A video generation module is used to generate the target action video of the target virtual character based on the target action sequence.

[0014] Optionally, the semantic tag processing module is used to:

[0015] Acquire a target text associated with the target virtual character, wherein the target text includes text corresponding to a plurality of character audios of the target virtual character, and the target audio is one of the plurality of character audios;

[0016] The target text and the target audio are determined as target guiding content associated with the target audio.

[0017] Optionally, the semantic tag processing module is used to:

[0018] Performing audio recognition on multiple character audios of the target virtual character to obtain the target text;

[0019] Alternatively, text associated with the target virtual character input through an input operation is obtained as the target text of the target audio.

[0020] Optionally, a diffusion module is used to:

[0021] Get the preset maximum time step value;

[0022] Determine the maximum value of the time step as the current time step, and determine the preset noise feature as the sequence to be denoised;

[0023] De-noising the sequence to be de-noised according to the current time step, the target audio feature and the semantic label feature using a diffusion model to obtain an intermediate de-noised sequence;

[0024] The current time step is decremented based on a preset interval, the intermediate denoising sequence is determined as the sequence to be denoised, and the diffusion model is used again to denoise the sequence to be denoised according to the decremented current time step, the target audio feature and the semantic label feature to obtain the intermediate denoising sequence, until the current time step returns to zero, and the intermediate denoising sequence when the current time step returns to zero is determined as the target action sequence.

[0025] Optionally, a diffusion module is used to:

[0026] Acquire initial motion data, and perform convolution on the initial motion data to obtain initial motion features;

[0027] The sequence to be denoised is denoised by using a diffusion model according to the current time step, the initial action feature, the target audio feature and the semantic label feature to obtain an intermediate denoised sequence.

[0028] Optionally, a diffusion module is used to:

[0029] Inputting the current time step, the initial action feature, the target audio feature and the semantic label feature into a diffusion model;

[0030] Fusing the initial action feature, the target audio feature, the sequence to be denoised, and the semantic label feature to obtain a first fused feature;

[0031] Performing a local attention operation on the first fusion feature according to a preset attention window size;

[0032] The initial action feature, the semantic label feature and the result of the local attention operation are fused to obtain a second fused feature, a self-attention operation is performed on the second fused feature, and a linear mapping is performed on the result of the self-attention operation to obtain a noise feature to be removed corresponding to the current time step, wherein the noise feature to be removed is a tensor with the same shape as the sequence to be denoised;

[0033] A first noise coefficient corresponding to the current time step is obtained, and the noise feature to be removed is subtracted from the sequence to be denoised according to the first noise coefficient to obtain the intermediate denoised sequence.

[0034] Optionally, a diffusion module is used to:

[0035] Acquire a second noise coefficient corresponding to each time step smaller than the current time step, wherein the time step is negatively correlated with the corresponding second noise coefficient, and the second noise coefficient is larger than the first noise coefficient;

[0036] A cumulative noise coefficient is obtained according to the product of the first noise coefficient and a plurality of the second noise coefficients, a first difference value of a reference value minus the cumulative noise coefficient is determined, and a square root of the first difference value is determined as a first square root value;

[0037] determining a second difference value of the reference value minus the first noise factor;

[0038] Determine a first quotient of the second difference and the first square root value, and obtain a scaled noise feature to be removed according to a product of the first quotient and the noise feature to be removed;

[0039] Determine a third difference value of the sequence to be denoised minus the scaled noise feature to be removed, and determine a first product of the third difference value and the inverse of the square root of the first noise coefficient;

[0040] Determine a second product of a first noise sequence and the second difference, and determine the sum of the first product and the second product as the intermediate denoising sequence, wherein the first noise sequence is obtained by sampling from a standard normal distribution, and a shape of the first noise sequence is the same as a shape of the sequence to be denoised.

[0041] Optionally, the action video generating device further includes a training module, and the training module is used to:

[0042] Acquire a sample action sequence of a sample virtual character, a sample audio associated with the sample virtual character, and a sample guidance content associated with the sample audio, wherein the sample action sequence includes action data of the sample virtual character in a plurality of action frames;

[0043] Encoding the sample audio into sample audio features, inputting the sample guide content and preset prompt words into a large language model for text prediction, and generating a sample label for describing the sample virtual character;

[0044] Adding noise to the sample action sequence in a plurality of time steps in sequence until the time step reaches the maximum value of the time step to obtain a sample noise sequence, and determining the noise feature added to the sample action sequence in each time step as a reference noise feature corresponding to the time step;

[0045] Encoding the sample label into a sample label feature, guiding the diffusion model to denoise the sample noise sequence in each of the time steps through the sample audio feature and the sample label feature, and determining a predicted noise feature predicted in each of the time steps, wherein the predicted noise feature is the noise removed from the sample noise sequence during the denoising process of the time step;

[0046] Based on the difference between the reference noise feature and the predicted noise feature corresponding to each of the time steps, a first loss function value is determined, and back propagation is performed based on the first loss function value to optimize the parameters of the diffusion model.

[0047] Optionally, the training module is used to:

[0048] For each of the time steps, respectively encoding the action data of the first n action frames in the sample action sequence as sample action features, where n increases as the time step decreases, and n is a positive integer;

[0049] The diffusion model is guided by the sample audio features, the sample action features, and the sample label features to denoise the sample noise sequence in each time step.

[0050] Optionally, the training module is used to:

[0051] Randomly masking the sample label feature and the sample action feature;

[0052] The diffusion model is guided by the sample audio features, the sample label features after random masking, and the sample action features after random masking to denoise the sample noise sequence in each of the time steps.

[0053] Optionally, the training module is used to:

[0054] For each of the time steps, calculating a fourth difference between the predicted noise feature corresponding to the time step and the reference noise feature corresponding to the time step;

[0055] The first loss function value is determined according to an L2 norm of the fourth difference.

[0056] Optionally, the training module is used to:

[0057] Setting the current time step to zero and determining the sample action sequence as a sequence to be noised;

[0058] Obtaining a first noise coefficient corresponding to the current time step, sampling a noise feature to be added corresponding to the current time step from a standard normal distribution, adding the noise feature to be added to the sequence to be noisy according to the first noise coefficient, and obtaining an intermediate noisy sequence corresponding to the current time step;

[0059] The current time step is incremented based on the preset interval, the intermediate noise-added sequence is determined as the sequence to be noise-added, the first noise coefficient is updated, the noise feature to be added corresponding to the current time step is sampled again from a standard normal distribution, and the noise feature to be added is added to the sequence to be noise-added according to the updated first noise coefficient until the current time step reaches the maximum value of the time step, thereby obtaining the sample noise sequence.

[0060] Optionally, the training module is used to:

[0061] weighting the sequence to be noisy according to the square root of the first noise coefficient;

[0062] Weighting the noise feature to be added according to the square root of the difference between the reference value and the noise coefficient;

[0063] The weighted sequence to be noisy and the weighted noise feature to be added are superimposed to obtain an intermediate noisy sequence corresponding to the current time step.

[0064] According to one aspect of the present disclosure, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned action video generation method when executing the computer program.

[0065] According to one aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the action video generating method as described above is implemented.

[0066] According to one aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer program. The computer program is read and executed by a processor of a computer device, so that the computer device executes the action video generation method as described above.

[0067] The action video generation method, device, electronic device and medium proposed in the present disclosure obtain target audio associated with the target virtual character and target guidance content associated with the target audio, and then input the target guidance content and preset prompt text into a large language model together, and use the semantic understanding ability of the large language model to extract semantic tags that can be used to describe the target virtual character from the target guidance content, and encode the semantic tags into semantic tag features, which can characterize the role attributes of the target virtual character itself. When using the diffusion model to generate actions for preset noise features, not only the corresponding target audio is used, but also the preset noise features are further combined with these semantic tag features that can characterize the role attributes of the target virtual character to perform denoising processing, because in the process of denoising the preset noise features using the diffusion model, the semantic tag features used to characterize the role attributes of the target virtual character are additionally injected, and the denoising process of the diffusion model is not only affected by the target audio features, but also by these semantic tags used to describe the role attributes of the target virtual character, which can make the target action sequence finally generated more accurately fit the role attributes of the target virtual character, thereby improving the accuracy and precision of the generated action data, and then improving the quality of the generated action video.

[0068] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0070] Figure 1 is a structural diagram of a system to which the action video generation method according to an embodiment of the present disclosure is applied;

[0071] Figure 2 A flowchart of a method for generating an action video provided by an embodiment of the present disclosure;

[0072] Figure 3 The implementation provided by the embodiment of the present disclosure Figure 2 A schematic diagram of an optional topological architecture of the action video generation method in FIG.

[0073] Figure 4 A schematic diagram of a target action sequence provided by an embodiment of the present disclosure;

[0074] Figure 5A schematic diagram of acquiring real-time speech and constructing corresponding target text provided by an embodiment of the present disclosure;

[0075] Figure 6 The implementation provided by the present disclosure Figure 2 Another optional topological architecture diagram of the action video generation method in ;

[0076] Figure 7 The implementation provided by the present disclosure Figure 2 An optional schematic diagram of step 203 in ;

[0077] Figure 8 A schematic diagram of performing attention calculation in combination with a position encoding matrix provided in an embodiment of the present disclosure;

[0078] Fig. 9 A schematic diagram of noise prediction combined with time step features provided in an embodiment of the present disclosure;

[0079] Fig.10 A flow chart of a training diffusion model is provided for the embodiment of the present disclosure;

[0080] Fig.11 Another flow chart of training a diffusion model is provided for an embodiment of the present disclosure;

[0081] Fig.12 A schematic diagram of the structure of an action video generating device provided by an embodiment of the present disclosure;

[0082] Fig.13 A partial structural block diagram of a terminal provided in an embodiment of the present disclosure;

[0083] Fig.14 A partial structural block diagram of a server provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0085] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0086] Large Language Models (LLMs): A deep learning model trained with a large amount of data. The core module of a large language model is the transformer module, which is often constructed by cascading multiple transformer modules. The transformer module can capture the correlation between each word in the input sequence through the attention mechanism, obtain the attention weights between the words, and predict the output corresponding to the input by weighting each word in the input sequence using the attention weights. After training with a large amount of data, the large language model can have a strong semantic understanding ability, so as to accurately understand and process the input problems. Commonly used large language models include the GPT series, ChatGLM, etc.

[0087] Diffusion model: A generative model that can be used for cross-modal data generation tasks, such as text-to-image, text-to-video, video-to-speech, speech-to-action, etc. A diffusion model is a generative model that converts raw data into noise data and then learns how to construct the required data from the input noise by learning the inverse diffusion process.

[0088] Virtual character modeling is widely used in games, live broadcasts and other scenarios. In these scenarios, it is often necessary to make the virtual character perform corresponding actions according to the audio emitted to improve the interactivity between players or viewers. With the development of technology, artificial intelligence technology can be used in related technologies to model the actions of virtual characters. However, when using artificial intelligence technology to model the actions of virtual characters, the action data of the virtual character is often generated only based on voice. Due to the limited amount of information carried by voice, the generated action data of the virtual character is insufficient in precision and accuracy, which reduces the quality of the generated action video.

[0089] Based on this, the present disclosure proposes a method for generating action videos, which can improve the precision and accuracy of the generated actions.

[0090] Figure 1 1 is a system architecture diagram used in the method for generating an action video according to an embodiment of the present disclosure, and includes: an object terminal 110 , the Internet 120 , a gateway 130 , and a server 140 .

[0091] The object terminal 110 is a device used by the object to input the target audio and target guidance content. It includes various forms such as desktop computers, laptops, PDAs (personal digital assistants), mobile phones, vehicle-mounted terminals, home theater terminals, and dedicated terminals. In addition, it can be a single device or a collection of multiple devices. For example, multiple devices are connected through a local area network and work together using a common display device to form a terminal. The object terminal 110 can also communicate with the Internet 120 in a wired or wireless manner to exchange data.

[0092] The gateway 130 is also called an internetwork connector or a protocol converter. The gateway 130 realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway 130 is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway 130 can also provide filtering and security functions. The message sent by the object terminal 110 to the server 140 is sent to the corresponding server 140 through the gateway 130. The message sent by the server 140 to the object terminal 110 is also sent to the corresponding object terminal 110 through the gateway 130.

[0093] The server 140 refers to a computer system that can provide action video generation services to the target terminal 110. Compared with the target terminal 110, the server 140 has higher requirements in terms of stability, security, performance, etc. The server 140 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc. The server 140 can also communicate with the Internet 120 in a wired or wireless manner to exchange data.

[0094] Exemplarily, the object terminal 110 sends the target audio to the server 140, and the server 140 can obtain the target audio associated with the target virtual character and encode the target audio into a target audio feature; the object terminal can send the target guidance content to the server 140, and the server 140 obtains the target guidance content associated with the target audio, inputs the target guidance content and the preset prompt text into the preset large language model for text prediction, generates at least one semantic tag for describing the target virtual character, and encodes the semantic tag into a semantic tag feature; the server 140 denoises the preset noise features according to the target audio features and the semantic tag features by utilizing a diffusion model to obtain a target action sequence; the server 140 generates a target action video of the target virtual character based on the target action sequence, and sends the target action video to the object terminal 110.

[0095] In addition, the server 140 may not send the target action video to the target terminal 110, but directly perform subsequent processing on the server 140, for example, directly loading the target action video into a program window to be displayed on the server 140.

[0096] The server 140 obtains the target audio associated with the target virtual character and the target guidance content associated with the target audio, and then inputs the target guidance content and the preset prompt text into the large language model, and uses the semantic understanding ability of the large language model to extract semantic tags that can be used to describe the target virtual character from the target guidance content, and encodes the semantic tags into semantic tag features, which can characterize the role attributes of the target virtual character itself. When using the diffusion model to generate actions for the preset noise features, not only the corresponding target audio is used, but also the preset noise features are further denoised in combination with these semantic tag features that can characterize the role attributes of the target virtual character, because in the process of denoising the preset noise features using the diffusion model, the semantic tag features used to characterize the role attributes of the target virtual character are additionally injected, and the denoising process of the diffusion model is not only affected by the target audio features, but also by these semantic tags used to describe the role attributes of the target virtual character, which can make the target action sequence finally generated more accurately fit the role attributes of the target virtual character, thereby improving the accuracy and precision of the generated action data, and then improving the quality of the generated action video.

[0097] It can be seen that compared with the action generation process that only uses audio to predict noise and denoises preset noise features, the method disclosed in the present invention can extract semantic labels used to describe the target virtual character by utilizing the semantic understanding ability of the large language model, and use this to guide the diffusion process, so that the generated target action video can be more in line with the characteristics of the target virtual character itself, thereby improving the refinement and accuracy of the generated target action video.

[0098] The action video generation method provided by the present disclosure can be applied to various scenarios, such as video creation, non-user role modeling, etc.

[0099] Reference Figure 2 , Figure 2 This is an optional flow chart of the action video generation method provided in the embodiment of the present disclosure. The action video generation method can be executed independently by the server, or independently by the terminal, or can be executed collaboratively by the terminal and the server, with the terminal and the server completing parts of the method respectively. The action video generation method can include but is not limited to the following steps:

[0100] Step 201, obtaining a target audio associated with a target virtual character, and encoding the target audio into a target audio feature;

[0101] The target virtual character is a virtual character that needs to generate corresponding actions. It can be a person, an animal, or an anthropomorphic animal, etc. It is understandable that the target virtual character is different in different application scenarios, and the target virtual character needs to be determined according to the actual application scenario. For example, in scenarios such as the metaverse and games, it is often necessary to set up a large number of non-player characters (NPC) to interact with users. In order to improve the experience of interacting with users, these non-player characters need to show certain body movements when interacting with users to improve the interactive experience. At this time, the target virtual character can be these non-player characters (NPC); in scenarios such as video creation and live broadcast, the target virtual character can be a virtual character used to replace the creator in the video. Using a virtual character to replace the creator can protect the creator's information from being leaked in the video while retaining interactivity.

[0102] The target audio is the audio corresponding to the target action video to be generated. The target audio is associated with the target virtual character, which means that the target audio can come from the target virtual character, that is, the audio emitted by the target virtual character itself, or it can come from the audio input by the user for emission by the target virtual character. In other words, the source of the target audio can be different in different application scenarios. For example, in the scenario of non-player character modeling, the target audio can be the audio emitted by the target virtual character after dubbing for the target virtual character in advance; in the scenario of video creation or video live broadcast, the target virtual character needs to generate corresponding actions according to the user's real-time audio, and the target audio can be the voice input by the user in real time.

[0103] In one embodiment, a target virtual character is assigned a corresponding role identity, the target audio corresponding to the target virtual character is stored in an audio file pool, and the role identity of the target virtual character is used to establish an association relationship between the audio files in the audio file pool and the target virtual character by means of a foreign key or the like.

[0104] In another possible implementation, a corresponding audio identity can be assigned to each target audio, and a corresponding database table can be set in the database for storing the target audio, and an audio identity item and a character identity item can be set in the database table. Then, when storing each target audio, the audio identity item corresponding to the target audio and the character identity item of the target virtual character are written into the audio identity item and the character identity item respectively to form corresponding entries stored in the database table, and the association relationship between the target audio and the target virtual character is determined based on each entry in the database table.

[0105] In another possible implementation, the target audio may also be stored in an external database and input into the model when the user needs to generate an action for the target virtual character. In this case, before generating the action, the user may first select the target virtual character for which the action needs to be generated, and then all the audio input by the user is determined as the target audio associated with the target virtual character.

[0106] After getting the target audio, refer to Figure 3 , the target audio can be encoded by a preset audio coding model, thereby extracting the features of each dimension of the target audio and encoding the target audio into the corresponding target audio features. In this embodiment, the audio coding model can be a wav2vec model or a wavLM model. The wav2vec model or the wavLM model can capture the rich features in the target audio and convert the target audio into a form suitable for model processing. The process can be recorded as ,in represents the target audio feature, Indicates the target audio, Indicates using the wav2vec model to process the input target audio.

[0107] Specifically, the target audio feature can be a sequence feature. In this embodiment, when encoding the target audio, the target audio can be first divided into multiple audio frames with equal frame lengths, and then each audio frame is encoded into a corresponding audio feature vector. Thereafter, the audio feature vectors of each audio frame are concatenated in sequence, and each audio feature vector is used as a matrix row, so that the corresponding target audio feature is obtained.

[0108] Step 202, obtaining target guidance content associated with the target audio, inputting the target guidance content and the preset prompt text into a preset large language model for text prediction, generating at least one semantic tag for describing the target virtual character, and encoding the at least one semantic tag into a semantic tag feature;

[0109] In step 202, the target guidance content is associated with the target virtual character, which means that the target guidance content can represent the character attributes of the target virtual character. The target guidance content can be pre-generated data composed of at least one modality that can directly or indirectly reflect the character attributes of the target virtual character. For example, the target guidance content can be the dialogue content generated by the target virtual character during the dialogue process, or the target guidance content can also be the description content input by the user to describe the target virtual character. The target guidance content can be a combination of one or more of text, voice, and video. The way to associate the target guidance content with the target virtual character can refer to the way to associate the target audio with the target virtual character in step 201, which will not be repeated here. Correspondingly, the semantic tag can specifically be a tag used to describe the character attributes of the target virtual character, and the number of semantic tags can also be one or more. A single semantic tag is used to represent the character attributes of the target virtual character in a single aspect; it can be understood that the character attributes described by these semantic tags are associated with the body movements made by the target virtual character when speaking the target audio. For example, the semantic tag can include but is not limited to tags of the character attributes of the target virtual character such as emotions, personality, age, and identity positioning. For example, the semantic tag includes the personality tag of the target virtual character. When the personality of the target virtual character is relatively stable, the body movements of the target virtual character when speaking are relatively few, and the movement range is also small; on the contrary, when the personality of the target virtual character is relatively lively, the body movements of the target virtual character when speaking are relatively many, and the movement range is large. By extracting these semantic tags from the target guidance content, using these semantic tags for noise prediction, and denoising the preset noise features, the precision and accuracy of the generated action data can be improved, so that the target action video finally generated is more in line with the role attributes of the target virtual character itself, and the interactive experience of the user when interacting with the target virtual character is improved.

[0110] Specifically, the character attributes of the target virtual character are often reflected in the target virtual character's dialogue process. Based on this, in this embodiment, the target guidance content can be the dialogue content generated by the target virtual character during the dialogue process. For example, a target virtual character with a relatively older age and a relatively calm personality often speaks after careful consideration during the dialogue, and the speech is relatively concise and concise; while a character with a relatively younger age and a relatively lively personality often shows a speaking mode of speaking freely and talking about everything during the dialogue process; by analyzing the dialogue content of the target virtual character, the target virtual character's personality, age and other attributes can be determined, thereby obtaining a semantic label for describing the character attributes of the target virtual character. Based on this, in this embodiment, the dialogue text or dialogue audio of the target virtual character can be used as the target guidance content. Specifically, in the scenario of non-player character modeling, the target virtual character is the non-player character, and the dialogue content of each non-player character is often pre-set by the user. Based on this, all dialogue texts or dialogue audios associated with the target virtual character can be directly obtained as target guidance content; in scenarios such as video creation and live broadcast, the target audio is the voice input by the user in real time. At this time, multiple historical target audios input by the user within a certain time range or historical dialogue texts obtained after voice recognition of multiple historical target audios can be saved, and these historical target audios or historical dialogue texts can be used as target guidance content.

[0111] After obtaining the target guidance content, the target guidance content and the preset prompt text are input into the large language model together, wherein the large language model can use a pre-trained large language model, and the specific large language model is not limited in this embodiment. The semantic understanding ability of the large language model can be used to analyze the target guidance content, and extract semantic tags that can reflect the role attributes of the target virtual character. Among them, the preset prompt text is a text used to prompt the large language model to generate the required semantic tags based on the input target guidance content. It can be understood that the target guidance content itself is only a piece of data associated with the target virtual character. When the target guidance content is input into the large language model, it is also necessary to make the large language model perceive the subsequent tasks to be processed. Based on this, in this embodiment, the target guidance content and the preset prompt text are input into the large language model together, so that the large language model determines the tasks to be processed based on the preset prompt text. Exemplarily, the preset prompt text can be "Please analyze the input content and identify the character characteristics, age, emotions and other role attributes of the target virtual character". Based on the preset prompt text, the large language model can perceive that the task to be processed is to determine the labels such as the character, age and emotions of the target virtual character by analyzing the target guidance content.

[0112] After obtaining the semantic tags used to describe the target virtual character, these semantic tags are essentially texts used to describe the attributes of the target virtual character in certain aspects. For example, the emotional tags are "excited", "depressed", etc., and the age tag can be a text describing the age group, such as "juvenile", "elderly", etc. At this time, these semantic tags can be encoded by a text modality encoder, and these semantic tags can be converted into a form suitable for model processing to obtain corresponding semantic tag features. Exemplarily, the text modality encoder can encode each semantic tag output by the large language model into a corresponding feature vector by means of one-hot or word embedding, and each feature vector is a semantic tag feature. These feature vectors are then concatenated and input into the diffusion model.

[0113] In a possible implementation, in step 202, target guidance content associated with target audio is obtained, including: obtaining target text associated with the target audio; and determining the target text and the target audio as target guidance content associated with the target audio.

[0114] Based on this, by obtaining the target text associated with the target virtual character, and inputting the target text and the target audio as the target guiding content into the large language model, the target text carries more semantic information than the target audio. Through the target text, some semantic labels corresponding to some relatively deep-level character attributes of the target virtual character itself that are relatively inconspicuous but not easy to change, such as personality labels, etc., can be more accurately captured; and the target audio can represent the role state of the target virtual character when speaking the target audio from more dimensions. Through the target audio, the semantic labels corresponding to relatively shallow-level character attributes that are obvious but easy to change, such as emotional labels, etc. can be better captured; the target audio and the target text are input into the large language model as the target guiding content, so that the semantic labels for describing the relatively fixed character attributes of the target virtual character itself can be extracted from the target text, and the semantic labels of the character attributes that the target virtual character only shows when speaking the target audio can be extracted from the target audio. Based on this, a variety of different character labels of the target virtual character can be captured more comprehensively, so as to perform noise prediction, and remove the predicted noise from the preset noise features, which can improve the denoising accuracy of the diffusion model, thereby improving the refinement and accuracy of the generated action data.

[0115] Specifically, the target audio is a segment of audio corresponding to the target action sequence to be generated, and the amount of semantic information it can carry is relatively limited. It is often difficult to accurately analyze the character, age, role positioning and other deep-level role attributes of the target virtual character through a single sentence of the target audio, which makes the semantic tags that can be extracted based on the target audio relatively small. In this embodiment, the target text associated with the target virtual character refers to the target text that can characterize the role attributes of the target virtual character. By obtaining the target text associated with the target virtual character, the target text can include texts corresponding to multiple role audios of the target virtual character, and the target audio is one of the multiple role audios. For example, in the scenario of non-player character modeling, the target virtual character is one of the non-player characters, and the character audio of these non-player characters is often set in advance. At this time, the text corresponding to all the voice audio of the target virtual character can be extracted from all the character audio of the target virtual character as the target text, or the character audio of these non-player characters is obtained by asking the user to dub according to a given text. At this time, the text provided to the dubbing staff for dubbing can be directly used as the target text; and in scenarios such as video creation or live broadcast, the target audio emitted by the target virtual character is the voice audio input by the user in real time. At this time, each time the real-time input voice is obtained, voice recognition can be performed to generate corresponding voice text, and the voice text within a certain time range or a certain context length can be used as the target text, for example, the text recognized from the 100 audios recently input by the user can be used as the target text. At this time, the target text not only contains the text corresponding to the target audio, but also contains the text corresponding to other character audios of the target virtual character. The target text can carry richer semantic information than the target audio. By taking the acquired target text as one of the inputs of the large language model and analyzing the target text using the semantic understanding ability of the large language model, the target virtual character's multiple role attributes can be more accurately analyzed from the target text, and more semantic tags can be generated to describe the target virtual character. Then, the semantic tag features and target audio features obtained based on these semantic tags are used to predict noise, and the predicted noise is removed from the preset noise features. This can make the generated target action video more accurately and finely reflect the role attributes of the target virtual character, thereby improving the fineness and accuracy of the target action video.

[0116] In one embodiment, the target text may also include text for describing the environment and atmosphere in which the target virtual character is located. It is understandable that when the environment and atmosphere in which the target virtual character is located changes, the actions made for the same voice will also change. For example, the body movements of the target virtual character when speaking in a relatively tense and depressing environment and atmosphere are often less than those when speaking in a more active environment and atmosphere. For example, in scenes such as the metaverse and games, corresponding background stories and scene settings are often set. These stories and scene settings determine the environment and atmosphere in which the non-player characters, i.e., the target virtual characters, are located. For example, in casual games, the target virtual characters are often active in a relatively relaxed and active environment and atmosphere; and in horror games, the target virtual characters are also active in a relatively tense and depressing environment and atmosphere. At this point, the target text can include the text part as the story background and scene setting, and the target text is processed using a large language model to extract the semantic tags corresponding to the environment and atmosphere in which the target virtual character is located, so that the target action finally generated can be more in line with the environment and atmosphere in which the target virtual character is located, and the precision and accuracy of the generated target action video can be improved.

[0117] In addition, it is understandable that although the target text carries richer semantic information than the target audio, the data structure of the text modality is relatively simple, and the dimensions of the features that can be represented are fewer than those of the audio modality data. For example, the data of the text modality cannot represent the features of dimensions such as rhythm and tempo, and the features of these dimensions can also reflect the role attributes of the target virtual character in certain aspects. For example, the rhythm, tempo and loudness of the audio can often reflect the emotions of the target virtual character when speaking the target audio. These role attributes will also affect the target actions that need to be generated. For example, when the target virtual character speaks the target audio in a relatively low mood, the corresponding body movements will be relatively few and the amplitude of the movements will be small; conversely, when the target virtual character speaks the target audio in a relatively high mood, the corresponding body movements will be richer and the amplitude of the movements will be relatively large. Based on this, in this embodiment, referring to Figure 3 , the target audio and target text can be determined together as the target guidance content and input into the large language model. The process can be expressed as:

[0118] in, represents a semantic label, Indicates the target audio, represents the target text, Indicates the preset prompt text.

[0119] In addition, the target audio itself may also have certain background sounds or environmental sounds, such as rain or waves. At this time, by taking the target audio and the target text as the input of the large language model, the large language model can also identify the environmental atmosphere of the target virtual character by identifying the background sounds or environmental sounds in the target audio, and generate semantic labels to describe the environmental atmosphere.

[0120] In one possible implementation, obtaining the target text associated with the target audio includes: performing audio recognition on the target audio to obtain the target text of the target audio; or obtaining text associated with the target virtual character input through an input operation as the target text of the target audio.

[0121] Based on this, when the input data does not contain the target text, the texts corresponding to the multiple character audios of the target virtual character can be automatically extracted as the target text by performing speech recognition on the character audio of the target virtual character, so that the target text can contain the semantic information carried in the multiple character audios of the target virtual character; at the same time, when there is a target text directly input through an input operation, the input text is directly obtained as the target text without the need to recognize the character audio of the target virtual character.

[0122] Specifically, there are various ways to obtain the target text. In some scenarios, the user directly inputs the target audio without inputting the text corresponding to the target audio. For example, in scenarios such as video creation or live broadcast, the target audio is the voice input by the user in real time. At this time, only the character audio associated with the target virtual character can be obtained. Therefore, it is necessary to perform voice recognition on each character audio of the target virtual character to obtain the audio text corresponding to each character audio, and construct the audio text into the target text according to certain conditions. For example, the audio text corresponding to all character audio associated with the target virtual character can be used as the target text, or the audio text corresponding to the character audio obtained within a specified time range can be used as the target text. For example, the audio text corresponding to the character audio obtained within 24 hours is used as the target text. Specifically, refer to Figure 5 After obtaining the input real-time voice each time, the obtained real-time voice can be recognized to obtain voice text, and then the voice text is added to a target text pool corresponding to the target virtual character as the target text.

[0123] When the user inputs the text through the input operation, the text input by the user is directly obtained as the target text. For example, in the scenario where non-player character modeling is required, such as the metaverse and games, the user often pre-sets the dialogue content of each non-player character, and then generates the target audio by dubbing the non-player character according to the target text. The user often records the preset dialogue content through text modal data, and these texts recording the dialogue content of the non-player character can be the target text. At this time, the target text for input through the input operation can be obtained, and the target text at this time is the original data for setting the dialogue content of the target virtual character. Compared with the target text obtained by speech recognition of the target audio, it will not be affected by audio noise or speech misrecognition. Compared with the target text obtained by speech recognition, it can more accurately represent the dialogue content of the target virtual character.

[0124] Step 203, using a diffusion model to denoise the preset noise features according to the target audio features and the semantic label features, to obtain a target action sequence;

[0125] In step 203, the preset noise feature can be a feature matrix composed entirely of random Gaussian noise, and the row dimension of the feature matrix is ​​consistent with the row dimension of the target audio feature. For example, the target audio feature is an M×N audio feature matrix, and the corresponding preset noise feature is an M×K matrix, wherein the values ​​of K and N are independent of each other. Thus, by denoising the preset noise feature, the row dimension of the target action sequence finally obtained can be consistent with the row dimension of the target audio feature, and each matrix row of the target action sequence can correspond one-to-one to the matrix row in the target audio feature, and the generated target action sequence and the input target audio feature can be consistent with each other. The frequency can be aligned in the time dimension, that is, each audio frame of the target audio has a corresponding action frame; the column dimension of the preset noise feature is consistent with the column dimension of the target action sequence to be generated, and the column dimension can be preset; the target action sequence includes the action data corresponding to multiple action frames of the target virtual character in the target action video to be generated, and the target action sequence is a feature matrix composed of action data corresponding to multiple action frames in the target action video corresponding to the continuous limb movements of the target virtual character when the target virtual character makes continuous limb movements corresponding to the target audio, and multiple action frames form the coherent movements of the target virtual character. Among them, these action data are data for controlling the actions of the target virtual character, for example, they can be kinematic features such as the spatial coordinates of each preset bone point of the target virtual character, the coordinates of each preset joint point of the target virtual character, the rotation angle of the preset joint point, etc. In the target action sequence, refer to Figure 4, each matrix row represents all action data corresponding to an action frame, and a single matrix column in the target action sequence can be used to represent the coordinates of a single preset bone point in each action frame or the rotation angle of a single preset joint point in each action frame. Based on these action data, the action of the target virtual character in each action frame can be constructed according to the target action sequence. Exemplarily, in the generated target action sequence, the elements of the fifth matrix row can be all action data corresponding to the fifth action frame, and the first 100 matrix columns are set to represent the spatial coordinates of each preset bone point respectively, and the spatial coordinates of each preset bone point are three-dimensional coordinates. When a preset bone point needs to use three elements in sequence as the x, y, and z coordinates of the preset bone point, then the elements of the 100th matrix column in the target action sequence are respectively used to represent the x-axis coordinates of the 34th preset bone point of the target virtual character in each action frame.

[0126] In addition, it is also possible to obtain an original action sequence, which includes action data of multiple action frames. The original action sequence can be obtained by obtaining a sample action video, sampling multiple action frames from the sample action video, and extracting the action data of the action frames. Then, a mask operation is performed on the action data at a specific position in the original action sequence to obtain a preset noise feature.

[0127] Moreover, the diffusion model outputs the target action sequence at one time through denoising, that is, it outputs the action data of multiple action frames at one time, and does not need to use autoregression to perform multiple rounds of denoising (that is, each round outputs the action data of an action frame, and then performs denoising for the next round based on the action data output in the current round). Therefore, it can effectively improve the denoising efficiency of the diffusion model, thereby improving the generation efficiency of the target action video.

[0128] It should be noted that the column dimensions of the preset noise features and the target action sequence may be inconsistent with the column dimensions of the semantic label features and the target audio features. It is understandable that in the preset noise features and the target action sequence, each matrix column is used to represent the coordinates of a preset bone point or the coordinates of a preset joint point or the rotation angle. The number of column dimensions of the preset noise features and the target action sequence actually depends on the number of key points such as preset bone points and preset joint points used to represent the action of the target virtual character, which makes the column dimensions of the preset noise features and the target action sequence often larger than the column dimensions of the semantic label features and the target audio features. At this time, in the process of diffusion processing, the preset noise features can be first linearly mapped through a linear layer, and the column dimensions are adjusted to be consistent with the column dimensions of the target audio features and the semantic label features before diffusion processing is performed. The target action sequence can also be obtained by linearly mapping the denoising result of the diffusion model through a linear layer. Exemplarily, the column dimensions of the preset noise feature and the target action sequence are both 1141, that is, the sequence to be noised and the intermediate noised sequence at each time step will also be processed into a matrix with a column dimension of 1141, while the column dimension of the semantic label feature and the target audio feature is 256. At this time, at each time step, the sequence to be noised at the current time step can be linearly mapped through a linear mapping layer and then input into the diffusion model for denoising. The linear mapping layer can be treated as a linear mapping matrix with a shape of 1141×256. The denoising sequence is linearly mapped to change the column dimension of the sequence to be denoised from 1141 to 256; at the same time, at each time step, the noise predicted by the diffusion model is a matrix with a column dimension of 256. After obtaining the noise, the noise predicted by the diffusion model is linearly mapped through a linear mapping matrix with a shape of 256×1141, and the column dimension of the predicted noise is changed from 256 to 1141. The noise is then removed from the sequence to be denoised to obtain the intermediate noise sequence of the current time step, which is used as the sequence to be denoised in the next time step.

[0129] In this embodiment, the target audio features and semantic label features are input into the diffusion model together as guiding features. Specifically, the target audio features and semantic label features can be spliced ​​or weighted summed to fuse the target audio features and semantic label features, and the fusion result is input into the diffusion model as a guiding feature. It should be noted that since the target audio features contain feature vectors corresponding to multiple audio frames, and the number of audio frames depends on the length of the target audio, when the length of the target audio is different, the shape of the target audio matrix will also be different. Based on this, in this embodiment, if the target audio features and semantic label features are mixed by weighted summation, it is necessary to transform the dimension of the semantic label features before mixing, so as to expand the semantic label features into a feature matrix with a consistent tensor shape with the target audio features; or, before processing the target audio, first divide the target audio into sub-audios of a specific length so that the audio features obtained based on each sub-audio have a consistent tensor shape with the semantic label features.

[0130] After the target audio features and semantic label features are input into the diffusion model, the diffusion model predicts multiple noises based on the input label semantic features and target audio features, and then gradually subtracts the predicted multiple noises from the preset noise features according to certain rules, so as to gradually convert the preset noise features composed of Gaussian noise into a target action sequence corresponding to the input target audio and target guidance content. In this embodiment, by inputting the semantic label features and target audio features used to describe the target virtual character into the diffusion model together, and injecting the information contained in each semantic label into the processing process of the diffusion model, the target action sequence finally generated by the diffusion model can be more consistent with the role attributes of the target virtual character, thereby improving the precision and accuracy of the target action sequence.

[0131] In a possible implementation, a diffusion model is used to denoise preset noise features according to target audio features and semantic label features to obtain a target action sequence, including: obtaining a preset time step maximum value; determining the time step maximum value as the current time step, and determining the preset noise feature as the sequence to be denoised; using the diffusion model to denoise the sequence to be denoised according to the current time step, the target audio features and the semantic label features to obtain an intermediate denoised sequence; decreasing the current time step based on a preset interval, determining the intermediate denoised sequence as the sequence to be denoised, and again using the diffusion model to denoise the sequence to be denoised according to the decreased current time step, the target audio features and the semantic label features to obtain an intermediate denoised sequence, until the current time step returns to zero, and determining the intermediate denoised sequence when the current time step returns to zero as the target action sequence.

[0132] Based on this, the preset noise features are denoised in multiple time steps in sequence according to the preset time step maximum value, and the noise that needs to be removed from the preset noise features in the current time step is predicted in each time step according to the current time step, the target audio features and the semantic label features. By removing the noise predicted according to the target audio features and the semantic label features from the preset noise features at each time step, the action details corresponding to the target audio features and the semantic label features are added to the preset noise features, and the preset noise features are gradually converted into a target action sequence corresponding to the target audio features and the semantic label features.

[0133] Specifically, the maximum time step value is a preset value, which is used to characterize the number of times the preset noise feature needs to be denoised during the processing of the diffusion model. The sequence to be denoised refers to the sequence that needs to be denoised at each time step, and the intermediate denoised sequence is the result of the diffusion model denoising the corresponding sequence to be denoised at each time step. It is not difficult to understand that in the diffusion model, the final output is generated by gradually denoising the preset noise features. Therefore, the denoising result of each time step is actually the sequence that needs to be denoised in the next time step.

[0134] It is understandable that when training the diffusion model, both the denoising process and the denoising process are divided into multiple time steps. In the denoising process, a random noise is sampled at each time step and gradually added to the given original action sequence, thereby gradually converting the original action sequence into a noise sequence; in the denoising process, the reverse process of the denoising process is learned, that is, how to predict the noise added to the original action sequence at each time step of the denoising process according to the given input, and gradually remove the predicted noise from the noise sequence, thereby restoring the noise sequence to the original action sequence. It is not difficult to understand that in order to ensure that the denoising process and the denoising process are mutually inverse, when denoising, the noise added at the last time step of the denoising process should be removed from the noise sequence first, and the noise sequence should be restored to the state before the noise corresponding to the last time step is added, and then the noise added at the penultimate time step of the denoising process is removed, and so on, so that the noise sequence is restored to the original action sequence by gradually denoising the noise sequence. Therefore, the time step in the denoising process should be decreasing from the maximum time step to zero. Based on this, in this embodiment, the maximum value of the time step is first used as the first current time step, and the preset sound feature is used as the sequence to be denoised corresponding to the first current time step; then, in each current time step, the noise corresponding to the current time step is predicted according to the target audio feature and the semantic label feature, and the noise predicted by the current time step is removed from the sequence to be denoised of the current time step to obtain the denoising result of the current time step, that is, the intermediate denoising sequence, and then the intermediate denoising sequence of the current time step is determined as a new sequence to be denoised, and the current time step is decremented according to the preset interval, that is, the current time step is decremented by 1, and then the denoising process of the next current time step is started, and the process is repeated until the current time step is decremented to 0. In this way, the preset noise feature can be gradually denoised according to the denoising method learned by the diffusion model during the training process, and a target action sequence corresponding to the target audio feature and the semantic label feature can be generated.

[0135] Specifically, the denoising process can be expressed as: ,in, represents the denoising result of the n+1th time step, represents the denoising result of the nth time step, represents the semantic label feature, represents the target audio feature, According to the semantic label features and target audio features right The denoising process is performed.

[0136] In a possible implementation, a diffusion model is used to denoise a sequence to be denoised according to a current time step, target audio features and semantic label features to obtain an intermediate denoised sequence, including: obtaining initial action data, convolving the initial action data to obtain initial action features; and using a diffusion model to denoise a sequence to be denoised according to a current time step, initial action features, target audio features and semantic label features to obtain an intermediate denoised sequence.

[0137] Based on this, by obtaining the initial action data corresponding to the target virtual character, the initial action features obtained based on the initial action data, the target audio features and the semantic label features are used together as the input of the diffusion model, and the initial action features are used to provide an initial action state for the target action sequence, so as to constrain that each action frame in the target action sequence is obtained after the evolution of several action frames starting from the initial action state. The initial action data is used as an anchor point to establish the connection between the action frames in the target action sequence, which can improve the coherence between the action frames in the generated target action sequence and improve the fluency of the generated target action video.

[0138] It can be understood that since the initial action feature indicates the initial action of the target action sequence to be generated, it constrains the corresponding parts of each action frame in the target action sequence to be generated to be obtained after several evolutions starting from the same initial action, thereby further strengthening the connection between each action frame in the target action sequence. Based on this, by performing noise prediction together with the initial action feature, the target audio feature and the semantic label feature, and removing the predicted noise from the preset noise feature, the coherence between the action data of each action frame contained in the generated target action sequence can be improved, and the smoothness of the generated target action video can be improved.

[0139] Specifically, the initial action data is used to characterize the starting action of the target virtual character when generating the target action. It can be composed of action data corresponding to several preset action frames. For example, in one embodiment, the initial action data can be composed of action data corresponding to 8 action frames. The action data of each action frame is an N-dimensional vector, and the initial action data is an 8×N matrix. It should be noted that the format of the action data corresponding to each action frame in the initial action data is consistent with the format of the action data corresponding to each action frame in the target action sequence to be generated, that is, in the target action sequence to be generated and the initial action data, the same matrix column is used to characterize the same kinematic feature data. For example, in the target action sequence to be generated, the 100th matrix column is used to characterize the x-axis coordinate of the 34th preset bone point of the target virtual character, then the 100th matrix column in the initial action data is also used to characterize the x-axis coordinate of the 34th preset bone point of the target virtual character.

[0140] It can be understood that the initial motion data is also used to characterize the motion data of the target virtual character in several motion frames. Similar to the preset noise features and the target motion sequence, each matrix column in the initial motion data is also used to characterize the three-dimensional coordinates of each preset bone point, the coordinates of the preset joint points, and the rotation angle and other data. This makes the column dimensions of the initial motion data often inconsistent with the column dimensions of the target audio features and the semantic label features. Therefore, it is impossible to directly splice the initial motion data with the target audio features and the semantic label features. In this embodiment, refer to Figure 6 , the initial action data can be convolved through an action encoder composed of several convolutional layers to convert the initial action data into a form suitable for input into the diffusion model together with the target audio features and semantic label features to obtain the initial action features. Figure 7 , the initial action features, target audio features and semantic label features are input into the diffusion model together to perform noise prediction, predict the corresponding noise features to be removed, and thus denoise the preset noise features.

[0141] In one possible implementation, refer to Figure 3 , the diffusion model includes a local attention mechanism, a global attention mechanism and a linear layer connected in sequence. The diffusion model is used to denoise the sequence to be denoised according to the current time step, the initial action feature, the target audio feature and the semantic label feature to obtain an intermediate denoised sequence, including: inputting the current time step, the initial action feature, the target audio feature and the semantic label feature into the diffusion model; fusing the initial action feature, the target audio feature, the sequence to be denoised and the semantic label feature to obtain a first fused feature; performing a local attention operation on the first fused feature according to a preset attention window size; fusing the initial action feature, the semantic label feature and the result of the local attention operation to obtain a second fused feature, performing a self-attention operation on the second fused feature, and performing a linear mapping on the result of the self-attention operation to obtain a noise feature to be removed corresponding to the current time step, wherein the noise feature to be removed is a tensor with the same shape as the sequence to be denoised; obtaining a first noise coefficient corresponding to the current time step, and subtracting the noise feature to be removed from the sequence to be denoised according to the first noise coefficient to obtain an intermediate denoised sequence.

[0142] Based on this, local attention calculation is performed on the input initial action features, target audio features, sequences to be denoised, and various semantic label features, so as to better integrate the features of these different dimensions. Then, the initial action features and semantic label features are integrated into the results of the local attention calculation to further strengthen the intervention of the initial action features and semantic label features on the diffusion process, so that the generated target action sequence is more consistent with the role attributes of the target virtual character represented by the semantic label, and at the same time, each action frame can have stronger coherence.

[0143] In one embodiment, the initial action features, the target audio features, the sequence to be denoised and the semantic label features can be fused by sequentially splicing the initial action features, the target audio features, the sequence to be denoised and the semantic label features. The matrix obtained after splicing the initial action features, the target audio features, the sequence to be denoised and the semantic label features is the first fused feature. It should be noted that, referring to the relevant description of the above embodiment, it can be known that the column dimension of the sequence to be noisy is often different from the column dimension of features such as the initial action feature, the target audio feature, and each semantic label feature. Therefore, before concatenating the sequence to be noisy with the initial action feature, the target audio feature, and each semantic label feature, it is necessary to linearly map the sequence to be noisy to map the column dimension of the sequence to be noisy into a style consistent with the column dimension of the initial action feature, the target audio feature, and each semantic label feature. For example, the column dimension of the sequence to be noisy is 1141, while the column dimension of the initial action feature, the target audio feature, and each semantic label feature is 256. At this time, it is necessary to reduce the column dimension of the sequence to be noisy from 1141 to 256 through a linear mapping matrix with a shape of 1141×256.

[0144] In another possible implementation, the initial action features, target audio features, the sequence to be denoised, and each semantic label feature can be fused by weighted summing them. It can be understood that when using the initial action features, target audio features, and each semantic label feature to guide the diffusion model to predict noise, the impact of these features on the final generated action is different in different situations. In this embodiment, the initial action features, target audio features, and each semantic label feature can be scored separately by a preset scoring model, and then the scores of the initial action features, target audio features, and each semantic label feature are normalized as corresponding weights, and then weighted summed according to the weights, so that the initial action features, target audio features, and each semantic label feature are fused into a fused feature.

[0145] After obtaining the first fusion feature, a local attention calculation is performed on the first fusion feature. The size of the attention window during the local attention calculation is preset. The attention window indicates the positions of other words that each word in the first fusion feature needs to pay attention to. By using the local attention mechanism, each word in the first fusion feature can pay attention to a specific part of the word. It can be understood that in the embodiment, the first fusion feature is obtained by fusing the initial action feature, the target audio feature, the sequence to be denoised, and the semantic label feature. The attention calculation of the first fusion feature can actually be regarded as a cross-attention calculation between the initial action feature, the target audio feature, the sequence to be denoised, and the semantic label feature. By performing local attention on the first fusion feature, these features can be better fused to obtain a better feature representation. In order to more accurately predict noise.

[0146] Specifically, when performing local attention calculations, the first fused feature is first multiplied with the query weight matrix, key weight matrix, and value weight matrix of the local attention mechanism to obtain the corresponding query matrix, key matrix, and value matrix. It can be understood that the query weight matrix, key weight matrix, and value weight matrix are used as parameters of the diffusion model and are determined during the training of the diffusion model; then, the query matrix and the transposed key matrix are matrix multiplied to obtain the corresponding attention score matrix, and at the same time, the attention score matrix is ​​masked according to the preset attention window size, and the attention scores of the masked positions are all set to negative infinity. Then, the attention score matrix is ​​exponentially normalized to obtain the attention weight matrix, and the value matrix is ​​weighted based on the attention weight matrix to obtain the result of the local attention operation. Specifically, the process of local attention processing can refer to the following formula:

[0147]

[0148]

[0149]

[0150]

[0151] in, represents the first fusion feature, , , The query weight matrix, key weight matrix and value weight matrix are represents the mask matrix determined according to the preset attention window size, Represents the dimension of the query matrix.

[0152] After obtaining the result of the local attention operation, the result of the local attention operation is fused with the initial action feature and the semantic label feature. Specifically, the initial action feature and the semantic label feature can be spliced ​​to the result of the local attention operation to obtain a second fused feature. Based on this, the information of the initial action feature and the semantic label feature can be injected into the diffusion processing process again, further strengthening the intervention of the initial action feature and the semantic label feature in the denoising process, so that the target action sequence finally generated by the diffusion model can more clearly reflect the role attributes of the target virtual character, and further improve the coherence between the action frames in the generated target action sequence.

[0153] After the initial action features and semantic label features are fused to the result of the local attention calculation to obtain the second fused feature, the second fused feature is subjected to self-attention calculation, and the required features for predicting noise are extracted from the second fused feature by means of self-attention calculation. The process of self-attention calculation can refer to the process of local attention processing in the above-mentioned embodiment, the difference being that the query weight matrix, key weight matrix and value weight matrix in the self-attention calculation process are independent of the query weight matrix, key weight matrix and value weight matrix of the local attention mechanism, and in the process of self-attention calculation, there is no need to generate a mask matrix according to the preset attention window size to mask the attention score matrix; the result of the self-attention calculation is then linearly mapped to predict the required noise features to be removed. Specifically, the process of predicting noise can be expressed as:

[0154]

[0155] in, represents the predicted noise feature to be removed, represents the linear mapping matrix, represents the self-attention mechanism, represents the local attention mechanism, [ ] represents matrix concatenation, represents the preset noise characteristics, represents the target audio feature, represents a random mask, represents the initial action feature, Represents semantic label features.

[0156] Furthermore, in one embodiment, referring to Figure 8Before performing local attention calculation on the first fused feature and self-attention calculation on the second fused feature, a position coding matrix for representing the position information of each feature in the first fused feature and the second fused feature can be generated, and then the position coding matrix is ​​superimposed on the first fused feature and the second fused feature, respectively, so as to inject position information into the first fused feature and the second fused feature. Specifically, the position coding matrix can be generated by relative position coding or rotation position coding.

[0157] In one embodiment, referring to Fig. 9 Before performing local attention processing on the first fusion feature, the serial number of the current time step can also be encoded into a corresponding time step feature, and the time step feature is superimposed on the first fusion matrix, so that the information used to characterize the current time step is injected into the diffusion processing process, so that the diffusion model can perceive which time step is currently being executed according to the time step feature, and the first fusion feature of each time step in the diffusion process changes relatively obviously through the time step feature.

[0158] In a possible implementation, subtracting a noise feature to be removed from a sequence to be denoised according to a first noise coefficient to obtain an intermediate denoised sequence, including: obtaining a second noise coefficient corresponding to each time step less than a current time step, wherein the time step and the corresponding second noise coefficient are negatively correlated, and the second noise coefficient is greater than the first noise coefficient; obtaining a cumulative noise coefficient according to the product of the first noise coefficient and a plurality of second noise coefficients, determining a first difference between a reference value and the cumulative noise coefficient, and determining a square root of the first difference as a first square root value; determining a second difference between the reference value and the first noise coefficient; determining a first quotient of the second difference and the first square root value, and obtaining a scaled noise feature to be removed according to the product of the first quotient and the noise feature to be removed; determining a third difference between the sequence to be denoised and the scaled noise feature to be removed, and determining a first product of the third difference and the inverse of the square root of the first noise coefficient; determining a second product of the first noise sequence and the second difference, and determining the sum of the first product and the second product as the intermediate denoised sequence, wherein the first noise sequence is obtained by sampling from a standard normal distribution, and the shape of the first noise sequence is the same as that of the sequence to be denoised.

[0159] Based on this, in the denoising process, after predicting the corresponding noise features to be removed at each time step, the first noise coefficient and the second noise coefficient corresponding to the time step are used to weight the noise features to be removed, and then denoising is performed based on the weighted noise features to be removed, so that the denoising process can be the inverse process of the denoising process in the training process, which is in line with the denoising paradigm learned by the diffusion model during the training process.

[0160] Specifically, the training process of the diffusion model includes a denoising process and a denoising process. During the training process, in order to ensure the smoothness and controllability of the denoising process, in the denoising process, the sampled random noise is weighted based on the first noise coefficient corresponding to the time step at each time step, and then added to the given original action sequence. The specific denoising process will be described in detail in the subsequent description of the training process of the diffusion model, and will not be repeated here. The denoising process and the denoising process are mutually inverse processes. Therefore, when training the diffusion model, each time step in the denoising process also needs to weight the predicted noise to be removed based on the first noise coefficient corresponding to each time step, and then perform denoising based on the weighted result to ensure the mutual inverse of the denoising process and the denoising process. After the training is completed, the diffusion model learns the denoising process based on the first noise coefficient corresponding to each time step. Therefore, in this embodiment, it is necessary to first obtain the second noise coefficient corresponding to each time step that is smaller than the current time step, and then weight the noise features to be removed based on each second noise coefficient and the first noise coefficient corresponding to the current time step, and then perform denoising based on the weighted noise to be removed.

[0161] Specifically, in this embodiment, the second noise coefficient refers to the noise coefficient corresponding to the time step whose sequence number is less than the sequence number of the current time step. For example, for the time step with sequence number t, the noise coefficients corresponding to the time steps with sequence numbers 1 to t-1 are all the second noise coefficients. It should be noted that the first noise coefficient corresponding to each time step is negatively correlated with the sequence number of the time step, that is, as the sequence number of the time step increases, the corresponding noise coefficient decreases.

[0162] After obtaining each second noise coefficient, each second noise coefficient and the first noise coefficient corresponding to the current time step are multiplied to obtain a cumulative noise coefficient, which can be expressed as: in represents the cumulative noise coefficient corresponding to the t-th time step, represents the first noise coefficient or the second noise coefficient corresponding to the i-th time step. Then, the first difference value of the base value minus the cumulative noise coefficient is determined, and the square root of the first difference value is determined. In this embodiment, the base value may be 1. Then, the second difference value obtained by subtracting the first noise coefficient corresponding to the current time step from the base value is calculated, that is, , then the second difference is divided by the first square root to obtain a first quotient value, which is the coefficient for weighting the noise feature to be removed, and the first quotient value is multiplied by the noise feature to be removed to obtain the weighted noise feature to be removed, that is, the scaled noise feature to be removed. It can be understood that the noise feature to be removed is a matrix with the same shape as the sequence to be denoised, and the result obtained by multiplying the matrix with a scalar is still a matrix. Then, the sequence to be denoised and the noise feature to be removed are subtracted, and the noise feature to be removed is removed from the sequence to be denoised, and the subtraction result is multiplied by the inverse of the square root of the first noise coefficient to obtain a first product. It can be understood that the first product is still a matrix with the same shape as the sequence to be denoised.

[0163] In addition, if the first product is directly used as the intermediate denoising sequence, the result of each denoising step is completely fixed, which is not conducive to the diversity of the diffusion model when generating actions, and is prone to overfitting in the process of training the diffusion model. Based on this, in this embodiment, an additional first noise sequence is introduced, and the first noise sequence is obtained by random sampling from a standard normal distribution. The first noise sequence is a matrix with the same shape as the sequence to be denoised. In this embodiment, the first noise sequence is weighted based on the second difference, and the weighted first noise sequence is added to the first product as the intermediate denoising sequence of the current time step. By introducing additional random noise in the denoising process, the diversity of the action sequence generated by the diffusion model is improved.

[0164] Specifically, the process of subtracting the noise feature to be removed from the sequence to be denoised according to the first noise coefficient to obtain the intermediate denoised sequence can be expressed as:

[0165]

[0166] in, represents the intermediate denoising sequence corresponding to the t-th time step, represents the first noise coefficient corresponding to the t-th time step, represents the denoised sequence corresponding to the t-th time step, represents the noise feature to be removed in the prediction of the tth time step, is the second difference, represents the first noise sequence.

[0167] Step 204: Generate a target action video of the target virtual character based on the target action sequence.

[0168] The target action video is a video of the physical movements that the target virtual character needs to make when speaking the target audio. For example, when the target virtual character says "turn right", the corresponding target action video is a video of the target virtual character raising his right hand and pointing his index finger to the right. After obtaining the target action sequence, the target action sequence indicates the action data of the target virtual character in a plurality of consecutive action frames. These action data can respectively represent the kinematic feature data of the target virtual character when making the corresponding action, for example, the coordinates of each preset bone point of the target virtual character in each action frame, the joint rotation angle, etc. The action posture of the target virtual character in each action frame can be determined through these action data. Rendering based on the target action sequence can generate a picture of the target virtual character making the target action represented by the target action sequence, thereby obtaining the corresponding target action video.

[0169] In the embodiment disclosed in steps 201 to 204, by acquiring the target audio associated with the target virtual character and the target guidance content associated with the target audio, the target guidance content and the preset prompt text are input into the large language model together, and the semantic understanding ability of the large language model is used to extract the semantic tags that can be used to describe the target virtual character from the target guidance content, and the semantic tags are encoded into semantic tag features, which can characterize the role attributes of the target virtual character itself. When the preset noise feature is used for action generation using the diffusion model, not only the corresponding target audio is used, but also the preset noise feature is further denoised in combination with these semantic tag features that can characterize the role attributes of the target virtual character, because the semantic tag features used to characterize the role attributes of the target virtual character are additionally injected in the process of denoising the preset noise feature using the diffusion model, the denoising process of the diffusion model is not only affected by the target audio features, but also by these semantic tags used to describe the role attributes of the target virtual character, which can make the target action sequence finally generated more accurately fit the role attributes of the target virtual character, thereby improving the accuracy and precision of the generated action data, and then improving the quality of the generated action video.

[0170] In a possible implementation, the training process of the diffusion model includes: obtaining a sample action sequence of a sample virtual character, a sample audio associated with the sample virtual character, and a sample guide content associated with the sample audio, wherein the sample action sequence includes action data of the sample virtual character in multiple action frames; encoding the sample audio into a sample audio feature, inputting the sample guide content and a preset prompt word into a large language model for text prediction, and generating a sample label for describing the sample virtual character; adding noise to the sample action sequence in multiple time steps in sequence until the time step reaches a time step maximum value, obtaining a sample noise sequence, and determining the noise feature added to the sample action sequence in each time step as a reference noise feature corresponding to the time step; encoding the sample label into a sample label feature, denoising the sample noise sequence in each time step using the diffusion model according to the sample audio feature and the sample label feature, and determining a predicted noise feature predicted in each time step, wherein the predicted noise feature is the noise removed from the sample noise sequence in the denoising process of the time step; determining a first loss function value based on the difference between the reference noise feature and the predicted noise feature corresponding to each time step, and performing back propagation based on the first loss function value to optimize the parameters of the diffusion model.

[0171] Based on this, the sample action sequence is denoised and the reference noise features added at each time step during the denoising process are recorded. Then, the sample noise sequence obtained after the multi-step denoising process is denoised in multiple steps, and the predicted noise features predicted by the diffusion model according to the sample audio features and the sample label features in each time step are recorded. The first loss function value is calculated according to the reference noise features and the predicted noise features corresponding to each time step, and the diffusion model is trained based on the first loss function value. In this way, the diffusion model can learn how to predict the predicted noise features that are close to the real reference noise features added during the denoising process according to the given sample label features and sample audio features, thereby learning how to generate the corresponding action sequence in random noise according to the label features and the audio features.

[0172] Specifically, the corresponding sample action sequence, sample audio, and sample guidance content can be represented as a tuple ( ), so as to construct a training data set. In each tuple, represents a sample action sequence, a sample action sequence It is used to represent the body movements of the sample virtual character when speaking the sample audio A. Represents sample audio, It is sample guide content, sample guide content It may consist of dialogue text or dialogue audio associated with a sample virtual character. Indicates the maximum time step. Later, when training the diffusion model, a tuple is randomly selected from the data set to obtain the corresponding sample action sequence, sample audio, and sample guidance content.

[0173] After obtaining the sample action sequence, sample audio and sample guidance content, the sample action sequence is first subjected to noise processing for multiple time steps. Fig.10 In the first time step of the noise addition process, random sampling is first performed from the Gaussian distribution to obtain the reference noise feature corresponding to the first time step It can be understood that the reference noise characteristic is a sequence of sample actions A matrix with the same shape, which will be referenced later on as the noise feature Add to sample action sequence In this way, the noise addition process of the first time step is completed, and the reference noise feature is added The sample action sequence after can be recorded as ; Then increment the current time step and randomly sample from the Gaussian distribution to obtain the reference noise feature corresponding to the second time step , the reference noise characteristic Noised result added to the first time step The noise result of the second time step is obtained , repeat the process until the current time step reaches the maximum time step , get the maximum time step The corresponding noise addition result , then we get the sample noise sequence, refer to Fig.10 In the noise adding process, the noise added to the sample action sequence at the tth time step will be determined as the reference noise feature corresponding to the time step .

[0174] While the sample action sequence is being denoised, the sample audio is input into a trained audio coding model for encoding to obtain sample audio features. At the same time, the sample guide content and the preset prompt text are input into a large language model, so that the large language model generates sample labels for describing the role attributes of the sample virtual character by analyzing the dialogue text or dialogue audio in the sample guide content, and encodes the sample labels into sample label features. This process can refer to the relevant steps of generating semantic labels for describing the target virtual character in the above step 202, which will not be elaborated here.

[0175] After obtaining the sample noise sequence, sample audio features, and sample label features, refer to Fig.10 , the sample noise sequence As the object of diffusion model denoising, the diffusion model is used to predict the predicted noise characteristics of each current time step based on the sample guidance content and sample audio, and the sample noise sequence is subjected to multi-step denoising. Specifically, refer to Fig.10 The time step of the denoising process is transferred from the denoising process. In the denoising process, the current time step is incremented to the maximum time step value, and then the denoising process of the sample action sequence is considered to be completed and the denoising of the sample action sequence is stopped. Therefore, in the denoising process, the denoising is started with the current time step with the maximum time step value as the starting point. By reusing the time step between the denoising process and the denoising process, on the one hand, it can ensure that the number of denoising steps is equal to the number of denoising steps, so that the multi-step denoising process is equivalent to the reverse process of the multi-step denoising process in form. On the other hand, it can also better correspond each time step of the denoising process to each time step of the denoising process, which is conducive to the diffusion model learning reverse denoising process. Fig.10 When performing denoising for multiple time steps, for the tth time step, the diffusion model generates the corresponding predicted noise features based on the sample speech features and sample label features. , and based on this predicted noise characteristics De-noise the denoising result of the previous time step, then reduce the current time step by 1 and remove the predicted noise feature of the current time step The denoising result obtained later Input to the diffusion model as the sample sequence to be denoised in the next time step, and repeat the process until the time step t decreases to 0. After that, the reference noise feature corresponding to each time step is used and predict noise characteristics The loss function is calculated based on the difference between them, and then back propagation is performed based on the calculated loss function value to optimize the parameters of the diffusion model to minimize the loss function value, so that the diffusion model can learn how to predict the corresponding noise based on the given audio features and label features, and generate an action sequence corresponding to the given audio features and label features in the noise sequence.

[0176] In a possible implementation, a diffusion model is used to denoise the sample noise sequence in each time step according to sample audio features and sample label features, including: for each time step, encoding the action data of the first n action frames in the sample action sequence into sample action features, where n increases as the time step decreases, and n is a positive integer; and a diffusion model is used to denoise the sample noise sequence in each time step according to sample audio features, sample action features, and sample label features.

[0177] Based on this, the action data of the first several action frames are extracted from the sample action sequence as sample action features, and then the sample action features, sample audio features and sample label features are input into the diffusion model together for noise prediction. During the training process, the diffusion model can learn how to combine the given initial action for noise prediction and generate an action sequence with a high degree of coherence with the given initial action.

[0178] Specifically, refer to Fig.11 The sample action feature is obtained by extracting and encoding the action data of the first several action frames in the sample action sequence. For example, if the sample action sequence contains N action frames, the action data of the first 8 action frames in the sample action sequence are intercepted, and the extracted action data is encoded by the action encoder to obtain the sample action feature. The details of the sample action feature itself can refer to the relevant description of the initial action feature in the above steps, which will not be repeated here.

[0179] Since the sample action feature is obtained by intercepting the action data of the first n action frames in the sample action sequence, the sample action feature and each action frame in the sample action sequence to be generated are a series of actions with coherence, and each action frame in the sample action sequence is actually obtained by evolving the action data contained in the sample action feature several times. In this embodiment, by taking the sample action feature, the sample label feature and the sample audio feature as the input of the diffusion model, the diffusion model can learn the connection between each action frame in the action sequence to be generated and the given initial action feature during the training process, because the action data of each action frame in the generated action sequence is obtained by evolving the action data of the same given initial action, so the connection between each action frame in the action sequence to be generated can be established, so that the diffusion model can learn how to generate an action sequence with high coherence between each action frame according to the given initial action, thereby improving the coherence between each action frame in the generated action sequence.

[0180] In one possible implementation, a diffusion model is used to denoise the sample noise sequence in each time step according to sample audio features, sample action features, and sample label features, including: randomly masking the sample label features and the sample action features; and using a diffusion model to denoise the sample noise sequence in each time step according to the sample audio features, the randomly masked sample label features, and the randomly masked sample action features.

[0181] Based on this, by randomly masking the sample label features and sample action features during the training process, the diffusion model can avoid over-reliance on the sample initial action or a certain sample label during the training process, thereby preventing the diffusion model from overfitting during the training process.

[0182] Specifically, refer to Figure 6 During the training process, a random mask module is set at the output end of the action encoder and the semantic label encoder, and the module randomly masks the sample action features and each sample label feature according to the preset mask probability. It should be noted that in the present embodiment, the masking operation is based on the entire sample label feature or sample action feature as the basic unit, that is, in the present embodiment, it is determined whether each sample label feature and sample action feature needs to be masked according to the set mask probability, and then the features that need to be masked are masked as a whole, and the masking method can be to set the values ​​of all elements in the corresponding feature vector to negative infinity. It should be noted that the masked sample label feature or sample action feature is only that all elements in the corresponding feature vector are set to negative infinity. In the subsequent process of predicting noise through the diffusion model, the masked sample label feature or sample action feature still needs to be input into the diffusion model to maintain the consistency of the format of the input of the diffusion model.

[0183] Exemplarily, in this embodiment, the sample label features, sample action features, sample audio features and sample noise sequences are concatenated and input into the diffusion model, wherein, in the matrix obtained after concatenation, the first X matrix rows are the feature vectors corresponding to each semantic label in turn, and during the training process, the random mask module masks the 10th sample semantic label. At this time, in the matrix obtained by concatenating the sample label features, sample action features, sample audio features and sample noise sequences, all the elements of the 10th matrix row are set to negative infinity.

[0184] Referring to the above embodiments, it can be seen that the diffusion model predicts noise by performing attention calculation on the input sample label features, sample action features, sample audio features and sample noise sequences through the attention mechanism; and in the process of attention calculation, it is necessary to multiply the input matrix with the given weight matrix respectively, convert the input matrix into the corresponding query matrix, key matrix and value matrix, and then calculate the result of the matrix multiplication between the query matrix and the transposed key matrix, and perform exponential normalization on the result of the matrix multiplication to obtain the attention weight matrix, and then perform weighted processing on the value matrix according to the attention weight matrix to obtain the result of the attention mechanism. After randomly masking the sample label features and sample action features, in the query matrix and the key matrix, all elements of the feature vectors corresponding to the masked sample label features or sample action features are set to negative infinity. When calculating the attention scores, the attention scores associated with the masked feature vectors are also negative infinity. In the attention weight matrix obtained after exponential normalization of the attention scores, the attention weights corresponding to the masked sample label features and sample action features are 0. At this time, the masked sample label features and sample action features will not have any effect on the output of the diffusion model, that is, the diffusion model is The output is independent of the masked sample label features or sample action features; at this time, if the diffusion model overfits during the training process and over-relies on a certain sample label feature or sample action feature to generate the output, when randomly masking, after the sample label feature or sample action feature is masked, the output quality of the diffusion model will be significantly reduced, resulting in an increase in the loss function. During the subsequent back propagation, the parameters of the diffusion model will be optimized to a large extent, so that the diffusion model reduces its dependence on the masked sample label features or sample action features when predicting noise, thereby preventing the diffusion model from overfitting during the training process.

[0185] In one possible implementation, a first loss function value is determined based on the difference between a reference noise feature and a predicted noise feature corresponding to each time step, including: for each time step, calculating a fourth difference between the predicted noise feature corresponding to the time step and the reference noise feature corresponding to the time step; and determining the first loss function value based on an L2 norm of the fourth difference.

[0186] Based on this, the L2 norm of the difference between the predicted noise feature and the reference noise feature corresponding to each time step is calculated as the loss function. Then, by minimizing the loss function, the diffusion model learns how to predict the noise that needs to be removed to restore the sample noise sequence to the sample action sequence based on the input of the model at each time step.

[0187] It can be understood that in the process of adding noise, a randomly sampled noise is added to the sample action sequence at each time step, thereby converting the sample action sequence into a sample noise sequence. If the sample noise sequence is to be restored to the sample action sequence, the sample noise sequence should be reversely denoised according to the reference noise feature added to the sample action sequence at each time step during the process of adding noise. For example, in the last time step, the reference noise feature is After adding to the sample action sequence, we get the sample noise sequence , then, in the sample noise sequence When denoising, we should first remove the sample noise sequence Remove the reference noise feature added in the last time step , restore the sample noise sequence to the state before the noise addition at the last time step. Similarly, for any time step, the predicted noise features that need to be removed from the sample noise sequence should be consistent with the reference noise features added to the sample action sequence at that time step. This makes the denoising process and the denoising process of the diffusion model inverse processes to each other, so that the diffusion model learns how to predict the corresponding noise based on the input audio features and label features, and converts the noise sequence into an action sequence corresponding to the given audio features and label features.

[0188] Specifically, the predicted noise feature and the reference noise feature are both matrices with the same shape as the given sample action sequence. At this time, the Euclidean norm between the predicted noise feature and the reference noise feature matrices can be calculated as the first loss function value. Specifically, first calculate the difference between the predicted noise feature and the reference noise feature corresponding to the step number. It can be understood that the predicted noise feature and the reference noise feature are matrices with the same shape. Calculating the difference between the two is to subtract the elements in the two one by one. The difference between the predicted noise feature and the reference noise feature is still a matrix. The difference between the two matrices cannot be intuitively determined through the matrix. For two relatively close matrices, each element in the two should be relatively close. Then each element in the difference matrix obtained by subtracting the two matrices should be close to 0, that is, each element in the difference matrix should be close to 0. The sum of the elements should be small. Similarly, for two matrices with large differences, the difference between the elements is large, so the sum of the elements in the difference matrix of the two will also be large. Based on this, the L2 norm of the difference between the predicted noise feature and the reference noise feature can be calculated to determine the difference between the predicted noise feature and the reference noise feature. Specifically, the L2 norm is to calculate the square of each element in the difference matrix of the predicted noise feature and the reference noise feature and sum them. At this time, by calculating the L2 norm, the difference between the predicted noise feature and the reference noise feature can be intuitively judged. Specifically, the value of the first loss function can be expressed as:

[0189]

[0190] in, is the first loss function value, represents the reference noise characteristic, represents the predicted noise characteristics predicted by the diffusion model at the tth time step, represents the intermediate denoising result corresponding to the t-th time step, Represents a sample action sequence The loss function calculated during training, It means calculating the square of L2 norm. It can be understood that the reference noise characteristic here is refers to Corresponding to the reference noise features of the same time step, the first loss function value is calculated once for each time step during the training process.

[0191] In a possible implementation, the sample action sequence is noised in multiple time steps in sequence until the time step reaches a maximum value of the time step to obtain a sample noise sequence, including: setting the current time step to zero and determining the sample action sequence as a sequence to be noised; obtaining a first noise coefficient corresponding to the current time step, sampling a noise feature to be added corresponding to the current time step from a standard normal distribution, adding the noise feature to be added to the sequence to be noised according to the first noise coefficient, and obtaining an intermediate noised sequence corresponding to the current time step; incrementing the current time step based on a preset interval, determining the intermediate noised sequence as the sequence to be noised, updating the first noise coefficient, sampling the noise feature to be added corresponding to the current time step again from a standard normal distribution, and adding the noise feature to be added to the sequence to be noised according to the updated first noise coefficient until the current time step reaches a maximum value of the time step to obtain a sample noise sequence.

[0192] Based on this, the process of adding noise to the sample action sequence is divided into multiple time steps. The noise result of each time step is obtained by adding noise to the noise result of the previous time step at the current time step. The noise result of each time step is only related to the noise result of the previous time step, but has nothing to do with other time steps. In this way, when performing reverse diffusion, the diffusion model only needs to learn how to predict the noise required to transform the sample noise sequence from the state of the current time step to the state of the next time step based on the sample audio features, sample label features and sample action features at each time step, without having to learn how to directly reconstruct the sample noise sequence into a sample action sequence based on the sample audio features, sample label features and sample action features. This simplifies the tasks that the diffusion model needs to learn during the training process, reduces the difficulty of diffusion model training, and improves the stability of the training process.

[0193] Specifically, when performing the noise addition process, the noise addition is started from the 0th time step. In each time step, random sampling is first performed from the Gaussian distribution to obtain the noise feature to be added corresponding to the current time step, wherein the noise feature to be added is a matrix with the same shape as the sequence to be noised. Then, the noise feature to be added is adjusted based on the first noise coefficient corresponding to the current time step, and the adjusted noise feature to be added is added to the sequence to be noised of the current time step, thereby completing the noise addition process of the current time step and obtaining the intermediate noised sequence of the current time step. Repeat this process until the current time step increases to the maximum value of the time step. At this time, since the noise feature to be added composed of random noise has been added to the sample action sequence many times, the sample action sequence no longer carries any meaningful information, and it can be considered that the sample noise sequence has been obtained.

[0194] It should be noted that the first noise coefficient in this embodiment is the same as the first noise coefficient in the step of subtracting the noise feature to be removed from the sequence to be denoised according to the first noise coefficient in the previous text, that is, the first noise coefficient corresponding to a single time step in the denoising process and the denoising process is consistent. This is because in the training process, the denoising process and the denoising process required to be learned by the diffusion model are mutually reverse processes, which requires ensuring the reversibility between the denoising process and the denoising process corresponding to each time step. In the denoising process, each time step adjusts the noise feature to be added according to the first noise coefficient corresponding to the time step and then adds it to the sample action sequence. Correspondingly, in the denoising process, each time step should reversely adjust the predicted noise feature according to the first noise coefficient corresponding to the time step when denoising, and then remove the adjusted predicted noise feature from the sample noise sequence, thereby ensuring the reversibility of the entire denoising process and denoising process in the training process.

[0195] In a possible implementation, the noise feature to be added is added to the sequence to be noisy according to the first noise coefficient to obtain an intermediate noisy sequence corresponding to the current time step, including: weighting the sequence to be noisy according to the square root of the first noise coefficient; weighting the noise feature to be added according to the square root of the difference between the reference value and the first noise coefficient; and superimposing the weighted sequence to be noisy and the weighted noise feature to be added to obtain the intermediate noisy sequence corresponding to the current time step.

[0196] Based on this, the first noise coefficient correlated with the sequence number of the time step is used to weight the sequence to be noisy and the noise to be added respectively, and then the weighted sequence to be noisy and the noise features to be added are superimposed. In this way, during the noisy process, the first noise coefficient can be used to control the noise amplitude of different time steps, and a small noise amplitude is maintained in the early stage of the noisy process to avoid causing more obvious damage to the information contained in the sequence to be noisy at one time; and in the noisy process, as the useful information in the sequence to be noisy becomes less and less, the noise amplitude is gradually increased to ensure that the noisy process can remain relatively smooth and controllable, so that the diffusion model can accurately learn how to generate actions corresponding to given label features and audio features through step-by-step denoising during the training process.

[0197] Specifically, in this embodiment, the first noise coefficient and the sequence number of the time step are negatively correlated, and in the process of adding noise, the time step gradually increases from zero to the maximum value of the time step. In each time step, the noise feature to be added is weighted based on the square root of the difference between the baseline value and the first noise coefficient. In the early stage of the process of adding noise, the time step is relatively small, and the corresponding first noise coefficient is larger. The difference obtained by subtracting the first noise coefficient from the baseline value is smaller. Therefore, in the early stage of the process of adding noise, the square root of the difference between the baseline value and the first noise coefficient is used to weight the noise feature to be added, which is equivalent to using a smaller The noise is added with a noise adding step, which can avoid adding too much noise to the sample action sequence in the early stage of the noise adding process, resulting in an unsmooth noise adding process; and as the time step increases, the corresponding first noise coefficient decreases, and the difference between the reference value and the first noise coefficient becomes larger and larger, which makes the noise adding amplitude gradually increase in the noise adding process. Therefore, even if more and more noise is added as the noise adding process progresses, the information carried in the intermediate noise adding sequence at this time is relatively small, but by continuously increasing the noise adding amplitude in the noise adding process, the information carried by the intermediate noise adding sequence can still be destroyed in the later stage of noise adding. Based on this, in this embodiment, at the beginning of the noise adding process, the intermediate noise adding sequence carries more effective information, and a smaller amplitude is used for noise adding to avoid removing too much effective information from the intermediate noise adding sequence at one time. As the noise adding process progresses, the effective information carried in the intermediate noise adding sequence becomes less and less, and the noise adding amplitude is gradually increased to ensure that the effective information in the intermediate noise adding sequence can still be effectively removed. Based on this, in the noise adding process, after the intermediate noise sequence is noised at each time step, the amount of effective information that can be removed is relatively balanced, and the entire noise adding process becomes relatively smooth and controllable, which is conducive to the diffusion model learning how to gradually generate an action sequence corresponding to a given input by denoising in multiple time steps during the training process.

[0198] Specifically, the noise adding process at each time step can be expressed as:

[0199]

[0200] in, represents the intermediate noise sequence at the t-th time step, represents the intermediate noise-added sequence at the t-1th time step, that is, the sequence to be noise-added at the tth time step, is the first noise coefficient corresponding to the t-th time step, Represents the noise features to be added at the tth time step.

[0201] Exemplarily, the action video generation method provided by the embodiments of the present disclosure can be used for modeling virtual characters in games. Specifically, when the virtual character controlled by the user interacts with a non-player character, the non-player character will emit a corresponding target audio according to the interactive content preset in the game. At this time, the non-player character is the target virtual character. The game application can generate a target action video of the non-player character based on the target audio currently emitted by the non-player character and obtain the corresponding target guidance content through the above-mentioned action video generation method, and then display the target action video in the game application, thereby achieving smooth and natural interaction between the virtual character controlled by the user and the non-player character.

[0202] For another example, the action video generation method provided in the embodiments of the present disclosure can also be used for modeling virtual characters in live broadcasts. Specifically, when a user starts to use the image of a live virtual character for live broadcasting, the live virtual character is the target virtual character. The live broadcast application will obtain the target audio currently emitted by the user during the live broadcast, as well as the corresponding target guidance content, and generate a target action video of the target virtual character through the above-mentioned action video generation method, and then display the target action video in the live broadcast application, thereby facilitating the user to use the live virtual character to interact with the audience smoothly and naturally.

[0203] For another example, the action video generation method provided in the embodiments of the present disclosure can also be used for modeling virtual characters in animation production. Specifically, when performing animation production, the user can input the target audio of the target virtual character through the animation production application. The animation application receives the target audio and obtains the corresponding target guidance content. The target action video of the target virtual character is generated through the above-mentioned action video generation method, and then the target action video is displayed in the animation production application, thereby achieving efficient and accurate animation production.

[0204] Reference Fig.12 , Fig.12 The structure diagram of the action video generation device provided in the embodiment of the present disclosure is as follows. The action video generation device 1200 includes:

[0205] The audio acquisition module 1201 is used to acquire the target audio associated with the target virtual character and encode the target audio into a target audio feature;

[0206] The semantic tag processing module 1202 is used to obtain target guidance content associated with the target virtual character, input the target guidance content and the preset prompt text into the preset large language model for text prediction, generate at least one semantic tag for describing the target virtual character, encode the at least one semantic tag into a semantic tag feature, and the preset prompt text is used to prompt the large language model to generate a semantic tag based on the target guidance content;

[0207] Diffusion module 1203, used to denoise preset noise features according to target audio features and semantic label features using a diffusion model to obtain a target action sequence, where the target action sequence includes action data of the target virtual character in multiple action frames;

[0208] The video generation module 1204 is used to generate a target action video of the target virtual character based on the target action sequence.

[0209] Optionally, the semantic tag processing module 1202 is used to:

[0210] Acquire a target text associated with a target virtual character, the target text including text corresponding to a plurality of character audios of the target virtual character, the target audio being one of the plurality of character audios;

[0211] The target text and the target audio are determined as target guiding contents associated with the target audio.

[0212] Optionally, the semantic tag processing module 1202 is used to:

[0213] Performing audio recognition on multiple character audios of a target virtual character to obtain a target text;

[0214] Alternatively, text associated with the target virtual character input through an input operation is obtained as the target text of the target audio.

[0215] Optionally, the diffusion module 1203 is used to:

[0216] Get the preset maximum time step value;

[0217] The maximum value of the time step is determined as the current time step, and the preset noise feature is determined as the sequence to be denoised;

[0218] The diffusion model is used to denoise the denoised sequence according to the current time step, target audio features and semantic label features to obtain an intermediate denoised sequence;

[0219] The current time step is decremented based on the preset interval, and the intermediate denoising sequence is determined as the sequence to be denoised. The diffusion model is used again to denoise the sequence to be denoised according to the decremented current time step, target audio features and semantic label features to obtain the intermediate denoising sequence until the current time step returns to zero. The intermediate denoising sequence when the current time step returns to zero is determined as the target action sequence.

[0220] Optionally, the diffusion module 1203 is used to:

[0221] Obtain initial action data, perform convolution on the initial action data, and obtain initial action features;

[0222] The diffusion model is used to denoise the denoised sequence according to the current time step, initial action features, target audio features and semantic label features to obtain an intermediate denoised sequence.

[0223] Optionally, the diffusion module 1203 is used to:

[0224] Input the current time step, initial action features, target audio features, and semantic label features into the diffusion model;

[0225] The initial action feature, the target audio feature, the sequence to be denoised and the semantic label feature are fused to obtain a first fused feature;

[0226] Performing a local attention operation on the first fused feature according to a preset attention window size;

[0227] The initial action feature, the semantic label feature and the result of the local attention operation are fused to obtain a second fused feature, a self-attention operation is performed on the second fused feature, and the result of the self-attention operation is linearly mapped to obtain a noise feature to be removed corresponding to the current time step, wherein the noise feature to be removed is a tensor with the same shape as the sequence to be denoised;

[0228] The first noise coefficient corresponding to the current time step is obtained, and the noise feature to be removed is subtracted from the sequence to be denoised according to the first noise coefficient to obtain an intermediate denoised sequence.

[0229] Optionally, the diffusion module 1203 is used to:

[0230] Obtain a second noise coefficient corresponding to each time step smaller than the current time step, wherein the time step and the corresponding second noise coefficient are negatively correlated, and the second noise coefficient is greater than the first noise coefficient;

[0231] A cumulative noise coefficient is obtained according to the product of the first noise coefficient and the plurality of second noise coefficients, a first difference value is determined by subtracting the cumulative noise coefficient from the reference value, and a square root of the first difference value is determined as a first square root value;

[0232] determining a second difference value of the reference value minus the first noise factor;

[0233] Determine a first quotient of the second difference and the first square root value, and obtain the scaled noise feature to be removed according to the product of the first quotient and the noise feature to be removed;

[0234] Determine a third difference of the denoised sequence minus the scaled noise feature to be removed, and determine a first product of the third difference and the inverse of the square root of the first noise coefficient;

[0235] A second product of the first noise sequence and the second difference is determined, and the sum of the first product and the second product is determined as an intermediate denoising sequence, wherein the first noise sequence is obtained by sampling from a standard normal distribution, and the shape of the first noise sequence is the same as the shape of the sequence to be denoised.

[0236] Optionally, the action video generating device 1200 further includes a training module, and the training module is used to:

[0237] Acquire a sample action sequence of a sample virtual character, a sample audio associated with the sample virtual character, and a sample guidance content associated with the sample audio, wherein the sample action sequence includes action data of the sample virtual character in a plurality of action frames;

[0238] Encode the sample audio into sample audio features, input the sample guide content and preset prompt words into the large language model for text prediction, and generate sample labels for describing the sample virtual character;

[0239] The sample action sequence is noised in multiple time steps in sequence until the time step reaches the maximum value of the time step to obtain a sample noise sequence, and the noise feature added to the sample action sequence in each time step is determined as the reference noise feature corresponding to the time step;

[0240] Encode the sample label into a sample label feature, use the diffusion model to denoise the sample noise sequence in each time step according to the sample audio feature and the sample label feature, and determine the predicted noise feature predicted in each time step. The predicted noise feature is the noise removed from the sample noise sequence during the denoising process of the time step;

[0241] Based on the difference between the reference noise feature and the predicted noise feature corresponding to each time step, a first loss function value is determined, and back propagation is performed based on the first loss function value to optimize the parameters of the diffusion model.

[0242] Optionally, the training module is used to:

[0243] For each time step, the action data of the first n action frames in the sample action sequence are encoded as sample action features, where n increases as the time step decreases, and n is a positive integer;

[0244] The diffusion model is used to denoise the sample noise sequence in each time step according to the sample audio features, sample action features, and sample label features.

[0245] Optionally, the training module is used to:

[0246] Randomly mask the sample label features and sample action features;

[0247] The diffusion model is used to denoise the sample noise sequence in each time step based on the sample audio features, the sample label features after random masking, and the sample action features after random masking.

[0248] Optionally, the training module is used to:

[0249] For each time step, calculating a fourth difference between the predicted noise feature corresponding to the time step and the reference noise feature corresponding to the time step;

[0250] A first loss function value is determined according to an L2 norm of the fourth difference.

[0251] Optionally, the training module is used to:

[0252] Set the current time step to zero and determine the sample action sequence as the sequence to be noised;

[0253] Obtain a first noise coefficient corresponding to the current time step, sample the noise feature to be added corresponding to the current time step from the standard normal distribution, add the noise feature to be added to the sequence to be noisy according to the first noise coefficient, and obtain an intermediate noisy sequence corresponding to the current time step;

[0254] The current time step is incremented based on a preset interval, the intermediate noise-added sequence is determined as the sequence to be noise-added, the first noise coefficient is updated, the noise feature to be added corresponding to the current time step is sampled again from the standard normal distribution, and the noise feature to be added is added to the sequence to be noise-added according to the updated first noise coefficient until the current time step reaches the maximum value of the time step to obtain the sample noise sequence.

[0255] Optionally, the training module is used to:

[0256] weighting the sequence to be noisy according to the square root of the first noise coefficient;

[0257] The noise feature to be added is weighted according to the square root of the difference between the baseline value and the noise coefficient;

[0258] The weighted sequence to be noisy and the weighted noise features to be added are superimposed to obtain the intermediate noisy sequence corresponding to the current time step.

[0259] The electronic device for executing the above-mentioned action video generation method provided in the embodiment of the present disclosure may be a terminal. Fig.13 , Fig.13 This is a partial structural block diagram of a terminal provided in an embodiment of the present disclosure, and the terminal includes: a camera assembly 1310, a first memory 1320, an input unit 1330, a display unit 1340, a sensor 1350, an audio circuit 1360, a wireless fidelity (WiFi) module 1370, a first processor 1380, and a first power supply 1390. Those skilled in the art can understand that Fig.13 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0260] The camera assembly 1310 can be used to capture images or videos. Optionally, the camera assembly 1310 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and the VR (Virtual Reality) shooting function or other fusion shooting functions.

[0261] The first memory 1320 may be used to store software programs and modules. The first processor 1380 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the first memory 1320 .

[0262] The input unit 1330 may be used to receive input digital or character information and generate key signal input related to the terminal's settings and function control. Specifically, the input unit 1330 may include a touch panel 1331 and other input devices 1332 .

[0263] The display unit 1340 may be used to display input information or provided information and various menus of the terminal. The display unit 1340 may include a display panel 1341 .

[0264] The audio circuit 1360 , the speaker 1361 , and the microphone 1362 may provide an audio interface.

[0265] The first power source 1390 may be alternating current, direct current, a disposable battery, or a rechargeable battery.

[0266] The number of sensors 1350 may be one or more, and the one or more sensors 1350 include but are not limited to: acceleration sensors, gyroscope sensors, pressure sensors, optical sensors, etc. Among them:

[0267] The acceleration sensor can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. For example, the acceleration sensor can be used to detect the components of gravity acceleration on the three coordinate axes. The first processor 1380 can control the display unit 1340 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor. The acceleration sensor can also be used for collecting game or user motion data.

[0268] The gyroscope sensor can detect the body direction and rotation angle of the terminal, and the gyroscope sensor can cooperate with the acceleration sensor to collect the user's 3D actions on the terminal. The first processor 1380 can implement the following functions based on the data collected by the gyroscope sensor: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0269] The pressure sensor can be set in the side frame of the terminal and / or the lower layer of the display unit 1340. When the pressure sensor is set in the side frame of the terminal, the user's holding signal of the terminal can be detected, and the first processor 1380 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor. When the pressure sensor is set in the lower layer of the display unit 1340, the first processor 1380 controls the operability controls on the UI interface according to the user's pressure operation on the display unit 1340. The operability control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0270] The optical sensor is used to collect the ambient light intensity. In one embodiment, the first processor 1380 can control the display brightness of the display unit 1340 according to the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 1340 is increased; when the ambient light intensity is low, the display brightness of the display unit 1340 is decreased. In another embodiment, the first processor 1380 can also dynamically adjust the shooting parameters of the camera assembly 1310 according to the ambient light intensity collected by the optical sensor.

[0271] In this embodiment, the first processor 1380 included in the terminal can execute the action video generating method of the previous embodiment.

[0272] The electronic device for executing the above-mentioned action video generation method provided in the embodiment of the present disclosure may also be a server, referring to Fig.14 , Fig.14Partial structural block diagram of a server provided in an embodiment of the present disclosure. The server may have relatively large differences due to different configurations or performances, and may include one or more second processors 1410 and a second memory 1430, and one or more storage media 1440 (e.g., one or more mass storage devices) storing application programs 1443 or data 1442. Among them, the second memory 1430 and the storage medium 1440 may be short-term storage or persistent storage. The program stored in the storage medium 1440 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the second processor 1410 may be configured to communicate with the storage medium 1440 to execute a series of instruction operations in the storage medium 1440 on the server.

[0273] The server may also include one or more second power supplies 1420, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1460, and / or one or more operating systems 1441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0274] The second processor 1410 in the server may be configured to execute the motion video generating method.

[0275] The embodiments of the present disclosure also provide a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the action video generation method of each of the aforementioned embodiments.

[0276] The present disclosure also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned action video generation method.

[0277] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate to describe the embodiments of the present disclosure, such as being able to be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0278] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0279] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood as not including the number, and above, below, within, etc. are understood as including the number.

[0280] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0281] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0282] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0283] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0284] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0285] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A method for generating an action video, characterized in that: include: Acquire target audio associated with a target virtual character, and encode the target audio into a target audio feature; Acquire target guidance content associated with the target virtual character, input the target guidance content and preset prompt text into a preset large language model for text prediction, generate at least one semantic tag for describing the target virtual character, encode at least one semantic tag into a semantic tag feature, and the preset prompt text is used to prompt the large language model to generate the semantic tag based on the target guidance content; De-noising the preset noise features according to the target audio features and the semantic label features using a diffusion model to obtain a target action sequence, wherein the target action sequence includes action data of each action frame of the target virtual character in the target action video to be generated; Generate the target action video of the target virtual character based on the target action sequence; The diffusion model is trained in the following way: Acquire a sample action sequence of a sample virtual character, a sample audio associated with the sample virtual character, and a sample guidance content associated with the sample audio, wherein the sample action sequence includes action data of the sample virtual character in a plurality of action frames; Encoding the sample audio into sample audio features, inputting the sample guide content and preset prompt words into a large language model for text prediction, and generating a sample label for describing the sample virtual character; Adding noise to the sample action sequence in a plurality of time steps in sequence until the time step reaches the maximum value of the time step to obtain a sample noise sequence, and determining the noise feature added to the sample action sequence in each time step as a reference noise feature corresponding to the time step; Encoding the sample label into a sample label feature, denoising the sample noise sequence in each of the time steps according to the sample audio feature and the sample label feature using the diffusion model, and determining a predicted noise feature predicted in each of the time steps, wherein the predicted noise feature is the noise removed from the sample noise sequence during the denoising process at the time step; Based on the difference between the reference noise feature and the predicted noise feature corresponding to each of the time steps, a first loss function value is determined, and back propagation is performed based on the first loss function value to optimize the parameters of the diffusion model.

2. The method for generating an action video according to claim 1, characterized in that: The step of obtaining target guidance content associated with the target virtual character includes: Acquire a target text associated with the target virtual character, wherein the target text includes text corresponding to a plurality of character audios of the target virtual character, and the target audio is one of the plurality of character audios; The target text and the target audio are determined as target guiding content associated with the target audio.

3. The action video generation method according to claim 2, characterized in that: The step of obtaining the target text associated with the target virtual character includes: Performing audio recognition on multiple character audios of the target virtual character to obtain the target text; Alternatively, text associated with the target virtual character input through an input operation is obtained as the target text of the target audio.

4. The method for generating an action video according to claim 1, characterized in that: The step of using the diffusion model to denoise the preset noise feature according to the target audio feature and the semantic label feature to obtain the target action sequence includes: Get the preset maximum time step value; Determine the maximum value of the time step as the current time step, and determine the preset noise feature as the sequence to be denoised; De-noising the sequence to be de-noised according to the current time step, the target audio feature and the semantic label feature using a diffusion model to obtain an intermediate de-noised sequence; The current time step is decremented based on a preset interval, the intermediate denoising sequence is determined as the sequence to be denoised, and the diffusion model is used again to denoise the sequence to be denoised according to the decremented current time step, the target audio feature and the semantic label feature to obtain the intermediate denoising sequence, until the current time step returns to zero, and the intermediate denoising sequence when the current time step returns to zero is determined as the target action sequence.

5. The method for generating an action video according to claim 4, characterized in that: The step of denoising the sequence to be denoised by using the diffusion model according to the current time step, the target audio feature and the semantic label feature to obtain an intermediate denoised sequence includes: Acquire initial motion data, and perform convolution on the initial motion data to obtain initial motion features; The sequence to be denoised is denoised by using a diffusion model according to the current time step, the initial action feature, the target audio feature and the semantic label feature to obtain an intermediate denoised sequence.

6. The method for generating an action video according to claim 5, characterized in that: The step of denoising the sequence to be denoised by using the diffusion model according to the current time step, the initial action feature, the target audio feature and the semantic label feature to obtain an intermediate denoised sequence includes: Inputting the current time step, the initial action feature, the target audio feature and the semantic label feature into a diffusion model; Fusing the initial action feature, the target audio feature, the sequence to be denoised, and the semantic label feature to obtain a first fused feature; Performing a local attention operation on the first fusion feature according to a preset attention window size; The initial action feature, the semantic label feature and the result of the local attention operation are fused to obtain a second fused feature, a self-attention operation is performed on the second fused feature, and a linear mapping is performed on the result of the self-attention operation to obtain a noise feature to be removed corresponding to the current time step, wherein the noise feature to be removed is a tensor with the same shape as the sequence to be denoised; A first noise coefficient corresponding to the current time step is obtained, and the noise feature to be removed is subtracted from the sequence to be denoised according to the first noise coefficient to obtain the intermediate denoised sequence.

7. The method for generating an action video according to claim 6, characterized in that: The step of subtracting the noise feature to be removed from the sequence to be denoised according to the first noise coefficient to obtain the intermediate denoised sequence comprises: Acquire a second noise coefficient corresponding to each time step smaller than the current time step, wherein the time step is negatively correlated with the corresponding second noise coefficient, and the second noise coefficient is larger than the first noise coefficient; A cumulative noise coefficient is obtained according to the product of the first noise coefficient and a plurality of the second noise coefficients, a first difference value of a reference value minus the cumulative noise coefficient is determined, and a square root of the first difference value is determined as a first square root value; determining a second difference value of the reference value minus the first noise factor; Determine a first quotient of the second difference and the first square root value, and obtain a scaled noise feature to be removed according to a product of the first quotient and the noise feature to be removed; Determine a third difference value of the sequence to be denoised minus the scaled noise feature to be removed, and determine a first product of the third difference value and the inverse of the square root of the first noise coefficient; Determine a second product of a first noise sequence and the second difference, and determine the sum of the first product and the second product as the intermediate denoising sequence, wherein the first noise sequence is obtained by sampling from a standard normal distribution, and a shape of the first noise sequence is the same as a shape of the sequence to be denoised.

8. The method for generating an action video according to claim 1, characterized in that: The step of using the diffusion model to denoise the sample noise sequence in each time step according to the sample audio features and the sample label features includes: For each of the time steps, respectively encoding the action data of the first n action frames in the sample action sequence as sample action features, where n increases as the time step decreases, and n is a positive integer; The diffusion model is used to denoise the sample noise sequence in each time step according to the sample audio features, the sample action features, and the sample label features.

9. The action video generation method according to claim 8, characterized in that: The step of using the diffusion model to denoise the sample noise sequence in each time step according to the sample audio feature, the sample action feature, and the sample label feature includes: Randomly masking the sample label feature and the sample action feature; The diffusion model is used to denoise the sample noise sequence in each of the time steps according to the sample audio features, the sample label features after random masking, and the sample action features after random masking.

10. The action video generation method according to claim 1, characterized in that: The determining of a first loss function value based on a difference between the reference noise feature and the predicted noise feature corresponding to each of the time steps comprises: For each of the time steps, calculating a fourth difference between the predicted noise feature corresponding to the time step and the reference noise feature corresponding to the time step; The first loss function value is determined according to an L2 norm of the fourth difference.

11. The method for generating an action video according to claim 1, characterized in that: The step of adding noise to the sample action sequence in a plurality of time steps in sequence until the time step reaches the maximum value of the time step to obtain a sample noise sequence includes: The current time step is set to zero, and the sample action sequence is determined as a sequence to be noised; Obtaining a first noise coefficient corresponding to the current time step, sampling a noise feature to be added corresponding to the current time step from a standard normal distribution, adding the noise feature to be added to the sequence to be noisy according to the first noise coefficient, and obtaining an intermediate noisy sequence corresponding to the current time step; The current time step is incremented based on the preset interval, the intermediate noise-added sequence is determined as the sequence to be noise-added, the first noise coefficient is updated, the noise feature to be added corresponding to the current time step is sampled again from a standard normal distribution, and the noise feature to be added is added to the sequence to be noise-added according to the updated first noise coefficient until the current time step reaches the maximum value of the time step, thereby obtaining the sample noise sequence.

12. The method for generating an action video according to claim 11, characterized in that: The step of adding the noise feature to be added to the sequence to be noisy according to the first noise coefficient to obtain an intermediate noisy sequence corresponding to the current time step includes: weighting the sequence to be noisy according to the square root of the first noise coefficient; Weighting the noise feature to be added according to the square root of the difference between the reference value and the noise coefficient; The weighted sequence to be noisy and the weighted noise feature to be added are superimposed to obtain an intermediate noisy sequence corresponding to the current time step.

13. An action video generating device, characterized in that: include: An audio acquisition module, used to acquire target audio associated with a target virtual character, and encode the target audio into a target audio feature; A semantic tag processing module, used to obtain target guidance content associated with the target virtual character, input the target guidance content and preset prompt text into a preset large language model for text prediction, generate at least one semantic tag for describing the target virtual character, encode at least one semantic tag into a semantic tag feature, and the preset prompt text is used to prompt the large language model to generate the semantic tag based on the target guidance content; A diffusion module, configured to denoise a preset noise feature according to the target audio feature and the semantic tag feature using a diffusion model to obtain a target action sequence, wherein the target action sequence includes action data of each action frame of the target virtual character in the target action video to be generated; A video generation module, used for generating the target action video of the target virtual character based on the target action sequence; The diffusion model is trained in the following way: Acquire a sample action sequence of a sample virtual character, a sample audio associated with the sample virtual character, and a sample guidance content associated with the sample audio, wherein the sample action sequence includes action data of the sample virtual character in a plurality of action frames; Encoding the sample audio into sample audio features, inputting the sample guide content and preset prompt words into a large language model for text prediction, and generating a sample label for describing the sample virtual character; Adding noise to the sample action sequence in a plurality of time steps in sequence until the time step reaches the maximum value of the time step to obtain a sample noise sequence, and determining the noise feature added to the sample action sequence in each time step as a reference noise feature corresponding to the time step; Encoding the sample label into a sample label feature, denoising the sample noise sequence in each of the time steps according to the sample audio feature and the sample label feature using the diffusion model, and determining a predicted noise feature predicted in each of the time steps, wherein the predicted noise feature is the noise removed from the sample noise sequence during the denoising process at the time step; Based on the difference between the reference noise feature and the predicted noise feature corresponding to each of the time steps, a first loss function value is determined, and back propagation is performed based on the first loss function value to optimize the parameters of the diffusion model.

14. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the action video generating method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the action video generation method according to any one of claims 1 to 12 is implemented. 16 . A computer program product, comprising a computer program, wherein the computer program is read and executed by a processor of a computer device, so that the computer device executes the action video generation method according to any one of claims 1 to 12.

Images