Digital human video generation method and device, storage medium and electronic equipment

Abstract

The invention relates to the field of artificial intelligence, in particular to a digital human video generation method, a digital human video generation device, a storage medium and electronic equipment. The digital human video generation method comprises the following steps: acquiring text description information, and generating a picture file and an audio file according to the text description information by using a pre-configured intelligent agent; extracting appearance features and key point features of a target person in the picture file, and extracting audio features of the audio file; and generating a picture frame sequence based on the appearance features, the key point features and the audio features, and synthesizing the picture frame sequence and the audio file to obtain a digital human video. The digital human video generation method provided by the invention can improve the sense of reality of the digital human video.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, specifically to a method for generating digital human videos, a device for generating digital human videos, a storage medium, and an electronic device. Background Technology

[0002] In recent years, AIGC (Artificial Intelligence Generated Content), 3D (3D) and metaverse-related technologies have developed rapidly. Digital human generation is one of the important technical aspects. How to use AIGC technology to efficiently and flexibly generate virtual character images with rich and natural movements and expressions and accurate lip movements is a key challenge.

[0003] Existing methods require input of voice, images, and videos, reusing the actions and postures of the videos, which is monotonous and does not match the script. Alternatively, they can directly generate actions and expressions from voice, which has problems such as computational complexity, many flaws, and unnaturalness.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a method, apparatus, storage medium, and electronic device for generating digital human videos, with the aim of improving the realism of digital human videos.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to one aspect of the present disclosure, a method for generating a digital human video is provided, comprising: acquiring text description information, and using a pre-configured intelligent agent to generate image files and audio files based on the text description information; extracting appearance features and key point features of a target person from the image files, and extracting audio features from the audio files; generating an image frame sequence based on the appearance features, the key point features and the audio features, and synthesizing the image frame sequence and the audio files to obtain a digital human video.

[0008] According to some embodiments of this disclosure, based on the foregoing scheme, the intelligent agent includes a language processing model, a text-to-image model, and a text-to-speech model. The step of generating image files and audio files using the pre-configured intelligent agent based on the text description information includes: extracting the image description information and spoken text information of the target person from the text description information based on the language processing model; generating the image file using the text-to-image model based on the image description information; and generating the audio file using the text-to-speech model based on the spoken text information.

[0009] According to some embodiments of this disclosure, based on the foregoing scheme, the step of extracting the appearance features and key point features of the target person in the image file, and extracting the audio features of the audio file, includes: inputting the image file into a face encoder to obtain the feature vector output by the face encoder as the appearance features; identifying the facial key points and torso key points of the target person in the image file based on a key point extraction model to obtain the key point features; and encoding the audio file using an audio encoder to obtain the audio features.

[0010] According to some embodiments of this disclosure, based on the foregoing scheme, generating an image frame sequence based on the appearance features, the key point features, and the audio features includes: generating an initial image frame sequence based on the key point features and the audio features; and rendering the initial image frame sequence and the appearance features to obtain the image frame sequence.

[0011] According to some embodiments of this disclosure, based on the foregoing scheme, generating an initial image frame sequence based on the keypoint features and the audio features includes: inputting the keypoint features and the audio features into a pre-trained generative pre-trained model to obtain the initial image frame sequence output by the generative pre-trained model.

[0012] According to some embodiments of this disclosure, based on the foregoing scheme, before inputting the keypoint features and the audio features into the generative pre-trained model, the method further includes: discretizing the keypoint features to obtain an initial keypoint representation, so as to update the keypoint features.

[0013] According to some embodiments of this disclosure, based on the foregoing scheme, the method further includes: when the image description information or spoken text information of the target person is not extracted based on the language processing model; searching for the image description information or spoken text information based on the association information of the text description information.

[0014] According to a second aspect of the present disclosure, a digital human video generation apparatus is provided, comprising: an acquisition module for acquiring textual description information and generating image files and audio files based on the textual description information using a pre-configured intelligent agent; an extraction module for extracting appearance features and key point features of a target person from the image files and extracting audio features from the audio files; and a synthesis module for generating an image frame sequence based on the appearance features, the key point features, and the audio features, and synthesizing the image frame sequence and the audio files to obtain a digital human video.

[0015] According to a third aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the digital human video generation method as described in the above embodiments.

[0016] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the digital human video generation method as described in the above embodiments.

[0017] The exemplary embodiments disclosed herein may have some or all of the following beneficial effects:

[0018] In the technical solutions provided by some embodiments of this disclosure, an intelligent agent is first used to divide text description information into image files and audio files. Feature extraction is performed on the image files to obtain the appearance features and key point features of the target person, and feature extraction is performed on the audio files to obtain audio features. Then, an image frame sequence is generated based on the appearance features, key point features, and audio features. Finally, the image frame sequence and audio files are synthesized to obtain a digital human video. On the one hand, only a short input text description is needed to automatically generate a matching digital human video, which improves the intelligence of digital human video generation. On the other hand, by using an intelligent agent to split the image and audio and extract features separately, the appearance, key points, audio, and other features given in the text description information can be fully utilized to generate digital human videos with natural movements, high matching accuracy, and accurate lip movements.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0021] Figure 1 The illustration shows a flowchart of a digital human video generation method according to an exemplary embodiment of the present disclosure;

[0022] Figure 2 This illustration schematically depicts a process for generating image and audio files using an intelligent agent according to an exemplary embodiment of the present disclosure.

[0023] Figure 3 The schematic diagram illustrates a flowchart of an action expression GPT model in an exemplary embodiment of the present disclosure;

[0024] Figure 4 The schematic diagram illustrates a flow chart of a VQ-VAE model in an exemplary embodiment of the present disclosure;

[0025] Figure 5 This schematic diagram illustrates a process diagram of a digital human video generation method in an exemplary embodiment of the present disclosure;

[0026] Figure 6 This schematic diagram illustrates the composition of a digital human video generation apparatus according to an exemplary embodiment of the present disclosure;

[0027] Figure 7 This schematic diagram illustrates a computer-readable storage medium according to an exemplary embodiment of the present disclosure;

[0028] Figure 8 The schematic diagram illustrates the structure of a computer system of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0029] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

[0030] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0033] In recent years, AIGC, 3D, and metaverse-related technologies have developed rapidly. Digital human generation is one of the important technological areas. How to use AIGC technology to efficiently and flexibly generate virtual character images with rich and natural movements and expressions, and accurate lip movements is a key challenge.

[0034] Existing technologies typically use methods such as Sadtalker, EMO, and Hallo to input voice and images, thereby generating talking face videos. However, these methods result in slightly moving heads and stiff facial expressions, making the generated videos unnatural. In addition, there are problems such as inconsistencies between the character's movements and voice style, as well as inconsistencies with the content. Alternatively, methods such as Wav2lip and DINet require video input and reuse video movements and postures, resulting in monotonous videos that do not match the script.

[0035] Therefore, in view of the problems in the prior art, this disclosure provides a digital human video generation method that enables digital human videos to have natural movements, high matching accuracy, and accurate lip movements.

[0036] The implementation details of the technical solutions of the embodiments of this disclosure are described in detail below.

[0037] Figure 1 This illustration schematically depicts a flowchart of a digital human video generation method according to an exemplary embodiment of this disclosure. Figure 1As shown, the digital human video generation method includes steps S101 to S103:

[0038] Step S101: Obtain text description information, and use a pre-configured intelligent agent to generate image files and audio files based on the text description information;

[0039] Step S102: Extract the appearance features and key point features of the target person from the image file, and extract the audio features from the audio file;

[0040] Step S103: Generate an image frame sequence based on the appearance features, the key point features, and the audio features, and synthesize the image frame sequence and the audio file to obtain a digital human video.

[0041] In the technical solutions provided by some embodiments of this disclosure, an intelligent agent is first used to divide text description information into image files and audio files. Feature extraction is performed on the image files to obtain the appearance features and key point features of the target person, and feature extraction is performed on the audio files to obtain audio features. Then, an image frame sequence is generated based on the appearance features, key point features, and audio features. Finally, the image frame sequence and audio files are synthesized to obtain a digital human video. On the one hand, only a short input text description is needed to automatically generate a matching digital human video, which improves the intelligence of digital human video generation. On the other hand, by using an intelligent agent to split the image and audio and extract features separately, the appearance, key points, audio, and other features given in the text description information can be fully utilized to generate digital human videos with natural movements, high matching accuracy, and accurate lip movements.

[0042] The steps of the digital human video method in this exemplary embodiment will now be described in more detail with reference to the accompanying drawings and embodiments.

[0043] In step S101, text description information is obtained, and a pre-configured intelligent agent is used to generate image files and audio files based on the text description information.

[0044] Specifically, it first needs to receive text descriptions input by the user, that is, to know the character's appearance, style, and words in the digital human video that the user expects to generate.

[0045] After receiving the text description information, the corresponding image and audio files are generated based on the text description information.

[0046] The intelligent agent is pre-configured and can be constructed using LLMs such as ChatGPT and Tongyi Qianwen, and implemented by calling tools such as StableDiffusion and CoquiTTS. In one embodiment of this disclosure, the intelligent agent includes a language processing model, a text-to-graph model, and a text-to-speech model.

[0047] Figure 2 This schematic diagram illustrates a process for generating image and audio files using an intelligent agent according to an exemplary embodiment of this disclosure, with reference to... Figure 2 As shown, the process of generating image files and audio files using an intelligent agent in step S101 is as follows: extracting the image description information and spoken text information of the target person from the text description information based on the language processing model; generating the image file based on the image description information using the text-to-image model; and generating the audio file based on the spoken text information using the text-to-speech model.

[0048] Specifically, the process begins by extracting the visual description and spoken text information of the target person from the textual description information based on the language processing model. The language processing model can be an LLM (Large Language Model), an advanced model built using deep learning techniques specifically designed to understand and generate natural language. In other words, the content of the textual description information is broken down into visual description and spoken text information using the LLM model.

[0049] Then, the image file is generated using the text-to-image model based on the image description information. A text-to-image model is a model that can automatically generate a corresponding image based on the input text description; examples include Stable Diffusion, DALL-E, and Imagen. Therefore, image description information can be used to generate an image file (Image) containing a person's image using the text-to-image model.

[0050] Simultaneously, the text-to-speech model is used to generate the audio file based on the spoken text information. A text-to-speech model is a system that uses artificial intelligence technology to convert text content into speech output. Therefore, spoken text information can be converted into an audio file (Audio) of spoken sound using a text-to-speech model.

[0051] It should be noted that this disclosure does not restrict the execution order of the two steps: converting the image description information into an image file (Image) and converting the spoken text information into an audio file (Audio). They can be executed one after the other or simultaneously.

[0052] In one embodiment of this disclosure, the method further includes: when the image description information or spoken text information of the target person is not extracted based on the language processing model; searching for the image description information or spoken text information based on the association information of the text description information.

[0053] In other words, when a language processing model processes textual descriptions, there may be missing information in the textual descriptions. Therefore, the language processing model may not be able to accurately analyze image files and audio files, or there may be missing information. In this case, a big data query can be performed based on the already separated image or audio files to collect another audio or image file that matches it, thereby completing the missing data.

[0054] In step S102, the appearance features and key point features of the target person in the image file are extracted, and the audio features of the audio file are extracted.

[0055] Specifically, image files are processed to obtain the facial features and key features of a person, and audio files are processed to obtain audio features.

[0056] In one embodiment of this disclosure, the image file processing in step S102 includes: inputting the image file into a face encoder to obtain the feature vector output by the face encoder as the appearance feature; identifying the facial key points and torso key points of the target person in the image file based on a key point extraction model to obtain the key point feature; and encoding the audio file using an audio encoder to obtain the audio feature.

[0057] Specifically, the Face Encoder is first used to extract appearance features of the person in the image file. The Face Encoder is a widely used deep learning model component in the field of face processing and recognition. It can automatically extract high-level feature vector representations from input face images. These features are crucial for face recognition and analysis, as they capture key information such as identity, expression, and pose, providing a foundation for subsequent tasks such as face recognition, face synthesis, and face animation. Appearance features can include one or more features from facial contours, facial feature distribution, proportions of the three sections of the face, facial expression, skin texture, and skin color; this disclosure does not impose specific limitations on these features.

[0058] Then, based on the keypoint extraction model, the facial and torso keypoints of the target person in the image file are identified to obtain the keypoint features. The keypoint extraction model aims to extract key points or feature points that are crucial for understanding and analysis tasks from data such as images, videos, or 3D point clouds. Therefore, the keypoint extraction model can be used to identify key points of people in image files, including facial and torso keypoints, to represent implicit features such as facial expressions, postures, and movements.

[0059] Simultaneously, the audio file is encoded using an audio encoder to obtain the audio features. The audio features of the audio file are extracted using an audio encoder, such as wav2vec / hubert.

[0060] It should be noted that this disclosure does not limit the execution order of the feature extraction process for image files and audio files; they can be executed one after the other or simultaneously.

[0061] In step S103, an image frame sequence is generated based on the appearance features, the key point features, and the audio features, and the image frame sequence and the audio file are combined to obtain a digital human video.

[0062] In one embodiment of this disclosure, the process of generating the image frame sequence in step S103 specifically includes: generating an initial image frame sequence based on the key point features and the audio features; and rendering the initial image frame sequence and the appearance features to obtain the image frame sequence.

[0063] First, the key point features and the audio features are input into a pre-trained generative pre-training model to obtain the initial image frame sequence output by the generative pre-training model.

[0064] Generative pre-trained models can be action-expression GPT models, which can transform keypoint features and audio features into an initial sequence of image frames. The initial sequence of image frames includes the lip features of the person, that is, the person's mouth, and facial expressions will change appropriately according to the audio features.

[0065] Figure 3 The illustration schematically depicts a flowchart of an action / expression GPT model according to an exemplary embodiment of this disclosure. Action / expression GPT models are typically based on a Generative Pre-Training (GPT) architecture, which consists of two stages: unsupervised pre-training and supervised downstream task fine-tuning. The core of the GPT model is the Transformer structure, particularly its Decoder part, which is specifically modified to adapt to different task requirements.

[0066] It should be noted that before inputting the keypoint features and the audio features into the generative pre-trained model, the method further includes: discretizing the keypoint features to obtain an initial keypoint representation, in order to update the keypoint features. Specifically, the keypoint features can be pre-processed, that is, the keypoint features can be discretized using a pre-trained VQ-VAE model to obtain an initial token representation.

[0067] Figure 4 The illustration schematically depicts a flowchart of a VQ-VAE model according to an exemplary embodiment of this disclosure. The VQ-VAE (Vector Quantized-Variational AutoEncoder) model is a generative model that combines variational autoencoder (VAE) and vector quantization (VQ) techniques. Its structure mainly includes three parts: an encoder, a vector quantizer, and a decoder, as well as an optional prior model (such as PixelCNN).

[0068] After obtaining the initial image frame sequence, the initial image frame sequence and Appearance features are input into the renderer model to obtain the image frame sequence.

[0069] Finally, the image frame sequence and audio file are combined to obtain a digital human video corresponding to the text description information.

[0070] Figure 5 This illustration schematically depicts a process diagram of a digital human video generation method according to an exemplary embodiment of this disclosure. (Reference) Figure 5 As shown, the user first inputs text description information, and the intelligent agent generates an image file (Image) and an audio file (Audio) based on the text description information. Then, the face encoder extracts the appearance features of the target person in the image file (Image), and the key point extraction model extracts facial key points or torso key points. At the same time, the audio features of the audio file (Audio) are extracted. Then, the key point features and audio features are input into the motion expression GPT model to obtain the motion expression sequence output by the model, which is the initial image frame sequence. Then, the appearance features and the image frame sequence are input into the Render model to obtain the digital human result image frames, which is the image frame sequence. Finally, the digital human result image frames and audio file are combined to form the digital human video.

[0071] Based on the above method, this disclosure uses a large model agent to generate image and audio files, uses a VQ-VAE model to represent actions and expressions, and then uses an action-expression GPT model to generate human actions and lip movements to create digital human videos. On the one hand, only a short text description is needed to automatically generate a matching digital human video, which can improve the intelligence of digital human video generation. On the other hand, by splitting the images and audio into parts and extracting features separately through the agent, the appearance, key points, audio and other features given in the text description can be fully utilized to generate digital human videos, resulting in digital human videos with natural movements, high matching, and accurate lip movements. Furthermore, the initial image frame sequence including lip movements generated by the action-expression GPT model has diversity and fidelity, thus avoiding the problems of high computational cost and poor controllability of the diffusion model in the prior art.

[0072] Figure 6 This schematic diagram illustrates the composition of a digital human video generation apparatus according to an exemplary embodiment of the present disclosure, such as... Figure 6 As shown, the digital human video generation device 600 may include an acquisition module 601, an extraction module 602, and a synthesis module 603. Wherein:

[0073] The acquisition module 601 is used to acquire text description information and generate image files and audio files based on the text description information using a pre-configured intelligent agent;

[0074] Extraction module 602 is used to extract the appearance features and key point features of the target person in the image file, and to extract the audio features of the audio file;

[0075] The synthesis module 603 is used to generate an image frame sequence based on the appearance features, the key point features and the audio features, and to synthesize the image frame sequence and the audio file to obtain a digital human video.

[0076] According to an exemplary embodiment of this disclosure, the intelligent agent includes a language processing model, a text-to-image model, and a text-to-speech model. The acquisition module 601 is further configured to extract the image description information and spoken text information of the target person from the text description information based on the language processing model; generate the image file based on the image description information using the text-to-image model; and generate the audio file based on the spoken text information using the text-to-speech model.

[0077] According to an exemplary embodiment of this disclosure, the extraction module 602 is further configured to input the image file into a face encoder to obtain the feature vector output by the face encoder as the appearance feature; identify the facial key points and torso key points of the target person in the image file based on the key point extraction model to obtain the key point feature; and encode the audio file using an audio encoder to obtain the audio feature.

[0078] According to an exemplary embodiment of the present disclosure, the synthesis module 603 is further configured to generate an initial image frame sequence based on the key point features and the audio features; and render the initial image frame sequence and the appearance features to obtain the image frame sequence.

[0079] According to an exemplary embodiment of the present disclosure, the synthesis module 603 is further configured to input the key point features and the audio features into a pre-trained generative pre-trained model to obtain an initial image frame sequence output by the generative pre-trained model.

[0080] According to an exemplary embodiment of the present disclosure, the synthesis module 603 is further configured to discretize the keypoint features to obtain an initial keypoint representation before inputting the keypoint features and the audio features into the generative pre-trained model, so as to update the keypoint features.

[0081] According to an exemplary embodiment of this disclosure, the acquisition module 601 is further configured to, when no image description information or spoken text information of the target person is extracted based on the language processing model, search for the image description information or spoken text information based on the association information of the text description information.

[0082] The specific details of each module in the aforementioned digital human video generation device 600 have been described in detail in the corresponding digital human video generation method, so they will not be repeated here.

[0083] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0084] In an exemplary embodiment of this disclosure, a storage medium capable of implementing the above-described method is also provided. Figure 7 This schematic diagram illustrates a computer-readable storage medium according to an exemplary embodiment of the present disclosure, such as... Figure 7As shown, a program product 700 for implementing the above-described method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a mobile phone. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0085] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided. Figure 8 The schematic diagram illustrates the structure of a computer system of an electronic device according to an exemplary embodiment of the present disclosure.

[0086] It should be noted that, Figure 8 The computer system 800 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0087] like Figure 8 As shown, the computer system 800 includes a Central Processing Unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 802 or programs loaded from storage section 808 into Random Access Memory (RAM) 803. The RAM 803 also stores various programs and data required for system operation. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An Input / Output (I / O) interface 805 is also connected to the bus 804.

[0088] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0089] In particular, according to embodiments of this disclosure, the processes described below with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs various functions defined in the system of this disclosure.

[0090] It should be noted that the computer-readable medium shown in the embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0092] The units described in the embodiments of this disclosure can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the unit itself.

[0093] In another aspect, this disclosure also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0094] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0095] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this disclosure.

[0096] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0097] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for generating digital human videos, characterized in that, include: Obtain text description information and use a pre-configured intelligent agent to generate image files and audio files based on the text description information; Extract the appearance features and key point features of the target person from the image file, and extract the audio features from the audio file; Based on the appearance features, key point features, and audio features, an image frame sequence is generated, and the image frame sequence and the audio file are combined to obtain a digital human video.

2. The digital human video generation method according to claim 1, characterized in that, The intelligent agent includes a language processing model, a text-to-image model, and a text-to-speech model. The step of generating image and audio files using the pre-configured intelligent agent based on the text description information includes: Based on the language processing model, extract the image description information and spoken text information of the target person from the text description information; The image file is generated using the text-based image model based on the image description information; and The audio file is generated based on the spoken text information using the text-to-speech model.

3. The digital human video generation method according to claim 1, characterized in that, The extraction of the appearance features and key point features of the target person in the image file, and the extraction of the audio features from the audio file, include: The image file is input into the face encoder to obtain the feature vector output by the face encoder as the appearance feature; The keypoint features are obtained by identifying facial and torso keypoints of the target person in the image file based on the keypoint extraction model; and The audio file is encoded using an audio encoder to obtain the audio features.

4. The digital human video generation method according to claim 1, characterized in that, The process of generating an image frame sequence based on the appearance features, the key point features, and the audio features includes: An initial image frame sequence is generated based on the key point features and the audio features; The initial image frame sequence and the appearance features are rendered to obtain the image frame sequence.

5. The digital human video generation method according to claim 4, characterized in that, The process of generating an initial image frame sequence based on the key point features and the audio features includes: The key point features and the audio features are input into a pre-trained generative pre-trained model to obtain the initial image frame sequence output by the generative pre-trained model.

6. The digital human video generation method according to claim 5, characterized in that, Before inputting the keypoint features and the audio features into the generative pre-trained model, the method further includes: The key point features are discretized to obtain an initial representation of the key points, which is then used to update the key point features.

7. The digital human video generation method according to claim 2, characterized in that, The method further includes: When the language processing model fails to extract the image description information or spoken text information of the target person; The image description or spoken text information is searched based on the associated information of the text description information.

8. A digital human video generation device, characterized in that, include: The acquisition module is used to acquire text description information and generate image files and audio files based on the text description information using a pre-configured intelligent agent; The extraction module is used to extract the appearance features and key point features of the target person in the image file, and to extract the audio features of the audio file; The synthesis module is used to generate a sequence of image frames based on the appearance features, the key point features, and the audio features, and to synthesize the sequence of image frames and the audio file to obtain a digital human video.

9. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the digital human video generation method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the digital human video generation method as described in any one of claims 1 to 7.

Images