Digital human video generation method and device, electronic equipment and storage medium

Abstract

The invention discloses a digital human video generation method and device, electronic equipment and a storage medium. The method comprises the following steps: determining a lip shape and a body language corresponding to a target audio, and generating an initial digital human video according to the lip shape, the body language and a predetermined target digital human; wherein the target digital figure is a digital figure determined according to a user demand; performing super-resolution processing on the initial digital human video through a generative facial prior generative adversarial network model to obtain a target video frame sequence; and generating a target digital human video according to the target video frame sequence and the target audio, and outputting the target digital human video according to a preset format. According to the technical scheme provided by the embodiment of the invention, the generated target digital human video has higher resolution and better video quality, and meanwhile, the facial details of the digital human can be recovered to keep the sense of reality of the image.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method, device, electronic equipment and storage medium for generating digital human videos. Background Art

[0002] Due to the development of artificial intelligence technology, digital humans, as an emerging technological product, are becoming more and more similar to real humans in appearance, behavior and language ability, bringing more convenience and innovation to people's lives and work. However, although digital humans are already highly similar to real humans, due to the drawbacks of technology and defects in artificial intelligence models, there are still deficiencies in the appearance characteristics and behavioral performance of digital humans, especially on high-resolution video playback devices, these deficiencies will be very obvious. Summary of the invention

[0003] The present invention provides a method, device, electronic device and storage medium for generating a digital human video. The generated target digital human video has higher resolution and better video quality, and can restore the facial details of the digital human and maintain the realism of the image.

[0004] According to one aspect of the present invention, a method for generating a digital human video is provided, the method comprising:

[0005] Determine the lip shape and body language corresponding to the target audio, and generate an initial digital human video according to the lip shape, the body language and a predetermined target digital human; wherein the target digital human is a digital human image determined according to user needs;

[0006] Performing super-resolution processing on the initial digital human video through a generative facial prior generative adversarial network model to obtain a target video frame sequence;

[0007] A target digital human video is generated according to the target video frame sequence and the target audio, and the target digital human video is output according to a preset format.

[0008] According to another aspect of the present invention, a digital human video generation device is provided, the device comprising:

[0009] An initial video generation module is used to determine the lip shape and body language corresponding to the target audio, and generate an initial digital human video according to the lip shape, the body language and a predetermined target digital human; wherein the target digital human is a digital human image determined according to user needs;

[0010] A target sequence determination module is used to perform super-resolution processing on the initial digital human video through a generative facial prior generative adversarial network model to obtain a target video frame sequence;

[0011] The target video generation module is used to generate a target digital human video according to the target video frame sequence and the target audio, and output the target digital human video according to a preset format.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the digital human video generation method described in any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the digital human video generation method described in any embodiment of the present invention when executed.

[0017] The technical solution of the embodiment of the present invention determines the lip shape and body language corresponding to the target audio, and generates an initial digital human video according to the lip shape, the body language and the predetermined target digital human; wherein the target digital human is a digital human image determined according to user needs; the initial digital human video is super-resolution processed by a generative facial prior generative adversarial network model to obtain a target video frame sequence; a target digital human video is generated according to the target video frame sequence and the target audio, and the target digital human video is output in a preset format. The technical solution of the embodiment of the present invention drives the target digital human by lip shape and body language according to the target audio to generate an initial digital human video, and generates a target digital human video according to the target video frame sequence and target audio obtained by super-resolution processing of the initial digital human video by the GPFGAN model. The generated target digital human video has higher resolution and better video quality, and can restore the facial details of the digital human and maintain the realism of the image.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flow chart of a method for generating a digital human video according to the first embodiment of the present invention;

[0021] Figure 2 is a flow chart of a method for generating a digital human video according to a second embodiment of the present invention;

[0022] Figure 3 is a structural schematic diagram of a digital human video generating device provided according to Embodiment 3 of the present invention;

[0023] Figure 4 It is a structural schematic diagram of an electronic device for implementing the digital human video generation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] Embodiment 1

[0027] Figure 1A flowchart of a method for generating a digital human video is provided for the first embodiment of the present invention. This embodiment is applicable to situations where a digital human video needs to be generated, typically, to situations where a high resolution is required for the generated digital human video. The method can be executed by a digital human video generating device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0028] S110, determining the lip shape and body language corresponding to the target audio, and generating an initial digital human video according to the lip shape, body language and a predetermined target digital human; wherein the target digital human is a digital human image determined according to user needs.

[0029] The target audio is the audio data to be used for digital human video generation, the lip shape corresponding to the target audio is the lip animation that matches the content of the target audio, and the body language corresponding to the target audio is the body movement animation that matches the content of the target audio. The target digital human is a digital human image determined according to user needs, which can meet the user's requirements for the appearance characteristics and style of the digital human.

[0030] In the embodiment of the present invention, after determining the target audio for generating the digital human video, the lip shape and body language corresponding to the target audio can be determined, and the lip shape and body language corresponding to the target audio can be bound to the predetermined target digital human, ensuring that the target digital human can accurately perform the lip animation and body movement animation corresponding to the target audio, and then, the target digital human bound to the lip shape and body language corresponding to the target audio is rendered using a three-dimensional rendering engine to generate an initial digital human video. By driving the target digital human by lip shape and body language according to the target audio, the interaction of the target digital human can be made more natural and vivid, and the obtained initial digital human video can bring users a stronger sense of reality and immersion.

[0031] Optionally, determining the lip shape and body language corresponding to the target audio includes: performing sentiment analysis on the target audio to obtain a sentiment analysis result corresponding to the target audio; performing speech recognition on the target audio to obtain a target text corresponding to the target audio; determining the lip shape corresponding to the target audio based on the target text, and determining the body language corresponding to the target audio based on the sentiment analysis result and the target text.

[0032] Among them, sentiment analysis is used to identify the emotions expressed in the target audio, and the sentiment analysis results can be positive, negative, neutral, or further divided into joy, sadness, anger, etc. Speech recognition is used to convert the speech signal of the target audio into target text, which can be performed using an automatic speech recognition (ASR) system or other speech recognition methods.

[0033] In an embodiment of the present invention, after determining the target audio for generating a digital human video, emotion analysis and speech recognition can be performed on the target audio respectively to obtain the emotion analysis result and target text corresponding to the target audio, and the target text can be used to determine the lip shape corresponding to the target audio, and the emotion analysis result and target text can be used to determine the body language corresponding to the target audio, so as to obtain lip animation and body movement animation that are consistent with the content of the target audio.

[0034] Optionally, the lip shape corresponding to the target audio is determined according to the target text, and the body language corresponding to the target audio is determined according to the sentiment analysis result and the target text, including: mapping the target text to a target pronunciation unit, and determining the lip shape corresponding to the target pronunciation unit through a preset lip shape determination model; and determining the corresponding body movement from a preset action library according to the target text and the sentiment analysis result.

[0035] In an embodiment of the present invention, the lip shape corresponding to the target audio is determined according to the target text. The target text can be first mapped to the target pronunciation unit (such as phoneme) using a pronunciation dictionary, and then the lip shape corresponding to the target pronunciation unit can be determined using a preset lip shape determination model. The lip animation corresponding to the target pronunciation unit can be generated according to the preset lip shape determination model. In an embodiment of the present invention, the body language corresponding to the target audio is determined according to the sentiment analysis results and the target text. Appropriate body movements can be selected or synthesized from a preset action library based on the sentiment analysis results and the text content. Among them, the action library can include various basic body movements and complex body movements, and the sentiment analysis results and text content corresponding to the target audio can be matched by combination, fine-tuning, etc.

[0036] S120, super-resolution processing is performed on the initial digital human video through a generative facial prior generative adversarial network model to obtain a target video frame sequence.

[0037] Among them, the Generative Facial Prioritization Generative Adversarial Networks (GPFGAN) model is a deep learning model based on adversarial networks, mainly used for face image generation and optimization. The GFPGAN model combines the advantages of facial detection and generative adversarial networks, and can perform high-quality image restoration and enhancement while maintaining facial features.

[0038] Among them, super-resolution processing, commonly known as super-resolution technology (SR), is the process of converting low-resolution images or video frames into high-resolution images or video frames through software or hardware means. Its principle is mainly based on image processing algorithms, which extract information from low-resolution images and use interpolation, machine learning or deep learning technologies to generate images with higher resolution and richer details.

[0039] In an embodiment of the present invention, after generating an initial digital human video based on lip shape, body language, and a predetermined target digital human, the initial video frame sequence of the initial digital human video can be super-resolution processed by the GPFGAN model to obtain a target video frame sequence with higher resolution and richer details. Thanks to the gating mechanism of the GPFGAN model that dynamically adjusts the weights in the generation process according to the frequency characteristics of the image, it is possible to avoid noise interference while retaining important high-frequency information of the image, thereby obtaining a clearer and more realistic image. Through the above process, the low-resolution digital human face image in the initial digital human video can be effectively converted into a high-resolution image, and the facial details of the digital human can be restored to maintain the realism of the image.

[0040] S130: Generate a target digital human video according to the target video frame sequence and the target audio, and output the target digital human video in a preset format.

[0041] In the embodiment of the present invention, after obtaining the target video frame sequence, a target digital human video with higher resolution and better video quality can be generated according to the target video frame sequence and the target audio, thereby improving the visual realism, breaking through the uncanny valley effect, and making the digital human closer to a real human. Afterwards, the target digital human video can be output in a preset format to meet the user's playback needs.

[0042] The technical solution of the embodiment of the present invention determines the lip shape and body language corresponding to the target audio, and generates an initial digital human video according to the lip shape, body language and a predetermined target digital human; wherein the target digital human is a digital human image determined according to user needs; the initial digital human video is super-resolution processed by a generative facial prior generative adversarial network model to obtain a target video frame sequence; the target digital human video is generated according to the target video frame sequence and the target audio, and the target digital human video is output in a preset format. The technical solution of the embodiment of the present invention drives the target digital human by lip shape and body language according to the target audio to generate an initial digital human video, and generates a target digital human video according to the target video frame sequence and the target audio obtained by super-resolution processing of the initial digital human video by the GPFGAN model. The generated target digital human video has higher resolution and better video quality, and can restore the facial details of the digital human and maintain the realism of the image.

[0043] Embodiment 2

[0044] Figure 2 This is a flow chart of a digital human video generation method provided by Embodiment 2 of the present invention. The embodiment of the present invention is optimized based on the above embodiment. For solutions not described in detail in the embodiment of the present invention, please refer to the above embodiment. Figure 2 As shown, the method includes:

[0045] S210: Displaying the digital human images in the digital human image library to the user through the user interface; wherein the digital human image library includes preset digital human images and user-defined digital human images.

[0046] Among them, the digital human image library can include user-defined digital human images, providing support for the personalized customization of digital humans. Users can define digital human images with specific appearance features and styles according to their own needs to meet the diverse requirements of different users for the appearance of digital humans.

[0047] In an embodiment of the present invention, the preset digital human images in the digital human image library and the user-defined digital human images can be displayed to the user through the user interface, so that the user can select the digital human image that meets his or her needs as the target digital human, so that the user can get a better experience.

[0048] S220: In response to a confirmation control on the user interface being triggered, determine the digital human image selected by the user as the target digital human.

[0049] In an embodiment of the present invention, a user can browse digital human images in a digital human image library through a user interface, and when a desired digital human image is browsed, the user can confirm the selected digital human image through a confirmation control set on the user interface. In response to the confirmation control on the user interface being triggered, the selected digital human image is determined as the target digital human. Optionally, in order to ensure that the user can correctly select the desired digital human image, a display style different from that of other digital human images is set for the currently browsed digital human image, such as highlight display, frame selection, zoom display, etc.

[0050] S230, determining the lip shape and body language corresponding to the target audio, and generating an initial digital human video according to the lip shape, body language and a predetermined target digital human; wherein the target digital human is a digital human image determined according to user needs.

[0051] S240: Determine an initial video frame sequence of an initial digital human video.

[0052] In the embodiment of the present invention, the initial video frame sequence is the digital human image of the initial digital human video arranged in the order of timestamps. After the initial digital human video is generated according to the lip shape, body language and the predetermined target digital human, the initial digital human video can be deframed by corresponding video processing software to obtain the initial video frame sequence of the initial digital human video.

[0053] S250, super-resolution processing is performed on each frame of the digital human image in the initial video frame sequence through a generative facial prior generative adversarial network model, and the initial video frame sequence after the processing is used as a target video frame sequence.

[0054] In an embodiment of the present invention, after determining the initial video frame sequence of the initial digital human video, each frame of the digital human image in the initial video frame sequence can be input into the GPFGAN model, and each frame of the digital human image is super-resolution processed by the GPFGAN model, and the high-resolution digital human images obtained after the processing are sorted according to the timestamp to obtain the target video frame sequence.

[0055] Optionally, the generative facial prior generative adversarial network model includes a generator and a discriminator; super-resolution processing is performed on each frame of the digital human image in the initial video frame sequence through the generative facial prior generative adversarial network model, including: identifying the digital human facial area in the digital human image, and repairing the digital human facial area through an adversarial training process between the generator and the discriminator.

[0056] Among them, the generator is responsible for converting the input low-resolution image into a high-resolution image, and the discriminator is used to evaluate whether the image generated by the generator is realistic. The generator and the discriminator compete through adversarial training to continuously improve their respective performance, so that the generator can gradually generate high-resolution images that are closer to real images.

[0057] In an embodiment of the present invention, after each frame of the digital human image in the initial video frame sequence is input into the GPFGAN model, the GPFGAN model first identifies the digital human facial area in the digital human image, and then through the adversarial training process between the generator and the discriminator, the generator generates a digital human image with higher resolution and richer details, and at the same time, completes the restoration of the digital human facial area.

[0058] S260: Generate a target digital human video according to the target video frame sequence and the target audio, and output the target digital human video in a preset format.

[0059] Optionally, a target digital human video is generated according to the target video frame sequence and the target audio, and the target digital human video is output according to a preset format, including: merging and synchronously processing the target video frame sequence and the target audio to generate the target digital human video; encoding and outputting the target digital human video according to a preset video encoding algorithm.

[0060] In an embodiment of the present invention, a target digital human video is generated according to a target video frame sequence and a target audio, and a video can be generated according to the target video frame sequence, and then the generated video and the target audio are synthesized and the audio and video are synchronized to obtain the target digital human video. Optionally, after the target digital human video is generated according to the target video frame sequence and the target audio, the generated target digital human video can be checked to ensure the video quality of the generated target digital human video. Afterwards, the target digital human video can be encoded and output according to a preset video encoding algorithm to meet the user's playback needs and facilitate storage and transmission.

[0061] The technical solution of the embodiment of the present invention displays the digital human image in the digital human image library to the user through the user interface; wherein the digital human image library includes preset digital human images and user-defined digital human images; in response to the confirmation control on the user interface being triggered, the digital human image selected by the user is determined as the target digital human; the lip shape and body language corresponding to the target audio are determined, and the initial digital human video is generated according to the lip shape, body language and the predetermined target digital human; wherein the target digital human is a digital human image determined according to user needs; the initial video frame sequence of the initial digital human video is determined; each frame of the digital human image in the initial video frame sequence is super-resolution processed by a generative facial prior generative adversarial network model, and the initial video frame sequence after processing is used as the target video frame sequence; the target digital human video is generated according to the target video frame sequence and the target audio, and the target digital human video is output according to a preset format. The technical solution of the embodiment of the present invention allows the user to select a preset or customized digital human image as the target digital human, which can meet the diverse requirements of different users for the appearance of digital humans. Furthermore, the target digital human is driven by lip shape and body language according to the target audio to generate an initial digital human video, and the target digital human video is generated according to the target video frame sequence and target audio obtained by super-resolution processing of the initial digital human video through the GPFGAN model. The generated target digital human video has higher resolution and better video quality, and at the same time, the facial details of the digital human can be restored to maintain the realism of the image.

[0062] Embodiment 3

[0063] Figure 3 This is a schematic diagram of the structure of a digital human video generation device provided by Embodiment 3 of the present invention. Figure 3 As shown, the device comprises:

[0064] The initial video generation module 310 is used to determine the lip shape and body language corresponding to the target audio, and generate an initial digital human video according to the lip shape, the body language and a predetermined target digital human; wherein the target digital human is a digital human image determined according to user needs;

[0065] A target sequence determination module 320 is used to perform super-resolution processing on the initial digital human video through a generative facial prior generative adversarial network model to obtain a target video frame sequence;

[0066] The target video generation module 330 is used to generate a target digital human video according to the target video frame sequence and the target audio, and output the target digital human video according to a preset format.

[0067] Optionally, the device further includes:

[0068] An image display module, used to display the digital human images in the digital human image library to the user through the user interface; wherein the digital human image library includes preset digital human images and user-defined digital human images;

[0069] The image determination module is used to determine the digital human image selected by the user as the target digital human in response to the confirmation control on the user interface being triggered.

[0070] Optionally, the initial video generation module 310 includes:

[0071] The emotion analysis unit is used to perform emotion analysis on the target audio to obtain the emotion analysis result corresponding to the target audio;

[0072] A speech recognition unit, used to perform speech recognition on the target audio to obtain a target text corresponding to the target audio;

[0073] An action determination unit is used to determine the lip shape corresponding to the target audio according to the target text, and to determine the body language corresponding to the target audio according to the emotion analysis result and the target text.

[0074] Optionally, the action determination unit includes:

[0075] A lip shape determination subunit, used for mapping the target text to a target pronunciation unit, and determining the lip shape corresponding to the target pronunciation unit through a preset lip shape determination model;

[0076] The body action determination subunit is used to determine the corresponding body action from a preset action library according to the target text and the sentiment analysis result.

[0077] Optionally, the target sequence determination module 320 includes:

[0078] An initial sequence determination unit, used to determine an initial video frame sequence of the initial digital human video;

[0079] The target sequence determination unit is used to perform super-resolution processing on each frame of the digital human image in the initial video frame sequence through a generative facial prior generative adversarial network model, and use the processed initial video frame sequence as the target video frame sequence.

[0080] Optionally, the generative facial prior generative adversarial network model includes a generator and a discriminator; the target sequence determination unit is specifically used to: identify the digital human facial area in the digital human image, and repair the digital human facial area through an adversarial training process between the generator and the discriminator.

[0081] Optionally, the target video generation module 330 includes:

[0082] A target video generation unit, used for merging and synchronously processing the target video frame sequence and the target audio to generate a target digital human video;

[0083] The target video output unit is used to encode and output the target digital human video according to a preset video encoding algorithm.

[0084] The digital human video generating device provided in the embodiment of the present invention can execute the digital human video generating method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the executing method.

[0085] Embodiment 4

[0086] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0087] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0088] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0089] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a digital human video generation method.

[0090] In some embodiments, the digital human video generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the digital human video generation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the digital human video generation method in any other appropriate manner (e.g., by means of firmware).

[0091] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0092] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0093] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0094] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0095] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0096] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0097] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0098] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a digital human video, characterized in that: The method comprises: Determine the lip shape and body language corresponding to the target audio, and generate an initial digital human video according to the lip shape, the body language and a predetermined target digital human; wherein the target digital human is a digital human image determined according to user needs; Performing super-resolution processing on the initial digital human video through a generative facial prior generative adversarial network model to obtain a target video frame sequence; A target digital human video is generated according to the target video frame sequence and the target audio, and the target digital human video is output according to a preset format.

2. The method according to claim 1, characterized in that Before determining the lip shape and body language of the target digital human based on the target audio and generating the initial digital human video, the following steps are included: Displaying the digital human images in the digital human image library to the user through the user interface; wherein the digital human image library includes preset digital human images and user-defined digital human images; In response to a confirmation control on the user interface being triggered, the digital human image selected by the user is determined as the target digital human.

3. The method according to claim 1, characterized in that Determine the lip shape and body language corresponding to the target audio, including: Performing sentiment analysis on the target audio to obtain a sentiment analysis result corresponding to the target audio; Performing speech recognition on the target audio to obtain a target text corresponding to the target audio; The lip shape corresponding to the target audio is determined according to the target text, and the body language corresponding to the target audio is determined according to the emotion analysis result and the target text.

4. The method according to claim 3, characterized in that: Determining the lip shape corresponding to the target audio according to the target text, and determining the body language corresponding to the target audio according to the emotion analysis result and the target text, including: Mapping the target text to a target pronunciation unit, and determining the lip shape corresponding to the target pronunciation unit through a preset lip shape determination model; According to the target text and the sentiment analysis result, a corresponding body movement is determined from a preset movement library.

5. The method according to claim 1, characterized in that The initial digital human video is subjected to super-resolution processing by a generative facial prior generative adversarial network model to obtain a target video frame sequence, including: Determining an initial video frame sequence of the initial digital human video; Super-resolution processing is performed on each frame of the digital human image in the initial video frame sequence through a generative facial prior generative adversarial network model, and the initial video frame sequence after processing is used as a target video frame sequence.

6. The method according to claim 5, characterized in that The generative facial prior generative adversarial network model includes a generator and a discriminator; Performing super-resolution processing on each frame of the digital human image in the initial video frame sequence through a generative facial prior generative adversarial network model, including: The digital human facial region in the digital human image is identified, and the digital human facial region is repaired through an adversarial training process between the generator and the discriminator.

7. The method according to claim 1, characterized in that Generating a target digital human video according to the target video frame sequence and the target audio, and outputting the target digital human video in a preset format, including: Merging and synchronously processing the target video frame sequence and the target audio to generate a target digital human video; The target digital human video is encoded and output according to a preset video encoding algorithm.

8. A digital human video generation device, characterized in that: The device comprises: An initial video generation module is used to determine the lip shape and body language corresponding to the target audio, and generate an initial digital human video according to the lip shape, the body language and a predetermined target digital human; wherein the target digital human is a digital human image determined according to user needs; A target sequence determination module is used to perform super-resolution processing on the initial digital human video through a generative facial prior generative adversarial network model to obtain a target video frame sequence; The target video generation module is used to generate a target digital human video according to the target video frame sequence and the target audio, and output the target digital human video according to a preset format.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the digital human video generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the digital human video generation method according to any one of claims 1 to 7 when executed.

Images