Video generation method and device, and electronic equipment

Abstract

The embodiment of the invention provides a video generation method. The video generation method comprises the following steps: acquiring a to-be-processed video containing a virtual digital human; adjusting the mouth area of the virtual digital human in the to-be-processed video through a face driving model for enhancing mouth shape supervision, and generating a mouth shape driving result; and rendering the video to be processed according to the mouth shape driving result to generate a broadcast video. According to the video generation method provided by the embodiment of the invention, the mouth area of the virtual digital human in the to-be-processed video is adjusted through the face driving model for enhancing mouth shape supervision, so that the overall fidelity of virtual digital human broadcasting is improved. The embodiment of the invention further provides a video generation device and electronic equipment.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video processing technology, and more specifically, to a video generation method, device, and electronic device. Background Art

[0002] The emergence of virtual humans has undoubtedly had a significant impact on the development of the livestreaming industry. Small and medium-sized enterprises are experiencing a growing demand for professional livestreamers who can deliver livestreams. However, factors such as the hardware requirements, the training cycle for livestreamers, and the economic costs of long broadcast periods in the traditional livestreaming industry significantly restrict the development of livestreaming services within these SMEs. Virtual human technology can effectively address these issues. AI-powered voice-activated virtual humans, as the core technology for virtual human applications, not only reduces production costs but also enables the meticulous development of the virtual human's hand-to-mouth coordination.

[0003] AI voice-driven lip-syncing technology primarily relies on two approaches: voice-driven 2D modeling and rendering, and voice-driven 3D modeling and rendering. 3D modeling offers advantages such as accurate motion and expression control and excellent detail rendering, but it also suffers from long production cycles and high customization costs. 2D modeling, on the other hand, can capture a streamer's motion and expression information from 2D audio and video, shortening the production cycle and reducing costs. However, it suffers from limitations such as lower accuracy and detail quality in motion and expression control. Summary of the Invention

[0004] In response to the problems existing in the above-mentioned prior art, the embodiments of the present application provide a video generation method, device, and electronic device. By strengthening the face-driven model for lip shape supervision, the mouth area of ​​the virtual digital human in the processed video is adjusted, thereby improving the overall realism of the virtual digital human's broadcast.

[0005] In a first aspect, an embodiment of the present application provides a video generation method, comprising the following steps:

[0006] Obtaining a video to be processed containing a virtual digital human;

[0007] By strengthening the face driving model of lip shape supervision, adjusting the mouth area of ​​the virtual digital human in the video to be processed to generate a lip shape driving result; and

[0008] Render the video to be processed according to the lip-sync driving result to generate a broadcast video.

[0009] Furthermore, the face driving model with enhanced lip-activated supervision is used to adjust the mouth area of ​​the virtual digital human in the video to be processed to generate a lip-activated result, including:

[0010] Get the face pre-training model;

[0011] Assigning mouth AU parameters to the mouth attention map of the virtual digital human in the video to be processed by strengthening the face-driven model with lip shape supervision; and

[0012] The lip-driven result is generated by the face pre-training model and the assigned mouth attention map.

[0013] Furthermore, the face-driven model with enhanced lip shape supervision assigns mouth AU parameters to the mouth attention map, including:

[0014] By strengthening the face-driven model with lip shape supervision, the mouth AU25 parameters are assigned to the mouth attention map.

[0015] Furthermore, before the face driven model with enhanced lip shape supervision assigns the mouth AU25 parameter to the mouth attention map of the virtual digital human in the video to be processed, the method further includes:

[0016] Extracting the AU25 parameters and mouth attention map of the virtual digital human in the video to be processed; and

[0017] By strengthening the face driving model with lip shape supervision, the AU25 parameters of the mouth of the virtual digital human in the video to be processed are assigned to the mouth attention map of the virtual digital human in the video to be processed.

[0018] Furthermore, before the face driven model with enhanced lip shape supervision assigns the mouth AU25 parameter to the mouth attention map of the virtual digital human in the video to be processed, the method further includes:

[0019] Extracting audio features from the video to be processed; and

[0020] Extracting driving lip-shape parameters of a virtual digital human corresponding to the audio features in the video to be processed, wherein the driving lip-shape parameters include the mouth attention map.

[0021] Furthermore, after assigning the mouth AU parameters to the mouth attention map by the face-driven model with enhanced lip shape supervision, the method further includes:

[0022] Acquiring facial expressions and / or facial gestures of the virtual digital human in the video to be processed; and

[0023] The broadcast video is rendered and generated by using the face pre-training model and the assigned mouth attention map, and in combination with the facial expressions and / or facial postures of the virtual digital human in the video to be processed.

[0024] Furthermore, obtaining a pre-trained face model includes:

[0025] Input audio and video data to train the speech-driven lip model and obtain a pre-trained face model.

[0026] In a second aspect, an embodiment of the present application further provides a video generation device, comprising:

[0027] A video acquisition module is used to acquire the video to be processed containing the virtual digital human;

[0028] A driving result generation module is used to adjust the mouth area of ​​the virtual digital human in the video to be processed by strengthening the face driving model of lip shape supervision to generate a lip shape driving result; and

[0029] The video generation module is used to render the video to be processed according to the lip-activated result to generate a broadcast video.

[0030] In a third aspect, an embodiment of the present application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the video generation method according to the first aspect above when executing the program.

[0031] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, wherein the computer program is used to implement the video generation method according to the first aspect above.

[0032] The embodiments of the present application bring the following beneficial effects:

[0033] In the video generation method provided in the embodiments of the present application, after obtaining a video to be processed that includes a virtual digital human, the mouth area of ​​the virtual digital human in the video to be processed is adjusted using a face-driven model that enhances lip-sync supervision, generating a lip-syncing result. Based on this lip-syncing result, the video to be processed is rendered to generate a broadcast video. Therefore, the video generation method provided in the embodiments of the present application can resolve the issue of inconsistency between the lip movements and voice of the virtual digital human in the video to be processed, thereby improving the overall realism of the virtual digital human's broadcast. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0035] Figure 1 A flowchart of a video generation method provided in an embodiment of the present application;

[0036] Figure 2 A structural block diagram of a video generation device provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0038] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments described in this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0040] In the specification and claims of this application and the above-mentioned drawings, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0041] Figure 1 FIG. 1 is a flow chart of a video generation method according to an embodiment of the present application. Figure 1 As shown, the video generation method of the embodiment of the present application includes the following steps:

[0042] S101: Obtain a video to be processed containing a virtual digital human;

[0043] Specifically, virtual humans are digitized characters that exist in the non-physical world and possess multiple human characteristics (physical features, performance abilities, and interaction abilities) created through computer graphics, image rendering, motion capture, deep learning, and speech synthesis. Existing virtual humans in videos often suffer from inconsistencies between their lip movements and voices, necessitating adjustments to the virtual humans in the videos being processed.

[0044] S102: adjusting the mouth area of ​​the virtual digital human in the video to be processed by strengthening the face driving model for lip shape supervision to generate a lip shape driving result; and

[0045] Specifically, a face-driven model uses computer vision and graphics techniques to capture and analyze dynamic changes in the human face, such as facial expressions and lip movements, to drive a virtual digital human or character. As mentioned above, the virtual digital human in a video often experiences inconsistencies between lip movements and voice. Therefore, adjustments need to be made to the virtual human in the video being processed. By strengthening the face-driven model with lip-sync supervision, adjustments can be made to the virtual human's mouth area in the video being processed, generating lip-sync-driven results.

[0046] S103: Rendering the video to be processed according to the lip-activated result to generate a broadcast video.

[0047] As described above, when the mouth area of ​​the virtual digital human in the video to be processed is adjusted by strengthening the face driving model of lip shape supervision and generating a lip shape driving result, the video to be processed can be 3D rendered according to the lip shape driving result to generate a broadcast video.

[0048] Therefore, in the video generation method provided in the embodiment of the present application, after obtaining a video to be processed that includes a virtual digital human, the mouth area of ​​the virtual digital human in the video to be processed is adjusted using a face-driven model that strengthens lip-activated supervision, generating a lip-activated result. Based on this lip-activated result, the video to be processed is rendered to generate a broadcast video. Therefore, the video generation method provided in the embodiment of the present application can solve the problem of inconsistency between the lip movements and voice of the virtual digital human in the video to be processed, thereby improving the overall realism of the virtual digital human's broadcast.

[0049] Furthermore, in some embodiments of the present application, the face driving model with enhanced lip-activated supervision is used to adjust the mouth area of ​​the virtual digital human in the video to be processed to generate a lip-activated result, including:

[0050] Get the face pre-training model;

[0051] Assigning mouth AU parameters to the mouth attention map of the virtual digital human in the video to be processed by strengthening the face-driven model with lip shape supervision; and

[0052] The lip-driven result is generated by the face pre-training model and the assigned mouth attention map.

[0053] Specifically, the face pre-training model is a model that uses deep learning technology to train with a large amount of face image data to improve the performance of face recognition and face feature extraction. The Facial Action Coding System (FACS) parameters of the mouth describe a series of action units (AU parameters) related to mouth movements. These action units can express different emotions and intentions. Mouth Attention Map refers to a visualization technique used to highlight the degree of attention that convolutional neural networks pay to the mouth area when processing images. This technology helps researchers or developers understand the importance of the mouth when the model processes specific tasks (such as face recognition, lip reading, etc.) by generating an attention map in which the color or brightness of the mouth area is emphasized according to the network's attention to the area.

[0054] Therefore, in the video generation method provided in the embodiment of the present application, after the mouth AU parameters are assigned to the mouth attention map of the virtual digital human in the video to be processed by strengthening the face driving model with lip shape supervision, the lip shape driving result can be generated through the face pre-training model and the assigned mouth attention map, thereby further improving the accuracy of face recognition and facial feature adjustment, so as to further improve the overall realism of the virtual digital human's broadcast.

[0055] Furthermore, in some embodiments of the present application, the face-driven model with enhanced lip shape supervision assigns mouth AU parameters to the mouth attention map, including:

[0056] By strengthening the face-driven model with lip shape supervision, the mouth AU25 parameters are assigned to the mouth attention map.

[0057] As mentioned above, the facial action coding system parameters of the mouth describe a series of expression action units related to mouth movement. These action units can express different emotions and intentions. Such movements are usually associated with certain emotions or expressions. The specific emotional expression needs to be comprehensively judged in combination with other AU parameters. Among them, the AU25 parameter describes a specific mouth movement, that is, the lips are pulled horizontally and become flat. Therefore, by strengthening the face-driven model with lip shape supervision, the mouth AU25 parameters are assigned to the mouth attention map, and the control effect of lip opening and closing is strengthened through the attention mechanism, which solves the common defect of not closing the mouth during silent moments in the current virtual human technology field.

[0058] Of course, the video generation method provided in the embodiment of the present application can also use other AU mouth-related single parameters or a combination of multiple parameters to achieve the effect of enhanced mouth supervision.

[0059] Furthermore, in some embodiments of the present application, before the face-driven model with enhanced lip shape supervision assigns the mouth AU25 parameter to the mouth attention map of the virtual digital human in the video to be processed, the method further includes:

[0060] Extracting the AU25 parameters and mouth attention map of the virtual digital human in the video to be processed; and

[0061] By strengthening the face driving model with lip shape supervision, the AU25 parameters of the mouth of the virtual digital human in the video to be processed are assigned to the mouth attention map of the virtual digital human in the video to be processed.

[0062] Specifically, after the mouth AU25 parameters and mouth attention map of the virtual digital human in the video to be processed are extracted, the mouth AU25 parameters are assigned to the mouth attention map through the face-driven model that strengthens lip shape supervision. The attention mechanism is used to enhance the control effect of lip opening and closing, solving the common defect of not closing the mouth in silent moments in the current field of virtual human technology.

[0063] Furthermore, in some embodiments of the present application, before the face-driven model with enhanced lip shape supervision assigns the mouth AU25 parameter to the mouth attention map of the virtual digital human in the video to be processed, the method further includes:

[0064] Extracting audio features from the video to be processed; and

[0065] Extracting driving lip-shape parameters of a virtual digital human corresponding to the audio features in the video to be processed, wherein the driving lip-shape parameters include the mouth attention map.

[0066] The driving lip shape parameters of the virtual digital human here include the mouth AU25 parameters, mouth Attention Map and facial parameters (ID, expression, posture, etc.) of the virtual digital human or virtual anchor in the video to be processed. The driving lip shape parameters of the corresponding virtual digital human are extracted based on the audio features to ensure that the virtual digital human's lip shape can match the audio features, thereby further improving the overall realism of the adjusted virtual digital human.

[0067] Furthermore, in some embodiments of the present application, after assigning the mouth AU parameters to the mouth attention map by the face-driven model with enhanced lip shape supervision, the method further includes:

[0068] Acquiring facial expressions and / or facial gestures of the virtual digital human in the video to be processed; and

[0069] The broadcast video is rendered and generated by using the face pre-training model and the assigned mouth attention map, and in combination with the facial expressions and / or facial postures of the virtual digital human in the video to be processed.

[0070] As described above, after extracting the mouth AU25 parameters and mouth attention map of the virtual digital human in the video to be processed, the mouth AU25 parameters are assigned to the mouth attention map through the face-driven model with enhanced lip shape supervision. This attention mechanism strengthens the control of lip opening and closing, solving the current problem of virtual human technology not closing the mouth during silent moments. By using this face pre-training model and the assigned mouth attention map, combined with the facial expressions and / or facial postures of the virtual digital human in the video to be processed, the overall realism of the virtual digital human can be further improved.

[0071] Furthermore, in some embodiments of the present application, obtaining a pre-trained face model includes:

[0072] Input audio and video data to train the speech-driven lip model and obtain a pre-trained face model.

[0073] Specifically, the face pre-training model is a model that uses deep learning technology to train a large amount of face image data to improve the performance of face recognition and face feature extraction. Therefore, the video generation method provided in the embodiment of the present application obtains a face pre-training model by inputting audio and video data to train a voice-driven lip model, thereby improving the accuracy of face recognition and face feature extraction.

[0074] Figure 2 2 is a structural block diagram of the video generation device 200 provided in an embodiment of the present application. Figure 2 As shown, the video generation device 200 of the embodiment of the present application includes: a video acquisition module 210, a drive result generation module 220 and a video generation module 230, wherein:

[0075] The video acquisition module 210 is used to acquire the video to be processed containing the virtual digital human;

[0076] A driving result generating module 220 is configured to adjust the mouth area of ​​the virtual digital human in the video to be processed by strengthening the face driving model of lip shape supervision to generate a lip shape driving result; and

[0077] The video generation module 230 is used to render the video to be processed according to the lip-activated result to generate a broadcast video.

[0078] In the video generation device provided in the embodiments of the present application, after obtaining a video to be processed containing a virtual digital human, the mouth area of ​​the virtual digital human in the video to be processed is adjusted using a face-driven model that enhances lip-activated monitoring, generating a lip-activated result. Based on this lip-activated result, the video to be processed is rendered to generate a broadcast video. Therefore, the video generation method provided in the embodiments of the present application can resolve the issue of inconsistency between the lip movements and voice of the virtual digital human in the video to be processed, thereby improving the overall realism of the virtual digital human's broadcast.

[0079] It should be noted that the specific implementation of the video generation device in the embodiment of the present application is similar to the specific implementation of the video generation method in the embodiment of the present application. Please refer to the description of the method part for details and will not be repeated here.

[0080] Figure 3 Schematic diagram of the structure of an electronic device 300 according to an embodiment of the present application.

[0081] like Figure 3 As shown, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 302 to the random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0082] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, and the like; an output section 307 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 308 including a hard disk; and a communication section 309 including a network interface card such as a LAN card or a modem. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. Removable media 311, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 310 as needed, so that computer programs read therefrom can be installed into the storage section 308 as needed.

[0083] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above-mentioned functions defined in the electronic device of the present application are executed.

[0084] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electronic device, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0085] In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction-executing electronic device, apparatus, or device. Furthermore, in this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction-executing electronic device, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0086] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the processing receiving device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the aforementioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based electronic device that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0087] The units or modules involved in the embodiments described in this application can be implemented by software or hardware. The units or modules described can also be set in a processor, and the processor is used to implement the video generation method when executing the program:

[0088] Obtaining a video to be processed containing a virtual digital human;

[0089] By strengthening the face driving model of lip shape supervision, adjusting the mouth area of ​​the virtual digital human in the video to be processed to generate a lip shape driving result; and

[0090] Render the video to be processed according to the lip-sync driving result to generate a broadcast video.

[0091] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently and not be incorporated into the electronic device. The computer-readable storage medium stores one or more programs, and when the aforementioned programs are used by one or more processors to execute the video generation method described in the present application:

[0092] Obtaining a video to be processed containing a virtual digital human;

[0093] By strengthening the face driving model of lip shape supervision, adjusting the mouth area of ​​the virtual digital human in the video to be processed to generate a lip shape driving result; and

[0094] Render the video to be processed according to the lip-sync driving result to generate a broadcast video.

[0095] As another aspect, the present application further provides a computer program product, which may be included in the electronic device described in the above embodiments; or may exist independently and not be incorporated into the electronic device. The above computer program product stores one or more programs, and when the above programs are used by one or more processors to execute the video generation method described in the present application:

[0096] Obtaining a video to be processed containing a virtual digital human;

[0097] By strengthening the face driving model of lip shape supervision, adjusting the mouth area of ​​the virtual digital human in the video to be processed to generate a lip shape driving result; and

[0098] Render the video to be processed according to the lip-sync driving result to generate a broadcast video.

[0099] The above description is only a preferred embodiment of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made based on the contents of the present application specification and drawings, or direct / indirect application in other related technical fields, are included in the patent protection scope of the present application.

Claims

1. A video generation method, characterized in that: The following steps are involved: Obtain a video to be processed containing a virtual digital person; By strengthening the face driving model of lip shape supervision, the mouth area of ​​the virtual digital human in the video to be processed is adjusted to generate a lip shape driving result; and The video to be processed is rendered according to the lip-activated result to generate a broadcast video.

2. The video generation method according to claim 1, characterized in that: The face driving model enhanced with lip shape supervision adjusts the mouth area of ​​the virtual digital human in the video to be processed to generate a lip shape driving result, including: Get the face pre-training model; By strengthening the face-driven model with lip shape supervision, assigning the mouth AU parameters to the mouth attention map of the virtual digital human in the video to be processed; and The lip-driven result is generated by using the face pre-training model and the assigned mouth attention map.

3. The video generation method according to claim 2, characterized in that: The face-driven model that strengthens lip shape supervision assigns mouth AU parameters to the mouth attention map, including: By strengthening the face-driven model with lip shape supervision, the mouth AU25 parameters are assigned to the mouth attention map.

4. The video generation method according to claim 3, characterized in that: Before the face driven model with enhanced lip shape supervision assigns the mouth AU25 parameter to the mouth attention map of the virtual digital human in the video to be processed, it also includes: Extracting the AU25 parameters and the mouth attention map of the virtual digital human of the video to be processed; and By strengthening the face driving model with lip shape supervision, the AU25 parameters of the mouth of the virtual digital human in the video to be processed are assigned to the mouth attention map of the virtual digital human in the video to be processed.

5. The video generation method according to claim 3, characterized in that: Before the face driven model with enhanced lip shape supervision assigns the mouth AU25 parameter to the mouth attention map of the virtual digital human in the video to be processed, it also includes: Extracting audio features from the video to be processed; and Extracting driving lip shape parameters of a virtual digital human corresponding to the audio features in the video to be processed, wherein the driving lip shape parameters include the mouth attention map.

6. The video generation method according to claim 2, characterized in that: After assigning the mouth AU parameters to the mouth attention map by the face driven model with enhanced lip shape supervision, the method further includes: Acquiring facial expressions and / or facial gestures of the virtual digital human in the video to be processed; and The broadcast video is rendered and generated by using the human face pre-training model and the assigned mouth attention map, and in combination with the facial expressions and / or facial gestures of the virtual digital human in the video to be processed.

7. The video generation method according to claim 1, characterized in that: The obtaining of the face pre-training model comprises: Input audio and video data to train the speech-driven lip model and obtain the pre-trained face model.

8. A video generating device, characterized in that: include: A video acquisition module, used to acquire a video to be processed containing a virtual digital human; A driving result generation module, used to adjust the mouth area of ​​the virtual digital human in the video to be processed by strengthening the face driving model of lip shape supervision, and generate a lip shape driving result; and The video generation module is used to render the video to be processed according to the lip-driven result to generate a broadcast video.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the video generating method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to implement the video generation method according to any one of claims 1-7.

Images