Digital human video generation method and device, medium and program product

Abstract

The invention discloses a digital human video generation method and device, a medium and a program product, and the method comprises the steps: firstly carrying out the data preprocessing of a digital human video material, and obtaining a preprocessed digital human sequence frame image; performing face recognition processing on the preprocessed digital human sequence frame image to obtain a face sequence frame image; the text input is converted into voice audio, the face sequence frame image and the voice audio are input into a lip shape synchronization model, and a first sequence frame synchronization image of lip motion accurately aligned with the voice audio is generated; performing face high-definition repair processing on the first sequence frame synchronous image by using a face repair model to obtain a second sequence frame synchronous image; and finally, performing video synthesis processing by using the second sequence frame synchronization image and the voice audio to obtain a digital human video. According to the invention, automatic and integrated digital human video generation supporting multiple languages is realized, the generation efficiency is effectively improved, and the generated digital human video has high sense of reality and fluency.

Description

Technical Field

[0001] The present invention relates to the field of digital media technology, and in particular to a method, device, medium and program product for generating digital human video. Background Art

[0002] Nowadays, digital human technology has been widely used in the fields of virtual anchors, online customer service, education and training. Existing digital human generation methods mainly rely on complex animation production processes, which usually involve 3D modeling, keyframe animation, facial capture technology and other steps. The core principle of these methods is to drive the expressions and movements of digital humans through pre-designed models and animations, or by capturing the facial movements of real people in real time.

[0003] Although the existing digital human video generation technology has achieved certain achievements in visual effects, it still has the following problems and shortcomings: 1. High cost: Traditional digital human generation methods require professional animators and facial capture equipment, which leads to high production costs. 2. Low efficiency: The complex animation production process and a large amount of manual participation limit the efficiency of digital human generation and make it difficult to meet the needs of large-scale applications. 3. Lack of multi-language support: Existing technologies often only support a single language, which has limitations for digital human applications in multi-language environments. 4. Insufficient video clarity: When generating digital human videos, due to the limitations of facial details and overall clarity, the generated videos often lack realism and attractiveness. 5. Lack of automation: The existing digital human generation process lacks automation and requires a lot of manual intervention, which not only increases costs but also limits generation efficiency. 6. Insufficient matting processing: When processing digital human videos, existing technologies often lack high-quality matting processing, resulting in unnatural fusion between the background of the video and the digital human, affecting the overall visual effect. 7. Difficulty in embedding dynamic materials: Existing technologies make it difficult to seamlessly embed dynamic materials such as video backgrounds, picture-in-picture, and animated images into digital human videos, limiting the richness and diversity of video content. 8. Lack of text-to-speech and audio generation functions: Existing technologies generally fail to integrate high-quality text-to-speech functions, limiting the voice generation and synchronization capabilities of digital human videos.

[0004] Therefore, it is urgent to invent an automated, integrated digital human high-definition video generation method with multi-language support to solve the problems of low efficiency, high cost, lack of multi-language support, low video clarity, low degree of automation, low quality of matte processing and low quality of audio generation in the existing digital human generation technology. Summary of the invention

[0005] In view of this, the embodiments of the present invention provide a method, device, medium and program product for generating a digital human video, which at least partially solve the problems existing in the prior art.

[0006] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0007] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:

[0008] According to a first aspect of an embodiment of the present invention, a method for generating a digital human video is provided, the method comprising:

[0009] Performing data preprocessing on the digital human video material to obtain preprocessed digital human sequence frame images;

[0010] Using the facial recognition model to perform facial recognition processing on the pre-processed digital human sequence frame images to obtain facial sequence frame images;

[0011] Convert text input into speech audio, extract speech time-frequency features of the audio data, and obtain a Mel-spectrogram corresponding to the speech audio;

[0012] Inputting the facial sequence frame images and the Mel-spectrogram corresponding to the speech audio into a lip synchronization model to generate a first sequence frame synchronization image of lip movements that is precisely aligned with the speech audio;

[0013] Using a facial restoration model, performing facial high-definition restoration processing on the first sequence of frame-synchronized images to obtain a second sequence of frame-synchronized images;

[0014] The second sequence of frame-synchronized images and the voice audio are used to perform video synthesis processing to obtain a digital human video.

[0015] Furthermore, the method further comprises:

[0016] The digital human video is interpolated based on the intelligent frame interpolation algorithm to obtain a high frame rate digital human video.

[0017] Furthermore, the digital human video material is preprocessed to obtain a preprocessed digital human sequence frame image, including:

[0018] Splitting the digital human video material frame by frame to obtain a first sequence of frame images;

[0019] Using a preset matting model to perform character matting processing on the first sequence of frame images to obtain a second sequence of frame images without background;

[0020] Based on the first and last frame smoothing and sequence frame inversion strategy, the second sequence frame image is seamlessly processed to obtain a third sequence frame image;

[0021] A fourth sequence of frame images of multiple specifications and scales is generated according to the size and configuration of the third sequence of frame images.

[0022] Further, converting the text input into speech audio includes:

[0023] Based on the TTS model, the text input is converted into corresponding voice audio according to the content of the text input and the preset voice characteristics.

[0024] Furthermore, the training process of the lip synchronization model includes:

[0025] Perform audio and video synchronization processing on the audio and video dataset containing multiple languages, and perform face detection and face centering processing on the training data to obtain a synchronized dataset;

[0026] Based on the synchronization data set, the GPU server is used to perform model training on the lip synchronization model to be trained until a convergence condition is met, thereby obtaining a trained lip synchronization model.

[0027] Furthermore, the method further comprises:

[0028] According to the HLS protocol, the generated digital human video is split into media segments of different specifications;

[0029] For each media segment, a corresponding index file is generated.

[0030] Further, the second sequence of frame-synchronized images and the voice audio are used to perform video synthesis processing to obtain a digital human video, including:

[0031] The second sequence of frame-synchronized images, the voice audio, and widgets are used to perform video synthesis to obtain a complete digital human video, wherein the widgets include image widgets, animated image widgets, and video widgets.

[0032] According to a second aspect of an embodiment of the present invention, there is provided a digital human video generation device, the device comprising: a processor and a memory;

[0033] The memory is used to store one or more program instructions;

[0034] The processor is used to run one or more program instructions to execute the steps of a digital human video generation method as described in any one of the above items.

[0035] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a method for generating a digital human video as described in any one of the above items are implemented.

[0036] According to a fourth aspect of an embodiment of the present invention, a computer program product is provided, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer implements the steps of a digital human video generation method as described in any one of the above items.

[0037] The embodiments of the present invention provide a method, device, medium and program product for generating digital human videos, which integrate multiple functions such as digital human material preprocessing, text-to-speech, audio generation, matting processing, lip synchronization, facial repair, and intelligent frame filling, and realize automated and integrated digital human video generation. The duration of video generation is not limited by the duration of the human material, and it can be seamlessly continued on the basis of the generated video, while supporting the embedding of dynamic materials such as video background, picture-in-picture, and animated images. The embodiments of the present invention significantly improve the generation efficiency of digital human videos, reduce the generation cost, support multi-language environments, and the final output digital human videos have a high sense of reality and fluency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0039] Figure 1 A schematic diagram of a flow chart of a method for generating a digital human video provided by an embodiment of the present invention;

[0040] Figure 2 A schematic block diagram of the principle of a method for generating a digital human video provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a system architecture corresponding to a digital human video generation method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] Figure 1 The figure shows a flow chart of a method for generating a digital human video according to an embodiment of the present invention.

[0045] like Figure 1 As shown, the digital human video generation method according to the embodiment of the present invention may include step S100, step S200, step S300, step S400, step S500 and step S600.

[0046] refer to Figure 2 In step S100, data preprocessing is performed on the digital human video material to obtain a preprocessed digital human sequence frame image.

[0047] Specifically, the above steps include:

[0048] First of all, the above-mentioned digital human video materials are shot according to the material shooting specifications set by the user. The above-mentioned material shooting specifications include shooting requirements, technical requirements, picture and background requirements, and character requirements. Among them, shooting requirements include video duration, content, data format, resolution, frame rate, etc.; technical requirements include color gamut, encoding, stability, aperture, etc.; picture and background requirements include the use of green screen, blue screen or real scene, and ensuring that the picture is level and the characters are clear; character requirements include light, movement, clothing, accessories, etc.

[0049] The embodiment of the present invention generates a digital human video based on the filmed digital human material, avoiding the use of professional animators and facial capture equipment, thereby greatly reducing the production cost of the digital human.

[0050] Then, the acquired digital human video material is split frame by frame to obtain a first sequence of frame images; the first sequence of frame images is subjected to frame by frame character matting processing using a preset special matting model to obtain a second sequence of frame images of pure characters without background; based on the first and last frame smoothing and sequence frame inversion strategies, the second sequence of frame images is seamlessly processed to obtain a third sequence of frame images; finally, according to the size and configuration of the third sequence of frame images, a fourth sequence of frame images of different specifications and sizes is generated.

[0051] Next, in step S200, the face recognition model is used to perform face recognition processing on the pre-processed digital human sequence frame images to obtain a face sequence frame image.

[0052] In step S300, the text input is converted into speech audio, and the speech time-frequency features of the audio data are extracted to obtain a Mel-spectrogram corresponding to the speech audio.

[0053] Specifically, the above steps include:

[0054] Based on the TTS model, the preset or real-time text input is converted into corresponding natural and fluent speech audio according to the content of the text input and the preset voice characteristics (language, timbre, etc.), providing synchronized audio for digital human video generation. During the conversion process, the large language model is used to polish the text to conform to the application habits in related scenarios.

[0055] In addition, the above-mentioned voice audio can be generated by fine-tuning or cloning the existing voice timbre.

[0056] The embodiments of the present invention significantly improve the speech generation and synchronization capabilities of digital human videos through high-quality text-to-speech and audio generation functions, thereby enhancing the naturalness of the video and the audience's sense of immersion.

[0057] Next, in step S400, the facial sequence frame images and the mel-spectrogram corresponding to the speech audio are input into the lip synchronization model to generate a first sequence frame synchronization image of the lip movement that is precisely aligned with the speech audio.

[0058] Specifically, the training process of the above lip synchronization model is as follows:

[0059] Use audio and video datasets containing multiple languages ​​for audio and video synchronization processing, and use the image tool library to perform face detection and face centering on the training video data to obtain a synchronized dataset, ensuring that the dataset can adapt to the lip synchronization requirements of different languages ​​and is suitable for a wider range of application scenarios.

[0060] Then, based on the synchronization data set, the GPU server is used to train the lip synchronization model to be trained until the model convergence conditions are met and the model reaches a usable level, thereby obtaining a trained lip synchronization model.

[0061] In step S500, a face restoration model is used to perform high-definition face restoration processing on a first sequence of frame-synchronized images to obtain a second sequence of frame-synchronized images.

[0062] The embodiment of the present invention performs facial restoration on the sequence frame images after lip synchronization through high-definition restoration technology, which can significantly improve the clarity and realism of the generated digital human video.

[0063] Next, in step S600, a video synthesis process is performed using the second sequence of frame-synchronized images and speech audio to obtain a digital human video.

[0064] Specifically, the above steps include:

[0065] The video synthesis library is used to perform video synthesis on the second sequence of frame-synchronized images, voice audio, and widgets to obtain a complete digital human video, wherein the widgets include image widgets, animated image widgets, video widgets, and the like.

[0066] The embodiment of the present invention supports embedding of dynamic materials such as video background, picture-in-picture, and animated images to generate richer and more diverse digital human videos.

[0067] Preferably, the digital human video generation method provided by the embodiment of the present invention further includes:

[0068] If the video frame rate needs to be increased, the digital human video is interpolated based on an intelligent frame interpolation algorithm to obtain a high frame rate digital human video.

[0069] The embodiment of the present invention performs intelligent frame interpolation processing on the digital human video through a frame interpolation algorithm, thereby improving the fluency and realism of the digital human video.

[0070] Preferably, the digital human video generation method provided by the embodiment of the present invention further includes:

[0071] According to the HLS protocol, the generated digital human video is split into media segments of different specifications, and a corresponding index file is generated for each media segment.

[0072] Figure 3A schematic diagram of the system architecture corresponding to a digital human video generation method provided in an embodiment of the present invention, wherein the system architecture includes a digital human middle-end system, a digital human back-end service and a service interface, wherein the digital human middle-end system is used to edit and plan all materials required for generating digital humans in a visual environment and push generation tasks to the task queue of the server; the service interface includes a digital human-related service interface, a digital human generation task creation / stop / query interface, a text language judgment / text-to-speech / index file merging interface and a character material preprocessing task creation / stop / query interface.

[0073] In addition, an embodiment of the present invention further provides a digital human video generation device, which includes: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of a digital human video generation method as described above.

[0074] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generating a digital human video as described above are implemented.

[0075] In addition, an embodiment of the present invention further provides a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the steps of the method for generating a digital human video as described above are implemented.

[0076] The embodiments of the present invention provide a method, device, medium and program product for generating digital human videos, which integrate multiple functions such as digital human material preprocessing, text-to-speech, audio generation, matting processing, lip synchronization, facial repair, and intelligent frame filling, and realize automated and integrated digital human video generation. The duration of video generation is not limited by the duration of the human material, and it can be seamlessly continued on the basis of the generated video, while supporting the embedding of dynamic materials such as video background, picture-in-picture, and animated images. The embodiments of the present invention significantly improve the generation efficiency of digital human videos, reduce the generation cost, support multi-language environments, and the final output digital human videos have a high sense of reality and fluency.

[0077] In the embodiment of the present invention, the processor may be an integrated circuit chip with signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware. The storage medium may be a memory, for example, a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM).The storage medium described in the embodiment of the present invention is intended to include but is not limited to these and any other suitable types of memory. Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by a combination of hardware and software. When the software is applied, the corresponding function can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication medium includes any medium that is convenient for transmitting a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access. Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to it on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed in the present invention.

Claims

1. A method for generating a digital human video, characterized in that: The method comprises: Performing data preprocessing on the digital human video material to obtain preprocessed digital human sequence frame images; Using the facial recognition model to perform facial recognition processing on the pre-processed digital human sequence frame images to obtain facial sequence frame images; Convert text input into speech audio, extract speech time-frequency features of the audio data, and obtain a Mel-spectrogram corresponding to the speech audio; Inputting the facial sequence frame images and the Mel-spectrogram corresponding to the speech audio into a lip synchronization model to generate a first sequence frame synchronization image of lip movements that is precisely aligned with the speech audio; Using a facial restoration model, performing facial high-definition restoration processing on the first sequence of frame-synchronized images to obtain a second sequence of frame-synchronized images; The second sequence of frame-synchronized images and the voice audio are used to perform video synthesis processing to obtain a digital human video.

2. A method for generating a digital human video according to claim 1, characterized in that: The method further comprises: The digital human video is interpolated based on the intelligent frame interpolation algorithm to obtain a high frame rate digital human video.

3. A method for generating a digital human video according to claim 1, characterized in that: Perform data preprocessing on the digital human video material to obtain preprocessed digital human sequence frame images, including: Splitting the digital human video material frame by frame to obtain a first sequence of frame images; Using a preset matting model to perform character matting processing on the first sequence of frame images to obtain a second sequence of frame images without background; Based on the first and last frame smoothing and sequence frame inversion strategy, the second sequence frame image is seamlessly processed to obtain a third sequence frame image; A fourth sequence of frame images of multiple specifications and scales is generated according to the size and configuration of the third sequence of frame images.

4. A method for generating a digital human video according to claim 1, characterized in that: Convert text input to speech audio, including: Based on the TTS model, the text input is converted into corresponding voice audio according to the content of the text input and the preset voice characteristics.

5. A method for generating a digital human video according to claim 1, characterized in that: The training process of the lip synchronization model includes: Perform audio and video synchronization processing on the audio and video dataset containing multiple languages, and perform face detection and face centering processing on the training data to obtain a synchronized dataset; Based on the synchronization data set, the GPU server is used to perform model training on the lip synchronization model to be trained until a convergence condition is met, thereby obtaining a trained lip synchronization model.

6. A method for generating a digital human video according to claim 1, characterized in that: The method further comprises: According to the HLS protocol, the generated digital human video is split into media segments of different specifications; For each media segment, a corresponding index file is generated.

7. A method for generating a digital human video according to claim 1, characterized in that: Performing video synthesis processing using the second sequence of frame-synchronized images and the voice audio to obtain a digital human video includes: The second sequence of frame-synchronized images, the voice audio, and widgets are used to perform video synthesis to obtain a complete digital human video, wherein the widgets include image widgets, animated image widgets, and video widgets.

8. A digital human video generation device, characterized in that: The device comprises: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of a digital human video generation method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a digital human video generation method as claimed in any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises computer program instructions, which, when executed by a processor, implement the steps of a method for generating a digital human video as claimed in any one of claims 1 to 7.

Images