Multi-element driving source-oriented 2D digital human generation method and device

Abstract

The invention discloses a 2D digital human generation method and device for a multi-element driving source, and the method comprises the steps: obtaining the driving sources of various elements, i.e., text, voice and video, and a to-be-driven virtual image; a lip-shaped driving source and an action driving source are obtained by using algorithms such as text-to-voice, audio feature extraction and video preprocessing in the driving source analysis module; migrating a posture action and an expression action in the action driving source to a to-be-driven virtual image to obtain a driving result synchronous with the action of the action driving source; the mouth shape in the action driving result is replaced with a new mouth shape according to the lip-shaped driving source, and a result synchronous with the lip-shaped driving source is obtained; and synthesizing the fused audio corresponding to the text and the voice signal with the dual-drive result of the action and the lip shape to obtain a 2D digital human generation result with sound and picture matching. According to the invention, a multi-element driving source is supported to carry out controllable 2D digital human drive generation.

Description

Technical Field

[0001] The present invention relates to the field of generative artificial intelligence, and in particular to a method and device for generating a 2D digital human oriented to multiple driving factors. Background Art

[0002] With the development of artificial intelligence, the concept of virtual digital human has emerged. 2D digital human is a two-dimensional virtual existence that exists in the digital world and is endowed with multiple human-like characteristics and abilities through advanced computer technology. These digital characters can express rich behaviors through language, facial expressions and body movements. They are not only widely used in commercial fields such as entertainment and brand promotion, but also show potential in cultural tourism, education, health and other fields.

[0003] 2D digital human generation technology mainly uses voice or driving video to realize the facial drive of virtual portraits, including lip shape drive and expression drive, so as to achieve the purpose of synchronizing voice or driving video with lip shape or expression movement. At present, most 2D digital human generation solutions use audio as the only driving source, replacing the lip shape of the character in the template video to generate the digital human video, which lacks the controllability of head movement and expression movement; a small number of solutions use video as the driving source, which lacks flexibility in lip shape changes. Therefore, how to realize a 2D digital human driven by multiple factors with a unified framework and flexible and controllable has become a valuable direction. Summary of the invention

[0004] The purpose of the present invention is to propose a 2D digital human generation method and device for multiple driving sources in order to address the shortcomings of the current 2D digital human generation scheme, which has a single driving source and is difficult to balance controllability and flexibility.

[0005] The present invention is implemented by the following technical solutions: On the one hand, the present invention discloses a 2D digital human generation method for multiple driving sources, the method comprising the following steps:

[0006] S1, obtaining images of a multi-factor driving source and a virtual image to be driven;

[0007] S2. Perform driver source analysis, including:

[0008] Convert the text signal into a noise-free first audio and extract features to serve as a first lip shape driving source;

[0009] The speech signal is converted into a noise-free second audio and features are extracted to serve as a second lip shape driving source;

[0010] Processing the video signal and the image of the virtual image to be driven by a video preprocessing algorithm to obtain a standardized action driving source and a standardized virtual image image, as well as a first cropping parameter saved during the processing;

[0011] S3, extracting the facial parameters of the action driving source and the virtual image, performing flow field modeling, repairing the facial parameters after image reconstruction based on the pixel points in the flow field, and obtaining the action driving result;

[0012] S4, performing a rectangular frame cropping on the action driving result, combining the obtained second cropping parameter with the standard virtual image, and saving and aligning the affine transformation matrix and the rectangular cropping frame;

[0013] S5, combining the first lip shape driving source and the second lip shape driving source into a third lip shape driving source, and combining the first audio and the voice signal into a third audio; mapping the lip shape information in the third lip shape driving source to the virtual image, and performing an inverse transformation according to the affine transformation matrix after obtaining the third result of the lip shape driving, and then replacing the image data according to the rectangular cropping frame corresponding to the second cropping parameter, to obtain a fourth result of the lip shape driving;

[0014] S6. Use the rectangular cropping frame corresponding to the first cropping parameter to replace the central area of ​​the original virtual image with the fourth result of the lip shape drive, synthesize the generated video and the third audio to obtain a 2D digital human generation result.

[0015] Furthermore, the analysis process of the first lip-shaped driving source and the second lip-shaped driving source in S2 is specifically as follows:

[0016] The acoustic model FastSpeech and the vocoder HifiGAN are used to quickly generate speech for the text signal to obtain a noise-free first audio; then the audio feature extractor is used to process the first audio into audio features, and the audio features are used as the first lip shape driving source; each frame of the speech signal is denoised using the denoising algorithm implemented by the noisereduce package to obtain a noise-free second audio after denoising; then the audio feature extractor is used to process the second audio into audio features, and the audio features are used as the second lip shape driving source.

[0017] Furthermore, the video preprocessing algorithm in S2 is specifically processed as follows:

[0018] Use the s3fd face detection algorithm to detect the facial key points in the video signal and the virtual image image, and obtain the facial key points of each image;

[0019] Locate the face position according to the triangle formed by the center of the left eye key point, the center of the right eye key point and the center of the mouth key point among the face key points;

[0020] Expand the triangle that locates the face outward to obtain a rectangular cropping frame, so that the face is located in the center of the rectangular cropping frame;

[0021] Each image is cropped according to the rectangular cropping frame to obtain an action driving source in a standard format and a virtual image image in a standard format, and the rectangular cropping frame of the virtual image image is saved as the first cropping parameter.

[0022] Furthermore, the extraction of the action driving source and the facial parameters of the virtual image and the flow field modeling are specifically as follows:

[0023] A first facial parameter is obtained from the action driving source using the Deep3DFaceRecon_pytorch model; the first facial parameter includes multiple facial parameters extracted from multiple images in the action driving source, and a second facial parameter is obtained from the image of the virtual image, the second facial parameter is a single facial parameter extracted from the virtual image;

[0024] The first facial parameter information is injected layer by layer into the second facial parameter through a pre-trained flow field predictor to complete the flow field modeling.

[0025] Furthermore, the facial parameters are repaired after image reconstruction based on the pixels in the flow field, specifically:

[0026] Based on the image reconstruction algorithm, the offset of each pixel point in the flow field is superimposed on the image coordinates of the virtual image; then, the bilinear interpolation method implemented by the numpy package is used to interpolate the image of the virtual image according to the generated coordinates, and finally, the target image is reconstructed according to the pixel values ​​obtained by the interpolation, and the image of the virtual image is distorted into an action-driven first result synchronized with the target action according to the flow field;

[0027] The motion-driven first result and the image of the virtual image are input into a pre-trained restoration network, and restoration is performed according to the first facial parameters to obtain a motion-driven second result synchronized with the driving source action.

[0028] Furthermore, S4 is specifically:

[0029] Using the s3fd face detection algorithm to detect the facial key points in the second result driven by the action, to obtain the facial key points of each image; then determining the first face position according to the left eye key point center, the right eye key point center and the nose lower key point in the facial key points;

[0030] Also use three coordinates to define the second face position

[0031] Calculate the affine transformation matrix based on the first face position and the second face position, and use the warpAffine algorithm in the cv2 package to obtain the rectangular cropping frame;

[0032] Each image of the second result of the action drive is cropped according to the rectangular cropping frame to obtain a standard image data set of the virtual object, and the rectangular cropping frame here is saved as the second cropping parameter.

[0033] Furthermore, the S5 is specifically:

[0034] Determine a lip shape driving source, connect the first lip shape driving source and the second lip shape driving source in a sequence specified by a user, and obtain a third lip shape driving source, wherein the specified sequence specifies an arrangement and combination sequence of each short Mel spectrum in the first lip shape driving source and the second lip shape driving source, and a sentence is used as the minimum unit for combination;

[0035] According to the lip shape driving source and the standardized virtual object image data set,

[0036] The lip shape information in the lip shape driving source is captured by using the pre-trained lip shape driving algorithm model, and the lip shape information is mapped to the RGB space of the virtual image, so that the original lip shape of the virtual image is modified into a new lip shape synchronized with the driving audio, and the third result of the lip shape driving is obtained;

[0037] The image data is inversely transformed using the affine transformation matrix, and the pixel area at the same position in the second result is replaced by the inversely transformed image data according to the second cropping parameter to obtain a fourth result synchronized with the lip driving source.

[0038] On the other hand, the specification also provides a 2D digital human generation system for multiple driving sources, the system comprising: a multiple driving source acquisition module, an image acquisition module, a driving source analysis module, an action driving module, a lip shape driving module, an audio and video synthesis module and a 2D digital human client display module;

[0039] The multi-factor driving source acquisition module is used to acquire one or more driving factors uploaded by the user.

[0040] The image acquisition module is used to acquire the face image of the virtual image to be driven.

[0041] The driving source analysis module is used to analyze the text signal and the voice signal into the lip shape driving source and analyze the video signal into the action driving source:

[0042] The action driving module is used to generate the virtual image action consistent with the action driving source; the lip shape driving module is used to generate the virtual image lip shape consistent with the lip shape driving source.

[0043] The audio and video synthesis module is used to merge a 2D digital human video with synchronized audio and video;

[0044] The 2D digital human client display module is used to display the 2D digital human image stream and play the corresponding 2D digital human voice.

[0045] On the other hand, the specification also provides a 2D digital human generation device for multiple-element driving sources, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the 2D digital human generation method for multiple-element driving sources.

[0046] On the other hand, the specification also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method for generating a 2D digital human for multiple driving sources is implemented.

[0047] The beneficial effects of the present invention are as follows:

[0048] 1. Supports multiple driving sources for 2D digital humans. One or more signal combinations of text signals, voice signals, and video signals can be used as driving factors for digital humans, which is applicable to a wider range of 2D digital human generation scenarios;

[0049] 2. The virtual image is driven by the action driving module and the lip shape driving module, which can simultaneously drive the head posture movement, facial expression movement and lip shape movement of the virtual image, making the generated 2D digital human image more controllable;

[0050] 3. Using a method of multi-level gradient cropping of image sets and saving cropping parameters, the virtual image and intermediate generation results are gradually cropped, which can reduce the resources required for calculation and increase the generation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a 2D digital human generation method for multiple driving factors provided by an embodiment of the present invention;

[0052] Figure 2 A flowchart of a method for generating a 2D digital human with multiple driving elements including text, voice and video provided in an embodiment of the present invention;

[0053] Figure 3 A flowchart of the implementation of a 2D digital human generation method for two lip shape driving elements, text and voice, provided in an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of the structure of a 2D digital human generation system for multiple driving sources provided by an embodiment of the present invention;

[0055] Figure 5 A schematic structural diagram of a 2D digital human generation device for multiple driving sources provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0057] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0058] The present invention mainly solves the 2D digital human generation method for multiple driving sources through the following approaches: the present invention obtains driving sources of multiple elements such as text, voice and video and the virtual image to be driven, and uses the text-to-speech, audio feature extraction and video preprocessing algorithms in the driving source analysis module to obtain the lip shape driving source and the action driving source; uses the action driving module to transfer the posture action and expression action in the action driving source to the virtual image to be driven, and obtains the driving result synchronized with the action driving source action; then uses the lip shape driving module to replace the lip shape in the action driving result with the new lip shape according to the lip shape driving source, and obtains the result synchronized with the lip shape driving source; finally, uses the audio and video synthesis module to synthesize the fused audio corresponding to the text and voice signals with the dual driving results of action and lip shape, and obtains the 2D digital human generation result that matches the sound and picture. Figure 1 shown.

[0059] First embodiment

[0060] like Figure 2 As shown, this embodiment provides a 2D digital human generation method for multiple driving sources, and the specific steps of the method are as follows:

[0061] S1. Acquire a multi-factor driving source, including: a text signal, a voice signal, and a video signal. The driving source may be one or more of them, and acquire an image of the virtual image to be driven.

[0062] Among them, the virtual image can be any role, including but not limited to teachers, doctors, cartoon characters, and virtual idols; the text signal and voice signal represent the information that needs to be "said" by the 2D digital human; and the characters in the video signal are preset to do some actions that do not cover the face, such as blinking, which is not specifically limited here, indicating the head posture and expression movements that need to be "performed" by the 2D digital human.

[0063] S2, perform drive source analysis, and analyze the multi-factor drive source into a lip drive source and a motion drive source that can be received by the subsequent drive module. Specifically:

[0064] Using the acoustic model FastSpeech and the vocoder HifiGAN to quickly generate speech from the text signal to obtain a noise-free first audio; then using an audio feature extractor to process the first audio into an audio feature, and using the audio feature as a first lip shape driving source;

[0065] Use the noise reduction algorithm implemented by the noisereduce package to reduce the noise of each frame of speech signal to obtain the noise-free second audio after noise reduction, and then use the audio feature extractor to process the second audio into audio features, which are used as the second lip shape driving source;

[0066] The video signal and the image of the virtual image to be driven are subjected to a video preprocessing algorithm to obtain a standardized action driving source and a standardized virtual image image.

[0067] The process of extracting audio features is specifically as follows:

[0068] First, the sampling rates of the first audio and the second audio are detected. If the sampling rates are not 16kHz, ffmpeg is used to resample them to 16kHz to obtain new first audio and second audio.

[0069] Then use the audio feature extractor implemented in the libsora package to convert the audio with a sampling rate of 16kHz into Mel spectrum features;

[0070] Then, the Mel-spectrogram feature is segmented according to the target frame rate of the final 2D digital human generated video to obtain a lip-shaped driving source composed of a series of short Mel-spectrograms, namely, a first lip-shaped driving source and a second lip-shaped driving source.

[0071] The specific process of the video preprocessing algorithm processing the video signal and the image of the virtual image to be driven is:

[0072] Firstly, the s3fd face detection algorithm is used to detect the facial key points in the video signal and the virtual image, and 68 facial key points are obtained for each image, where each key point consists of an x-coordinate and a y-coordinate, and the coordinate system takes the upper left corner of the image as the origin, the upper edge of the image as the x-axis, and the left edge of the image as the y-axis;

[0073] Then, the face position is located according to the triangle formed by the center of the left eye key point (the 36th to 39th key points), the center of the right eye key point (the 42nd to 45th key points) and the center of the mouth key point (the 48th to 54th key points) among the 68 face key points;

[0074] Then expand the triangle that locates the face outward to obtain a rectangular cropping frame, so that the face is located in the center of the rectangular cropping frame;

[0075] Finally, each image is cropped according to the rectangular cropping frame to obtain an action driving source with a standard format and a virtual image with a standard format, and the rectangular cropping frame of the virtual image is saved as the first cropping parameter.

[0076] Step S3, using the 3DMM model to obtain a first facial parameter from the action driving source, the first facial parameter including multiple facial parameters extracted from multiple images in the action driving source; similarly, using the 3DMM model to obtain a second facial parameter from the image of the virtual image, the second facial parameter being only a single facial parameter extracted from the virtual image;

[0077] Among them, the 3DMM model refers to the three-dimensional facial information reconstruction model implemented by Deep3DFaceRecon_pytorch, and the extracted facial parameters include identity parameters, texture parameters, expression parameters, etc.

[0078] Performing flow field modeling on the second facial parameter according to the first facial parameter to obtain a flow field storing action information in the action driving source and identity information in the virtual image;

[0079] The flow field modeling is to inject the first facial parameter information into the second target parameter layer by layer through a pre-trained flow field predictor, and the obtained flow field can depict the migration path of each RGB pixel point in the virtual image image.

[0080] Based on the image reconstruction algorithm, the image of the virtual image is distorted into a motion-driven first result that is synchronized with the target motion according to the flow field.

[0081] The action-driven first result and the image of the virtual image are input into a pre-trained repair network (such as the Editing network in PIRedener), and repaired according to the first facial parameters to obtain the action-driven second result, which is the result synchronized with the driving source action.

[0082] The distortion operation is specifically as follows: first, the offset of each pixel point in the flow field is superimposed on the image coordinates of the virtual image; then, a bilinear interpolation method is used to interpolate the virtual image according to the generated coordinates, and finally, a target image is reconstructed according to the pixel values ​​obtained by interpolation. The bilinear interpolation method can be implemented by the numpy package.

[0083] S4, performing rectangular frame cropping on the action-driven result, combining the obtained cropping parameters with the standard virtual image, and saving and aligning the affine transformation matrix and the rectangular cropping frame;

[0084] The processing of the video preprocessing algorithm here is different from the previous steps. Specifically:

[0085] The facial key points in the second result driven by the action are detected using the s3fd face detection algorithm to obtain 68 facial key points of each image; then the first face position is determined according to the left eye key point center (36th to 39th key points), the right eye key point center (42nd to 45th key points) and the lower nose key point (33rd key point) among the 68 facial key points;

[0086] The second face position is also defined using three coordinates: (17, 20), (58, 20) and (37.5, 40). These coordinates are based on an image size of 256*256.

[0087] Calculate the affine transformation matrix based on the first face position and the second face position, and use the warpAffine algorithm in the cv2 package to obtain the rectangular cropping frame;

[0088] The second result of the action drive is aligned and cropped through a video preprocessing algorithm to obtain a standardized image data set of the virtual object, and the affine transformation matrix and rectangular cropping frame in the alignment and cropping process are saved, where the rectangular cropping frame is the second cropping parameter.

[0089] S5. Determine a lip shape driving source, connect the first lip shape driving source and the second lip shape driving source in a sequence specified by the user, obtain a third lip shape driving source, and use the third lip shape driving source as a direct input of the lip shape synchronization module, and save the specified sequence as a connection parameter. The specified sequence specifies the sequence of permutations and combinations of the short mel spectra in the first lip shape driving source and the second lip shape driving source, and the combination is performed with a sentence as the minimum unit.

[0090] According to the lip shape driving source and the standardized virtual object image data set, the pre-trained lip shape driving algorithm model is used to capture the lip shape information in the driving source, and the lip shape information is mapped to the RGB space of the virtual image, so that the original lip shape of the virtual image is modified into a new lip shape synchronized with the driving audio, and the third result of the lip shape driving is obtained.

[0091] Since the image data sets of the third result are all standard aligned images, it is necessary to use the affine transformation matrix to inversely transform the image data, and replace the pixel area at the same position in the second result with the inversely transformed image data according to the second cropping parameters to obtain a fourth result synchronized with the lip drive source, and the image size of the fourth result is consistent with the second result driven by action.

[0092] The first audio signal and the speech signal are arranged and combined according to the connection parameter to obtain a third audio signal.

[0093] S6. Since the size of the image data set of the fourth result is a part of the original virtual image, it is necessary to use the rectangular cropping frame corresponding to the first cropping parameter to replace the pixel area at the same position in the original virtual image with the image data of the fourth result, so as to obtain an image data set consistent with the size of the original image; based on the generated image data set and the third audio, use the ffmpeg tool to merge the audio and video, set the target frame rate to 25FPS and the target audio sampling rate to 16kHz, and obtain a 2D digital human generation result with matching sound and picture.

[0094] Second embodiment

[0095] like Figure 3 As shown, the method provides an embodiment of a missing video signal for assisting understanding. When the video signal is missing, a 2D digital human generation method for text and voice driven elements is provided. The specific steps of the method are as follows:

[0096] S1, obtaining two driving elements, a text signal and a voice signal, and obtaining a video of a virtual image whose lip shape is to be driven; wherein the virtual image can be any role, including but not limited to teachers, doctors, cartoon characters, and virtual idols, and the video of the lip shape to be driven is a video of the virtual image containing the complete face; the virtual image can perform some actions that do not cover the face in the video by default, such as blinking, which is not specifically limited here.

[0097] S2, perform driving source analysis, and analyze the multi-factor driving source into a lip shape driving source and a motion driving source that can be received by the subsequent driving module. Convert the text signal into a noise-free first audio through the TTS algorithm, use the audio feature extractor to process the first audio into an audio feature, and use the audio feature as the first lip shape driving source; convert the speech signal into a noise-free second audio through the audio noise reduction algorithm, use the audio feature extractor to process the second audio into an audio feature, and use the audio feature extractor to process the second audio into an audio feature, and use the audio feature as the second lip shape driving source;

[0098] The process of processing the text signal and the voice signal is similar to that in step S2 of the first embodiment, and will not be described in detail here.

[0099] S3, determining a lip shape driving source, connecting the first lip shape driving source and the second lip shape driving source in a sequence specified by the user, obtaining a third lip shape driving source as a direct input of the lip shape synchronization module, and saving the specified sequence as a connection parameter. The specified sequence specifies the permutation and combination sequence of each short mel spectrum in the first lip shape driving source and the second lip shape driving source, and the combination is performed with a sentence as the minimum unit.

[0100] S4, using a video preprocessing algorithm to align and crop the virtual image video to obtain a standardized virtual object image data set, and save the affine transformation matrix and cropping parameters in this process.

[0101] The video preprocessing process is similar to S4 in the first embodiment and will not be described in detail here.

[0102] Step S5, based on the third lip shape driving source and the image data set of the standardized virtual object, use the pre-trained lip shape driving algorithm model to capture the lip shape information in the driving source, and map the lip shape information to the RGB space of the virtual image, so that the original lip shape of the virtual image is modified to a new lip shape synchronized with the driving audio, and the first lip shape driving result is obtained.

[0103] The affine transformation matrix is ​​used to inversely transform the lip driving result, and the inversely transformed image data is used to replace the pixel area at the same position in the original virtual image video according to the cropping parameters to obtain a second result synchronized with the lip driving source.

[0104] The first audio signal and the speech signal are arranged and combined according to the connection parameter to obtain a third audio signal.

[0105] Step S6, based on the third audio and the second result synchronized with the lip shape driving source, use the ffmpeg tool to merge the audio and video, set the target frame rate to 25FPS and the target audio sampling rate to 16kHz, and obtain a 2D digital human generation result with matching sound and picture.

[0106] Third embodiment

[0107] Another embodiment of the present invention also discloses a device, which includes: a multi-factor driving source acquisition module written using a file upload framework, used to collect one or more driving factors uploaded by a user, the driving factors including text signals, voice signals and video signals; an image acquisition module written using a file upload framework, used to collect facial images of virtual images to be driven; a driving source parsing module including an acoustic model FastSpeech, a vocoder HifiGAN and a video preprocessing algorithm, used to parse text signals and voice signals into lip driving sources and to parse video signals into action driving sources; an action driving module using a 3DMM model and an expression migration model, used to generate a lip-shaped driving source according to the lip shape; and a lip-shaped driving source parsing module using a lip-shaped driving source parsing module. The action driving source controls the head posture and facial expression of the virtual image, so that the action of the virtual image is consistent with the action driving source; the lip shape driving module is based on the pre-trained lip shape driving model, and is used to change the lip shape of the virtual character in the output result of the action driving module according to the lip shape driving source, so that the lip shape of the virtual image is consistent with the lip shape driving source; the audio and video synthesis module is written using ffmpeg, and is used to merge the output result of the lip shape driving module with the audio corresponding to the text signal and the voice signal into a 2D digital human video with synchronized picture and audio; the 2D digital human client display module is written based on Vue3, and is used to display the 2D digital human image stream and play the corresponding 2D digital human voice;

[0108] Fourth embodiment

[0109] Corresponding to the aforementioned embodiment of a 2D digital human generation method for multiple driving sources, the present invention further provides an embodiment of a 2D digital human generation device for multiple driving sources.

[0110] See also Figure 5 An embodiment of the present invention provides a 2D digital human generation device for multiple driving sources, including a memory and one or more processors. The memory stores executable codes. When the processor executes the executable codes, it is used to implement a 2D digital human generation method for multiple driving sources in the above embodiment.

[0111] The embodiment of a 2D digital human generation device for multiple driving sources provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 5 As shown, it is a hardware structure diagram of a 2D digital human generation device for multiple driving sources provided by the present invention, in which any device with data processing capability is located, except Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0112] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0113] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0114] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a 2D digital human generation method for multiple driving elements in the above embodiment is implemented.

[0115] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0116] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method for generating a 2D digital human oriented to multiple driving sources is implemented.

[0117] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0118] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A 2D digital human generation method for multiple driving factors, characterized in that: The method comprises the following steps: S1, obtaining images of a multi-factor driving source and a virtual image to be driven; S2. Perform driver source analysis, including: Convert the text signal into a noise-free first audio and extract features to serve as a first lip shape driving source; The speech signal is converted into a noise-free second audio and features are extracted to serve as a second lip shape driving source; Processing the video signal and the image of the virtual image to be driven by a video preprocessing algorithm to obtain a standardized action driving source and a standardized virtual image image, and using a rectangular cropping frame of the virtual image image as a first cropping parameter; S3, extracting the facial parameters of the action driving source and the virtual image, performing flow field modeling, repairing the facial parameters after image reconstruction based on the pixel points in the flow field, and obtaining the action driving result; S4, performing rectangular frame cropping on the action-driven result, combining the obtained cropping parameters with the standard virtual image image, and saving and aligning the affine transformation matrix and the rectangular cropping frame; S5, combining the first lip shape driving source and the second lip shape driving source into a third lip shape driving source, and combining the first audio and the voice signal into a third audio; mapping the lip shape information in the third lip shape driving source to the virtual image, and performing an inverse transformation according to the affine transformation matrix after obtaining the third result of the lip shape driving, and then replacing the image data according to the rectangular cropping frame to obtain a fourth result of the lip shape driving; S6. Replace the image data in the original virtual image cropping frame with the fourth result according to the first cropping parameter, synthesize the fourth result driven by the lip shape and the third audio to obtain a 2D digital human generation result.

2. The method for generating a 2D digital human based on multiple driving factors according to claim 1, characterized in that: The analysis process of the first lip-shaped driving source and the second lip-shaped driving source in S2 is specifically as follows: The acoustic model FastSpeech and the vocoder HifiGAN are used to quickly generate speech for the text signal to obtain a noise-free first audio; then the audio feature extractor is used to process the first audio into audio features, and the audio features are used as the first lip shape driving source; each frame of the speech signal is denoised using the denoising algorithm implemented by the noisereduce package to obtain a noise-free second audio after denoising; then the audio feature extractor is used to process the second audio into audio features, and the audio features are used as the second lip shape driving source.

3. The method for generating a 2D digital human with multiple driving factors according to claim 1, characterized in that: The video preprocessing algorithm in S2 is specifically as follows: Use the s3fd face detection algorithm to detect the facial key points in the video signal and the virtual image image, and obtain the facial key points of each image; The face position is located according to the triangle formed by the center of the left eye key point, the center of the right eye key point and the center of the mouth key point in the face key points; Expand the triangle that locates the face outward to obtain a rectangular cropping frame, so that the face is located in the center of the rectangular cropping frame; Each image is cropped according to the rectangular cropping frame to obtain an action driving source in a standard format and a virtual image image in a standard format, and the rectangular cropping frame of the virtual image image is saved as the first cropping parameter.

4. The method for generating a 2D digital human with multiple driving factors according to claim 1, characterized in that: The extraction of the action driving source and the facial parameters of the virtual image and the flow field modeling are specifically as follows: A first facial parameter is obtained from the action driving source using the Deep3DFaceRecon_pytorch model; the first facial parameter includes multiple facial parameters extracted from multiple images in the action driving source, and a second facial parameter is obtained from the image of the virtual image, the second facial parameter is a single facial parameter extracted from the virtual image; The first facial parameter information is injected layer by layer into the second facial parameter through a pre-trained flow field predictor to complete the flow field modeling.

5. The method for generating a 2D digital human with multiple driving factors according to claim 1, characterized in that: The facial parameters are repaired after image reconstruction based on the pixel points in the flow field, specifically: Based on the image reconstruction algorithm, the offset of each pixel point in the flow field is superimposed on the image coordinates of the virtual image; then, the bilinear interpolation method implemented by the numpy package is used to interpolate the image of the virtual image according to the generated coordinates, and finally, the target image is reconstructed according to the pixel values ​​obtained by the interpolation, and the image of the virtual image is distorted into an action-driven first result synchronized with the target action according to the flow field; The motion-driven first result and the image of the virtual image are input into a pre-trained restoration network, and restoration is performed according to the first facial parameters to obtain a motion-driven second result synchronized with the driving source action.

6. The method for generating a 2D digital human with multiple driving factors according to claim 1, characterized in that: The S4 is specifically: Using the s3fd face detection algorithm to detect the facial key points in the second result driven by the action, to obtain the facial key points of each image; then determining the first face position according to the left eye key point center, the right eye key point center and the nose lower key point in the facial key points; Also use three coordinates to define the second face position Calculate the affine transformation matrix based on the first face position and the second face position, and use the warpAffine algorithm in the cv2 package to obtain the rectangular cropping frame; Each image of the second result of the action drive is cropped according to the rectangular cropping frame to obtain a standard image data set of the virtual object, and the rectangular cropping frame here is saved as the second cropping parameter.

7. The method for generating a 2D digital human with multiple driving factors according to claim 1, characterized in that: The S5 is specifically: Determine a lip shape driving source, connect the first lip shape driving source and the second lip shape driving source in a sequence specified by a user, and obtain a third lip shape driving source, wherein the specified sequence specifies an arrangement and combination sequence of each short Mel spectrum in the first lip shape driving source and the second lip shape driving source, and a sentence is used as the minimum unit for combination; According to the lip shape driving source and the standardized virtual object image data set, The lip shape information in the lip shape driving source is captured by using the pre-trained lip shape driving algorithm model, and the lip shape information is mapped to the RGB space of the virtual image, so that the original lip shape of the virtual image is modified into a new lip shape synchronized with the driving audio, and the third result of the lip shape driving is obtained; The image data is inversely transformed using the affine transformation matrix, and the pixel area at the same position in the second result is replaced by the inversely transformed image data according to the second cropping parameter to obtain a fourth result synchronized with the lip driving source.

8. A system for implementing the method according to any one of claims 1 to 7, characterized in that: The system includes: a multi-factor driving source acquisition module, an image acquisition module, a driving source analysis module, a motion driving module, a lip shape driving module, an audio and video synthesis module and a 2D digital human client display module; The multi-factor driving source acquisition module is used to acquire one or more driving factors uploaded by the user. The image acquisition module is used to acquire the face image of the virtual image to be driven. The driving source analysis module is used to analyze the text signal and the voice signal into the lip shape driving source and analyze the video signal into the action driving source: The action driving module is used to generate the virtual image action consistent with the action driving source; the lip shape driving module is used to generate the virtual image lip shape consistent with the lip shape driving source. The audio and video synthesis module is used to merge a 2D digital human video with synchronized audio and video; The 2D digital human client display module is used to display the 2D digital human image stream and play the corresponding 2D digital human voice.

9. A 2D digital human generation device for multiple element driving sources, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a 2D digital human generation method for multiple driving sources according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a 2D digital human generation method for multiple driving sources according to any one of claims 1 to 7 is implemented.

Images