A multi-element driving source-oriented 2D digital human generation method and device

Abstract

The application discloses a kind of 2D digital person generation method and device for multi-element driving source, including obtaining text, voice and video various elements driving source and virtual image to be driven, and using text-to-speech, audio feature extraction and video preprocessing algorithm in driving source analysis module to obtain lip shape driving source and action driving source;The posture action and expression action in action driving source are migrated to the virtual image to be driven, to obtain the driving result synchronized with the action of action driving source;And according to the lip shape driving source, the mouth shape in action driving result is replaced with new mouth shape, to obtain the result synchronized with the lip shape driving source;The fusion audio corresponding to text and voice signal is synthesized with the double driving result of action and lip shape, to obtain the 2D digital person generation result of sound picture matching.The application supports the more controllable 2D digital person generation of multi-element driving source.

Description

Technical Field

[0001] The present invention relates to the field of generative artificial intelligence, and in particular to a method and device for generating a 2D digital human oriented to multiple driving factors. Background Art

[0002] With the development of artificial intelligence, the concept of virtual digital humans has emerged. 2D digital humans are two-dimensional beings that exist in the digital world, endowed with diverse human-like characteristics and abilities through advanced computer technology. These digital characters can express themselves through language, facial expressions, and body movements. They have broad applications in commercial fields such as entertainment and brand promotion, and also show potential in areas such as culture, tourism, education, and healthcare.

[0003] 2D digital human generation technology primarily uses voice or video to drive the virtual human's face, including lip shape and expression, to synchronize the voice or video with the lip shape or expression. Currently, most 2D digital human generation solutions use audio as the sole driving source, replacing the character's lip shape in a template video to generate the digital human video. This lacks controllability of head movement and expression. A few solutions use video as the driving source, but lack flexibility in lip shape changes. Therefore, achieving a unified framework for 2D digital humans driven by multiple factors, flexible and controllable, has become a valuable research direction. Summary of the Invention

[0004] The purpose of the present invention is to address the shortcomings of current 2D digital human generation solutions, such as a single driving source and difficulty in balancing controllability and flexibility, and to propose a 2D digital human generation method and device with multiple driving sources.

[0005] The present invention is achieved through the following technical solutions: On the one hand, the present invention discloses a method for generating a 2D digital human oriented to multiple driving sources, the method comprising the following steps:

[0006] S1. Acquire images of a multi-factor driving source and a virtual image to be driven;

[0007] S2. Perform driver source analysis, including:

[0008] Converting the text signal into a noise-free first audio and extracting features to serve as a first lip shape driving source;

[0009] The speech signal is converted into a noise-free second audio and features are extracted to serve as a second lip shape driving source;

[0010] The video signal and the image of the virtual image to be driven are processed by a video preprocessing algorithm to obtain a standardized action driving source and a standardized virtual image image, as well as a first cropping parameter saved during the processing;

[0011] S3, extract the action driving source and the facial parameters of the virtual image, perform flow field modeling, and repair the facial parameters after image reconstruction according to the pixel points in the flow field to obtain the action driving result;

[0012] S4, performing rectangular frame cropping on the action driving result, combining the obtained second cropping parameter and the standard virtual image, and saving and aligning the affine transformation matrix and the rectangular cropping frame;

[0013] S5, combining the first lip shape driving source and the second lip shape driving source into a third lip shape driving source, combining the first audio and the speech signal into a third audio, mapping the lip shape information in the third lip shape driving source to the virtual image, obtaining a third result driven by the lip shape, then performing inverse transformation according to the affine transformation matrix, and then replacing the image data according to the rectangular cropping frame corresponding to the second cropping parameter to obtain a fourth result driven by the lip shape;

[0014] S6, replacing the central region of the original virtual image with the fourth result driven by the lip shape using the rectangular cropping frame corresponding to the first cropping parameter, and synthesizing the generated video and the third audio to obtain a 2D digital human generation result.

[0015] Further, the parsing process of the first lip shape driving source and the second lip shape driving source in S2 is specifically:

[0016] Using the acoustic model FastSpeech and the vocoder HifiGAN to quickly generate speech from the text signal to obtain a noiseless first audio; then using an audio feature extractor to process the first audio into an audio feature, which is used as the first lip shape driving source; using a noise reduction algorithm implemented by the noisereduce package to reduce noise for each frame of speech signal to obtain a noise-reduced noiseless second audio, and then using an audio feature extractor to process the second audio into an audio feature, which is used as the second lip shape driving source.

[0017] Further, the video preprocessing algorithm in S2 is specifically:

[0018] Using the s3fd face detection algorithm to detect the face key points in the video signal and the virtual image to obtain the face key points of each image;

[0019] Positioning the face position according to the triangle formed by the left eye key point center, the right eye key point center and the mouth key point center in the face key points;

[0020] Extending the triangle positioning the face outward to obtain a rectangular cropping frame, so that the face is in the center of the rectangular cropping frame;

[0021] According to the rectangular cropping frame, each image is cropped to obtain a format standard action driving source and a format standard virtual image, and the rectangular cropping frame of the virtual image is saved as a first cropping parameter.

[0022] Further, the face parameters of the action driving source and the virtual image are extracted, and the flow field modeling is specifically:

[0023] The first face parameter is obtained from the action driving source using the Deep3DFaceRecon_pytorch model; the first face parameter includes multiple face parameters extracted from multiple images in the action driving source, and the second face parameter is obtained from the image of the virtual image, and the second face parameter is a single face parameter extracted from the virtual image;

[0024] The first face parameter information is injected into the second face parameter layer by layer through the pre-trained flow field predictor to complete the flow field modeling.

[0025] Further, the face parameters are repaired after the image reconstruction according to the pixel points in the flow field, and the specific process is:

[0026] Based on the image reconstruction algorithm, the offset of each pixel point in the flow field is superimposed on the image coordinates of the virtual image; then the bilinear interpolation method realized by the numpy package is used to perform interpolation on the virtual image according to the generated coordinates, and finally the target image is reconstructed according to the pixel value obtained by interpolation, and the image of the virtual image is distorted into an action driving first result synchronized with the target action according to the flow field;

[0027] The action driving first result and the image of the virtual image are input into the pre-trained repair network, and the first face parameter is repaired to obtain an action driving second result synchronized with the driving source action.

[0028] Further, S4 is specifically:

[0029] The face key points in the action driving second result are detected using the s3fd face detection algorithm to obtain the face key points of each image; then the first face position is determined according to the left eye key point center, the right eye key point center and the nose lower key point in the face key points;

[0030] Similarly, three coordinates are used to define the second face position

[0031] The affine transformation matrix is calculated according to the first face position and the second face position, and the rectangular cropping frame is obtained using the warpAffine algorithm in the cv2 package.

[0032] According to the rectangular clipping frame, each image of the action-driven second result is clipped to obtain a set of image data of a standard virtual object, and the rectangular clipping frame is saved as a second clipping parameter.

[0033] Further, the S5 is specifically:

[0034] A lip driving source is determined, and the first lip driving source and the second lip driving source are connected according to a user-specified order to obtain a third lip driving source, wherein the specified order defines the arrangement and combination order of each short mel-frequency spectrum in the first lip driving source and the second lip driving source, and the combination is performed in a sentence as a minimum unit;

[0035] According to the lip driving source and the set of image data of the standard virtual object,

[0036] The lip shape information in the lip driving source is captured by using a pre-trained lip driving algorithm model, the lip shape information is mapped to the RGB space of the virtual image, so that the original lip shape of the virtual image is modified to a new lip shape synchronized with the driving audio, and a third result of lip driving is obtained.

[0037] The image data is inversely transformed using the affine transformation matrix, and the inversely transformed image data is replaced with a pixel region at the same position in the second result according to the second clipping parameter, to obtain a fourth result synchronized with the lip driving source.

[0038] On the other hand, the specification also provides a 2D digital human generation system for a multi-element driving source, which comprises a multi-element driving source acquisition module, an image acquisition module, a driving source analysis module, an action driving module, a lip driving module, an audio-video synthesis module and a 2D digital human client display module.

[0039] The multi-element driving source acquisition module is used to acquire one or more driving elements uploaded by a user,

[0040] The image acquisition module is used to acquire a face image of a virtual image to be driven,

[0041] The driving source analysis module is used to analyze text signals and voice signals into lip driving sources and analyze video signals into action driving sources.

[0042] The action driving module is used to generate virtual image actions consistent with the action driving source; the lip driving module is used to generate virtual image lip shapes consistent with the lip driving source,

[0043] The audio-video synthesis module is used to merge an audio-video synchronized 2D digital human video.

[0044] The 2D digital human client display module is used to display a 2D digital human image stream and play corresponding 2D digital human voice.

[0045] In another aspect, the specification also provides a 2D digital human generation device for a multi-element driving source, comprising a memory and one or more processors, wherein the memory stores executable code, and the processor executes the executable code to implement the 2D digital human generation method for the multi-element driving source.

[0046] In another aspect, the specification also provides a computer-readable storage medium having a program stored thereon, wherein the program is executed by a processor to implement the 2D digital human generation method for the multi-element driving source.

[0047] The beneficial effects of the present application are as follows:

[0048] 1. The 2D digital human driving source supports multiple elements, and one or more combinations of text signals, voice signals and video signals can be used as driving elements of the digital human to perform driving, and the 2D digital human generation scene is more suitable;

[0049] 2. The virtual image is driven by the action driving module and the lip driving module, and the head posture action, the facial expression action and the mouth shape action of the virtual image can be driven at the same time, so that the generated 2D digital human image is more controllable;

[0050] 3. The method of using multi-level gradient clipping image set and saving clipping parameters can gradually clip the virtual image and the intermediate generation result, which can reduce the required resources and improve the generation speed. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A 2D digital human generation method for a multi-element driving source provided by an embodiment of the present application is shown in the figure;

[0052] Figure 2 An implementation flowchart of the 2D digital human generation method for text, voice and video multi-driving elements provided by an embodiment of the present application is shown in the figure;

[0053] Figure 3 An implementation flowchart of the 2D digital human generation method for text and voice two kinds of lip driving elements provided by an embodiment of the present application is shown in the figure;

[0054] Figure 4 A structure schematic diagram of a 2D digital human generation system for a multi-element driving source provided by an embodiment of the present application is shown in the figure;

[0055] Figure 5 A structure schematic diagram of a 2D digital human generation device for a multi-element driving source provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0057] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0058] The present invention mainly solves the 2D digital human generation method for multiple driving sources through the following approaches: the present invention obtains driving sources of multiple elements of text, voice and video and the virtual image to be driven, and uses the text-to-speech, audio feature extraction and video preprocessing algorithms in the driving source analysis module to obtain the lip shape driving source and the action driving source; uses the action driving module to transfer the posture action and expression action in the action driving source to the virtual image to be driven, and obtains the driving result synchronized with the action driving source action; then uses the lip shape driving module to replace the lip shape in the action driving result with the new lip shape according to the lip shape driving source, and obtains the result synchronized with the lip shape driving source; finally uses the audio and video synthesis module to synthesize the fused audio corresponding to the text and voice signals with the dual driving results of action and lip shape, and obtains the 2D digital human generation result that matches the sound and picture. Figure 1 shown.

[0059] First embodiment

[0060] like Figure 2 As shown, this embodiment provides a method for generating a 2D digital human based on multiple driving factors. The specific steps of the method are as follows:

[0061] S1. Acquire a multi-factor driving source, including: a text signal, a voice signal, and a video signal. The driving source can be one or more of them, and acquire an image of the virtual image to be driven.

[0062] Among them, the virtual image can be any role, including but not limited to teachers, doctors, cartoon characters, and virtual idols; text signals and voice signals represent the information that needs to be "said" by the 2D digital human; and the characters in the video signal are preset to perform some actions that do not cover the face, such as blinking, which is not specifically limited here, indicating the head posture and expression movements that need to be "performed" by the 2D digital human.

[0063] S2: Analyze the driving source and analyze the multi-factor driving source into the lip driving source and action driving source that can be received by the subsequent driving module.

[0064] The text signal is quickly converted into speech by using an acoustic model FastSpeech and a vocoder HifiGAN to obtain a first audio without noise; then the first audio is processed into audio features by using an audio feature extractor, and the audio features are used as a first lip shape driving source;

[0065] The noise reduction algorithm implemented by the noisereduce package is used to reduce noise for each frame of speech signal to obtain a second audio without noise after noise reduction, and then the second audio is processed into audio features by using an audio feature extractor, and the audio features are used as a second lip shape driving source;

[0066] The video signal and the image of the virtual image to be driven are processed by a video preprocessing algorithm to obtain a motion driving source with a standard format and a standard virtual image image.

[0067] The process of extracting audio features is specifically:

[0068] First, the sampling rate of the first audio and the second audio is detected, and if the sampling rate is not 16 kHz, ffmpeg is used to resample it to 16 kHz to obtain new first audio and second audio;

[0069] Then, the audio with a sampling rate of 16 kHz is converted into mel spectrum features by using the audio feature extractor implemented by the libsora package;

[0070] Then, the mel spectrum features are divided according to the target frame rate of the final 2D digital human generation video to obtain a series of short mel spectrum composed of lip shape driving sources, i.e., first lip shape driving source and second lip shape driving source.

[0071] The process of processing the video signal and the image of the virtual image to be driven by the video preprocessing algorithm is specifically:

[0072] First, the s3fd face detection algorithm is used to detect the face key points in the video signal and the virtual image image to obtain 68 face key points for each image, wherein each key point is composed of an x coordinate and a y coordinate, and the coordinate system is with the left upper corner of the image as the origin, with the upper edge of the image as the x axis, and with the left edge of the image as the y axis;

[0073] Then, the face position is located according to the triangle composed of the left eye key point center (36th to 39th key points), the right eye key point center (42nd to 45th key points) and the mouth key point center (48th to 54th key points) in the 68 face key points;

[0074] Then, the triangle locating the face is expanded outward to obtain a rectangular cropping frame, so that the face is located in the center of the rectangular cropping frame;

[0075] Finally, according to the rectangular cropping frame, the action driving source and the virtual image in a standard format are obtained, and the rectangular cropping frame of the virtual image is saved as the first cropping parameter.

[0076] In step S3, the first face parameter is obtained from the action driving source using the 3DMM model, and the first face parameter includes multiple face parameters extracted from multiple images in the action driving source. Similarly, the second face parameter is obtained from the image of the virtual image using the 3DMM model, and the second face parameter is only a single face parameter extracted from the virtual image.

[0077] The 3DMM model refers to a three-dimensional face information reconstruction model implemented by Deep3DFaceRecon_pytorch, and the extracted face parameters include identity parameters, texture parameters, expression parameters, etc.

[0078] According to the first face parameter, the second face parameter is modeled by a flow field, and a flow field storing action information in the action driving source and identity information in the virtual image is obtained.

[0079] The flow field modeling is achieved by injecting the first face parameter information into the second target parameter layer by layer through a pre-trained flow field predictor, and the obtained flow field can depict the migration path of each RGB pixel point in the virtual image.

[0080] Based on the image reconstruction algorithm, the image of the virtual image is warped into an action driving first result synchronized with the target action according to the flow field.

[0081] The action driving first result and the image of the virtual image are input into a pre-trained repair network (such as the Editing network in PIRedener), and are repaired according to the first face parameter to obtain an action driving second result, which is a result synchronized with the driving source action.

[0082] The warping operation specifically includes: first, adding the offset of each pixel point in the flow field to the coordinates of the virtual image; then using a bilinear interpolation method to perform interpolation on the virtual image according to the generated coordinates; finally, reconstructing the target image according to the pixel values obtained by interpolation. The bilinear interpolation method can be realized by a numpy package.

[0083] In step S4, the action driving result is cropped by a rectangular frame, the obtained cropping parameter and the standard virtual image are combined, and the affine transformation matrix and the rectangular cropping frame are saved and aligned.

[0084] The processing process of the video preprocessing algorithm is different from the foregoing steps, and specifically includes:

[0085] The s3fd human face detection algorithm is used to detect the human face key points in the action-driven second result, to obtain 68 human face key points of each image; then the first human face position is determined according to the left eye key point center (36th to 39th key points), right eye key point center (42nd to 45th key points) and lower nose key point (33rd key point) in the 68 human face key points;

[0086] The second human face position is also defined by three coordinates: (17, 20), (58, 20) and (37.5, 40), which are all based on the image size of 256*256;

[0087] The affine transformation matrix is calculated according to the first human face position and the second human face position, and the warpAffine algorithm in the cv2 package is used to obtain the rectangular clipping frame;

[0088] The action-driven second result is aligned and clipped by the video preprocessing algorithm to obtain a standardized virtual object image data set, and the affine transformation matrix and the rectangular clipping frame in the alignment and clipping process are saved, and the rectangular clipping frame is the second clipping parameter.

[0089] S5, determine the lip shape driving source, connect the first lip shape driving source and the second lip shape driving source according to the user specified order to obtain the third lip shape driving source as the direct input of the lip shape synchronization module, and save the specified order as the connection parameter. Wherein, the specified order specifies the arrangement and combination order of each short mel spectrum in the first lip shape driving source and the second lip shape driving source, and a sentence is taken as the minimum unit for combination.

[0090] According to the lip shape driving source and the standardized virtual object image data set, the pre-trained lip shape driving algorithm model is used to capture the lip shape information in the driving source, and the lip shape information is mapped to the RGB space of the virtual image, so that the original lip shape of the virtual image is modified to a new lip shape synchronized with the driving audio, to obtain the lip shape driven third result.

[0091] Since the image data set of the third result is all standardized aligned images, the affine transformation matrix is used to perform inverse transformation on the image data, and the inverse transformed image data is replaced with the pixel area at the same position in the second result according to the second clipping parameter, to obtain the fourth result synchronized with the lip shape driving source, and the image size of the fourth result is consistent with that of the action-driven second result.

[0092] The first audio and the speech signal are arranged and combined according to the connection parameter to obtain the third audio.

[0093] S6, since the size of the fourth result image data set is part of the original virtual image, it is necessary to replace the fourth result image data with the same position pixel area in the original virtual image using the rectangular clipping frame corresponding to the first clipping parameter to obtain an image data set consistent with the size of the original image; based on the generated image data set and the third audio, use the ffmpeg tool to merge audio and video, set the target frame rate to 25FPS and the target audio sampling rate to 16kHz, to obtain a 2D digital human generation result with matching sound and picture.

[0094] Second embodiment

[0095] As Figure 3 shown, the method provides an embodiment of a missing video signal for assisting understanding, a 2D digital human generation method facing text and voice driven elements, the specific steps of the method are as follows:

[0096] S1, obtain two driving elements of text signal and voice signal, and obtain the video of the virtual image to be driven mouth shape; wherein the virtual image can be any role, including but not limited to teachers, doctors, cartoon characters, virtual idols, and the video to be driven mouth shape is a video containing a complete face of the virtual image; the virtual image can be preset to make some actions that do not block the face, such as blinking, which is not limited here.

[0097] S2, perform driving source analysis to analyze the multi-element driving source into a lip shape driving source and an action driving source that can be received by the subsequent driving module. The text signal is converted into a noise-free first audio by a TTS algorithm, and the first audio is processed into an audio feature using an audio feature extractor, which is used as a first lip shape driving source; the voice signal is converted into a noise-free second audio by an audio noise reduction algorithm, and the second audio is processed into an audio feature using an audio feature extractor, which is used as a second lip shape driving source;

[0098] The processing process of the text signal and the voice signal is similar to step S2 in the first embodiment, which will not be repeated here.

[0099] S3, determine the lip shape driving source, connect the first lip shape driving source and the second lip shape driving source according to the user specified order to obtain a third lip shape driving source as the direct input of the lip shape synchronization module, and save the specified order as the connection parameter. Wherein, the specified order defines the arrangement and combination order of each short mel frequency spectrum in the first lip shape driving source and the second lip shape driving source, and a sentence is used as the minimum unit for combination.

[0100] S4, use a video preprocessing algorithm to align and crop the virtual image video to obtain a standard virtual object image data set, and save the affine transformation matrix and the clipping parameter in this process.

[0101] The video pre-processing process is similar to S4 in the first embodiment, and will not be described here.

[0102] In step S5, according to the third lip-driven source and the image data set of the specified virtual object, the lip-driven algorithm model is used to capture the mouth shape information in the driving source, and the mouth shape information is mapped to the RGB space of the virtual image, so that the original mouth shape of the virtual image is modified to a new mouth shape synchronized with the driving audio, and a first result of lip driving is obtained.

[0103] The lip-driven result is inversely transformed using the affine transformation matrix, and the inversely transformed image data is replaced with the pixel region at the same position in the original virtual image video according to the cropping parameters, to obtain a second result synchronized with the lip-driven source.

[0104] The first audio and the speech signal are arranged and combined according to the connection parameters to obtain a third audio.

[0105] In step S6, based on the third audio and the second result synchronized with the lip-driven source, the ffmpeg tool is used to merge the audio and video, the target frame rate is set to 25FPS, and the target audio sampling rate is set to 16kHz, to obtain a 2D digital human generation result with sound and picture matching.

[0106] Third embodiment

[0107] Another embodiment of the present application also discloses a device, which comprises: a multi-element driving source acquisition module written using a file upload framework, used to acquire one or more driving elements uploaded by a user, the driving elements including a text signal, a speech signal and a video signal; an image acquisition module written using a file upload framework, used to acquire a face image of a virtual image to be driven; a driving source analysis module including an acoustic model FastSpeech, a vocoder HifiGAN and a video preprocessing algorithm, used to analyze the text signal and the speech signal into a lip-driven source and analyze the video signal into a motion-driven source; a motion driving module using a 3DMM model and an expression transfer model, used to control the head posture action and the expression action of the virtual image according to the motion-driven source, so that the action of the virtual image is consistent with the motion-driven source; a lip-driven module based on a pre-trained lip-driven model, used to change the mouth shape of the virtual character in the output result of the motion driving module according to the lip-driven source, so that the mouth shape of the virtual image is consistent with the lip-driven source; an audio and video synthesis module written using ffmpeg, used to combine the result output by the lip-driven module and the audio corresponding to the text signal and the speech signal into a 2D digital human video with picture and audio synchronization; and a 2D digital human client display module written based on Vue3, used to display a 2D digital human image stream and play corresponding 2D digital human speech.

[0108] Fourth embodiment

[0109] Corresponding to the foregoing embodiment of the method for generating a 2D digital human facing a multi-element driving source, the application also provides an embodiment of a device for generating a 2D digital human facing a multi-element driving source.

[0110] Referring to Figure 5 The device for generating a 2D digital human facing a multi-element driving source provided by the embodiment of the application comprises a memory and one or more processors, the memory stores executable code, and the processor executes the executable code to implement the method for generating a 2D digital human facing a multi-element driving source in any of the foregoing embodiments.

[0111] The embodiment of the device for generating a 2D digital human facing a multi-element driving source provided by the application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logically meaningful device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for execution by the processor of the device with data processing capability where the device is located. From the hardware level, as shown in Figure 5 As shown in the figure, in addition to the processor, the memory, the network interface, and the non-volatile memory shown in the figure, the device with data processing capability where the device is located in the embodiment usually also comprises other hardware according to the actual functions of the device with data processing capability, and details are not described herein. Figure 5

[0112] The implementation process of the functions and roles of each unit in the device is specifically described in the implementation process of the corresponding steps in the method, and details are not described herein.

[0113] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part is described in the part of the method embodiment. The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the application scheme. Those skilled in the art can understand and implement it without creative labor.

[0114] ​The embodiment of the present application also provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the 2D digital human generation method for multiple-element driving sources.

[0115] The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory, of any of the preceding embodiments. The computer readable storage medium can also be an external storage device of any data processing capable device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both an internal storage unit and an external storage device of any data processing capable device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0116] The present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the 2D digital human generation method for multiple-element driving sources.

[0117] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The specification and examples given are exemplary and are not intended to limit the scope of the application. The true scope of the application is indicated by the claims.

[0118] It should be understood that the general description above and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. The present application is not limited to the precise structures described and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is only limited by the appended claims.

Claims

1. A 2D digital human generation method for multiple driving factors, characterized by: The method comprises the following steps: S1. Acquire images of a multi-factor driving source and a virtual image to be driven; S2. Perform driver source analysis, including: Converting the text signal into a noise-free first audio and extracting features to serve as a first lip shape driving source; The speech signal is converted into a noise-free second audio and features are extracted to serve as a second lip shape driving source; The video signal and the image of the virtual image to be driven are processed by a video preprocessing algorithm to obtain a standardized action driving source and a standardized virtual image, and a rectangular cropping frame of the virtual image is used as a first cropping parameter; S3. Extract the facial parameters of the action driving source and the virtual image, perform flow field modeling, and perform image reconstruction based on the pixels in the flow field to repair the facial parameters. Specifically: Based on an image reconstruction algorithm, the offset of each pixel in the flow field is superimposed on the image coordinates of the virtual image; then, a bilinear interpolation method implemented in the numpy package is used to interpolate the generated coordinates on the image of the virtual image; finally, the target image is reconstructed based on the interpolated pixel values, and the image of the virtual image is warped according to the flow field to a motion-driven first result that is synchronized with the target action; Inputting the motion-driven first result and the image of the virtual image into a pre-trained restoration network, and performing restoration based on the first facial parameters to obtain a motion-driven second result synchronized with the driving source motion; S4, performing a rectangular frame cropping on the action-driven result, combining the obtained cropping parameters with the standard avatar image, and saving and aligning the affine transformation matrix with the rectangular cropping frame; S5. Combining the first lip shape driving source and the second lip shape driving source into a third lip shape driving source, and combining the first audio signal and the voice signal into a third audio signal; mapping the lip shape information in the third lip shape driving source to a virtual image, and then performing an inverse transformation according to an affine transformation matrix to obtain a third lip shape driving result; and then replacing the image data according to a rectangular cropping frame to obtain a fourth lip shape driving result; S6. Replace the image data in the original virtual image cropping frame with the fourth result according to the first cropping parameter, synthesize the fourth result driven by the lip shape and the third audio to obtain a 2D digital human generation result.

2. The method for generating a 2D digital human based on multiple driving factors according to claim 1, characterized in that: The analysis process of the first lip-shaped driving source and the second lip-shaped driving source in S2 is specifically as follows: The acoustic model FastSpeech and the vocoder HifiGAN are used to quickly generate speech from the text signal to obtain a noise-free first audio. The first audio is then processed into audio features using an audio feature extractor, and the audio features are used as the first lip shape driving source. Each frame of the speech signal is denoised using the noise reduction algorithm implemented by the noisereduce package to obtain a noise-free second audio after denoising. The second audio is then processed into audio features using an audio feature extractor, and the audio features are used as the second lip shape driving source.

3. The method for generating a 2D digital human based on multiple driving factors according to claim 1, characterized in that: The video preprocessing algorithm in S2 is specifically as follows: Use the s3fd face detection algorithm to detect facial key points in the video signal and the virtual image, and obtain the facial key points of each image; Locate the face position based on the triangle formed by the center of the left eye key point, the center of the right eye key point and the center of the mouth key point in the face key points; Expand the triangle that locates the face outward to obtain a rectangular cropping frame, so that the face is centered in the rectangular cropping frame; Each image is cropped according to the rectangular cropping frame to obtain a standardized action driving source and a standardized virtual image, and the rectangular cropping frame of the virtual image is saved as a first cropping parameter.

4. The method for generating a 2D digital human with multiple driving factors according to claim 1, characterized in that: The extraction of the action driving source and the facial parameters of the virtual image and the flow field modeling is specifically as follows: A first facial parameter is obtained from the action-driven source using the Deep3DFaceRecon_pytorch model; the first facial parameter includes multiple facial parameters extracted from multiple images in the action-driven source, and a second facial parameter is obtained from the image of the avatar, where the second facial parameter is a single facial parameter extracted from the avatar; The first facial parameter information is injected layer by layer into the second facial parameter through a pre-trained flow field predictor to complete the flow field modeling.

5. The method for generating a 2D digital human with multiple driving factors according to claim 1, characterized in that: The S4 is specifically: Using the s3fd face detection algorithm to detect facial key points in the second result of the action drive, the facial key points of each image are obtained; then, the first face position is determined based on the left eye key point center, the right eye key point center, and the nose lower key point in the facial key points; Also use three coordinates to define the second face position Calculate the affine transformation matrix based on the first and second face positions, and use the warpAffine algorithm in the cv2 package to obtain a rectangular cropping frame; Each image of the second result of the action drive is cropped according to the rectangular cropping frame to obtain a standard image data set of the virtual object, and the rectangular cropping frame here is saved as the second cropping parameter.

6. The method for generating a 2D digital human with multiple driving factors according to claim 5, characterized in that: The S5 is specifically: Determining a lip shape driving source, and connecting the first lip shape driving source and the second lip shape driving source in a user-specified order to obtain a third lip shape driving source, wherein the specified order specifies the order of permutation and combination of short mel spectra in the first lip shape driving source and the second lip shape driving source, with a sentence as the minimum unit for combination; According to the lip shape driving source and the standardized virtual object image data set, The pre-trained lip-driven algorithm model is used to capture the lip shape information in the lip-driven source, and the lip shape information is mapped to the RGB space of the virtual image, so that the original lip shape of the virtual image is modified into a new lip shape that is synchronized with the driving audio, thereby obtaining the third result of lip-driven. The image data is inversely transformed using the affine transformation matrix, and the pixel area at the same position in the second result is replaced by the inversely transformed image data according to the second cropping parameter to obtain a fourth result synchronized with the lip driving source.

7. A system for implementing the method according to any one of claims 1 to 6, characterized in that: The system includes: multi-factor driving source acquisition module, image acquisition module, driving source analysis module, action driving module, lip shape driving module, audio and video synthesis module and 2D digital human client display module; The multi-factor driving source acquisition module is used to collect one or more driving factors uploaded by the user. The image acquisition module is used to acquire the face image of the virtual image to be driven, The driving source analysis module is used to analyze the text signal and the voice signal into the lip driving source and analyze the video signal into the action driving source: The action driving module is used to generate the virtual image action consistent with the action driving source; the lip shape driving module is used to generate the virtual image lip shape consistent with the lip shape driving source. The audio and video synthesis module is used to merge a 2D digital human video with synchronized audio and video; The 2D digital human client display module is used to display the 2D digital human image stream and play the corresponding 2D digital human voice.

8. A 2D digital human generation device for multiple driving sources, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a 2D digital human generation method for multiple driving sources according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a 2D digital human generation method for multiple driving sources according to any one of claims 1 to 6 is implemented.

Images