Digital human generation method and device, program product and medium
By converting long text into multiple audio bands in parallel to generate digital human videos, the problem of long text generation in the existing technology is solved, and the processing efficiency is improved.
Patent Information
- Application Number
- CN202411071805.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-03
AI Technical Summary
The existing solution for generating digital people based on text is too long to generate long texts and it is difficult to meet production needs.
By converting the target text information into audio information, and dividing it into multiple audio frame sequences according to the preset image sequence length for parallel processing, the digital human image sequence is generated separately, and the target digital human video is finally synthesized.
It effectively reduces the time for long text to generate digital people, improves processing efficiency, and meets production needs.
Smart Images

Figure CN119071597A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, program product and medium for generating a digital human. Background Art
[0002] Existing text-based digital human generation solutions are basically only aimed at the generation of short texts, usually using frame-by-frame or small batch inference. The synthesis speed is limited, and the generation of long texts is time-consuming, which makes it difficult to meet production needs. Summary of the invention
[0003] The present invention provides a digital human generation method, device, program product and medium, which solves the problem that it takes too long to generate a digital human based on a long text in the prior art.
[0004] In order to solve the above technical problems, in a first aspect, the present invention provides a method for generating a digital human, comprising:
[0005] Generate audio information corresponding to the target text information;
[0006] The audio information is segmented according to the length of a preset image sequence to obtain at least two audio frame sequences, wherein the image frames in the preset image sequence include a digital human image;
[0007] Based on the preset image sequence, the at least two audio frame sequences are processed in parallel to obtain a digital human image sequence corresponding to each audio frame sequence;
[0008] The audio information and the digital human image sequences respectively corresponding to the at least two audio frame sequences are synthesized to obtain a target digital human video.
[0009] Optionally, the processing of the at least two audio frame sequences in parallel based on the preset image sequence to obtain a digital human image sequence corresponding to each of the audio frame sequences includes:
[0010] In the process of processing each of the audio frame sequences, obtaining driving parameters corresponding to the audio frame sequence;
[0011] According to the driving parameters, driving processing is performed on the preset image sequence to obtain a first image sequence corresponding to the audio frame sequence;
[0012] Based on the first image sequence, a digital human image sequence corresponding to the audio frame sequence is obtained.
[0013] Optionally, the parallel processing of the at least two audio frame sequences and the processing of obtaining a digital human image sequence corresponding to the audio frame sequence based on the first image sequence are performed asynchronously.
[0014] Optionally, obtaining a digital human image sequence corresponding to the audio frame sequence based on the first image sequence includes:
[0015] Performing correction processing on each image frame in the first image sequence to obtain a second image sequence corresponding to the first image sequence;
[0016] Super-resolution processing is performed on a second image sequence corresponding to the first image sequence to obtain a digital human image sequence corresponding to the audio frame sequence, wherein the resolution of the digital human image sequence is higher than that of the second image sequence.
[0017] Optionally, performing correction processing on each image frame in the first image sequence to obtain a second image sequence corresponding to the first image sequence includes:
[0018] In the correction processing process of each first image frame in the first image sequence, a target face image frame corresponding to the first image frame is obtained, wherein the first image frame is any image frame in the first image sequence, and the target face image frame includes a frontal perspective of the face image in the first image frame;
[0019] Performing key point detection on the target face image frame to obtain coordinates of key points in the target face image frame, wherein the key points include facial feature points of the face in the target face image frame;
[0020] According to the coordinates of the key points in the target face image frame, the first image frame is corrected to obtain a first target image frame corresponding to the first image frame;
[0021] When each of the first image frames has been corrected, a first target image frame corresponding to at least one of the first image frames is determined as the second image sequence.
[0022] Optionally, acquiring a target face image frame corresponding to the first image frame includes:
[0023] Acquire coordinate information corresponding to a face frame of a second image frame, where the second image frame is: an image frame in the preset image sequence whose order is consistent with that of the first image frame in the first image sequence;
[0024] Cropping the first image frame according to coordinate information corresponding to the face frame of the second image frame to obtain a face image frame corresponding to the first image frame;
[0025] The angle of the face in the first image frame is adjusted to obtain a target face image frame corresponding to the first image frame.
[0026] Optionally, the synthesizing the audio information and the digital human image sequences respectively corresponding to the at least two audio frame sequences to obtain a target digital human video includes:
[0027] According to the order of the at least two audio frame sequences from front to back in the audio information, the digital human image sequence corresponding to each of the audio frame sequences is sorted;
[0028] The sorted digital human image sequence and the audio information are synthesized to obtain a target digital human video.
[0029] In a second aspect, an embodiment of the present invention further provides a device, comprising: a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the digital human generation method as described in the first aspect.
[0030] In a third aspect, an embodiment of the present invention further provides a digital human generation device, the device comprising:
[0031] A first processing module, used to generate audio information corresponding to the target text information;
[0032] An audio segmentation module, used to segment the audio information according to the length of a preset image sequence to obtain at least two audio frame sequences, wherein the image frames in the preset image sequence include a digital human image;
[0033] A second processing module, configured to process the at least two audio frame sequences in parallel based on the preset image sequence, and obtain a digital human image sequence corresponding to each of the audio frame sequences;
[0034] The synthesis processing module is used to synthesize the audio information and the digital human image sequences corresponding to the at least two audio frame sequences to obtain a target digital human video.
[0035] Optionally, the second processing module includes:
[0036] A first driving submodule, configured to obtain driving parameters corresponding to the audio frame sequence during the processing of each audio frame sequence;
[0037] A second driving submodule, configured to drive the preset image sequence according to the driving parameters to obtain a first image sequence corresponding to the audio frame sequence;
[0038] The first processing submodule is used to obtain a digital human image sequence corresponding to the audio frame sequence based on the first image sequence.
[0039] Optionally, the parallel processing of the at least two audio frame sequences and the processing of obtaining a digital human image sequence corresponding to the audio frame sequence based on the first image sequence are performed asynchronously.
[0040] Optionally, the first processing submodule includes:
[0041] a correction processing unit, configured to perform correction processing on each image frame in the first image sequence to obtain a second image sequence corresponding to the first image sequence;
[0042] The super-resolution processing unit is used to perform super-resolution processing on the second image sequence corresponding to the first image sequence to obtain a digital human image sequence corresponding to the audio frame sequence, wherein the resolution of the digital human image sequence is higher than that of the second image sequence.
[0043] Optionally, the correction processing unit includes:
[0044] A first processing subunit is configured to obtain a target face image frame corresponding to each first image frame in the first image sequence during the correction processing of the first image frame, wherein the first image frame is any image frame in the first image sequence, and the target face image frame includes a frontal perspective of the face image in the first image frame;
[0045] A key point detection subunit, configured to perform key point detection on the target face image frame to obtain coordinates of key points in the target face image frame, wherein the key points include facial feature points of the face in the target face image frame;
[0046] A correction processing subunit, configured to perform correction processing on the first image frame according to the coordinates of the key points in the target face image frame, so as to obtain a first target image frame corresponding to the first image frame;
[0047] The second processing sub-unit is configured to, when each of the first image frames has been corrected, determine a first target image frame corresponding to at least one of the first image frames as the second image sequence.
[0048] Optionally, the first processing subunit includes:
[0049] a third processing sub-unit, configured to obtain coordinate information corresponding to a face frame of a second image frame, wherein the second image frame is an image frame in the preset image sequence, the order of which is consistent with the order of the first image frame in the first image sequence;
[0050] an image cropping subunit, configured to crop the first image frame according to coordinate information corresponding to a face frame of the second image frame, to obtain a face image frame corresponding to the first image frame;
[0051] The angle adjustment subunit is used to adjust the angle of the face in the first image frame to obtain a target face image frame corresponding to the first image frame.
[0052] Optionally, the synthesis processing module includes:
[0053] A sequence sorting submodule, used to sort the digital human image sequence corresponding to each audio frame sequence according to the order of the at least two audio frame sequences from front to back in the audio information;
[0054] The synthesis processing submodule is used to perform synthesis processing on the sorted digital human image sequence and the audio information to obtain the target digital human video.
[0055] In a fourth aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the method for generating a digital human as described in the first aspect.
[0056] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the method for generating a digital human as described in the first aspect above are implemented.
[0057] The beneficial effects of the above technical solution of the present invention are as follows:
[0058] In the embodiment of the present invention, after the audio information corresponding to the target text information is generated, the audio information can be segmented according to the length of the preset image sequence to obtain at least two audio frame sequences, and then the at least two audio frame sequences are processed in parallel based on the preset image sequence to obtain the digital human image sequence corresponding to each audio frame sequence, and finally the audio information and the digital human image sequence corresponding to the at least two audio frame sequences are synthesized to obtain the target digital human video. In this way, for the scenario of generating a digital human from a long text, the audio converted from the long text can be segmented into multiple audio segments, so that the multiple audio segments can be processed in parallel in the subsequent processing process, thereby effectively reducing the time of generating a digital human based on the long text, improving the processing efficiency, and better meeting the production needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0060] Figure 1A schematic diagram showing a flow chart of a method for generating a digital human according to an embodiment of the present invention;
[0061] Figure 2 A schematic diagram showing a digital human generating device according to an embodiment of the present invention executing a digital human generating method;
[0062] Figure 3 A processing schematic diagram showing a digital human generation module according to an embodiment of the present invention;
[0063] Figure 4 A flowchart showing the generation of key points according to an embodiment of the present invention;
[0064] Figure 5 A schematic diagram showing the pipeline processing of a digital human generation module according to an embodiment of the present invention;
[0065] Figure 6 A schematic diagram showing thread pool processing according to an embodiment of the present invention;
[0066] Figure 7 A schematic diagram showing the structure of a digital human generating device according to an embodiment of the present invention;
[0067] Figure 8 A schematic diagram showing the structure of a device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0068] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as specific configurations and components are provided only to help fully understand the embodiments of the present invention. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. In addition, for clarity and brevity, the description of known functions and structures is omitted.
[0069] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present invention. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0070] In various embodiments of the present invention, it should be understood that the order of execution of the following processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. In addition, the terms "system" and "network" are often used interchangeably in this article.
[0071] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.
[0072] In the embodiment of the present invention, the user terminal may be a mobile phone (or cell phone), or other devices capable of sending or receiving wireless signals, including user equipment, personal digital assistants (PDAs), wireless modems, wireless communication devices, handheld devices, laptop computers, cordless phones, wireless local loop (WLL) stations, CPE (Customer Premise Equipment) capable of converting mobile signals into WiFi signals, or mobile smart hotspots, smart home appliances, or other devices that can spontaneously communicate with a mobile communication network without human operation.
[0073] like Figure 1 As shown, an embodiment of the present invention provides a method for generating a digital human, comprising the following steps:
[0074] Step 11: Generate audio information corresponding to the target text information.
[0075] Here, the target text information may be a long text, for example, a text information of more than 10,000 words. In the prior art, it takes too long to generate a digital human based on such a long text.
[0076] Step 12: Segment the audio information according to the length of a preset image sequence to obtain at least two audio frame sequences, wherein the image frames in the preset image sequence contain a digital human image.
[0077] Here, the preset image sequence is a preset template, which is used to generate the target digital human video. Assuming that the length of the preset image sequence is L (that is, the preset image sequence includes L image frames), the audio information generated according to the target text information is sliced (divided) according to the length L, and the obtained n audio frame sequences (that is, audio segments) are respectively recorded as: V1, V2, V3..., V n , n audio frame sequences can be arranged in sequence according to chronological order, wherein each audio frame sequence includes L audio frames, and the L audio frames are arranged in sequence according to chronological order.
[0078] Step 13: Based on the preset image sequence, the at least two audio frame sequences are processed in parallel to obtain a digital human image sequence corresponding to each of the audio frame sequences.
[0079] Here, the processing efficiency can be effectively improved by processing at least two audio frame sequences obtained by segmentation in parallel.
[0080] Step 14: synthesizing the audio information and the digital human image sequences corresponding to the at least two audio frame sequences to obtain a target digital human video.
[0081] In some optional embodiments, step 14 may specifically include the following steps: sorting the digital human image sequence corresponding to each audio frame sequence according to the order of the at least two audio frame sequences from front to back in the audio information; synthesizing the sorted digital human image sequence and the audio information to obtain the target digital human video.
[0082] For example, the n audio frame sequences obtained by segmenting the audio information are respectively: V1, V2, V3..., V n , the digital human image sequences corresponding to the n audio frame sequences are: O1, O2, ..., O n-1 , O n , then the sorted digital human image sequence is the sequence: O1 O2...O n-1 O n .
[0083] In this embodiment, after the audio information corresponding to the target text information is generated, the audio information can be segmented according to the length of the preset image sequence to obtain at least two audio frame sequences, and then the at least two audio frame sequences are processed in parallel based on the preset image sequence to obtain the digital human image sequence corresponding to each audio frame sequence, and finally the audio information and the digital human image sequence corresponding to the at least two audio frame sequences are synthesized to obtain the target digital human video. In this way, for the scenario of generating a digital human from a long text, the audio converted from the long text can be segmented into multiple audio segments, so that multiple audio segments can be processed in parallel in the subsequent processing process, thereby effectively reducing the time for generating a digital human based on a long text, improving processing efficiency, and better meeting production needs.
[0084] like Figure 2 As shown, in a specific embodiment of the present invention, a digital human generation device can execute the digital human generation method provided in the embodiment of the present invention, and the device can mainly include the following modules: input module, TTS voice generation module, voice distribution module, digital human generation module and video synthesis module. When the device executes the digital human generation method provided in the embodiment of the present invention, the specific process is as follows:
[0085] Step 1: The input module can obtain the long text (i.e., target text information) passed in by the business end;
[0086] Step 2: The TTS speech generation module can generate audio information corresponding to the target text information through text-to-speech (TTS) (for example, the target text information can be converted into a speech file with a preset tone);
[0087] Step 3: The voice distribution module can segment the audio information obtained in the previous step according to the length of a preset template (that is, a preset image sequence), and then distribute each segmented voice (that is, an audio frame sequence) to the digital human generation module in a polling manner.
[0088] Step 4: The digital human generation module is mainly used to generate a digital human. Specifically, the module can input personalized voice, preset image template data (i.e., preset image sequence), etc., and can output a digital human video (i.e., a digital human image sequence corresponding to at least two audio frame sequences). Here, the output digital human video can be a silent video without audio information. Furthermore, the digital human generation module can mainly include: a driving module mainly responsible for driving processing, and a super-resolution module mainly responsible for super-resolution processing and post-processing.
[0089] It should be noted that the digital human generation module can deploy multiple instances according to its computing resources, and these instances can be processed in parallel, thereby speeding up the digital human generation speed. In addition, in some optional examples, the component method can also be used to configure the number of driver modules and super-resolution modules in proportion according to the computing resources of the digital human generation device. For example, the ratio of the number of driver modules to super-resolution modules can be 1:1 or 2:3, etc., so that the inference concurrency can be flexibly increased or decreased.
[0090] like Figure 3 As shown, the voice distribution module slices the obtained audio segments (i.e., the audio frame sequences V1, V2, V3..., V n-1 、V n ) are distributed to the digital human generation module for processing in a polling manner, wherein for each audio frame sequence, the digital human generation module generates a corresponding image sequence segment (i.e., digital human image sequence O1, O2, ..., O n-1 , O n ).
[0091] Step 5: The video synthesis module can synthesize the audio information converted from the target text information and the digital human video generated by the digital human generation module in sequence into a target digital human video with sound.
[0092] That is to say, the above digital human image sequence O1, O2, ..., O n-1 , O nThe target digital human video is synthesized with the audio information converted from the target text information to obtain the target digital human video.
[0093] In this embodiment, the audio file converted from the target text information is sliced by the voice distribution module, and the obtained audio frame sequence is distributed to the digital human generation module, so that the concurrency of a processing cycle can be increased.
[0094] In some embodiments, step 13, based on the preset image sequence, processing the at least two audio frame sequences in parallel to obtain a digital human image sequence corresponding to each audio frame sequence, respectively, includes:
[0095] Step 1301, in the process of processing each of the audio frame sequences, obtaining driving parameters corresponding to the audio frame sequence;
[0096] Step 1302: Drive the preset image sequence according to the drive parameters to obtain a first image sequence corresponding to the audio frame sequence.
[0097] Here, the driving processing is mainly used to perform deep learning reasoning on the audio frame sequence and the preset image sequence (that is, the picture template data of the preset image, which can be a video template), and then replace the reasoning results in the preset image sequence according to the position information, and then perform post-processing operations such as rotation, affine, and translation, so as to achieve the effect of audio driving the video template.
[0098] It should be noted that, by driving the preset image sequence through the audio frame sequence, the mouth shape of the character in the image frame of the first image sequence obtained corresponds to the pronunciation of the corresponding audio frame. For example, assuming that the length of the preset image sequence is L, each audio frame sequence includes L frames. For any audio frame sequence V n , use the first frame of the preset image sequence to drive V n The first frame in the image sequence is used to drive V n The second frame in the preset image sequence is used to drive V n The Lth frame in n The first frame of the sound is "O", then the V n In the corresponding first image sequence, the mouth shape of the character in the first image frame is the mouth shape when making the sound "O".
[0099] Step 1303: Based on the first image sequence, obtain a digital human image sequence corresponding to the audio frame sequence.
[0100] It should be noted that in step 1303, super-resolution processing and post-processing and other steps can be performed on the first image sequence to obtain an image sequence corresponding to the first image sequence. Assuming that the audio information is divided into n audio frame sequences, the first image sequence corresponding to these n audio frame sequences can be determined as the digital human image sequence corresponding to the audio frame sequence.
[0101] In some embodiments, the parallel processing of the at least two audio frame sequences and the processing of obtaining a digital human image sequence corresponding to the audio frame sequence based on the first image sequence are performed in an asynchronous manner.
[0102] For example, the driving module or super-resolution module in the digital human generation device may be processed in a pipeline manner. Figure 5 As shown, for the driver module, after completing the driving process for the first slice V1, the driver module will then process the second slice V2. At the same time, the super-resolution module will also process the first slice after the driving process (i.e., S1). In this way, when the internal processing time is constant, the pipeline method can speed up the entire digital human generation process and reduce the processing time. The granularity of the pipeline can be determined according to the actual situation. In this way, by adopting the pipeline technology, the processing waiting time of each link can be reduced, and the speed of digital human generation can be effectively improved.
[0103] It should also be noted that in the driver module or super-resolution module in the above-mentioned digital human generation device, a thread pool for asynchronous processing can be set inside each process. Here, it is still assumed that the length of the preset image sequence is L, then there are L picture frames in each audio frame sequence obtained by segmentation according to L that need to be processed concurrently, such as Figure 6 As shown, a thread pool containing K threads (such as Thread1, Thread2, ..., Threadk) can be opened to process L picture frames. In this way, the asynchronous thread pool method can realize multi-process parallel and asynchronous multi-threaded processing, so that computer resources can be better utilized and tasks can be completed more quickly and efficiently.
[0104] In some embodiments, step 1303, obtaining a digital human image sequence corresponding to the audio frame sequence based on the first image sequence, includes:
[0105] Step 13031: perform correction processing on each image frame in the first image sequence to obtain a second image sequence corresponding to the first image sequence.
[0106] In a specific example, step 13031 may specifically include the following steps:
[0107] (1) During the correction processing of each first image frame in the first image sequence, a target face image frame corresponding to the first image frame is obtained, wherein the first image frame is any image frame in the first image sequence, and the target face image frame includes a frontal perspective of the face image in the first image frame.
[0108] In some embodiments, obtaining a target face image frame corresponding to the first image frame includes:
[0109] (1) Obtaining coordinate information corresponding to a face frame of a second image frame, where the second image frame is an image frame in the preset image sequence whose order is consistent with that of the first image frame in the first image sequence.
[0110] It should be noted that, in the embodiment of the present invention, in order to speed up the entire digital human generation process, face detection can be performed on each image frame in the preset image sequence in advance to obtain the face detection results of each image frame in the preset image sequence (that is, the coordinate information corresponding to the face frame of the image frame), and the coordinate information corresponding to the face frame of each image frame in the preset image sequence can be further stored in a json file as preset face frame data. In this way, the coordinate information corresponding to the face frame of each image frame in the preset image sequence can be loaded.
[0111] Exemplarily, the structure of the coordinate information corresponding to the face frame can be defined as follows:
[0112] {
[0113] "1":[[x1,y1],[x1,y2],[x2,y2],[x2,y1]],
[0114] "2":[[x1,y1],[x1,y2],[x2,y2],[x2,y1]], ...
[0116] "n":[[x1,y1],[x1,y2],[x2,y2],[x2,y1]],
[0117] }
[0118] Among them, in "1", "2", ..., "n", the numerical value in quotation marks represents the sequence number of the image frame in the preset image sequence. For example, the 1 in "1" represents the first image frame in the preset image sequence, and so on. [x1, y1] is the coordinate of the upper left point of the face frame, [x1, y2] is the coordinate of the lower left point of the face frame, [x2, y2] is the coordinate of the lower right point of the face frame, and [x2, y1] is the coordinate of the upper right point of the face frame. Here, x1, y1, x2 and y2 can be normalized results.
[0119] It is understandable that the coordinates of the face frames corresponding to images with different serial numbers may be the same or different. For example, the value of x1 in "1":[[x1,y1],[x1,y2],[x2,y2],[x2,y1]] may be different from the value of x1 in "2":[[x1,y1],[x1,y2],[x2,y2],[x2,y1]].
[0120] (2) Cropping the first image frame according to coordinate information corresponding to the face frame of the second image frame to obtain a face image frame corresponding to the first image frame.
[0121] (3) Adjusting the angle of the face in the first image frame to obtain a target face image frame corresponding to the first image frame.
[0122] Specifically, after cropping the image frame to obtain the face image frame corresponding to the first image frame, the Umeyama algorithm can be used to calculate the minimum variance estimate of the similarity transformation parameters for the four coordinate points corresponding to the face frame (i.e., the coordinates corresponding to the upper left point, lower left point, lower right point, and upper right point of the face frame, respectively) and the reference point [[0.0,0.0], [0.0,1.0], [1.0,1.0], [1.0,0.0]], and the result is a first matrix, which is recorded. The first matrix can be further used to adjust the angle of the face in the first image frame, that is, to turn the face in the face image frame to the right. The similarity transformation can specifically include one or more of rotation, translation, and scaling.
[0123] (ii) performing key point detection on the target face image frame to obtain coordinates of key points in the target face image frame, wherein the key points include facial feature points of the face in the target face image frame.
[0124] Here, specifically, the key point detection algorithm built into OpenCV can be used for key point detection, or the Google Facemesh algorithm can be used for key point detection, which is not specifically limited here.
[0125] (iii) performing correction processing on the first image frame according to the coordinates of the key points in the target face image frame to obtain a first target image frame corresponding to the first image frame.
[0126] like Figure 4 As shown, after key point detection is performed on the target face image frame, the key points in the target face image frame can be returned to the face image frame according to the first matrix, thereby obtaining the coordinates of the key points on the face image frame.
[0127] (iv) when each of the first image frames has been corrected, determining a first target image frame corresponding to at least one of the first image frames as the second image sequence.
[0128] Here, taking the key point detection algorithm built into OpenCV as an example, the process of correcting the image is described as follows:
[0129] (1) Using the Umeyama algorithm, the minimum variance estimate of the similarity transformation parameters is calculated for the generated key point coordinates (i.e., the coordinates of the key points in the target face image frame) and the preset reference points [[0.0,0.0], [0.0,1.0], [1.0,1.0], [1.0,0.0]], and the calculation result is a matrix, which is the transformation matrix from the global space to the local alignment space (denoted as mat). The similarity transformation may specifically include one or more of rotation, translation, and scaling.
[0130] (2) Use the transformation matrix mat to calculate the affine transformation matrix mat and the normalized affine transformation matrix uni_mat from the image space (large image, i.e. the first image frame) to the face space (small image, i.e. the target face image frame). Specifically, the coordinate points (3 in total) of the reference points in the face space after transformation in the global space can be calculated first, and then the affine transformation matrix mat and the normalized affine transformation matrix uni_mat can be calculated with the three coordinate points in the global space according to the method in OpenCV.
[0131] (3) Use the affine transformation matrix mat to normalize the first image frame, and the normalized image (i.e., the first target image frame) is recorded as: face_align_img.
[0132] (4) Use the convexHull method to find the convex hull of each pixel value (x, y) in face_align_img. Then, use the fillConvexPoly method to set the inside of the convex hull to 255 and the outside of the convex hull to 0 to obtain the convex hull marked image (referred to as face_align_lmrks_mask_img), and record the size of the convex hull marked image.
[0133] Step 13032: Perform super-resolution processing on the second image sequence corresponding to the first image sequence to obtain a digital human image sequence corresponding to the audio frame sequence, wherein the resolution of the digital human image sequence is higher than that of the second image sequence.
[0134] In some specific optional embodiments, the image frames in the second image sequence obtained after the correction processing can be input into the super-resolution network model, and the super-resolution network model can be used for super-resolution processing. The super-resolution network model can output: the super-resolution image (denoted as face_swap_img) and the mask of the super-resolution image (denoted as face_swap_mask_img). After that, further post-processing can be performed to finally obtain the digital human image sequence corresponding to the audio frame sequence. Here, the specific process of post-processing is introduced as follows:
[0135] Step 1: Overlay maskface_swap_mask_img (i.e., the super-resolved image) and face_align_lmrks_mask_img (convex hull marked image) to obtain the first mask image (denoted as face_mask).
[0136] Step 2: Perform affine transformation on face_mask according to uni_mat (i.e., the normalized affine transformation matrix), and upsample the transformed image to the [0.0, 1.0] interval.
[0137] Step 3: Perform affine transformation on face_swap_img (i.e. the super-resolution image) according to the uni_mat matrix, and then scale the transformed image to the original size to obtain the contour image (denoted as frame_face_swap_img).
[0138] Step 4: Superimpose the preset face_mask_t (which can be obtained by performing facial cutout on the original frame_image) with face_mask (i.e., the first mask image), and upsample the superimposed result image to the [0.0, 1.0] interval. Then, scale the sampled image to the original image size, and finally perform dilation and erosion to obtain frame_face_mask.
[0139] Step 5: According to the following formula:
[0140] merged_frame=ne.evaluate('frame_image*(1.0-frame_face_mask)+
[0141] frame_image*frame_face_mask*0.01+frame_face_swap_img*frame_face_mask*0.99')
[0142] The final image frame (merged_frame) in the digital human image sequence is calculated.
[0143] In the embodiment of the present invention, after the target text information is converted into audio information, the audio information can be segmented according to the length of the preset image sequence to obtain at least two audio frame sequences, and then the at least two audio frame sequences are processed in parallel based on the preset image sequence to obtain the digital human image sequence corresponding to each audio frame sequence, and finally the audio information and the digital human image sequence corresponding to the at least two audio frame sequences are synthesized to obtain the target digital human video. In this way, for the scenario of generating a digital human from a long text, the audio converted from the long text can be segmented into multiple audio segments, so that the multiple audio segments can be processed in parallel in the subsequent processing process, thereby effectively reducing the time of generating a digital human based on the long text, improving the processing efficiency, and better meeting the production needs.
[0144] like Figure 7 As shown, the embodiment of the present invention further provides a digital human generation device, comprising:
[0145] The first processing module 71 is used to generate audio information corresponding to the target text information;
[0146] An audio segmentation module 72, configured to segment the audio information according to the length of a preset image sequence to obtain at least two audio frame sequences, wherein the image frames in the preset image sequence include a digital human image;
[0147] A second processing module 73 is used to process the at least two audio frame sequences in parallel based on the preset image sequence to obtain a digital human image sequence corresponding to each audio frame sequence;
[0148] The synthesis processing module 74 is used to perform synthesis processing on the audio information and the digital human image sequences corresponding to the at least two audio frame sequences to obtain a target digital human video.
[0149] Optionally, the second processing module 73 includes:
[0150] A first driving submodule, configured to obtain driving parameters corresponding to the audio frame sequence during the processing of each audio frame sequence;
[0151] A second driving submodule, configured to drive the preset image sequence according to the driving parameters to obtain a first image sequence corresponding to the audio frame sequence;
[0152] The first processing submodule is used to obtain a digital human image sequence corresponding to the audio frame sequence based on the first image sequence.
[0153] In this embodiment, after the audio information corresponding to the target text information is generated, the audio information can be segmented according to the length of the preset image sequence to obtain at least two audio frame sequences, and then the at least two audio frame sequences are processed in parallel based on the preset image sequence to obtain the digital human image sequence corresponding to each audio frame sequence, and finally the audio information and the digital human image sequence corresponding to the at least two audio frame sequences are synthesized to obtain the target digital human video. In this way, for the scenario of generating a digital human from a long text, the audio converted from the long text can be segmented into multiple audio segments, so that multiple audio segments can be processed in parallel in the subsequent processing process, thereby effectively reducing the time for generating a digital human based on a long text, improving processing efficiency, and better meeting production needs.
[0154] Optionally, the parallel processing of the at least two audio frame sequences and the processing of obtaining a digital human image sequence corresponding to the audio frame sequence based on the first image sequence are performed asynchronously.
[0155] Optionally, the first processing submodule includes:
[0156] a correction processing unit, configured to perform correction processing on each image frame in the first image sequence to obtain a second image sequence corresponding to the first image sequence;
[0157] The super-resolution processing unit is used to perform super-resolution processing on the second image sequence corresponding to the first image sequence to obtain a digital human image sequence corresponding to the audio frame sequence, wherein the resolution of the digital human image sequence is higher than that of the second image sequence.
[0158] Optionally, the correction processing unit includes:
[0159] A first processing subunit is configured to obtain a target face image frame corresponding to each first image frame in the first image sequence during the correction processing of the first image frame, wherein the first image frame is any image frame in the first image sequence, and the target face image frame includes a frontal perspective of the face image in the first image frame;
[0160] A key point detection subunit, configured to perform key point detection on the target face image frame to obtain coordinates of key points in the target face image frame, wherein the key points include facial feature points of the face in the target face image frame;
[0161] A correction processing subunit, configured to perform correction processing on the first image frame according to the coordinates of the key points in the target face image frame, so as to obtain a first target image frame corresponding to the first image frame;
[0162] The second processing sub-unit is configured to, when each of the first image frames has been corrected, determine a first target image frame corresponding to at least one of the first image frames as the second image sequence.
[0163] Optionally, the first processing subunit includes:
[0164] a third processing sub-unit, configured to obtain coordinate information corresponding to a face frame of a second image frame, wherein the second image frame is an image frame in the preset image sequence, the order of which is consistent with the order of the first image frame in the first image sequence;
[0165] an image cropping subunit, configured to crop the first image frame according to coordinate information corresponding to a face frame of the second image frame, to obtain a face image frame corresponding to the first image frame;
[0166] The angle adjustment subunit is used to adjust the angle of the face in the first image frame to obtain a target face image frame corresponding to the first image frame.
[0167] Optionally, the synthesis processing module 74 includes:
[0168] A sequence sorting submodule, used to sort the digital human image sequence corresponding to each audio frame sequence according to the order of the at least two audio frame sequences from front to back in the audio information;
[0169] The synthesis processing submodule is used to perform synthesis processing on the sorted digital human image sequence and the audio information to obtain the target digital human video.
[0170] The device provided in the embodiment of the present invention can execute the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0171] like Figure 8 As shown, an embodiment of the present invention also provides a device, including: a processor 81; and a memory 83 connected to the processor 81 through a bus interface 82, wherein the memory 83 is used to store programs and data used by the processor 81 when performing operations, and the processor 81 calls and executes the programs and data stored in the memory 83.
[0172] The transceiver 84 is connected to the bus interface 82 and is used to receive and send data under the control of the processor 81 ; the processor 81 is used to read the program in the memory 83 .
[0173] In this embodiment, after the audio information corresponding to the target text information is generated, the audio information can be segmented according to the length of the preset image sequence to obtain at least two audio frame sequences, and then the at least two audio frame sequences are processed in parallel based on the preset image sequence to obtain the digital human image sequence corresponding to each audio frame sequence, and finally the audio information and the digital human image sequence corresponding to the at least two audio frame sequences are synthesized to obtain the target digital human video. In this way, for the scenario of generating a digital human from a long text, the audio converted from the long text can be segmented into multiple audio segments, so that multiple audio segments can be processed in parallel in the subsequent processing process, thereby effectively reducing the time for generating a digital human based on a long text, improving processing efficiency, and better meeting production needs.
[0174] Specifically, the processor 81 is used to execute:
[0175] Generate audio information corresponding to the target text information;
[0176] The audio information is segmented according to the length of a preset image sequence to obtain at least two audio frame sequences, wherein the image frames in the preset image sequence include a digital human image;
[0177] Based on the preset image sequence, the at least two audio frame sequences are processed in parallel to obtain a digital human image sequence corresponding to each audio frame sequence;
[0178] The audio information and the digital human image sequences respectively corresponding to the at least two audio frame sequences are synthesized to obtain a target digital human video.
[0179] Optionally, when the processor 81 processes the at least two audio frame sequences in parallel based on the preset image sequence to obtain a digital human image sequence corresponding to each of the audio frame sequences, the processor 81 is specifically configured to execute:
[0180] In the process of processing each of the audio frame sequences, obtaining driving parameters corresponding to the audio frame sequence;
[0181] According to the driving parameters, driving processing is performed on the preset image sequence to obtain a first image sequence corresponding to the audio frame sequence;
[0182] Based on the first image sequence, a digital human image sequence corresponding to the audio frame sequence is obtained.
[0183] Optionally, the parallel processing of the at least two audio frame sequences and the processing of obtaining a digital human image sequence corresponding to the audio frame sequence based on the first image sequence are performed asynchronously.
[0184] Optionally, when the processor 81 obtains the digital human image sequence corresponding to the audio frame sequence based on the first image sequence, it is specifically configured to execute:
[0185] Performing correction processing on each image frame in the first image sequence to obtain a second image sequence corresponding to the first image sequence;
[0186] Super-resolution processing is performed on a second image sequence corresponding to the first image sequence to obtain a digital human image sequence corresponding to the audio frame sequence, wherein the resolution of the digital human image sequence is higher than that of the second image sequence.
[0187] Optionally, when the processor 81 performs correction processing on each image frame in the first image sequence to obtain a second image sequence corresponding to the first image sequence, the processor 81 is specifically configured to execute:
[0188] In the correction processing process of each first image frame in the first image sequence, a target face image frame corresponding to the first image frame is obtained, wherein the first image frame is any image frame in the first image sequence, and the target face image frame includes a frontal perspective of the face image in the first image frame;
[0189] Performing key point detection on the target face image frame to obtain coordinates of key points in the target face image frame, wherein the key points include facial feature points of the face in the target face image frame;
[0190] According to the coordinates of the key points in the target face image frame, the first image frame is corrected to obtain a first target image frame corresponding to the first image frame;
[0191] When each of the first image frames has been corrected, a first target image frame corresponding to at least one of the first image frames is determined as the second image sequence.
[0192] Optionally, when acquiring the target face image frame corresponding to the first image frame, the processor 81 is specifically configured to execute:
[0193] Acquire coordinate information corresponding to a face frame of a second image frame, where the second image frame is: an image frame in the preset image sequence whose order is consistent with that of the first image frame in the first image sequence;
[0194] Cropping the first image frame according to coordinate information corresponding to the face frame of the second image frame to obtain a face image frame corresponding to the first image frame;
[0195] The angle of the face in the first image frame is adjusted to obtain a target face image frame corresponding to the first image frame.
[0196] Optionally, when the processor 81 performs synthesis processing on the audio information and the digital human image sequences respectively corresponding to the at least two audio frame sequences to obtain the target digital human video, it is specifically configured to execute:
[0197] According to the order of the at least two audio frame sequences from front to back in the audio information, the digital human image sequence corresponding to each of the audio frame sequences is sorted;
[0198] The sorted digital human image sequence and the audio information are synthesized to obtain a target digital human video.
[0199] It should be noted that in Figure 8 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 81 and various circuits of memory represented by memory 83 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 84 may be a plurality of components, namely, a transmitter and a transceiver, providing a unit for communicating with various other devices on a transmission medium. For different terminals, the user interface 85 may also be an interface capable of externally connecting or internally connecting required devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, etc. The processor 81 is responsible for managing the bus architecture and general processing, and the memory 83 may store data used by the processor 81 when performing operations.
[0200] The device provided in the embodiment of the present invention can execute the above-mentioned digital human generation method embodiment, and its implementation principle and technical effects are similar, which will not be repeated in this embodiment.
[0201] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.
[0202] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing relevant hardware through a computer program, wherein the computer program includes instructions for executing part or all of the steps of the above method; and the computer program may be stored in a readable storage medium, and the storage medium may be any form of storage medium.
[0203] In addition, a specific embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the above-mentioned digital human generation method are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0204] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0205] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0206] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.
[0207] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary personnel in this technical field, several improvements and modifications can be made without departing from the principles described in the present invention. These improvements and modifications are also within the scope of protection of the present invention.
Claims
1. A method for generating a digital human, characterized in that: include: Generate audio information corresponding to the target text information; The audio information is segmented according to the length of a preset image sequence to obtain at least two audio frame sequences, wherein the image frames in the preset image sequence include a digital human image; Based on the preset image sequence, the at least two audio frame sequences are processed in parallel to obtain a digital human image sequence corresponding to each audio frame sequence; The audio information and the digital human image sequences respectively corresponding to the at least two audio frame sequences are synthesized to obtain a target digital human video.
2. The method according to claim 1, characterized in that The step of processing the at least two audio frame sequences in parallel based on the preset image sequence to obtain a digital human image sequence corresponding to each of the audio frame sequences comprises: In the process of processing each of the audio frame sequences, obtaining driving parameters corresponding to the audio frame sequence; According to the driving parameters, driving processing is performed on the preset image sequence to obtain a first image sequence corresponding to the audio frame sequence; Based on the first image sequence, a digital human image sequence corresponding to the audio frame sequence is obtained.
3. The method according to claim 2, characterized in that The parallel processing of the at least two audio frame sequences and the processing of obtaining a digital human image sequence corresponding to the audio frame sequence based on the first image sequence are performed in an asynchronous manner.
4. The method according to claim 2, characterized in that: The step of obtaining a digital human image sequence corresponding to the audio frame sequence based on the first image sequence includes: Performing correction processing on each image frame in the first image sequence to obtain a second image sequence corresponding to the first image sequence; Super-resolution processing is performed on a second image sequence corresponding to the first image sequence to obtain a digital human image sequence corresponding to the audio frame sequence, wherein the resolution of the digital human image sequence is higher than that of the second image sequence.
5. The method according to claim 4, characterized in that The performing correction processing on each image frame in the first image sequence to obtain a second image sequence corresponding to the first image sequence includes: In the correction processing process of each first image frame in the first image sequence, a target face image frame corresponding to the first image frame is obtained, wherein the first image frame is any image frame in the first image sequence, and the target face image frame includes a frontal perspective of the face image in the first image frame; Performing key point detection on the target face image frame to obtain coordinates of key points in the target face image frame, wherein the key points include facial feature points of the face in the target face image frame; According to the coordinates of the key points in the target face image frame, the first image frame is corrected to obtain a first target image frame corresponding to the first image frame; When each of the first image frames has been corrected, a first target image frame corresponding to at least one of the first image frames is determined as the second image sequence.
6. The method according to claim 5, characterized in that The step of acquiring a target face image frame corresponding to the first image frame includes: Acquire coordinate information corresponding to a face frame of a second image frame, where the second image frame is: an image frame in the preset image sequence whose order is consistent with that of the first image frame in the first image sequence; Cropping the first image frame according to coordinate information corresponding to the face frame of the second image frame to obtain a face image frame corresponding to the first image frame; The angle of the face in the first image frame is adjusted to obtain a target face image frame corresponding to the first image frame.
7. The method according to claim 1, characterized in that The synthesizing process of the audio information and the digital human image sequences respectively corresponding to the at least two audio frame sequences to obtain the target digital human video includes: According to the order of the at least two audio frame sequences from front to back in the audio information, the digital human image sequence corresponding to each of the audio frame sequences is sorted; The sorted digital human image sequence and the audio information are synthesized to obtain a target digital human video.
8. A device, characterized in that include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the program.
9. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
