Digital human image generation method, apparatus and device, and readable storage medium

Abstract

The invention provides a digital human image generation method, device and equipment and a readable storage medium, and the method comprises the steps: generating a plurality of frames of intermediate images matched with driving voice data through model driving, and combining a pre-trained tooth generation network with the first K frames of correction images for completing tooth reconstruction of the intermediate images, according to the invention, tooth reconstruction is carried out on the current frame of intermediate image, the tooth area on the intermediate image is masked, and the corrected image corresponding to the intermediate image is fused with the intermediate image according to the masked image, so that the consistency matching of the facial expression on the intermediate image and the lip mouth shape with the driving voice data is ensured, and the user experience is improved. According to the method, the optimization of the tooth area on the intermediate image is realized, so that the teeth of the digital human are more clear and vivid visually and present a natural and reasonable arrangement state, and the overall sense of reality of the digital human is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and readable storage medium for generating a digital human image. Background Art

[0002] In the current process of digital human model construction and rendering, the relevant technologies have problems such as blurred teeth, abnormal multiple rows of teeth, unstable tooth positions and unnatural displacement when generating digital human teeth, which creates a sense of visual dissonance, reduces the visual realism of the digital human, and affects the digital human's expressiveness and audience immersion. Summary of the invention

[0003] In view of this, in order to solve the above technical problems, the present application provides a method, device, equipment and readable storage medium for generating a digital human image.

[0004] Specifically, the present application is implemented through the following technical solutions:

[0005] According to a first aspect of an embodiment of the present application, a method for generating a digital human image is provided, the method comprising:

[0006] Based on the initial image containing the digital human, the model is used to drive the facial expression and lip shape of the digital human to generate multiple intermediate images that match the driving voice data;

[0007] For each intermediate image, teeth reconstruction is performed on the intermediate image according to a pre-trained teeth generation network to obtain a corrected image corresponding to the intermediate image; wherein the teeth reconstruction of the intermediate image is based on the previous K frames of corrected images obtained by completing teeth reconstruction before the intermediate image;

[0008] Locating the tooth region of the digital person from each frame of the intermediate image, and constructing a mask image for the tooth region;

[0009] According to the mask image, the intermediate image and its corresponding corrected image are fused to obtain a digital human image corresponding to each frame of the intermediate image.

[0010] Optionally, reconstructing teeth on the intermediate image according to a pre-trained tooth generation network comprises:

[0011] Extracting feature information of tooth regions in the intermediate image and the first K frames of corrected images before the intermediate image through the tooth generation network;

[0012] According to the similarity of the tooth area between the intermediate image and the first K frame corrected image, a temporal consistency attention mechanism is applied to fuse and represent the feature information to obtain a fused feature;

[0013] According to the fusion features corresponding to the intermediate image, a corrected image output by the tooth generation network is obtained.

[0014] Optionally, reconstructing teeth on the intermediate image according to a pre-trained tooth generation network comprises:

[0015] Locating the teeth region of the digital person from the intermediate image, and cutting out a local image including the teeth region from the intermediate image;

[0016] Perform teeth reconstruction on the partial image according to a pre-trained teeth generation network.

[0017] Optionally, the method further comprises a step of training a tooth generation network:

[0018] Acquire a training sample set, each training sample includes M consecutive frames of sample images, and the tooth area of ​​the object in the sample image meets the preset tooth quality requirements;

[0019] Perform defect introduction processing on the tooth area of ​​the sample images of the last N frames in the M frames as the samples to be predicted; wherein N is a positive integer and is less than M;

[0020] Input the previous (MN) frame sample image and the next N frames of samples to be predicted into the tooth generation network for tooth reconstruction, and obtain the N frames of predicted images of its predicted output;

[0021] The loss function is calculated using the N frames of predicted images and the corresponding N frames of sample images, and the parameters of the tooth generation network are adjusted to minimize the loss function; the loss function at least includes pixel level difference loss and image temporal coherence loss.

[0022] Optionally, constructing a mask map for the tooth region includes:

[0023] Creating a blank image; wherein the pixels of the blank image correspond one-to-one to the pixels at corresponding positions on the intermediate image;

[0024] Determining the tooth region and the non-tooth region on the blank image according to the tooth region on the intermediate image;

[0025] Pixel values ​​are set for the pixel points in the tooth area and the non-tooth area respectively, and a mask image corresponding to the intermediate image is generated.

[0026] Optionally, locating the tooth region of the digital human from each frame of the intermediate image includes:

[0027] Determine the key points of the contour of the digital human's lips in the intermediate image; determine the boundary of the tooth area according to the key points of the contour that identify the inner lip boundary;

[0028] Alternatively, the intermediate image is input into a pre-trained tooth region segmentation network to obtain an intermediate image with marked tooth regions.

[0029] Optionally, the method further comprises:

[0030] After obtaining the plurality of intermediate images, detecting whether the lips of the digital human in the intermediate images are in a closed state;

[0031] When it is detected that the lips of the digital human in the intermediate image are in a closed state, the intermediate image is directly used as the corresponding digital human image.

[0032] Optionally, the method further comprises:

[0033] Acquire a standard tooth image preset for the digital human and conforming to the image design of the digital human;

[0034] The standard tooth image and the intermediate image are input into the tooth generation network together, so as to perform tooth reconstruction on the intermediate image according to the tooth generation network.

[0035] According to a second aspect of an embodiment of the present application, a digital human image generating device is provided, the device comprising:

[0036] A model driving module, for driving the facial expression and lip shape of the digital human using a model based on an initial image containing the digital human, and generating a plurality of intermediate images matching the driving voice data;

[0037] A tooth reconstruction module is used to perform tooth reconstruction on each intermediate image according to a pre-trained tooth generation network to obtain a corrected image corresponding to the intermediate image; wherein the tooth reconstruction of the intermediate image is based on the first K frames of corrected images obtained by completing tooth reconstruction before the intermediate image;

[0038] A mask image construction module, used to locate the tooth region of the digital human from each frame of the intermediate image and construct a mask image for the tooth region;

[0039] The fusion generation module is used to fuse the intermediate image and its corresponding corrected image according to the mask image to obtain a digital human image corresponding to each frame of the intermediate image.

[0040] Optionally, the tooth reconstruction module is specifically used for:

[0041] Extracting feature information of tooth regions in the intermediate image and the first K frames of corrected images before the intermediate image through the tooth generation network;

[0042] According to the similarity of the tooth area between the intermediate image and the first K frame corrected image, a temporal consistency attention mechanism is applied to fuse and represent the feature information to obtain a fused feature;

[0043] According to the fusion features corresponding to the intermediate image, a corrected image output by the tooth generation network is obtained.

[0044] Optionally, the device further comprises a training step of a tooth generation network:

[0045] Acquire a training sample set, each training sample includes M consecutive frames of sample images, and the tooth area of ​​the object in the sample image meets the preset tooth quality requirements;

[0046] Perform defect introduction processing on the tooth area of ​​the sample images of the last N frames in the M frames as the samples to be predicted; wherein N is a positive integer and is less than M;

[0047] Input the previous (MN) frame sample image and the next N frames of samples to be predicted into the tooth generation network for tooth reconstruction, and obtain the N frames of predicted images of its predicted output;

[0048] The loss function is calculated using the N frames of predicted images and the corresponding N frames of sample images, and the parameters of the tooth generation network are adjusted to minimize the loss function; the loss function at least includes pixel level difference loss and image temporal coherence loss.

[0049] Optionally, the mask image construction module is specifically used to:

[0050] Creating a blank image; wherein the pixels of the blank image correspond one-to-one to the pixels at corresponding positions on the intermediate image;

[0051] Determining the tooth region and the non-tooth region on the blank image according to the tooth region on the intermediate image;

[0052] Pixel values ​​are set for the pixel points in the tooth area and the non-tooth area respectively, and a mask image corresponding to the intermediate image is generated.

[0053] Optionally, the mask image construction module is specifically used to:

[0054] Determine the key points of the contour of the digital human's lips in the intermediate image; determine the boundary of the tooth area according to the key points of the contour that identify the inner lip boundary;

[0055] Alternatively, the intermediate image is input into a pre-trained tooth region segmentation network to obtain an intermediate image with marked tooth regions.

[0056] Optionally, the device further comprises:

[0057] After obtaining the plurality of intermediate images, detecting whether the lips of the digital human in the intermediate images are in a closed state;

[0058] When it is detected that the lips of the digital human in the intermediate image are in a closed state, the intermediate image is directly used as the corresponding digital human image.

[0059] Optionally, the device further comprises:

[0060] Acquire a standard tooth image preset for the digital human and conforming to the image design of the digital human;

[0061] The standard tooth image and the intermediate image are input into the tooth generation network together, so as to perform tooth reconstruction on the intermediate image according to the tooth generation network.

[0062] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the above-mentioned digital human image generation method by calling the computer program.

[0063] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned digital human image generation method is implemented.

[0064] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0065] In the technical solution provided by the present application, multiple frames of intermediate images matching the driving voice data are generated by model driving, and the teeth of each intermediate image are reconstructed by combining the previous K frames of corrected images before each intermediate image through a pre-trained tooth generation network. The tooth area in each frame of the image can be finely processed and the consistency of the tooth morphology can be maintained in the time dimension. Through masking of the tooth area on the intermediate image, the corrected image corresponding to the intermediate image is fused with the intermediate image according to the mask image. While ensuring the consistency of the facial expression and lip shape on the intermediate image with the driving voice data, the tooth area on the intermediate image is optimized. By combining the model-driven digital human with the image processing of the tooth area, the teeth of the digital human are visually clearer and more realistic, presenting a natural and reasonable arrangement, thereby improving the overall realism of the digital human and enhancing the audience's sense of immersion and the expressiveness of the digital human.

[0066] It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. In addition, any embodiment in the present application does not need to achieve all the above effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0068] Figure 1A The exemplary embodiment shows an example diagram of a tooth region being blurred in the related art;

[0069] Figure 1B is an exemplary diagram showing a multi-row tooth problem in the related art;

[0070] Figure 1C The exemplary embodiment shows an example diagram of an unnatural displacement of teeth in two consecutive frames in the related art;

[0071] Figure 2A It is an overall framework diagram of a method for generating a digital human image shown in an exemplary embodiment of the present application;

[0072] Figure 2B It is a schematic diagram of the steps of a method for generating a digital human image according to an exemplary embodiment of the present application;

[0073] Figure 3 is a flowchart of a tooth reconstruction step using a tooth generation network shown in an exemplary embodiment of the present application;

[0074] Figure 4 is an example diagram of a training process using a tooth generation network shown in an exemplary embodiment of the present application;

[0075] Figure 5A is a schematic diagram of a mask diagram shown in an exemplary embodiment of the present application;

[0076] Figure 5B is another schematic diagram of a mask diagram shown in an exemplary embodiment of the present application;

[0077] Figure 6 is a schematic diagram of the lip shape state of any two intermediate frames of images shown in an exemplary embodiment of the present application;

[0078] Figure 7 is an example flow chart of a digital human tooth reconstruction and image fusion process shown in an exemplary embodiment of the present application;

[0079] Figure 8 is a structural schematic diagram of a digital human image generation device shown in an exemplary embodiment of the present application;

[0080] Fig. 9 It is a hardware schematic diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0081] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. It should be understood that the present application may use the terms first, second, third, etc. to describe various information, but these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other.

[0082] With the rapid iterative development of artificial intelligence technology, 2D digital humans have been widely used in many fields such as virtual anchors, virtual customer service, film and television production and other digital human interaction scenarios. The generation and driving of this digital human usually relies on machine learning models. Based on the reference image that describes the initial expression state of the digital human, the digital human's facial expressions and lip movements are driven by input audio or motion data, enabling the digital human to express and move naturally according to text or audio input, with a high degree of realism.

[0083] However, the inventors discovered during the study of digital human driving that in the current digital human model construction and rendering process, the generated digital human's tooth area has many defects and deficiencies, such as unclear teeth, abnormal multiple rows of teeth, unstable tooth position and unnatural displacement.

[0084] For example, see Figure 1A The example image shows a blurred tooth area. The generated digital human image simply distinguishes the upper and lower teeth and blurs them as a row of white blocks. It lacks the outline and texture of the two rows of teeth, making the digital human lack of realism in visual effect. Figure 1B The example diagram exemplifies the problem of multiple rows of teeth. In some frames of the generated digital human image, the teeth are generated incorrectly, resulting in the visual abnormality of multiple rows of teeth, which affects the authenticity of the visual effect. Figure 1C The example diagram shows unnatural displacement of teeth in two consecutive frames. The lip shape changes slightly in the two consecutive frames, and the teeth are unnaturally displaced in the previous and next frames, resulting in discontinuity in the positions of the front and back teeth, causing visual disharmony, affecting the expressiveness of the digital human and the audience's sense of immersion.

[0085] In view of this, the present application provides a method for generating a digital human image. In the process of generating an image by using a model to drive the digital human's facial expressions and lip shapes, the digital human's tooth area in the generated image is individually optimized. The individual optimization of the tooth area is achieved through a tooth generation network. The tooth generation network is trained using a specific data set during a pre-training process to learn the morphology and details of teeth under different lip shapes. By masking and fusing the optimized tooth area with the image generated by the model drive, the digital human's teeth finally presented are visually clearer and more realistic, showing a natural and reasonable arrangement, thereby enhancing the overall realism of the digital human.

[0086] join Figure 2A The overall framework diagram of the digital human image generation method provided in the present application is exemplarily shown. For the initial image and driving voice data that provide reference for the digital human's facial expression and lip shape, a corresponding plurality of intermediate images are obtained by model driving, and the tooth region of each intermediate image is reconstructed using a trained tooth generation network to obtain a corrected image, and then the corrected image is fused with the intermediate image using image fusion processing to obtain a digital human image with optimized tooth region.

[0087] The digital human image generation method can be applied to various scenarios involving lip shape driven generation of digital human images, such as virtual digital human teaching, virtual live broadcast, etc., and can also be extended to any other applicable scenarios, which are not limited in this application. The digital human involved in the embodiments of the present application includes a two-dimensional digital human, which may include but is not limited to a simulation type virtual digital human and a cartoon virtual digital human.

[0088] Next, combine Figure 2B-Figure 9 The digital human image generation method provided in the present application is described. The method can be executed by a terminal device or a server. The terminal device may include but is not limited to smart devices such as mobile phones, wireless handheld devices, tablet computers or personal computers, and the server may include but is not limited to edge servers, cloud servers and multi-server clusters.

[0089] See also Figure 2B The method step flow chart schematically shows that the method may include at least the following steps:

[0090] S201, based on the initial image containing the digital human, using the model to drive the facial expression and lip shape of the digital human, and generating multiple intermediate images matching the driving voice data;

[0091] The initial image containing the digital human is used to provide an initial state reference of the digital human's image, facial expression, and lip shape. It can be a pre-recorded or generated static image containing the digital human's image, movements, and expressions, or it can be a continuous image sequence from the initial video, such as a continuous frame of a 10-second video of the digital human not speaking. For example, in the scenario of generating a digital human speaking video, the initial image can be a static image of the digital human standing silently or a continuous video.

[0092] The driving voice data is used to store the content of the digital human's speech, which may include a text-to-speech audio file or a user-recorded audio file, as a driving source for generating the digital human's synchronized facial expressions and lip movements. Taking the above-mentioned scenario of generating a digital human speech video as an example, the audio corresponding to the digital human's speech content such as "Hello, I am a virtual anchor" is the driving voice data of the digital human.

[0093] The intermediate image is generated based on the image, action and expression of the digital human provided by the initial image, corresponding to a specific time point and voice content in the driving voice data, and the facial expression and lip movement of the digital human presented in the intermediate image match the voice content at the specific time point. Still taking the above-mentioned scenario of generating a digital human speech video as an example, different intermediate images correspond to different speech contents, and the facial expression and lip movement of the digital human in each frame of the intermediate image move naturally with the change of the speech content. For example, if the driving voice data is "Hello, I am a virtual anchor", multiple intermediate images include images of facial expressions and lip shapes of the digital human when he says "you", "good", "I", "yes", "virtual", "simulated", "host", "anchor", and pauses in tone.

[0094] Regarding using the model to drive the facial expressions and lip shapes of the digital human to generate an intermediate image, the features of the digital human can be first extracted based on the initial image containing the digital human, and the structural features of the face, the relative position and proportion of the facial features in the digital human image and other feature information can be analyzed to at least obtain facial features representing the shape and position information of key parts such as the eyes, facial contours, and lips; then the driving voice data can be analyzed to extract acoustic features in the voice, such as the order of phonemes, pitch, intonation, speaking speed, etc.; next, the changes in the facial expressions and lip shapes of the digital human can be predicted based on the acoustic features, and the facial features of the digital human can be reconstructed by adjusting the feature vector of the digital human to obtain an intermediate image that conforms to the driving voice data.

[0095] The model used to drive the facial expression and lip shape of the digital human can be a model that is pre-trained based on an existing digital human model framework such as a steady-state diffusion model and can generate overall facial expressions and lip shape synchronization. During the training process, through a large amount of pre-training data, the model can learn the mapping and conversion relationship between facial expressions and lip movements, as well as the speech features in the same sample speech data. Among them, the pre-training data may include different sample speech data, and the real image corresponding to each sample speech data representing the facial expression and lip movement of the digital human. The loss function is obtained by performing a difference loss calculation between the predicted image output by the model and the real image corresponding to the sample speech data, and the parameters of the model are adjusted by using back propagation and the gradient descent of the loss function to minimize the loss function to obtain a trained model. The pre-trained model learns how to adjust facial features and lip movements according to changes in speech features to achieve a synchronization effect. Therefore, when the model is used to drive the digital human's facial expressions and lip shapes to generate intermediate images, the model can output images of facial expressions and lip movements that are synchronized with the driving speech data. The intermediate image maintains the original features of the digital human in the initial image, and makes corresponding adjustments to the digital human's facial expressions and lip shapes according to changes in the speech data.

[0096] In this embodiment, the model driving the facial expression and lip shape of the digital human can be a diffusion model, so that the parameters and settings in the diffusion model can be adjusted to achieve fine control over the generated facial expressions and lip movements, generate highly realistic facial expressions and lip movements, and make the performance of the digital human more natural and vivid. When the diffusion model is used for driving, the digital human features extracted from the initial image and the acoustic features extracted from the driving voice data can be used as inputs, and an intermediate image sequence that matches the driving voice data and synchronizes the facial expressions and lip movements can be generated through the reverse diffusion process of the diffusion model.

[0097] S202, for each intermediate image, reconstruct teeth for the intermediate image according to a pre-trained tooth generation network to obtain a corrected image corresponding to the intermediate image; wherein the teeth reconstruction of the intermediate image is based on the first K frames of corrected images obtained by completing the teeth reconstruction before the intermediate image;

[0098] The previous K frame correction image refers to the K frame intermediate image before the current frame intermediate image, and the corresponding K frame correction image obtained by completing the tooth reconstruction. The current frame intermediate image is the intermediate image currently input to the tooth generation network for tooth reconstruction. Regarding the value of K, in order to reduce the amount of calculation, an upper limit can be set. For example, the maximum value of K can be 5. When the number of all intermediate images before the current frame intermediate image is less than the upper limit:

[0099] If the number of all intermediate images before the current frame intermediate image is 0, that is, the input starting frame intermediate image belongs to the first frame image of the multiple intermediate images generated in step S201, K is set to 0 by default, and only the first frame intermediate image can be input into the tooth generation network for tooth reconstruction;

[0100] If the number of all intermediate images before the intermediate image of the current frame is not 0 and is less than the upper limit of the value, the corrected images corresponding to all the intermediate images before the intermediate image of the current frame can be used as the first K frames of corrected images before the intermediate image of the current frame, and as the basis for the tooth generation network to reconstruct the teeth of the intermediate image of the current frame.

[0101] When the number of all intermediate images before the intermediate image of the current frame is greater than or equal to the upper limit of the value, a corrected image obtained by dental reconstruction from the intermediate images of the previous K frames adjacent to the intermediate image of the current frame can be selected as the corrected image of the previous K frames before the intermediate image of the current frame, and as the basis for the dental generation network to reconstruct the teeth of the intermediate image of the current frame.

[0102] The tooth generation network is a pre-trained deep learning model specifically designed to generate or reconstruct images of teeth. The network can be built based on a generative adversarial network, a convolutional neural network, or other deep learning architectures. The network learns the features and patterns of a large number of tooth images and can generate realistic tooth images or reconstruct and repair teeth based on input images.

[0103] Dental reconstruction refers to the process of repairing, reconstructing or enhancing the image of teeth using image processing technology or deep learning models. It may include but is not limited to the adjustment and optimization of tooth shape, color, texture, etc. to optimize the visual effect of the digital human tooth image in the intermediate image.

[0104] When the teeth of the intermediate image are reconstructed according to the pre-trained teeth generation network, the intermediate image and the first K frame corrected image obtained by completing the teeth reconstruction before the intermediate image can be input into the teeth generation network together, so that the teeth generation network reconstructs the teeth of the intermediate image according to the first K frame corrected image before the intermediate image, and outputs the corrected image corresponding to the intermediate image. It can be understood that step S202 describes the processing process of an intermediate image. For multiple intermediate images matching the driving voice data, repeating step S202 can obtain the corrected image corresponding to each intermediate image.

[0105] In order to improve processing efficiency, teeth reconstruction can be performed by inputting multiple consecutive frames of intermediate images into a pre-trained tooth generation network each time until corrected images corresponding to all intermediate images that match the driving speech data are obtained; in this case where multiple intermediate images are input each time, the teeth reconstruction of the multiple consecutive frames of intermediate images input into the tooth generation network is based on the first K frames of corrected images that have been completed through tooth reconstruction before the input starting frame intermediate image.

[0106] In this embodiment, the tooth generation network can predict the correction image corresponding to at least one intermediate image at a time. Therefore, according to the image order of the multiple intermediate images generated in step S201, one intermediate image or multiple consecutive frame intermediate images can be input into the tooth generation network each time, and the corresponding first K frame correction images are determined based on the input starting frame intermediate image, and input into the tooth generation network together, so that the tooth generation network reconstructs the teeth of the digital human in the one or more consecutive frames of intermediate images input into the network this time according to the one or more consecutive frames of intermediate images input into the network this time and the first K frame correction images before the input starting frame intermediate image, and obtains the correction image corresponding to the intermediate image, and repeats the above process until the teeth reconstruction processing is completed for the multiple intermediate images. For example, when generating 20 intermediate images matching the driving voice, one intermediate image and the first K frame correction images corresponding to the intermediate image can be input into the tooth generation network each time; or, four intermediate images can be input into the tooth generation network each time, so that the tooth generation network performs batch processing based on the four intermediate images input each time and outputs the corresponding four correction images.

[0107] Assume that P intermediate images matching the driving voice data are generated, and the upper limit of K is 3. Taking the example of inputting one intermediate image each time, when the first intermediate image matching the driving voice data is used as the starting frame intermediate image input to the tooth generation network, the corresponding first K frame correction images are empty, that is, the first K frame intermediate images corresponding to the intermediate image are not input; when the second intermediate image matching the driving voice data is used as the starting frame intermediate image input to the tooth generation network, based on the fact that it only includes the first intermediate image before, which is less than the upper limit of K 3, the corresponding first K corrected images can be taken as the corrected images corresponding to the first intermediate image; in the third intermediate image When the intermediate image is used as the starting frame intermediate image input to the tooth generation network, its corresponding first K corrected images are taken as the corrected images corresponding to the 1st-2nd intermediate images; when the 4th intermediate image is used as the starting frame intermediate image input to the tooth generation network, its corresponding first K corrected images are taken as the corrected images corresponding to the 1st-3rd intermediate images; when the 3+yth intermediate image is used as the starting frame intermediate image input to the tooth generation network, its corresponding first K corrected images are taken as the corrected images corresponding to the y-(2+y)th intermediate image. For example, the first K corrected images corresponding to the 9th intermediate image are the corrected images corresponding to the 6th, 7th, and 8th intermediate images.

[0108] S203, locating the tooth region of the digital human from each frame of the intermediate image, and constructing a mask image for the tooth region;

[0109] The tooth region described in this embodiment refers to the region within the inner contour of the upper and lower lips of the digital human, including components such as teeth, tongue, and gums. The tooth region can be determined by determining the contour key points of the digital human's lips in the intermediate image, and determining the boundary of the tooth region based on the contour key points that identify the inner lip boundary, wherein the contour key points of the digital human's lips can be obtained by methods based on machine learning or deep learning, edge detection, contour extraction, etc., and then the detected contour key points can be screened and sorted so that they can form the lip contour in the correct order, and the contour key points can be fitted using appropriate algorithms such as the least squares method, Bezier curves, etc. to form a smooth boundary curve, and the inner lip region, that is, the boundary of the tooth region, is extracted based on the fitted boundary curve.

[0110] Alternatively, the tooth region can also be realized by inputting the intermediate image into a pre-trained tooth region segmentation network to obtain an intermediate image with a marked tooth region. The tooth region segmentation network is used to accurately locate the tooth region from the input image, and the training samples used in the pre-training process include sample images with tooth region calibration. The tooth region predicted by the tooth region segmentation network and the real calibrated tooth region in the sample image are calculated by position difference loss, so as to realize parameter adjustment and training of the tooth region segmentation network.

[0111] The mask image is used to distinguish and process different areas in the image to specify areas that need to be retained or paid special attention to during the image processing process. In this embodiment, the mask image is used to mark the tooth area of ​​the digital human so that the tooth area information on the corrected image can be retained in the subsequent image fusion process, while the non-tooth area information on the intermediate image can be retained to achieve the optimization of the teeth of the digital human in the intermediate image without changing the facial expression and lip shape of the digital human.

[0112] S204: According to the mask image, the intermediate image and its corresponding corrected image are fused to obtain a digital human image corresponding to each frame of the intermediate image.

[0113] Fusion based on the mask image can retain all features in the intermediate image except the tooth area, and at the same time use the result of tooth reconstruction to improve the performance of the digital human tooth area, thereby obtaining a more realistic and natural digital human image.

[0114] That is, fusing the intermediate image and its corresponding corrected image means fusing the intermediate image with a partial image of the tooth area in the corrected image. That is, based on the digital human tooth area identified by the mask image, the tooth area in the corrected image and the image area other than the tooth area in the corresponding intermediate image are fused to obtain the final generated digital human image. The image information other than the tooth area such as facial expression and lip shape in the digital human image comes from the intermediate image, while the image information of the tooth area comes from the corrected image output by the intermediate image after being processed by the tooth generation network. The mask image is used to achieve seamless fusion between the tooth area in the corrected image and the image area other than the tooth area in the corresponding intermediate image, thereby achieving the purpose of optimizing the digital human teeth in the intermediate image.

[0115] By fusing each intermediate image with its corresponding corrected image in the above manner, the tooth area of ​​the digital person contained in the corrected image can be effectively and accurately fused to the intermediate image while maintaining the background and non-tooth area information of the intermediate image, so that the tooth details of the digital person are perfectly integrated with the intermediate image, presenting a realistic visual effect.

[0116] By synthesizing each frame of the digital human image in the order of the intermediate image frames, a digital human target video matching the driving voice data can be generated. The facial expression, lip shape and tooth position of the digital human in the digital human target video change in real time with the content of the driving voice data, presenting a dynamic effect that is highly synchronized with the voice content.

[0117] In the disclosed embodiment, multiple frames of intermediate images matching the driving voice data are generated through model driving, and the teeth of each frame of the intermediate image are reconstructed through a pre-trained tooth generation network combined with the first K frames of corrected images before each frame of the intermediate image. The tooth area in each frame of the image can be finely processed, and the consistency of the tooth morphology can be maintained in the time dimension. Through masking of the tooth area on the intermediate image, the corrected image corresponding to the intermediate image is fused with the intermediate image according to the mask image. While ensuring the consistency of the facial expression and lip shape on the intermediate image with the driving voice data, the teeth of the digital human in the intermediate image are optimized without changing the facial expression and lip shape of the digital human. This method uses a model to drive a digital human to generate an intermediate image that matches the driving voice data, combines the image-level tooth reconstruction processing of the digital human's tooth area image on the intermediate image, and performs mask processing to fuse the intermediate image with the corrected image of tooth reconstruction. Without significantly adjusting the training process of the model that generates the matching driving voice data, the method can quickly optimize the digital human's tooth area, improve processing efficiency, and reduce maintenance and development costs, making the digital human's teeth visually clearer and more realistic, presenting a natural and reasonable arrangement state, and enhancing the overall realism of the digital human.

[0118] In some embodiments, regarding the tooth reconstruction of the intermediate image according to the pre-trained tooth generation network described in step S202 in the above embodiment, the intermediate image can be directly input into the tooth generation network, and the tooth generation network identifies the tooth region of the intermediate image, and performs tooth reconstruction processing on the tooth region, and keeps the original image pixels of the non-tooth region, so as to output a corrected image corresponding to the intermediate image. Alternatively, the tooth region of the digital human is located from the intermediate image, and a local image containing the tooth region is cropped from the intermediate image; the tooth reconstruction of the local image is performed according to the pre-trained tooth generation network, that is, the local image is input into the tooth generation network, so that the tooth generation network performs tooth reconstruction processing on the local image.

[0119] Regarding the tooth reconstruction of the intermediate image according to the pre-trained tooth generation network described in step S202 in the aforementioned embodiment, this embodiment introduces a temporal consistency attention mechanism to optimize the processing process of the tooth generation network to optimize the image reconstruction effect of the digital human tooth area on the intermediate image.

[0120] like Figure 3 The tooth reconstruction process flow chart is shown as an example. This method can be implemented in the following way:

[0121] S301, extracting feature information of tooth regions in the intermediate image and the first K frames of corrected images before the intermediate image through the tooth generation network;

[0122] For each intermediate image and the first K corrected images before the intermediate image, the feature information of the tooth area may include but is not limited to key information such as tooth contour, position, inclination angle, color change, size, shape ratio, surface texture, etc., which can be represented in the form of feature vectors. These feature information together constitute the basis for the tooth generation network to reconstruct the teeth.

[0123] S302, according to the similarity of the tooth area between the intermediate image and the first K frames of corrected image, a temporal consistency attention mechanism is applied to fuse and represent the feature information to obtain a fused feature;

[0124] Temporal consistency attention mechanism refers to a deep learning technique for processing time series data or information with a temporal dimension. In the field of image processing, when processing temporally related images, this mechanism can help the tooth generation network identify and focus on information that remains consistent or changes significantly between different time points.

[0125] For each intermediate image, the similarity of the tooth area between the intermediate image of the frame and the corrected image of the previous K frames is calculated. The vector similarity calculation can be performed through the extracted feature information, such as by calculating the dot product, cosine similarity or other measurement methods between the feature vectors. The result of the similarity calculation can be a similarity matrix, which contains the similarity scores between all feature vector pairs; then, the similarity matrix can be normalized by a softmax function or other methods, and the similarity scores can be converted into attention weights in the form of probability distribution to ensure that the sum of all attention weights is 1 or a fixed constant. The size of the attention weight reflects the importance of the feature information in the fusion process. Feature information with high similarity is characterized as being consistent between different time points, containing more stable and reliable information, and is given a higher weight. Next, according to the normalized attention weights, the feature information in the intermediate image and the previous K frame corrected image is weighted summed or weighted averaged to obtain the fusion feature corresponding to each frame of the image. The fusion feature contains the comprehensive information from the intermediate image and its corresponding previous K frame corrected image, and highlights the features that remain consistent or change significantly between different time points, to ensure that the fusion feature can accurately reflect the changing trend of the digital human teeth in the time dimension.

[0126] By applying the temporal consistency attention mechanism to fuse the tooth area feature information of the intermediate image and the corrected image obtained by tooth reconstruction of the first K frames before the intermediate image, a more comprehensive and accurate tooth feature description can be obtained, thereby improving the accuracy of tooth reconstruction corresponding to each intermediate image.

[0127] S303, obtaining a corrected image output by the tooth generation network according to the fusion feature corresponding to the intermediate image.

[0128] The fusion features are passed to the pre-trained tooth generation network. Based on the training of a large number of tooth images, the tooth generation network learns the morphology, structure and change rules of teeth. The tooth generation network uses the learned knowledge to decode and reconstruct the fusion features and generate a new tooth image as a corrected image. The corrected image is clear and realistic in terms of tooth arrangement, shape, contour, etc., while maintaining consistency with the changes in tooth contour, size, shape, position, etc. of the previous K frame corrected images before the intermediate image in the time series.

[0129] In the disclosed embodiment, by introducing a time-consistent attention mechanism, the tooth generation network not only pays attention to the information of the current frame when processing each frame of the intermediate image, but also considers the association with the previous frame, i.e., the previous K frames of corrected images before the intermediate image, in the time dimension. By fusing the feature information of different time points, the generated corrected image is ensured to be continuous in time, avoiding sudden changes or inconsistencies in tooth morphology during the reconstruction process, improving the reconstruction accuracy, and making the generated tooth image more natural and realistic.

[0130] In some embodiments, after receiving the input intermediate image and the intermediate K-frame corrected image corresponding to it, the tooth generation network may first locate and crop the local area containing the lip shape of the digital human in the intermediate image and the K-frame corrected image to obtain the first local image corresponding to the intermediate image and the second local image corresponding to the K-frame corrected image. Next, extract the feature information of the tooth area in the first local image corresponding to the intermediate image and the second local image corresponding to the K-frame corrected image. Then, according to the similarity of the tooth area between each of the first local image and the second local image, the time-consistent attention mechanism may be applied to fuse and represent the extracted feature information to obtain the fused feature; and according to the fused feature corresponding to the first local image, the corrected image output by the tooth generation network may be obtained. At this time, the corrected image may be an image of the local area containing the lip shape after tooth reconstruction, that is, the corrected image may only include the lip shape and the tooth area after tooth reconstruction.

[0131] In some embodiments, the tooth generation network used in the above-mentioned tooth reconstruction has a strong feature extraction capability and can accurately capture the detailed information of the tooth area of ​​the digital human in the input image. It incorporates the attention mechanism of time series processing to ensure that the tooth area image in the corrected image after tooth reconstruction remains coherent in the time series. Based on this, this embodiment provides the training steps of the tooth generation network, see Figure 4 The exemplary training process diagram shown exemplarily may include at least the following processes a1-a4:

[0132] a1, obtaining a training sample set, each training sample includes M consecutive frames of sample images, and the tooth area of ​​the object in the sample image meets the preset tooth quality requirements;

[0133] The tooth area of ​​the object in the sample image meets the preset tooth quality requirements, that is, the contour and texture of the tooth area presented are clear, there are no abnormalities of multiple rows of teeth or unnatural tooth displacement, and it conforms to the tooth shape of the actual character during speaking and matches the changes in the mouth shape.

[0134] Regarding the training sample, a real person speaking video can be collected, in which the lip shape changes and the corresponding tooth area of ​​the person during the speaking process can be clearly captured, and M consecutive frames of images can be captured from the speaking video as training samples.

[0135] a2, performing defect introduction processing on the tooth area of ​​the sample images of the last N frames in the M frames as the sample to be predicted; wherein N is a positive integer and is less than M;

[0136] Defect introduction processing refers to the degradation operation of the tooth area of ​​the sample image to simulate or introduce defects in the tooth area such as missing teeth, misaligned teeth, abnormal tooth color, multiple rows of teeth, unclear tooth contours, etc., which is used to train the tooth generation network to repair the tooth area defects in the sample image in order to learn how to reconstruct teeth.

[0137] For each training sample, based on the fact that the tooth area in each frame of sample image included in the training sample meets the requirements, that is, the contour and texture of the presented tooth area are clear and consistent with the tooth shape of the actual person image during speaking, the last N frames of sample images are processed into samples to be predicted by the tooth generation network through defect introduction processing, and the previous (MN) frames of sample images that have not been processed with defects will be used as historical reference images of the samples to be predicted in the time series, and will be input into the tooth generation network together with the samples to be predicted. The training tooth generation network can accurately learn the mapping relationship between lip shape and tooth shape, and learn the consistency of changes in the tooth area in the time dimension. For example, assuming that M is 4 and N is 2, that is, the training sample includes 4 frames of sample images, then the last 2 frames can be introduced with defects as samples to be predicted.

[0138] a3, input the previous (MN) frame sample image and the next N frames of samples to be predicted into the tooth generation network for tooth reconstruction, and obtain the N frames of predicted images of its predicted output;

[0139] That is, the samples to be predicted in the latter N frames with defects and the sample images of the tooth area in the previous (MN) frame that meet the requirements are input into the tooth generation network together, and the tooth generation network is trained to learn to accurately predict the tooth area corresponding to the samples to be predicted in the latter N frames while maintaining the consistency of the tooth area in the time dimension, and obtain the corresponding N frames of predicted images.

[0140] See also Figure 4 The example diagram of the training process is shown exemplarily. After obtaining the previous (MN) frame sample image and the next N frames of samples to be predicted corresponding to each training sample, both are used as inputs of the tooth generation network. The tooth generation network sequentially outputs the predicted images corresponding to the next N frames of samples to be predicted. Compared with the samples to be predicted, the tooth area of ​​the predicted images is reconstructed and optimized.

[0141] a4, using the N frames of predicted images and the corresponding N frames of sample images to calculate the loss function, and adjusting the parameters of the tooth generation network to minimize the loss function; the loss function at least includes pixel level difference loss and image temporal continuity loss.

[0142] The pixel level difference loss is used to measure the difference between the predicted image and the sample image at the pixel level, which can be achieved by calculating the mean square error or mean absolute error between the two. In this embodiment, after obtaining N frames of predicted images, the N frames of predicted images and their corresponding N frames of sample images can be subjected to pixel difference loss calculation such as L1 loss or L2 loss to obtain the corresponding pixel level difference loss.

[0143] Image temporal coherence loss is used to measure the coherence of changes in the tooth region in the predicted image and the input previous (MN) frame sample image in the time series, that is, whether the changes in the tooth region between adjacent frames are smooth and consistent, so that the tooth region in the generated predicted image is more natural and coherent in the dynamic process, avoiding sudden displacement of the tooth position. The image temporal coherence loss can be calculated by using the optical flow method to calculate the motion vector of the tooth region between adjacent frames, and compare the difference in motion vectors between adjacent images between the generated predicted image and the previous MN frame sample image; or, deep learning models such as convolutional neural networks or recurrent neural networks can be used to extract image features and calculate the similarity or difference between adjacent frame features, such as cosine similarity or L2 distance. Furthermore, the attention mechanism can be combined to emphasize important temporal information by calculating the attention weights between different frames, and the loss is calculated accordingly.

[0144] Combine the image temporal coherence loss with the pixel-level difference loss, such as weighted fusion of the two losses by set weights, to obtain the final loss function. Adjust the parameters of the tooth generation network through the back propagation algorithm to minimize the loss function, and repeat the above steps a3-a4 until the preset training convergence condition is reached, such as the loss function no longer significantly decreases or the set iteration round is reached.

[0145] In the disclosed embodiment, by using a training set containing M consecutive frames of sample images, the tooth generation network can learn the subtle changes and common features of the tooth region between different frames, so that the tooth generation network can capture the time dependency in dynamic scenes. The loss function simultaneously considers the pixel-level difference loss and the image temporal coherence loss, and incorporates the attention mechanism of time series processing, so that the network can consider the correlation and continuity between the previous and next frames when generating predicted images. By minimizing the comprehensive loss, the finally trained tooth generation network can generate tooth region images that are both accurate and coherent, thereby achieving tooth region optimization.

[0146] In some embodiments, regarding the construction of the mask image for the tooth area in step S203 in the above embodiment, the mask processing can be directly performed on the tooth area of ​​the intermediate image to obtain the mask image, that is, the mask image retains all non-tooth area image information of the intermediate image, and the tooth area is marked by the mask processing as needing to be fused and replaced. For example, see Figure 5A A schematic diagram of a mask image is exemplarily shown, and the mask image is obtained by performing mask processing on the teeth area of ​​the digital person in the intermediate image, and replacing the teeth area with a black mask.

[0147] Alternatively, for the mask image constructed for the tooth area in step S203 in the above embodiment, the mask image may also be constructed as a binary image independent of the intermediate image. The mask image construction method described in this embodiment may at least include the following steps:

[0148] Create a blank image; correspond the pixels of the blank image to the pixels at corresponding positions on the intermediate image one by one; determine the tooth area and non-tooth area on the blank image based on the tooth area on the intermediate image; set pixel values ​​for the pixels in the tooth area and non-tooth area respectively, and generate a mask image corresponding to the intermediate image.

[0149] Among them, the image size and resolution of the blank image are the same as those of the intermediate image. Based on the image size and image resolution of the intermediate image, a blank image is created, and the pixel values ​​of all pixels of the blank image can be initialized to pixel values ​​(0,0,0) or to grayscale value 0.

[0150] By mapping the position information of the tooth area in the intermediate image to the blank image, the position information coordinates of the tooth area can be directly applied to the corresponding position of the blank image based on the same image size and resolution as the intermediate image. Based on the position information coordinates of the tooth area, the tooth area and the non-tooth area can be accurately identified on the blank image.

[0151] In this embodiment, since the mask image is used to indicate that the non-tooth area of ​​the intermediate image is retained during the image fusion process, when the blank image is set in the form of a grayscale image, the pixel value of the tooth area can be set to the maximum grayscale value of 255, and the pixel value of the non-tooth area can be set to the minimum grayscale value of 0; alternatively, the target mask image can be set in the form of a binary mask image, and the pixel value of the non-tooth area can be set to a value of 0 to indicate that it is retained, and the pixel value of the tooth area can be set to a value of 1 to indicate the area that needs to be replaced or modified.

[0152] Through the above pixel value setting method, the image effect presented by the mask image is an image including two black and white areas, wherein the black area represents the non-tooth area and the white area represents the tooth area. Figure 5B The mask image shown as an example sets black and white pixels for the non-tooth area and the tooth area respectively to generate the mask image shown in the figure. The white area boundary in the figure is the boundary of the tooth area, which matches the tooth area of ​​the digital person in the intermediate image.

[0153] In the embodiment of the present disclosure, by accurately detecting the tooth area of ​​the digital human on the intermediate image, and based on the tooth area, determining the tooth area and the non-tooth area on a pre-set blank image, and then setting different pixel values ​​to identify the non-tooth area on the intermediate image during image fusion and updating the tooth area, the construction of the mask map is realized, providing effective fusion boundary information for the fusion of the intermediate image and the corrected image.

[0154] In some embodiments, the intermediate images obtained by driving the digital human do not have missing teeth, that is, the lips of the digital human in these images are in a naturally closed state at certain moments so that no teeth are exposed. In order to further optimize the realism and coherence of the digital human image and reduce unnecessary computing overhead, this embodiment proposes an intelligent selection and optimization processing method, which focuses on the fine processing of the tooth area and fully considers the changing state of the digital human's lip shape to achieve more efficient and natural digital human image generation. The method may include the following steps:

[0155] After obtaining the plurality of intermediate images, it is detected whether the lips of the digital human in the intermediate images are in a closed state; when it is detected that the lips of the digital human in the intermediate images are in a closed state, the intermediate images are directly used as the corresponding digital human images.

[0156] That is, for each intermediate image, image recognition or machine learning technology can be used to identify whether the digital human's lip shape is in a naturally closed state. Figure 6 In any two intermediate images shown as examples, the lip shape of the digital human in the left image A is in a naturally closed state, while the lip shape of the digital human in the right image B is not in a closed state.

[0157] When it is detected that the lips of a digital human are in a closed state in a certain intermediate image, since the teeth are usually not exposed when the lips are closed, in order to save computing resources, no additional calculation or modification may be performed on this type of intermediate image, that is, the intermediate image does not need further processing of the teeth area, and the intermediate image itself can be directly used as the final output digital human image.

[0158] In the disclosed embodiment, by detecting the state of the lip shape, additional calculations for intermediate images that do not require tooth area processing can be avoided, thereby saving computing resources and reducing unnecessary computing overhead, thereby improving overall processing efficiency and image realism.

[0159] In some embodiments, in order to improve the realism, consistency and generation efficiency of the digital human image, this embodiment provides a set of standard tooth images preset for the digital human and in line with its image design as a reconstruction reference during the tooth reconstruction process. During the tooth reconstruction process, for images with a lack of tooth information in the intermediate image, such as blurred tooth color and no tooth contour, this embodiment uses the generalization ability of the tooth generation network combined with the standard tooth image as a reference to generate a corrected image that is more in line with the digital human image, so as to ensure that the corrected digital human tooth image is highly matched with the overall image of the digital human.

[0160] Based on this, before the teeth are reconstructed, a standard tooth image preset for the digital human and in accordance with the design of the digital human image can be obtained. The standard tooth image represents the dental appearance of the digital human in an ideal state, including the shape, size, color, arrangement and other characteristics of the teeth. The neatness, color uniformity and coordination with the face of the teeth are consistent with the image of the digital human. In order to provide more adequate tooth features, the standard tooth image can expose as many teeth as possible while keeping the lip shape within the speaking shape. For example, the standard tooth image can be an image of the digital human showing 8 teeth when simulating a human smile. The standard tooth image can be produced by professional modeling software or image editing tools to ensure that they are consistent with the overall image of the digital human.

[0161] Based on the standard tooth image, during the process of tooth reconstruction for any intermediate image, the standard tooth image together with the frame of intermediate image can be input into the tooth generation network to perform tooth reconstruction for the intermediate image according to the tooth generation network.

[0162] That is, the standard tooth image, as one of the inputs of the tooth generation network, provides the network with clear guidance on the appearance of digital human teeth. By comparing and analyzing the tooth regions in the standard tooth image and the intermediate images of consecutive frames, the tooth generation network can learn more accurate tooth generation rules and features. For images with a lack of tooth information in the intermediate image, such as blurred tooth color and no tooth contour, or for digital human teeth with special features such as canines and fangs, the standard tooth image is used as a reference image to improve the generalization ability of the network, enabling it to generate more natural, realistic and digital human-like tooth images.

[0163] In the disclosed embodiment, by introducing a standard tooth image into the tooth generation network, the tooth generation network can automatically infer and generate the missing tooth parts according to the input image information while maintaining coordination with the overall image of the digital human. The generated corrected image adds tooth parts that are highly matched with the digital human image, thereby improving the accuracy of tooth reconstruction, thereby greatly improving the realism and naturalness of the digital human image.

[0164] Next, in order to enable those skilled in the art to better understand the digital human image generation method provided by the present application, see Figure 7 An exemplary process of digital human tooth reconstruction and image fusion is shown. This embodiment gives an example of the processing process of an intermediate image. The processing process of other intermediate images is the same as this process:

[0165] The figure marked as 701 represents an initial image of a digital human, which is used to provide facial features of the digital human in the model driving process. The initial image is driven by the model to obtain an intermediate image, and a mask processing is performed on the tooth area on the intermediate image. The tooth area is replaced with a black mask to obtain a mask image marked as 702;

[0166] The local area including the lip shape on the intermediate image is cropped to obtain a local image marked as 703, and the position information of the local image on the intermediate image is recorded. The local image and the corresponding previous K-frame corrected image are input into the tooth generation network, and the corrected image 704 is obtained by the tooth generation network for tooth reconstruction;

[0167] According to the position information of the corrected image 704 on the intermediate image, the mask image 702 that has been subjected to mask processing is subjected to image fusion processing with the corrected image 704 to obtain a digital human tooth image 705 that will be output eventually.

[0168] In the disclosed embodiment, after the intermediate image is generated by model-driven generation, a high-quality tooth image generated by the tooth generation network for the tooth area of ​​the intermediate image is fused with other facial images except the tooth area in the intermediate image to ensure the natural connection of the teeth and the overall realistic effect, so that the final output of the optimized digital human has clear and coherent teeth, which are naturally integrated with the mouth area, ensuring consistency and realism in the animation.

[0169] Corresponding to the above-mentioned embodiment of the method for generating a digital human image, see Figure 8 As shown, the present application also provides an embodiment of a digital human image generation device, the device comprising:

[0170] The model driving module 801 is used to drive the facial expression and lip shape of the digital human using the model according to the initial image containing the digital human, and generate multiple intermediate images matching the driving voice data;

[0171] The tooth reconstruction module 802 is used to perform tooth reconstruction on each intermediate image according to a pre-trained tooth generation network to obtain a corrected image corresponding to the intermediate image; wherein the tooth reconstruction of the intermediate image is based on the previous K frame corrected images obtained by completing tooth reconstruction before the intermediate image;

[0172] A mask image construction module 803 is used to locate the tooth region of the digital person from each frame of the intermediate image and construct a mask image for the tooth region;

[0173] The fusion generation module 804 is used to fuse the intermediate image and its corresponding corrected image according to the mask image to obtain a digital human image corresponding to each frame of the intermediate image.

[0174] In some embodiments, the tooth reconstruction module is specifically used to:

[0175] Extracting feature information of tooth regions in the intermediate image and the first K frames of corrected images before the intermediate image through the tooth generation network;

[0176] According to the similarity of the tooth area between the intermediate image and the first K frame corrected image, a temporal consistency attention mechanism is applied to fuse and represent the feature information to obtain a fused feature;

[0177] According to the fusion features corresponding to the intermediate image, a corrected image output by the tooth generation network is obtained.

[0178] In some embodiments, the apparatus further comprises a step of training a tooth generation network:

[0179] Acquire a training sample set, each training sample includes M consecutive frames of sample images, and the tooth area of ​​the object in the sample image meets the preset tooth quality requirements;

[0180] Perform defect introduction processing on the tooth area of ​​the sample images of the last N frames in the M frames as the samples to be predicted; wherein N is a positive integer and is less than M;

[0181] Input the previous (MN) frame sample image and the next N frames of samples to be predicted into the tooth generation network for tooth reconstruction, and obtain the N frames of predicted images of its predicted output;

[0182] The loss function is calculated using the N frames of predicted images and the corresponding N frames of sample images, and the parameters of the tooth generation network are adjusted to minimize the loss function; the loss function at least includes pixel level difference loss and image temporal coherence loss.

[0183] In some embodiments, the mask image construction module is specifically used to:

[0184] Creating a blank image; wherein the pixels of the blank image correspond one-to-one to the pixels at corresponding positions on the intermediate image;

[0185] Determining the tooth region and the non-tooth region on the blank image according to the tooth region on the intermediate image;

[0186] Pixel values ​​are set for the pixel points in the tooth area and the non-tooth area respectively, and a mask image corresponding to the intermediate image is generated.

[0187] In some embodiments, the mask image construction module is specifically used to:

[0188] Determine the key points of the contour of the digital human's lips in the intermediate image; determine the boundary of the tooth area according to the key points of the contour that identify the inner lip boundary;

[0189] Alternatively, the intermediate image is input into a pre-trained tooth region segmentation network to obtain an intermediate image with marked tooth regions.

[0190] In some embodiments, the apparatus further comprises:

[0191] After obtaining the plurality of intermediate images, detecting whether the lips of the digital human in the intermediate images are in a closed state;

[0192] When it is detected that the lips of the digital human in the intermediate image are in a closed state, the intermediate image is directly used as the corresponding digital human image.

[0193] In some embodiments, the apparatus further comprises:

[0194] Acquire a standard tooth image preset for the digital human and conforming to the image design of the digital human;

[0195] The standard tooth image and the intermediate image are input into the tooth generation network together, so as to perform tooth reconstruction on the intermediate image according to the tooth generation network.

[0196] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0197] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. Ordinary technicians in this field can understand and implement it without creative work.

[0198] The present application also provides an electronic device. The structural diagram of the electronic device is as follows: Fig. 9 As shown, the electronic device 900 includes at least one processor 901, a memory 902 and a bus 903, and the at least one processor 901 is electrically connected to the memory 902; the memory 902 is configured to store at least one computer-executable instruction, and the processor 901 is configured to execute the at least one computer-executable instruction, thereby performing the steps of any digital human image generation method provided in any embodiment or any optional implementation manner in the present application.

[0199] Furthermore, the processor 901 may be a Field-Programmable Gate Array (FPGA) or other devices with logic processing capabilities, such as a Microcontroller Unit (MCU) or a Central Process Unit (CPU).

[0200] An embodiment of the present application further provides another readable storage medium storing a computer program, which is used to implement the steps of any digital human image generation method provided in any embodiment or any optional implementation manner of the present application when executed by a processor.

[0201] The readable storage medium provided in the embodiments of the present application includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the readable storage medium includes any medium that can store or transmit information in a readable form by a device (e.g., a computer).

[0202] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0203] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for generating a digital human image, characterized in that: The method comprises: Based on the initial image containing the digital human, the model is used to drive the facial expression and lip shape of the digital human to generate multiple intermediate images that match the driving voice data; For each intermediate image, teeth reconstruction is performed on the intermediate image according to a pre-trained teeth generation network to obtain a corrected image corresponding to the intermediate image; wherein the teeth reconstruction of the intermediate image is based on the previous K frames of corrected images obtained by completing teeth reconstruction before the intermediate image; Locating the tooth region of the digital person from each frame of the intermediate image, and constructing a mask image for the tooth region; According to the mask image, each frame of the intermediate image and its corresponding corrected image are fused to obtain a digital human image corresponding to each frame of the intermediate image.

2. The method according to claim 1, characterized in that The step of reconstructing teeth from the intermediate image according to a pre-trained teeth generation network comprises: Extracting feature information of tooth regions in the intermediate image and the first K frames of corrected images before the intermediate image through the tooth generation network; According to the similarity of the tooth area between the intermediate image and the first K frame corrected image, a temporal consistency attention mechanism is applied to fuse and represent the feature information to obtain a fused feature; According to the fusion features corresponding to the intermediate image, a corrected image output by the tooth generation network is obtained.

3. The method according to claim 1, characterized in that The step of reconstructing teeth from the intermediate image according to a pre-trained teeth generation network comprises: Locating the teeth region of the digital person from the intermediate image, and cutting out a local image including the teeth region from the intermediate image; Perform teeth reconstruction on the partial image according to a pre-trained teeth generation network.

4. The method according to claim 1, characterized in that: The method further comprises the step of training a tooth generation network: Acquire a training sample set, each training sample includes M consecutive frames of sample images, and the tooth area of ​​the object in the sample image meets the preset tooth quality requirements; Perform defect introduction processing on the tooth area of ​​the sample images of the last N frames in the M frames as the samples to be predicted; wherein N is a positive integer and is less than M; Input the previous (MN) frame sample image and the next N frames of samples to be predicted into the tooth generation network for tooth reconstruction, and obtain the N frames of predicted images of its predicted output; The loss function is calculated using the N frames of predicted images and the corresponding N frames of sample images, and the parameters of the tooth generation network are adjusted to minimize the loss function; the loss function at least includes pixel level difference loss and image temporal coherence loss.

5. The method according to claim 1, characterized in that The constructing a mask map for the tooth region includes: Creating a blank image; wherein the pixels of the blank image correspond one-to-one to the pixels at corresponding positions on the intermediate image; Determining the tooth region and the non-tooth region on the blank image according to the tooth region on the intermediate image; Pixel values ​​are set for the pixel points in the tooth area and the non-tooth area respectively, and a mask image corresponding to the intermediate image is generated.

6. The method according to claim 1, characterized in that The step of locating the tooth region of the digital person from each frame of the intermediate image comprises: Determine the key points of the contour of the digital human's lips in the intermediate image; determine the boundary of the tooth area according to the key points of the contour that identify the inner lip boundary; or, The intermediate image is input into a pre-trained tooth region segmentation network to obtain an intermediate image with a marked tooth region.

7. The method according to claim 1, characterized in that The method further comprises: After obtaining the plurality of intermediate images, detecting whether the lips of the digital human in the intermediate images are in a closed state; When it is detected that the lips of the digital human in the intermediate image are in a closed state, the intermediate image is directly used as the corresponding digital human image.

8. The method according to claim 1, characterized in that The method further comprises: Acquire a standard tooth image preset for the digital human and conforming to the image design of the digital human; The standard tooth image and the intermediate image are input into the tooth generation network together, so as to perform tooth reconstruction on the intermediate image according to the tooth generation network.

9. A digital human image generation device, characterized in that: The device comprises: A model driving module, for driving the facial expression and lip shape of the digital human using a model based on an initial image containing the digital human, and generating a plurality of intermediate images matching the driving voice data; A tooth reconstruction module is used to reconstruct teeth of each intermediate image according to a pre-trained tooth generation network to obtain a corrected image corresponding to the intermediate image; wherein the tooth reconstruction of the intermediate image is based on the first K frames of corrected images obtained by completing tooth reconstruction before the intermediate image; A mask image construction module, used to locate the tooth region of the digital human from each frame of the intermediate image and construct a mask image for the tooth region; The fusion generation module is used to fuse the intermediate image and its corresponding corrected image according to the mask image to obtain a digital human image corresponding to each frame of the intermediate image.

10. An electronic device, characterized in that: include: Memory, processor; The memory is used to store computer programs; The processor is used to call the computer program to implement the method according to any one of claims 1-8.

11. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images