Human body image restoration method and product based on diffusion model

Abstract

The invention provides a human body image restoration method and product based on a diffusion model, and belongs to the technical field of image processing. The method comprises the steps that after an original image is acquired, a three-dimensional human body model of a target human body is generated according to the target human body in the original image; obtaining a depth map of the target human body according to the three-dimensional human body model of the target human body; and inputting the depth map of the target human body into a ControlNet model, and controlling an Inpainting Stable Diffusion model to redraw the image of the target human body in the original image by the trained ControlNet model so as to repair the image of the target human body in the original image. The invention aims to improve the generation quality of the human body image.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of image processing, and in particular, to a method and product for restoring human body images based on a diffusion model. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technologies, deep learning-based methods have made significant progress in the field of image generation and editing. As one of the cutting-edge technologies, Text-to-Image Diffusion Models can generate highly realistic images based on input text descriptions.

[0003] However, in practical applications, the model may be distorted when dealing with complex human structures. Current human image generation technology often has distortion problems, including unnatural positions and angles of human joints (such as elbows, knees, shoulders, etc.), imbalanced proportions between various parts of the human body, stiff or uncoordinated postures and movements in human images, distorted or asymmetrical facial features, and errors in the number, length and shape of fingers or toes, resulting in low quality of human image generation. Summary of the invention

[0004] The embodiments of the present application provide a method and product for restoring human images based on a diffusion model, aiming to improve the generation quality of human images.

[0005] In a first aspect, an embodiment of the present application provides a method for restoring a human body image based on a diffusion model, the method comprising: After acquiring the original image, generating a three-dimensional human body model of the target human body according to the target human body in the original image; Obtaining a depth map of the target human body according to the three-dimensional human body model of the target human body; The depth map of the target human body is input into the ControlNet model, and the trained ControlNet model controls the Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image, so as to repair the image of the target human body in the original image.

[0006] Optionally, generating a three-dimensional human body model of the target human body according to the target human body in the original image includes: Identify the key points of any target human body in the original image by estimating human body posture in the original image, wherein the key points of the human body include key joint points and joint connection areas; According to the key points of the target human body, a three-dimensional human body model of the target human body is obtained by reconstructing the three-dimensional human body model.

[0007] Optionally, obtaining a depth map of the target human body according to the three-dimensional human body model of the target human body includes: According to the three-dimensional human body model of the target human body, a depth map and a human body mask of the target human body are obtained through a depth map renderer.

[0008] Optionally, the depth map of the target human body is input into a ControlNet model, and the trained ControlNet model controls an Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image, including: Inputting the depth map of the target human body into the ControlNet model as control information; Inputting the original image after the human body masking into the Inpainting Stable Diffusion diffusion model; The ControlNet model controls the Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image after the human body mask is applied.

[0009] Optionally, the method further comprises: Build a training dataset; According to the training data set, the ControlNet model and the Inpainting Stable Diffusion diffusion model are jointly trained to obtain the trained ControlNet model and the Inpainting Stable Diffusion diffusion model.

[0010] Optionally, constructing a training dataset includes: Obtaining an image sample and a normal human posture of any human body in the image sample; Depth map samples are obtained by rendering according to the normal human posture of any human body; and binarizing the depth map samples to form human body mask samples; The image sample, the depth map sample and the human body mask sample are taken as a group of training samples.

[0011] Optionally, according to the training data set, jointly training a ControlNet model and an Inpainting Stable Diffusion diffusion model to obtain a trained ControlNet model and an Inpainting Stable Diffusion diffusion model, including: Input the depth map sample of any training sample into the ControlNet model as control information; Inputting a mask image obtained by subjecting the image sample to the human body mask sample into the Inpainting StableDiffusion diffusion model, wherein the mask image includes other background information in the image sample except the human body area; The Inpainting Stable Diffusion diffusion model iteratively optimizes the random noise vector of the human body area in the mask image according to the depth map samples through DDIM sampling to generate an output image after redrawing the human body image; The ControlNet model and the Inpainting Stable Diffusion diffusion model are iteratively updated through a joint loss function consisting of reconstruction loss, perception loss and local smoothness loss to obtain the trained ControlNet model and Inpainting Stable Diffusion diffusion model.

[0012] In a second aspect, an embodiment of the present application provides a human body image restoration device based on a diffusion model, the device comprising: A three-dimensional human body model generation module is used to generate a three-dimensional human body model of a target human body according to the target human body in the original image after acquiring the original image; A depth map generating module, used for obtaining a depth map of the target human body according to the three-dimensional human body model of the target human body; The redrawing module is used to input the depth map of the target human body into the ControlNet model, and the trained ControlNet model controls the Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image to repair the image of the target human body in the original image.

[0013] In a third aspect, an embodiment of the present application provides a computer device, comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the human body image restoration method based on the diffusion model described in the first aspect of the embodiment is executed.

[0014] In a fourth aspect, an embodiment of the present application provides a non-volatile readable storage medium, wherein the non-volatile readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the human body image restoration method based on the diffusion model described in the first aspect of the embodiment is executed.

[0015] Beneficial effects: The human body image restoration method based on the diffusion model provided in this embodiment generates a three-dimensional human body model of the target human body in the original image after acquiring the original image; obtains a depth map of the target human body based on the three-dimensional human body model of the target human body; inputs the depth map of the target human body into a ControlNet model, and controls the Inpainting Stable Diffusion diffusion model controlled by the trained ControlNet model to redraw the image of the target human body in the original image, so as to restore the image of the target human body in the original image.

[0016] In this way, human body images with human body deformities in the original images can be redrawn, and the generation quality of human body images can be improved. This not only improves the efficiency of human body image restoration, but also has applicability and practicality in different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 is a flowchart of the steps of a human body image restoration method based on a diffusion model proposed in an embodiment of the present application; Figure 2 is a schematic diagram of executing a human body image restoration method based on a diffusion model proposed in an embodiment of the present application; Figure 3 is a schematic diagram of executing a training process provided by an embodiment of the present application; Figure 4 It is a functional module diagram of a human body image restoration device based on a diffusion model proposed in an embodiment of the present application; Figure 5 is a schematic diagram of a computer device proposed in one embodiment of the present application; Figure 6 is a schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application; Figure 7 It is a schematic diagram of a computer program product proposed in one embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] ControlNet is a neural network structure that can guide the diffusion model through additional conditions (such as edge maps, segmentation maps, etc.), allowing training and fine-tuning of pre-trained models on small datasets.

[0021] ‌Stable Diffusion's Inpainting function is a local repainting technology, which is mainly used for local modification and improvement of images. Through the Inpainting function, users can select specific areas in the image for repainting to add or delete elements, adjust colors, and correct postures. This technology is particularly suitable for correcting imperfect parts in the image, such as adjusting details such as hands and faces.

[0022] With the rapid development of computer vision and artificial intelligence technologies, deep learning-based methods have made significant progress in the field of image generation and editing. As one of the cutting-edge technologies, Text-to-Image Diffusion Models can generate highly realistic images based on input text descriptions.

[0023] However, in practical applications, the model may be distorted when dealing with complex human structures. Current human image generation technology often has distortion problems, including unnatural positions and angles of human joints (such as elbows, knees, shoulders, etc.), imbalanced proportions between various parts of the human body, stiff or uncoordinated postures and movements in human images, distorted or asymmetrical facial features, and errors in the number, length, and shape of fingers or toes.

[0024] In order to improve the quality and practicality of human body image generation and ensure a good user interaction experience, this embodiment proposes a human body image restoration method based on a diffusion model, which can automatically detect and repair human body structure distortion caused by the image generation model, and ensure its practicality and reliability in various application scenarios by improving the naturalness and aesthetics of the generated image.

[0025] Reference Figure 1 , shows a flowchart of a method for restoring a human image based on a diffusion model in an embodiment of the present application, and the method may specifically include the following steps: S101: After acquiring an original image, a three-dimensional human body model of the target human body is generated according to the target human body in the original image.

[0026] The original image includes any image containing a human body. If a human body image in an image is distorted, the image can be used as the original image. The distortion of the human body image includes unnatural positions and angles of human joints, such as elbows, knees, shoulders, etc., the proportions between various parts of the human body may be unbalanced, the posture and movement in the human body image may appear stiff or uncoordinated, the facial features of the person may be distorted or asymmetrical, and the number, length and shape of fingers or toes may be wrong.

[0027] After acquiring the original image, firstly, a three-dimensional human body model of the target human body in the original image is generated. Specifically, the process of generating the three-dimensional human body model of the target human body according to the target human body in the original image includes the following steps: A1: Identify the key points of any target human body in the original image by estimating the human body posture in the original image, wherein the key points of the human body include key joint points and joint connection areas.

[0028] In the actual implementation process, two-dimensional human posture estimation can be used to identify the human key points of any target human body in the original image, including but not limited to human key points such as facial features, neck, fingers, wrists, elbows, knees, shoulders, ankles, etc.; human key points can characterize the relative position and proportion of the key points of the target human body.

[0029] In two-dimensional human posture estimation, the main focus is on identifying and locating the key points of the human body in the two-dimensional image and obtaining the human body posture information. Specifically, the position of each joint in the RGB image of the original image is estimated through the two-dimensional coordinates (x, y).

[0030] For example, a top-down approach can be used to first detect each person using an object detection algorithm, and then estimate the pose of each person based on a single person image; or a bottom-up approach can be used to directly detect all key points in the image, and then estimate the pose of the person by combining these key points; In the actual implementation process, the key points of the human body can also be identified based on the deep learning model for human posture estimation. The deep learning model for human posture estimation can be the DeepPose model (a model for two-dimensional posture estimation based on convolutional neural networks) and HRNet. HRNet is an efficient network architecture that can process high-resolution feature maps and low-resolution feature maps at the same time to improve the accuracy of posture estimation.

[0031] A2: Obtain a three-dimensional human body model of the target human body through a three-dimensional human body reconstruction model according to the human body key points of the target human body.

[0032] After the key points of any target human body in the original image are identified through human body posture estimation, a three-dimensional human body model of the target human body in the current human body posture is reconstructed through a three-dimensional human body reconstruction model.

[0033] For example, if the key points of the target human body are represented as standing, a three-dimensional human body model of the target human body in a standing state can be reconstructed; if the key points of the target human body are represented as sitting, a three-dimensional human body model of the target human body in a sitting state can be reconstructed.

[0034] The three-dimensional human body reconstruction model can adopt any existing three-dimensional human body reconstruction model and can be selected according to the needs of actual application. This embodiment does not impose any restrictions. Common three-dimensional human body reconstruction models are learned from a large amount of normal human body data, so even based on the key joint points obtained from the distorted human body image in the current original image, a complete, distortion-free three-dimensional model of the human body posture can be constructed. For example, even if the current original image shows that there are only 4 fingers in the human key points of the target human body through human body posture estimation, it can still be reshaped into a three-dimensional human body model with 5 fingers based on the three-dimensional human body reconstruction model.

[0035] S102: Obtaining a depth map of the target human body according to the three-dimensional human body model of the target human body.

[0036] Specifically, in the process of obtaining the depth map of the target human body according to the three-dimensional human body model of the target human body, the depth map and the human body mask of the target human body can be obtained according to the three-dimensional human body model of the target human body through a depth map renderer.

[0037] The depth map represents the shape and spatial position information of the human body in the current human body posture. The spatial position information includes the distance between the pixel point corresponding to any human body key point and the camera, and its main purpose is to control the human body posture.

[0038] The human body mask is based on the depth of the target human body Figure 2 The valued human mask is used to separate the human image and non-human image in the original image.

[0039] S103: inputting the depth map of the target human body into a ControlNet model, and controlling the trained ControlNet model to control an Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image, so as to repair the image of the target human body in the original image.

[0040] ControlNet is a neural network structure that can guide the diffusion model through additional conditions. The Inpainting Stable Diffusion diffusion model is used for local modification and improvement of the image. In this method, the depth map of the target human body is used as the control information of the ControlNet model, so that the ControlNet model guides the Inpainting Stable Diffusion diffusion model to redraw the distorted target human body image in the original image.

[0041] In a feasible implementation, the depth map of the target human body is input into a ControlNet model, and the trained ControlNet model controls an Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image, including the following steps: B1: Input the depth map of the target human body into the ControlNet model as control information.

[0042] The depth map of the target human body can control the posture of the human body in the human body image. After the key points of the human body in the original image are reconstructed in three dimensions through the three-dimensional human body reconstruction model, the depth map of the target human body rendered according to the three-dimensional human body model of the target human body can describe a complete human body without distortion, and the posture of the target human body described by the depth map is consistent with the posture of the target human body in the original image.

[0043] B2: Inputting the original image after the human body mask into the Inpainting Stable Diffusion diffusion model.

[0044] After the original image and the human body mask, the mask image of the original image is obtained. The mask image of the original image includes non-human body images except the human body image. After the mask image is input into the Inpainting Stable Diffusion diffusion model, the Inpainting Stable Diffusion diffusion model can know that the area to be redrawn is the human body area.

[0045] For example, in the mask image of the original image, the human image is set to 0, and the non-human image is set to 1.

[0046] B3: The ControlNet model controls the Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image after the human body mask is applied.

[0047] After the Inpainting Stable Diffusion diffusion model determines that the redrawn area is a human body image according to the mask image of the original image, the human body image is redrawn according to the depth map of the target human body to obtain the original image after the target human body image is restored.

[0048] Reference Figure 2 , shows a schematic diagram of executing a human image restoration method based on a diffusion model provided in an embodiment of the present application, the method comprising the following processes: First, obtain the original image; Then, the key points of any target human body in the original image are determined through human body posture estimation; Next, a three-dimensional human body model of the target human body is generated according to the key points of the human body through the three-dimensional human body reconstruction model; Then, according to the three-dimensional human body model of the target human body, a depth map of the target human body is obtained through a depth map renderer; Next, the depth map is binarized to obtain the human body mask; Finally, the depth map of the target human body is input into the ControlNet model, and the original image after the human body mask is input into the Inpainting Stable Diffusion diffusion model. The ControlNet model controls the Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image after the human body mask is applied, and outputs the original image after the human body image is repaired.

[0049] In a feasible implementation, the method further includes: constructing a training data set, and then jointly training a ControlNet model and an Inpainting Stable Diffusion diffusion model according to the training data set to obtain trained ControlNet model and Inpainting Stable Diffusion diffusion model.

[0050] Since ControlNet trained using Midas depth estimation is trained on a paired dataset of full-image depth, using the ControlNet model to control the Inpainting Stable Diffusion diffusion model may cause artifacts in the human body and a sense of separation between the human body and the scene. Therefore, it is necessary to construct a training dataset to retrain ControlNet.

[0051] For example, the process of constructing a training data set includes the following steps for constructing any training sample: C1: Obtain image samples and the normal human posture of any human body in the image samples; C2: Depth map samples are rendered based on the normal human posture of any human body; C3: binarizing the depth map samples to form human mask samples; C4: The image sample, depth map sample and human mask sample are taken as a set of training samples.

[0052] In the process of jointly training the ControlNet model and the Inpainting StableDiffusion diffusion model according to the training data set to obtain the trained ControlNet model and the Inpainting Stable Diffusion diffusion model, the purpose is to train ControlNet through large-scale normal human body data so that it can finely control Inpainting Stable Diffusion to redraw the human body area.

[0053] Specifically, the training process includes the following steps: D1: Input the depth map sample of any training sample as control information into the ControlNet model; D2: Input the mask image obtained after the image sample is acted upon by the human body mask sample into the Inpainting StableDiffusion diffusion model, wherein the mask image contains other background information in the image sample except the human body area; D3: The Inpainting Stable Diffusion diffusion model performs iterative optimization on the random noise vector of the human body area in the mask image according to the depth map samples through DDIM sampling, and generates an output image after redrawing the human body image; D4: The ControlNet model and the Inpainting Stable Diffusion diffusion model are iteratively updated through a joint loss function consisting of reconstruction loss, perception loss, and local smoothing loss to obtain the trained ControlNet model and the Inpainting Stable Diffusion diffusion model.

[0054] Specifically, image reconstruction loss is a commonly used loss function in image generation or reconstruction tasks, which measures the difference between the generated image and the original image. Image reconstruction loss is used to guide the model to generate an image that is as similar as possible to the original image. For example, common image reconstruction losses including pixel-level reconstruction loss can be selected according to the needs of actual applications, so that the image after repairing the human body image is consistent with the original image in pixel dimension.

[0055] Perceptual loss does not directly compare pixel values, but compares the difference between the generated image and the original image through the intermediate layer feature map of the model, which can capture the high-level semantic features of the image and make the generated image clearer and more realistic.

[0056] The local smoothness loss is used to make the boundary between human images and non-human images in the restored images more consistent.

[0057] In the actual implementation process, the choice of loss function can be selected according to the needs of the actual application, and this embodiment does not impose any limitation.

[0058] Reference Figure 3 , shows a schematic diagram of the execution of the training process provided by the embodiment of the present application. In the process of jointly training the ControlNet model and the Inpainting Stable Diffusion diffusion model, for any training sample, the following process is included: First, the depth map sample of any training sample is input into the ControlNet model as control information; Next, the mask image obtained after the image sample is acted upon by the human mask sample is input into the InpaintingStable Diffusion diffusion model; Then, the Inpainting Stable Diffusion diffusion model iteratively optimizes the random noise vector of the human body area in the mask image according to the depth map samples through DDIM sampling, and generates an output image after redrawing the human body image.

[0059] There are currently many problems in generating human body images, mainly including structural distortion, disproportion and detail errors. The generated human body images often have problems such as uncoordinated limb proportions, unnatural joint positions and angles, stiff postures, distorted or asymmetric facial features, and incorrect numbers of fingers or toes. In addition, when a person is partially occluded, the performance of the generated model decreases significantly, and it is difficult to accurately reconstruct the occluded part. In complex backgrounds, existing models have difficulty distinguishing between targets and backgrounds, which can easily lead to erroneous generation. These problems not only affect the aesthetics of the image, but also limit the practicality and reliability of the generated images in application scenarios such as virtual fitting, film and television production, and virtual reality.

[0060] The human body image restoration method based on the diffusion model provided in this embodiment can regenerate a three-dimensional human body model of the target human body in the original image; obtain a depth map of the target human body based on the three-dimensional human body model of the target human body; input the depth map of the target human body into the ControlNet model, and the trained ControlNet model controls the Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image, so as to restore the image of the target human body in the original image.

[0061] In this way, human body images with human body deformities in the original images can be redrawn, and the generation quality of human body images can be improved. This not only improves the efficiency of human body image restoration, but also has applicability and practicality in different application scenarios.

[0062] In the process of image restoration, the restored human body image is made more realistic and natural, artifacts and scene fragmentation are reduced, and the structural and visual consistency between the restored image and the original image is ensured.

[0063] This method can be applied to scenarios including virtual fitting, social media and creative content generation, advertising and marketing, and virtual reality (VR) and augmented reality (AR). For example, in the creative content generation scenario, personalized and unique art images are generated with the help of AI art models. However, if there are distorted human images in the generated images, this method can automatically identify and mark the portrait area in the image, and repair the distorted human images.

[0064] Reference Figure 4 , shows a functional module diagram of a human image restoration device based on a diffusion model provided in an embodiment of the present application, the device comprising: The three-dimensional human body model generation module 100 is used to generate a three-dimensional human body model of a target human body according to the target human body in the original image after acquiring the original image; A depth map generating module 200 is used to obtain a depth map of the target human body according to the three-dimensional human body model of the target human body; The redrawing module 300 is used to input the depth map of the target human body into the ControlNet model, and control the trained ControlNet model to control the Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image to repair the image of the target human body in the original image.

[0065] Optionally, the three-dimensional human body model generation module includes: A key point recognition unit, used to recognize the key points of any target human body in the original image by estimating human body posture in the original image, wherein the key points of the human body include key joint points and joint connection areas; The three-dimensional reconstruction unit is used to obtain a three-dimensional human body model of the target human body through a three-dimensional human body reconstruction model according to the human body key points of the target human body.

[0066] Optionally, the depth map generation module includes: The depth map generating unit is used to obtain the depth map and the human body mask of the target human body through a depth map renderer according to the three-dimensional human body model of the target human body.

[0067] Optionally, the redrawing module redrawing unit is used to: Inputting the depth map of the target human body into the ControlNet model as control information; Inputting the original image after the human body masking into the Inpainting Stable Diffusion diffusion model; The ControlNet model controls the Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image after the human body mask is applied.

[0068] Optionally, the device further comprises: Construction module, used to construct training data set; The training module is used to jointly train the ControlNet model and the InpaintingStable Diffusion diffusion model according to the training data set to obtain the trained ControlNet model and the Inpainting StableDiffusion diffusion model.

[0069] Optionally, the building blocks include: A first construction unit is used to obtain an image sample and a normal human posture of any human body in the image sample; A second construction unit is used for rendering a depth map sample according to a normal human posture of any human body; A third construction unit is used to binarize the depth map sample to form a human body mask sample; The fourth construction unit is configured to use the image sample, the depth map sample, and the human body mask sample as a group of training samples.

[0070] Optionally, the training module includes: A first training unit is used to input a depth map sample of any training sample as control information into a ControlNet model; A second training unit is used to input a mask image obtained by subjecting the image sample to the human body mask sample into an Inpainting Stable Diffusion diffusion model, wherein the mask image includes other background information in the image sample except the human body area; A third training unit is used for the Inpainting Stable Diffusion diffusion model to iteratively optimize the random noise vector of the human body area in the mask image according to the depth map sample through DDIM sampling, so as to generate an output image after redrawing the human body image; The fourth training unit is used to iteratively update the ControlNet model and the Inpainting Stable Diffusion diffusion model through a joint loss function consisting of reconstruction loss, perception loss and local smoothing loss to obtain the trained ControlNet model and Inpainting Stable Diffusion diffusion model.

[0071] Reference Figure 5 , shows a schematic diagram of a computer device provided in an embodiment of the present application, comprising: at least one processor 501, and a memory 502, wherein the memory 502 stores a computer program that can be run on the processor 501, wherein when the processor 501 executes the computer program, the human body image restoration method based on the diffusion model described in the embodiment is executed.

[0072] When the processor 501 executes the computer program to perform the human body image restoration method based on the diffusion model described in the embodiment, it includes generating a three-dimensional human body model of the target human body according to the target human body in the original image after acquiring the original image; obtaining a depth map of the target human body according to the three-dimensional human body model of the target human body; inputting the depth map of the target human body into the ControlNet model, and controlling the Inpainting Stable Diffusion diffusion model by the trained ControlNet model to redraw the image of the target human body in the original image to restore the image of the target human body in the original image.

[0073] Reference Figure 6 , showing a schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application, wherein the non-volatile readable storage medium 600 stores a computer program 601, wherein when the computer program 601 is executed by a processor, the human body image restoration method based on the diffusion model described in the embodiment is executed.

[0074] When the computer program 601 is executed by the processor to perform the human body image restoration method based on the diffusion model described in the embodiment, it includes generating a three-dimensional human body model of the target human body according to the target human body in the original image after acquiring the original image; obtaining a depth map of the target human body according to the three-dimensional human body model of the target human body; inputting the depth map of the target human body into the ControlNet model, and controlling the Inpainting Stable Diffusion diffusion model by the trained ControlNet model to redraw the image of the target human body in the original image to restore the image of the target human body in the original image.

[0075] Reference Figure 7 , shows a schematic diagram of a computer program product provided in an embodiment of the present application, wherein the computer program product 700 includes a computer program / instruction 701, which, when executed by a processor, implements a human body image restoration method based on a diffusion model as described in the embodiment.

[0076] When the computer program / instruction 701 is executed by the processor to implement the human body image restoration method based on the diffusion model as described in the embodiment, it includes generating a three-dimensional human body model of the target human body according to the target human body in the original image after acquiring the original image; obtaining a depth map of the target human body according to the three-dimensional human body model of the target human body; inputting the depth map of the target human body into the ControlNet model, and controlling the Inpainting Stable Diffusion diffusion model by the trained ControlNet model to redraw the image of the target human body in the original image to restore the image of the target human body in the original image.

[0077] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0078] It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, devices or computer program products. Therefore, the embodiments of the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0079] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0082] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0083] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0084] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A human image restoration method based on a diffusion model, characterized in that: The method comprises: After acquiring the original image, generating a three-dimensional human body model of the target human body according to the target human body in the original image; Obtaining a depth map of the target human body according to the three-dimensional human body model of the target human body; The depth map of the target human body is input into the ControlNet model, and the trained ControlNet model controls the Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image, so as to repair the image of the target human body in the original image.

2. The method according to claim 1, characterized in that Generating a three-dimensional human body model of the target human body according to the target human body in the original image includes: Identify the key points of any target human body in the original image by estimating the human body posture in the original image, wherein the key points of the human body include key joint points and joint connection areas; According to the key points of the target human body, a three-dimensional human body model of the target human body is obtained by reconstructing the three-dimensional human body model.

3. The method according to claim 1, characterized in that Obtaining a depth map of the target human body according to the three-dimensional human body model of the target human body, including: According to the three-dimensional human body model of the target human body, a depth map and a human body mask of the target human body are obtained through a depth map renderer.

4. The method according to claim 3, characterized in that The depth map of the target human body is input into the ControlNet model, and the trained ControlNet model controls the Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image, including: Inputting the depth map of the target human body into the ControlNet model as control information; Inputting the original image after the human body masking into the Inpainting Stable Diffusion diffusion model; The ControlNet model controls the Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image after the human body mask is applied.

5. The method according to claim 1, characterized in that The method further comprises: Build a training dataset; According to the training data set, the ControlNet model and the Inpainting Stable Diffusion diffusion model are jointly trained to obtain the trained ControlNet model and the Inpainting Stable Diffusion diffusion model.

6. The method according to claim 5, characterized in that Building a training dataset includes: Obtaining an image sample and a normal human posture of any human body in the image sample; Depth map samples are obtained by rendering according to the normal human posture of any human body; and binarizing the depth map samples to form human body mask samples; The image sample, the depth map sample and the human body mask sample are taken as a group of training samples.

7. The method according to claim 6, characterized in that According to the training data set, the ControlNet model and the Inpainting Stable Diffusion diffusion model are jointly trained to obtain the trained ControlNet model and the Inpainting Stable Diffusion diffusion model, including: Input the depth map sample of any training sample into the ControlNet model as control information; Inputting a mask image obtained by subjecting the image sample to the human body mask sample into the Inpainting StableDiffusion diffusion model, wherein the mask image includes other background information in the image sample except the human body area; The Inpainting Stable Diffusion diffusion model iteratively optimizes the random noise vector of the human body area in the mask image according to the depth map samples through DDIM sampling to generate an output image after redrawing the human body image; The ControlNet model and the Inpainting Stable Diffusion diffusion model are iteratively updated through a joint loss function consisting of reconstruction loss, perception loss and local smoothness loss to obtain the trained ControlNet model and Inpainting Stable Diffusion diffusion model.

8. A human image restoration device based on a diffusion model, characterized in that: The device comprises: A three-dimensional human body model generation module is used to generate a three-dimensional human body model of a target human body according to the target human body in the original image after acquiring the original image; A depth map generating module, used for obtaining a depth map of the target human body according to the three-dimensional human body model of the target human body; The redrawing module is used to input the depth map of the target human body into the ControlNet model, and the trained ControlNet model controls the Inpainting Stable Diffusion diffusion model to redraw the image of the target human body in the original image to repair the image of the target human body in the original image.

9. A computer device, characterized in that: include: At least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the human body image restoration method based on the diffusion model according to any one of claims 1-7 is executed.

10. A non-volatile readable storage medium, characterized in that: The non-volatile readable storage medium stores a computer program, wherein the computer program, when executed by a processor, executes the human body image restoration method based on a diffusion model according to any one of claims 1 to 7.

Images