Image restoration method, and image restoration model training method and device

Abstract

The embodiment of the invention provides an image restoration method and device, electronic equipment and a storage medium, and belongs to the technical field of computer vision. The method comprises the following steps: acquiring a target detection frame and a redrawn mask of a to-be-restored image; cutting the to-be-restored image according to the target detection frame to obtain a local image; generating a guide vector according to the local image; inputting the guide vector and the redrawn mask into a diffusion model to obtain a target feature; and inputting the target feature into an image decoder for decoding to obtain a redrawing image. According to the embodiment of the invention, the distorted hand region in the portrait can be effectively repaired, the image quality is improved, the style consistency of the repair result is kept, and the repair effect is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image restoration method, an image restoration model training method and a device. Background Art

[0002] Image generation technology is an important research direction in the field of computer vision and has broad application prospects. However, when the generated image involves the hand, the current mainstream text-based image model often suffers from distortion in this area, resulting in a decrease in image quality, which seriously restricts the promotion of practical applications.

[0003] In related technologies, common image restoration algorithms are mostly used for quality improvement tasks such as image denoising and old photo restoration. However, the restoration effect on hand distortion (such as twisted fingers, incorrect number of fingers, etc.) in the Wensheng graph model is limited, and it is difficult to generate a restoration result with a reasonable hand structure and inconsistent with the background style.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose an image restoration method, an image restoration model training method and a device, aiming to repair the hand area in a portrait and improve the image quality and restoration effect.

[0006] To achieve the above object, an embodiment of the present application provides an image restoration method, which comprises the following steps: Get the target detection frame and redrawing mask of the image to be repaired; Cropping the image to be repaired according to the target detection frame to obtain a partial image; generating a guidance vector according to the local image; Inputting the guide vector and the redrawing mask into a diffusion model to obtain a target feature; The target feature is input into an image decoder for decoding to obtain a redrawn image.

[0007] In some embodiments, obtaining the target detection frame and the redrawing mask of the image to be repaired includes: Input the image to be repaired into the hand detection model to obtain the initial detection frame; Expanding the initial detection frame according to a preset multiple to obtain a target detection frame; An initial mask is generated according to the target detection frame, and an edge feathering operation is performed on the initial mask to obtain a redrawing mask.

[0008] In some embodiments, generating a guidance vector according to the local image includes: Performing three-dimensional reconstruction according to the partial image to obtain a three-dimensional model of a human hand, and rendering the three-dimensional model into a depth map; Inputting the local image into a human skeleton extractor to obtain a skeleton image of the hand; Inputting the local image into a pre-trained style encoder to obtain a style feature vector; Inputting the depth map into a depth control network to obtain a depth feature vector; Inputting the skeleton graph into a skeleton control network to obtain a skeleton feature vector; The depth feature vector, the skeleton feature vector and the style feature vector are input into a feature fusion network for fusion to obtain a guide vector.

[0009] In some embodiments, inputting the guide vector and the redrawing mask into a diffusion model to obtain a target feature comprises: Get text prompt words; Input the text prompt word into a text encoder to obtain a text vector; Inputting the local image into an image encoder to obtain original image features; Inputting the text vector, the guide vector, the original image feature and the redrawing mask into a diffusion model to obtain a denoising feature; The denoising feature and the original image feature are weighted according to the redrawing mask to obtain a target feature.

[0010] In some embodiments, the method further comprises: The redrawn image replaces the local image in the image to be repaired to obtain a target repaired image.

[0011] An aspect of an embodiment of the present application provides a method for training an image restoration model, wherein the image restoration model includes a diffusion model, a style encoder, a depth control network, a skeleton control network, a feature fusion network, a text encoder, an image encoder, and an image decoder. The method includes the following steps: Acquire a training data set, wherein the training data set includes a left-hand-right-hand paired data set and a normal-distorted paired data set; Training the image restoration model using the training data set; Calculating the denoising loss of the diffusion model and obtaining a loss threshold; The image restoration model is optimized by the denoising loss, and the step of training the image restoration model by the training data set is returned to until the denoising loss is less than or equal to the loss threshold, so as to obtain the trained image restoration model, which is used to perform image restoration by the above-mentioned image restoration method.

[0012] In some embodiments, the left-hand-right-hand paired data set includes a left-hand-right-hand image pair, the normal-distorted paired data set includes a normal-distorted image pair, and the training of the image restoration model using the training data set includes: Inputting a first image of the left-hand-right-hand image pair into the depth control network, the skeleton control network and the image encoder, and inputting a second image of the left-hand-right-hand image pair into the style encoder; The normal image in the normal-distorted image pair is input into the depth control network, the skeleton control network and the image encoder, and the distorted image in the left-hand-right-hand image pair is input into the style encoder.

[0013] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides an image restoration device, the device comprising: The detection frame acquisition module is used to obtain the target detection frame and redrawing mask of the image to be repaired; An image cropping module, used for cropping the image to be repaired according to the target detection frame to obtain a partial image; A vector generation module, used for generating a guide vector according to the local image; A feature processing module, used for inputting the guide vector and the redrawing mask into a diffusion model to obtain a target feature; The decoding module is used to input the target feature into an image decoder for decoding to obtain a redrawn image.

[0014] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0015] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0016] The embodiments of the present application include at least the following beneficial effects: the present application provides an image restoration method and device, an electronic device and a storage medium, the scheme obtains a target detection frame and a redrawing mask of the image to be restored; crops the image to be restored according to the target detection frame to obtain a local image; generates a guide vector according to the local image; inputs the guide vector and the redrawing mask into a diffusion model to obtain target features, maintains the style consistency of the restoration result, and improves the restoration effect; inputs the target features into an image decoder for decoding to obtain a redrawn image, effectively repairs the distorted hand area in the portrait, and improves the image quality.

[0017] The present application also provides an image restoration model training method, which obtains a training data set, and the training data set includes a left-hand-right-hand paired data set and a normal-distorted paired data set; trains the image restoration model through the training data set; calculates the denoising loss of the diffusion model and obtains a loss threshold; optimizes the image restoration model through the denoising loss, returns to the step of training the image restoration model through the training data set until the denoising loss is less than or equal to the loss threshold, and obtains a trained image restoration model. The image restoration model is used to perform image restoration through the above-mentioned image restoration method, which improves the generalization ability of the model. The image restoration model can effectively repair the distorted hand area in the portrait and improve the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of an image restoration method provided by an embodiment of the present application; Figure 2 yes Figure 1 Flow chart of step S101 in FIG. Figure 3 yes Figure 1 Flow chart of step S103 in FIG. Figure 4 yes Figure 1 Flow chart of step S104 in FIG. Figure 5 is a flow chart of the image restoration model training method provided in an embodiment of the present application; Figure 6 is a flowchart of preparing a training data set provided in an embodiment of the present application; Figure 7 is a flow chart of the image restoration method provided in the embodiment of the present application applied to the image restoration system; Figure 8 It is a redrawing flow chart provided in an embodiment of the present application; Fig. 9 is a schematic structural diagram of an image restoration device provided in an embodiment of the present application; Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0020] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0021] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0023] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0024] 1) DWPose, a human skeleton extractor, is used to detect and extract key bone points of the human body from images or videos, so as to realize the recognition and analysis of human posture.

[0025] 2) CLIP (Contrastive Language-Image Pre-training), a multimodal model developed by OpenAI that can understand and associate images with natural language descriptions.

[0026] 3) Diffusion model, a deep learning model based on probability generation, which is inspired by the molecular diffusion process in physics. The diffusion model generates high-quality data by simulating the forward diffusion process of gradually adding noise to the data, and the reverse diffusion process of reverse step-by-step denoising using the deep learning model.

[0027] 4) Generative Adversarial Network (GAN), a deep learning framework. The GAN consists of two main parts: the generator and the discriminator. Through the adversarial training process, the generator competes with each other to generate data close to the real data distribution.

[0028] In recent years, image generation technology has been an important research direction in the field of computer vision and has broad application prospects. However, when the generated image involves the hand, the current mainstream text-based image model often suffers from distortion in this area, resulting in a decrease in image quality, which seriously restricts the promotion of practical applications.

[0029] In related technologies, common image restoration algorithms are mostly used for quality improvement tasks such as image denoising and old photo restoration. However, the restoration effect on hand distortion (such as twisted fingers, incorrect number of fingers, etc.) in the Wensheng graph model is limited, and it is difficult to generate a restoration result with a reasonable hand structure and inconsistent with the background style.

[0030] In summary, the technical problems existing in the relevant technologies need to be improved.

[0031] In view of this, an image restoration method, apparatus, device and medium are provided in an embodiment of the present application. The scheme obtains a target detection frame and a redrawing mask of an image to be restored; crops the image to be restored according to the target detection frame to obtain a local image; generates a guide vector according to the local image; inputs the guide vector and the redrawing mask into a diffusion model to obtain target features, maintains the style consistency of the restoration result, and improves the restoration effect; inputs the target features into an image decoder for decoding to obtain a redrawn image, effectively repairs the distorted hand area in the portrait, and improves the image quality.

[0032] The present application also provides an image restoration model training method, which obtains a training data set, wherein the training data set includes a left-hand-right-hand paired data set and a normal-distorted paired data set; trains an image restoration model through the training data set; calculates the denoising loss of a diffusion model and obtains a loss threshold; optimizes the image restoration model through the denoising loss, and returns to the step of training the image restoration model through the training data set until the denoising loss is less than or equal to the loss threshold, thereby obtaining a trained image restoration model. The image restoration model is used to perform image restoration through the above-mentioned image restoration method, thereby improving the generalization ability of the model. The image restoration model can effectively repair the distorted hand area in the portrait and improve the image quality.

[0033] The image restoration method and image restoration model training method provided in the embodiments of the present application relate to the field of computer vision technology. The image restoration method and image restoration model training method provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the image restoration method, etc., but is not limited to the above forms.

[0034] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0035] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0036] Figure 1is an optional flow chart of the image restoration method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0037] Step S101, obtaining the target detection frame and redrawing mask of the image to be repaired. In some embodiments, the image to be repaired is input into a hand detection model to obtain an initial detection frame; the initial detection frame is expanded according to a preset multiple to obtain a target detection frame; an initial mask is generated according to the target detection frame, and the initial mask is subjected to edge feathering to obtain a redrawing mask.

[0038] The center coordinates of the detection frame remain unchanged before and after the expansion, and if the detection frame after the expansion exceeds the image boundary, the pixels in the exceeding part are filled with black. Optionally, the target detection frame is a square.

[0039] Specifically, a mask with the same size as the detection box after expansion is initialized to all zeros, and the detection box area before expansion is assigned to The mask is then feathered to blend the redrawn content and the background content harmoniously.

[0040] In this embodiment, obtaining the target detection frame and redrawing mask of the image to be repaired prepares for the subsequent cropping of the image.

[0041] Step S102: cropping the image to be repaired according to the target detection frame to obtain a partial image. In some embodiments, the image to be repaired is cropped according to the target detection frame to obtain a partial image, wherein the location information of the partial image is saved to prepare for subsequent image repair.

[0042] Step S103: generating a guide vector according to the local image.

[0043] Specifically, the guidance vector is used to guide the model to redraw a reasonable hand structure and a consistent image style.

[0044] In some embodiments, the steering vector is generated based on feature information of the local image.

[0045] Optionally, features of the local image are extracted and mapped into a vector space.

[0046] In some embodiments, a three-dimensional model of a human hand is obtained by performing three-dimensional reconstruction based on a local image, and the three-dimensional model is rendered as a depth map; the local image is input into a human skeleton extractor to obtain a skeleton map of the hand; the local image is input into a pre-trained style encoder to obtain a style feature vector; the depth map is input into a depth control network to obtain a depth feature vector; the skeleton map is input into a skeleton control network to obtain a skeleton feature vector; the depth feature vector, the skeleton feature vector and the style feature vector are input into a feature fusion network for fusion to obtain a guidance vector.

[0047] In this embodiment, a guidance vector is generated according to a local image, which provides guidance for subsequent image restoration, making the restoration result more natural and coherent, and improving the quality and effect of image restoration.

[0048] Step S104: input the guide vector and the redrawing mask into the diffusion model to obtain the target feature. In some embodiments, the guide vector and the redrawing mask are input into the diffusion model, and the diffusion model generates a target feature that matches the local image to fill the missing area. Optionally, the target feature includes depth information, skeleton information, style, skin color, texture and other information of the local image.

[0049] Optionally, obtain text prompt words; input the text prompt words into a text encoder to obtain a text vector; input the local image into an image encoder to obtain original image features; input the text vector, the guide vector, the original image features and the redrawing mask into a diffusion model to obtain denoising features; and weight the denoising features and the original image features according to the redrawing mask to obtain the target features.

[0050] In this embodiment, the guide vector and the redrawing mask are input into the diffusion model to obtain the target feature. The diffusion model can produce a restoration result that is more consistent with the image context, reduce the inconsistency between the restored part and the unrestored part, and make the restoration result more natural and coherent.

[0051] Step S105: input the target feature into an image decoder for decoding to obtain a redrawn image.

[0052] In some embodiments, the target feature is input into an image decoder for decoding to obtain a redrawn image.

[0053] Optionally, a self-attention mechanism or Transformer technology can be applied in the image decoder to improve the effect and accuracy of image restoration.

[0054] Furthermore, the redrawn image replaces the local image in the image to be repaired to obtain a target repaired image.

[0055] In this embodiment, the target features are input into an image decoder for decoding to obtain a redrawn image, which effectively repairs the distorted hand area in the portrait, improves the image quality, and maintains the style consistency of the repair result, thereby improving the repair effect.

[0056] In steps S101 to S105 shown in the embodiment of the present application, a target detection frame and a redrawing mask are obtained for the image to be repaired; a local image is obtained by cropping the image to be repaired according to the target detection frame; a guide vector is generated according to the local image; the guide vector and the redrawing mask are input into a diffusion model to obtain target features, thereby maintaining the style consistency of the repair result and improving the repair effect; the target features are input into an image decoder for decoding to obtain a redrawn image, thereby effectively repairing the distorted hand area in the portrait and improving the image quality.

[0057] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S203: Step S201: input the image to be repaired into a hand detection model to obtain an initial detection frame.

[0058] In step S201 of some embodiments, the image to be repaired is identified by a hand detection model to obtain an initial detection frame.

[0059] Step S202, expanding the initial detection frame according to a preset multiple to obtain a target detection frame.

[0060] In step S202 of some embodiments, the initial detection frame is expanded by a preset multiple to obtain a target detection frame. The center coordinates remain unchanged before and after the expansion, and the shape of the target detection frame is a square. If the expanded detection frame exceeds the image boundary, the pixels in the excess portion are filled with black.

[0061] It is understandable that the preset multiple can be predetermined or dynamically adjusted according to the application scenario and user needs.

[0062] Specifically, the calculation method of the detection box expansion is as follows:

[0063] in, and They are the coordinates of the upper left corner of the detection box before and after expansion, and They are the coordinates of the lower right corner of the detection box before and after expansion.

[0064] Step S203, generating an initial mask according to the target detection frame, performing an edge feathering operation on the initial mask to obtain a redrawing mask.

[0065] In some embodiments, an all-zero mask with the same size as the target detection box is initialized, and an edge feathering operation is performed on the initial mask to obtain a redrawing mask.

[0066] The edge is smoothed by edge feathering. Specifically, the edge of the mask is blurred to reduce the sharp transition area, so that the edge of the mask is more natural and smooth.

[0067] In step S203 of some embodiments, the expression of the redraw mask is as follows:

[0068]

[0069] in, is the segmentation mask before feathering, is the segmentation mask after feathering (i.e., redrawing mask), is the Gaussian kernel function, Represents a convolution operation.

[0070] See also Figure 3 In some embodiments, step S103 may include but is not limited to steps S301 to S306: Step S301 , performing three-dimensional reconstruction according to the local image to obtain a three-dimensional model of the human hand, and rendering the three-dimensional model into a depth map.

[0071] In step S301 of some embodiments, a 3D model of a human hand is obtained by performing 3D reconstruction on the partial image, and the model is rendered as a depth map, wherein the depth map depicts a non-distorted and reasonable human hand structure from the perspective of depth information.

[0072] Step S302, inputting the local image into a human skeleton extractor to obtain a skeleton image of the hand.

[0073] In step S302 of some embodiments, the local image is input into a human skeleton extractor to obtain a skeleton graph of the hand, wherein the skeleton graph describes a non-distorted and reasonable human hand structure from the perspective of skeleton information.

[0074] Optionally, the human skeleton extractor includes but is not limited to DWPose, OpenPose, and DeepCut.

[0075] Step S303: input the local image into a pre-trained style encoder to obtain a style feature vector.

[0076] In step S303 of some embodiments, the local image is input into a pre-trained style encoder to obtain a style feature vector, wherein the style feature vector includes information such as style, skin color, and texture.

[0077] Optionally, the style encoder includes the image encoder part of CLIP, the mapping network part of StyleGAN2, and AdaIN, etc.

[0078] Step S304: input the depth map into the depth control network to obtain a depth feature vector.

[0079] Among them, the depth feature vector is extracted from the depth map through the depth control network.

[0080] Step S305, inputting the skeleton image into the skeleton control network to obtain a skeleton feature vector.

[0081] Among them, the skeleton feature vector is extracted from the skeleton graph through the skeleton control network.

[0082] Step S306: Input the depth feature vector, the skeleton feature vector and the style feature vector into a feature fusion network for fusion to obtain a guidance vector.

[0083] In step S306 of some embodiments, the feature vectors are fused through a feature fusion network.

[0084] Among them, the deep feature vector, skeleton feature vector and style feature vector are fused through the feature fusion network to obtain the guidance vector.

[0085] See also Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S405: Step S401, obtaining text prompt words.

[0086] Specifically, the text prompt words are used to describe the image to be repaired.

[0087] In step S401 of some embodiments, a text prompt word input by a user is obtained.

[0088] Step S402: input the text prompt word into a text encoder to obtain a text vector.

[0089] In step S402 of some embodiments, a text vector is extracted from the text prompt word by a text encoder.

[0090] Step S403: input the local image into an image encoder to obtain original image features.

[0091] In step S403 of some embodiments, original image features are extracted from the local image by an image encoder.

[0092] Step S404: input the text vector, the guide vector, the original image feature and the redrawing mask into the diffusion model to obtain the denoising feature.

[0093] In step S404 of some embodiments, the text vector, the guide vector, the original image feature and the redrawing mask are input into a diffusion model, and the diffusion model adds Gaussian noise to the image feature and then performs an iterative denoising operation to obtain a denoised feature.

[0094] Step S405 , weighting the denoised features and the original image features according to the redrawing mask to obtain the target features.

[0095] In step S405 of some embodiments, at each time step t, the denoised features and the original image features are weighted according to the redrawing mask to obtain the target features, and the expression of the target features is as follows:

[0096] in, represents an element-wise multiplication operation, represents a text vector, represents the original image features, represents the guide vector, Represents the redraw mask, represents the target feature at time step t, represents the target feature at time step t-1.

[0097] Figure 5 is an optional flowchart of the image restoration model training method provided in the embodiment of the present application. Figure 5 The method may include but is not limited to steps S501 to S504.

[0098] Step S501, obtaining a training data set.

[0099] Specifically, the image restoration model includes a diffusion model, a style encoder, a deep control network, a skeleton control network, a feature fusion network, a text encoder, an image encoder and an image decoder, and the training dataset includes a left-hand-right-hand paired dataset and a normal-distorted paired dataset.

[0100] In step S501 of some embodiments, the training set is obtained by processing the original image-text pair data set, and includes two parts: a left-hand-right-hand paired data set and a normal-distorted paired data set. The purpose of the left-hand-right-hand paired data set is to encourage the model to focus on style-related information when extracting and applying style vectors, and to avoid introducing interference information that is irrelevant to the style as much as possible. Human body detection and hand detection are performed on the images in the original data set. If they contain the left and right hands of the same person, the partial images of the left and right hands are cropped out to form an image pair, otherwise the data is discarded. The purpose of the normal-distorted paired data set is to simulate the reasoning scene, encourage the model to learn to extract effective style information from the distorted input image, and achieve alignment of training and reasoning. The construction method is to redraw the hand area of ​​the normal image without control conditions. At this time, the redrawing result is often distorted, and then the hand areas of the normal image and the redrawn image are cropped out to form an image pair.

[0101] Exemplarily, the flowchart for preparing the training data set is as follows Figure 6 shown.

[0102] Step S502: training an image restoration model using a training data set.

[0103] Specifically, the left-hand-right-hand paired dataset includes left-hand-right-hand image pairs, and the normal-distorted paired dataset includes normal-distorted image pairs.

[0104] In step S502 of some embodiments, the first image in the left-hand-right-hand image pair is input into the depth control network, the skeleton control network and the image encoder, and the second image in the left-hand-right-hand image pair is input into the style encoder. It can be understood that since the left and right hands of the same person often have the same style and different hand postures, this difference helps to prompt the model to focus on style-related information when extracting and applying style features.

[0105] Furthermore, the normal image in the normal-distorted image pair is input into the depth control network, the skeleton control network and the image encoder, and the distorted image in the left-hand-right-hand image pair is input into the style encoder.

[0106] In this embodiment, the image restoration model is trained by using a training data set, which helps to achieve alignment between model training and reasoning.

[0107] Step S503: Calculate the denoising loss of the diffusion model and obtain a loss threshold.

[0108] In step S503 of some embodiments, the denoising loss of the diffusion model is calculated, and the loss function expression of the denoising loss is as follows:

[0109] in, represents the loss function of denoising loss, represents a text vector, represents the guide vector, Represents the redraw mask, represents the time step, Express expectations, represents the real noise, represents the prediction noise of the model.

[0110] Step S504, optimizing the image restoration model by denoising loss, returning to the step of training the image restoration model by using the training data set, until the denoising loss is less than or equal to the loss threshold, thereby obtaining a trained image restoration model.

[0111] In step S504 of some embodiments, the image restoration model is iteratively optimized through denoising loss.

[0112] It is understandable that the loss threshold may be predetermined or dynamically adjusted according to application scenarios or user needs.

[0113] It should be noted that in the above training process, only the depth control network, skeleton control network and feature fusion network are optimized, and the remaining model parameters are frozen.

[0114] Figure 7 is an optional flow chart of an image restoration method provided in an embodiment of the present application applied to an image restoration system, wherein the image restoration system includes a detection module, a condition extraction module and a redrawing module. Figure 7 The method may include but is not limited to the following steps.

[0115] Step 1: Get the hand detection frame and redrawing mask of the image to be repaired.

[0116] It should be noted that the front end obtains the text prompt word input by the user through the user interface, and then calls the text-based image model of the back end to generate an image, and then obtains the optimized image through the image restoration process of this embodiment, and finally returns the optimized image to the front end through the interface and provides it to the user.

[0117] In some embodiments, obtaining a hand detection frame and a redrawing mask of an image to be repaired includes the following steps: 1. Input the image to be repaired into a hand detection model (such as YOLO) to obtain a detection frame (i.e., the initial detection frame).

[0118] 2. Expand the detection frame, keep the center coordinates unchanged, and expand the length and width to the original times ( ). The purpose of this step is to provide sufficient context information for redrawing the distorted area. At the same time, considering that the mainstream diffusion model used for redrawing is often based on a large number of To better utilize the diffusion model, the detection frame is further expanded into a square, while keeping the center coordinates unchanged before and after the expansion. If the expanded detection frame (i.e., the target detection frame) exceeds the image boundary, the pixels in the excess part are filled with black. The calculation method for the above detection frame expansion is as follows:

[0119] in, and They are the coordinates of the upper left corner of the detection box before and after expansion, and They are the coordinates of the lower right corner of the detection box before and after expansion, respectively.

[0120] 3. Initialize a mask of all zeros with the same size as the detection box after expansion, and assign the detection box area before expansion inside it to The mask is then feathered to blend the redrawn content and the background harmoniously:

[0121]

[0122] in, is the segmentation mask before feathering, is the segmentation mask after feathering (i.e., redrawing mask), is the Gaussian kernel function, Represents a convolution operation.

[0123] Step 2: Extract the hand depth map and skeleton map to extract the style features of the hand image.

[0124] In some embodiments, extracting a hand depth map and a skeleton map, and extracting style features of a hand image comprises the following steps: 1. A local image is obtained by cropping according to the detection frame, and a 3D model of the human hand is obtained by performing 3D reconstruction on it, which is then rendered into a depth map. The depth map depicts a non-distorted and reasonable human hand structure from the perspective of depth information.

[0125] 2. Input the local image into a human skeleton extractor (such as DWPose) to obtain a skeleton map of the hand. Similarly, the skeleton map describes a non-distorted and reasonable human hand structure from the perspective of skeleton information.

[0126] 3. Input the local image into a pre-trained style encoder (such as the image encoder part of CLIP) to obtain its style feature vector, which contains information such as style, skin color, and texture.

[0127] Step 3: Redraw the hand area in the detection frame.

[0128] In some embodiments, redrawing the hand area in the detection frame includes the following steps: 1. Redraw the flowchart as follows Figure 8 As shown, the hand depth map and skeleton map are respectively input into the depth control network and the skeleton control network to obtain the depth feature vector and the skeleton feature vector. These two feature vectors are input into the feature fusion network together with the style feature vector to obtain the fused feature vector as the final guidance vector, which is used to guide the model to redraw a reasonable hand structure and a consistent image style.

[0129] 2. Redraw the hand area of ​​the local image determined by the detection frame, such as Figure 8 First, the text prompt word is input into the text encoder to obtain the text vector , the local image is input into the image encoder to obtain the original image features . Then, the text vector, guide vector , original image features and hand region mask Enter the diffusion model together The diffusion model adds Gaussian noise to the image features and then performs iterative denoising. , the updated image features are obtained by weighting the features processed by the diffusion model and the original features according to the mask:

[0130] in, represents an element-wise multiplication operation, represents a text vector, represents the original image features, represents the guide vector, Represents the redraw mask, represents the target feature at time step t, represents the target feature at time step t-1.

[0131] Preset After the denoising process, the image features The image is input to the image decoder for decoding to obtain the redrawn image. The redrawn image is pasted back to the image to be repaired according to the position determined by the detection frame to replace the original partial image, and the final repaired image can be obtained.

[0132] This embodiment designs a restoration method based on guided redrawing, which guides the diffusion model to redraw a reasonable hand structure by extracting two hand control conditions, namely the depth map and the skeleton map; introduces style features as guidance, and designs a feature fusion module to adaptively fuse multiple guidance information, so as to maintain the style consistency of the restoration result; This embodiment provides a method for constructing a training data set, which can effectively support the training of a repair model.

[0133] The embodiment of the present application obtains a target detection frame and a redrawing mask of an image to be repaired; crops the image to be repaired according to the target detection frame to obtain a local image; generates a guidance vector according to the local image; inputs the guidance vector and the redrawing mask into a diffusion model to obtain target features, maintains the style consistency of the repair result, and improves the repair effect; inputs the target features into an image decoder for decoding to obtain a redrawn image, effectively repairs the distorted hand area in the portrait, and improves the image quality.

[0134] See also Fig. 9 The embodiment of the present application also provides an image restoration device, which can implement the above-mentioned image restoration method, and the device includes: A detection frame acquisition module 901 is used to acquire a target detection frame and a redrawing mask of the image to be repaired; An image cropping module 902 is used to crop the image to be repaired according to the target detection frame to obtain a partial image; A vector generation module 903, used to generate a guide vector according to a local image; A feature processing module 904 is used to input the guide vector and the redrawing mask into the diffusion model to obtain the target feature; The decoding module 905 is used to input the target feature into the image decoder for decoding to obtain a redrawn image.

[0135] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0136] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image restoration method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0137] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes: The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the image restoration method of the embodiment of this application; Input / output interface 1003, used to implement information input and output; Communication interface 1004, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 ); The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0139] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned image restoration method when executed by a processor.

[0140] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0141] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The image restoration method, image restoration device, electronic device and storage medium provided by the embodiments of the present application obtain a target detection frame and a redrawing mask of an image to be restored; crop the image to be restored according to the target detection frame to obtain a local image; generate a guide vector according to the local image; input the guide vector and the redrawing mask into a diffusion model to obtain target features, maintain the style consistency of the restoration result, and improve the restoration effect; input the target features into an image decoder for decoding to obtain a redrawn image, effectively restore the distorted hand area in the portrait, and improve the image quality.

[0143] The image restoration model training method provided by the present application obtains a training data set, and the training data set includes a left-hand-right-hand paired data set and a normal-distorted paired data set; trains the image restoration model through the training data set; calculates the denoising loss of the diffusion model and obtains the loss threshold; optimizes the image restoration model through the denoising loss, returns to the step of training the image restoration model through the training data set until the denoising loss is less than or equal to the loss threshold, and obtains a trained image restoration model. The image restoration model is used to improve the generalization ability of the model by performing image restoration. The image restoration model can effectively repair the distorted hand area in the portrait and improve the image quality.

[0144] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0145] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0146] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0148] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0149] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0150] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0151] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0154] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An image restoration method, characterized in that: The method comprises the following steps: Get the target detection frame and redrawing mask of the image to be repaired; Cropping the image to be repaired according to the target detection frame to obtain a partial image; generating a guidance vector according to the local image; Inputting the guide vector and the redrawing mask into a diffusion model to obtain a target feature; The target feature is input into an image decoder for decoding to obtain a redrawn image.

2. The method according to claim 1, characterized in that The step of obtaining the target detection frame and the redrawing mask of the image to be repaired includes: Input the image to be repaired into the hand detection model to obtain the initial detection frame; Expanding the initial detection frame according to a preset multiple to obtain a target detection frame; An initial mask is generated according to the target detection frame, and an edge feathering operation is performed on the initial mask to obtain a redrawing mask.

3. The method according to claim 1, characterized in that The step of generating a guidance vector according to the local image comprises: Performing three-dimensional reconstruction according to the partial image to obtain a three-dimensional model of a human hand, and rendering the three-dimensional model into a depth map; Inputting the local image into a human skeleton extractor to obtain a skeleton image of the hand; Inputting the local image into a pre-trained style encoder to obtain a style feature vector; Inputting the depth map into a depth control network to obtain a depth feature vector; Inputting the skeleton graph into a skeleton control network to obtain a skeleton feature vector; The depth feature vector, the skeleton feature vector and the style feature vector are input into a feature fusion network for fusion to obtain a guide vector.

4. The method according to claim 1, characterized in that The step of inputting the guide vector and the redrawing mask into a diffusion model to obtain a target feature comprises: Get text prompt words; Input the text prompt word into a text encoder to obtain a text vector; Inputting the local image into an image encoder to obtain original image features; Inputting the text vector, the guide vector, the original image feature and the redrawing mask into a diffusion model to obtain a denoising feature; The denoising feature and the original image feature are weighted according to the redrawing mask to obtain a target feature.

5. The method according to claim 1, characterized in that The method further comprises: The redrawn image replaces the local image in the image to be repaired to obtain a target repaired image.

6. A method for training an image restoration model, characterized in that: The image restoration model includes a diffusion model, a style encoder, a depth control network, a skeleton control network, a feature fusion network, a text encoder, an image encoder and an image decoder. The method includes the following steps: Acquire a training data set, wherein the training data set includes a left-hand-right-hand paired data set and a normal-distorted paired data set; Training the image restoration model using the training data set; Calculating the denoising loss of the diffusion model and obtaining a loss threshold; The image restoration model is optimized by the denoising loss, and the step of training the image restoration model by the training data set is returned to until the denoising loss is less than or equal to the loss threshold, so as to obtain the trained image restoration model, which is used for image restoration by the method described in any one of claims 1 to 5.

7. The method according to claim 6, characterized in that The left-hand-right-hand paired data set includes left-hand-right-hand image pairs, the normal-distorted paired data set includes normal-distorted image pairs, and training the image restoration model by using the training data set includes: Inputting a first image of the left-hand-right-hand image pair into the depth control network, the skeleton control network and the image encoder, and inputting a second image of the left-hand-right-hand image pair into the style encoder; The normal image in the normal-distorted image pair is input into the depth control network, the skeleton control network and the image encoder, and the distorted image in the left-hand-right-hand image pair is input into the style encoder.

8. An image restoration device, characterized in that: The device comprises: The detection frame acquisition module is used to obtain the target detection frame and redrawing mask of the image to be repaired; An image cropping module, used for cropping the image to be repaired according to the target detection frame to obtain a partial image; A vector generation module, used for generating a guide vector according to the local image; A feature processing module, used for inputting the guide vector and the redrawing mask into a diffusion model to obtain a target feature; The decoding module is used to input the target feature into an image decoder for decoding to obtain a redrawn image.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images