Image restoration method, device, electronic device and storage medium

Abstract

The present disclosure relates to the field of image processing technology, and discloses an image restoration method, device, electronic device and storage medium, the method comprising: identifying a deformed object in an image to be restored, determining a deformed image region in the image to be restored where the deformed object exists, the image to be restored being an image generated based on a text description; based on the text description, using a diffusion model, redrawing the deformed image region to generate a redrawn image; using the redrawn image to replace the deformed image region in the image to be restored, to obtain a restored target image. The embodiments of the present disclosure can improve the quality and accuracy of image restoration.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image restoration method, device, electronic device and storage medium. Background Art

[0002] Image generation technology based on artificial intelligence has made significant progress, and its application is becoming more and more extensive. By using image generation technology, corresponding images can be generated according to the text theme. This technology has broad development prospects in the fields of digital art, poster making, etc.

[0003] The principle of AI-based image generation technology is to collect a large number of existing images, and then use a combination of deep learning, computer vision, and image processing technologies to combine these images to generate images corresponding to the text topic.

[0004] At present, images generated based on text are prone to deformed and blurred image areas, such as deformed faces, which affect the overall beauty of the image. Repairing deformed objects in images is a complex problem. In related technologies, the deformed objects in the image are reconstructed to obtain a reconstructed image, and the reconstructed image is filled into the original image. However, the style of the reconstructed image generated by this method is not fixed, and it often does not match the style of the original image, and cannot be well integrated with the original image, resulting in poor image restoration effect. Summary of the invention

[0005] The present invention proposes an image restoration technology solution.

[0006] According to one aspect of the present disclosure, there is provided an image restoration method, comprising:

[0007] Performing deformed object recognition on the image to be repaired, and determining a deformed image region where the deformed object exists in the image to be repaired, wherein the image to be repaired is an image generated based on a text description;

[0008] Based on the text description, the deformed image area is redrawn using a diffusion model to generate a redrawn image;

[0009] The redrawn image is used to replace the deformed image area in the image to be repaired to obtain a repaired target image.

[0010] In a possible implementation, the step of identifying a deformed object on the image to be repaired and determining a deformed image region where the deformed object exists in the image to be repaired includes:

[0011] Performing target detection on the image to be repaired to determine the rectangular area where the deformed object is located;

[0012] Perform pixel-level image segmentation on the rectangular area to segment out the deformed image area where the deformed object is located.

[0013] In a possible implementation, the deformed object is a human body part, and the performing target detection on the image to be repaired to determine a rectangular area where the deformed object is located includes:

[0014] Performing human body key point detection on the image to be repaired to obtain the area where the human body is located;

[0015] Based on the detected key points of the human body, determine the rectangular area where the human body parts are located;

[0016] Detect key points of human body parts in the rectangular area;

[0017] When the ratio between the detected key points of the parts exceeds the normal ratio range, the rectangular area where the key points of the parts are located is determined as the rectangular area where the deformed object is located.

[0018] In a possible implementation, redrawing the deformed image region based on the text description using a diffusion model to generate a redrawn image includes:

[0019] Dividing the deformed image area into blocks to obtain a plurality of image blocks;

[0020] Perform semantic recognition on each image block to obtain the semantics of the image block;

[0021] When the semantics of the image block is consistent with the text description, the image block is redrawn based on the text description using a diffusion model to generate a redrawn image block;

[0022] A redrawn image is obtained based on the redrawn image block.

[0023] In a possible implementation, redrawing the deformed image region based on the text description using a diffusion model to generate a redrawn image includes:

[0024] Based on the text description and the non-deformed image region in the image to be repaired, the deformed image region is redrawn using a diffusion model.

[0025] In a possible implementation, redrawing the deformed image area based on the text description and the non-deformed image area in the image to be repaired by using a diffusion model includes:

[0026] Based on the text description and the non-deformed image region in the image to be repaired, noise is iteratively added to the deformed image region and gradually removed until a redrawn image that conforms to the text description and blends naturally with the background of the non-deformed image region is generated.

[0027] In a possible implementation, redrawing the deformed image region based on the text description using a diffusion model to generate a redrawn image includes:

[0028] Performing resolution enlargement processing on the deformed image to obtain an enlarged image;

[0029] Based on the text description and the non-deformed image region in the image to be repaired, the region of the enlarged image is redrawn using a diffusion model to generate an initial redrawn image;

[0030] The initial redrawn image is subjected to resolution reduction processing to obtain a redrawn image.

[0031] In a possible implementation, the step of replacing the deformed image region in the image to be repaired with the redrawn image to obtain a repaired target image includes:

[0032] Fusing the redrawn image and the non-deformed image region to obtain an initial fused image, wherein the redrawn image and the non-deformed image region are jaggedly spliced ​​at the pixel level;

[0033] Color matching and illumination adjustment operations are performed on the initial fused image to obtain a restored target image.

[0034] According to one aspect of the present disclosure, there is provided an image restoration device, comprising:

[0035] An identification module is used to identify deformed objects in the image to be repaired and determine a deformed image region where the deformed object exists in the image to be repaired, wherein the image to be repaired is an image generated based on a text description;

[0036] A redrawing module, used to redraw the deformed image area based on the text description and using a diffusion model to generate a redrawn image;

[0037] The replacement module is used to replace the deformed image area in the image to be repaired with the redrawn image to obtain a repaired target image.

[0038] In a possible implementation, the identification module is used to:

[0039] Performing target detection on the image to be repaired to determine the rectangular area where the deformed object is located;

[0040] Perform pixel-level image segmentation on the rectangular area to segment out the deformed image area where the deformed object is located.

[0041] In a possible implementation, the deformed object is a human body part, and the recognition module is used to:

[0042] Performing human body key point detection on the image to be repaired to obtain the area where the human body is located;

[0043] Based on the detected key points of the human body, determine the rectangular area where the human body parts are located;

[0044] Detect key points of human body parts in the rectangular area;

[0045] When the ratio between the detected key points of the parts exceeds the normal ratio range, the rectangular area where the key points of the parts are located is determined as the rectangular area where the deformed object is located.

[0046] In a possible implementation, the redrawing module is used to:

[0047] Dividing the deformed image area into blocks to obtain a plurality of image blocks;

[0048] Perform semantic recognition on each image block to obtain the semantics of the image block;

[0049] When the semantics of the image block is consistent with the text description, the image block is redrawn based on the text description using a diffusion model to generate a redrawn image block;

[0050] A redrawn image is obtained based on the redrawn image block.

[0051] In a possible implementation, the redrawing module is used to:

[0052] Based on the text description and the non-deformed image region in the image to be repaired, the deformed image region is redrawn using a diffusion model.

[0053] In a possible implementation, the redrawing module is used to:

[0054] Based on the text description and the non-deformed image region in the image to be repaired, noise is iteratively added to the deformed image region and gradually removed until a redrawn image that conforms to the text description and blends naturally with the background of the non-deformed image region is generated.

[0055] In a possible implementation, the redrawing module is used to:

[0056] Performing resolution enlargement processing on the deformed image to obtain an enlarged image;

[0057] Based on the text description and the non-deformed image region in the image to be repaired, the region of the enlarged image is redrawn using a diffusion model to generate an initial redrawn image;

[0058] The initial redrawn image is subjected to resolution reduction processing to obtain a redrawn image.

[0059] In a possible implementation, the replacement module is used to:

[0060] Fusing the redrawn image and the non-deformed image region to obtain an initial fused image, wherein the redrawn image and the non-deformed image region are jaggedly spliced ​​at the pixel level;

[0061] Color matching and illumination adjustment operations are performed on the initial fused image to obtain a restored target image.

[0062] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0063] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0064] In the disclosed embodiment, by identifying the deformed object in the image to be repaired, the deformed image area where the deformed object exists in the image to be repaired is determined, and the image to be repaired is an image generated based on the text description; based on the text description and the non-deformed image area in the image to be repaired, the diffusion model is used to redraw the deformed image area to generate a redrawn image; the redrawn image is used to replace the deformed image area in the image to be repaired to obtain a repaired target image. Thus, by automatically identifying the deformed image area in the image to be repaired, and redrawing the deformed image area based on the text description, the style of the generated redrawn image can be consistent with the style of the image to be repaired, then the redrawn image can be well integrated with the image to be repaired, and the repaired target image obtained is more realistic and has a better repair effect. In addition, since the diffusion model is used for the redrawing operation, a high-quality image can be generated, thereby improving the quality and accuracy of the repair effect.

[0065] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0067] Figure 1 A flowchart of an image restoration method according to an embodiment of the present disclosure is shown.

[0068] Figure 2 A block diagram of an image restoration apparatus according to an embodiment of the present disclosure is shown.

[0069] Figure 3 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0070] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0071] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0072] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0073] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0074] In related technologies, images generated based on text descriptions often have deformed or blurred image areas, and image restoration technology still faces many challenges. Below, we take the restoration of deformed faces in images as an example to explain the challenges in image restoration in detail.

[0075] First, for missing facial features, many methods try to make up for this defect by reconstructing the facial image. However, the style of the face constructed by this method is often not fixed and cannot match the style of the background well, resulting in poor restoration effect.

[0076] Secondly, some technologies use image editing tools for manual repair, which requires professional personnel to operate and is very time-consuming. This method is obviously not applicable for large-scale face data or real-time applications. In addition, the effect of manual repair depends largely on the skills and experience of the operator, so the repair effect is unstable and the quality is difficult to guarantee.

[0077] In the disclosed embodiment, by identifying the deformed object in the image to be repaired, the deformed image area where the deformed object exists in the image to be repaired is determined, and the image to be repaired is an image generated based on the text description; based on the text description and the non-deformed image area in the image to be repaired, the diffusion model is used to redraw the deformed image area to generate a redrawn image; the redrawn image is used to replace the deformed image area in the image to be repaired to obtain a repaired target image. Thus, by automatically identifying the deformed image area in the image to be repaired, and redrawing the deformed image area based on the text description, the style of the generated redrawn image can be consistent with the style of the image to be repaired, then the redrawn image can be well integrated with the image to be repaired, and the repaired target image obtained is more realistic and has a better repair effect. In addition, since the diffusion model is used for the redrawing operation, a high-quality image can be generated, thereby improving the quality and accuracy of the repair effect.

[0078] Figure 1 A flowchart of an image restoration method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes:

[0079] In step S11, deformed objects are identified on the image to be repaired, and a deformed image region where a deformed object exists in the image to be repaired is determined, wherein the image to be repaired is an image generated based on a text description;

[0080] The image to be repaired is an image that needs to be repaired, that is, an image that needs to be repaired because of some defects or problems. For example, the mouth or eyes in a face image may be deformed. In this case, these deformed parts need to be repaired to make it look more natural and normal; in addition, the image to be repaired can also be an image with unclear parts, for example, the eyes in the face are partially blurred. In this case, the unclear area needs to be repaired to make it look clearer and sharper.

[0081] The image to be repaired may be an image generated based on a text description, for example, an image automatically generated based on a text description using artificial intelligence technology. These images may have unsatisfactory generation results or defects and need to be repaired. The text description is a description of the image to be generated, usually including information such as objects, colors, shapes, sizes, and positions. For example, the text description may be "generate a portrait of a girl holding flowers".

[0082] A deformed object can be an abnormal or unclear object in the image to be repaired. For example, in a face image, if the position or size of one eye is abnormal, then this eye is a deformed object. Alternatively, if one eye is not clear enough, then this eye can also be considered a deformed object.

[0083] Then, the deformed image region is the region occupied by the deformed object in the image to be repaired. For example, if a nose is deformed in the image, then the region where the nose is located is the deformed image region.

[0084] Deformed objects are identified on the image to be repaired to determine the deformed objects in the image to be repaired. This process can be achieved by target detection technology. After detecting the target deformed object, the area occupied by the deformed object is determined, which can be achieved by performing pixel-level image segmentation on the detected deformed object. Specifically, each pixel point can be analyzed to determine whether it belongs to a deformed object, thereby determining the area occupied by the entire deformed object.

[0085] In step S12, based on the text description, the deformed image area is redrawn using a diffusion model to generate a redrawn image;

[0086] In order to make the redrawn image consistent with the style of the image to be repaired and the transition natural, when redrawing the deformed image area, it can be drawn based on the text description. Since the image to be repaired is obtained based on the text description, the deformed image area is redrawn based on the text description to keep the content and style of the redrawn image consistent with the image to be repaired.

[0087] When redrawing the deformed image area, a diffusion model can be used for drawing. The diffusion model is a deep learning model used to convert random noise into a structured output, such as converting random noise into an image. In the disclosed embodiment, the diffusion model is used to redraw the deformed image area according to the text description to obtain a redrawn image.

[0088] In step S13, the deformed image area in the image to be repaired is replaced with the redrawn image to obtain a repaired target image.

[0089] After obtaining the redrawn image, the deformed image area is replaced by the redrawn image to obtain the repaired image. This step can be achieved by performing pixel-level operations on the original image to be repaired. The pixel values ​​of the replaced deformed image area are replaced with the pixel values ​​of the corresponding area in the redrawn image, thereby obtaining the final repaired target image.

[0090] In addition, when replacing, some image processing operations can be performed on the redrawn image to merge the redrawn image with the background so that the joint between the redrawn image and the image to be repaired has a natural transition, thereby improving the effect of repairing the image. For details, please refer to the possible implementation methods provided in the present disclosure, which will not be elaborated here.

[0091] In the disclosed embodiment, by identifying the deformed object in the image to be repaired, the deformed image area where the deformed object exists in the image to be repaired is determined, and the image to be repaired is an image generated based on the text description; based on the text description and the non-deformed image area in the image to be repaired, the diffusion model is used to redraw the deformed image area to generate a redrawn image; the redrawn image is used to replace the deformed image area to obtain a repaired target image. Thus, by automatically identifying the deformed image area in the image to be repaired, and redrawing the deformed image area based on the text description and the non-deformed image area, the style of the generated redrawn image can be consistent with the style of the image to be repaired, then the redrawn image can be well integrated with the image to be repaired, and the repaired target image obtained is more realistic and has a better repair effect. In addition, since the diffusion model is used for the redrawing operation, a high-quality image can be generated, thereby improving the quality and accuracy of the repair effect.

[0092] In a possible implementation, the deformed object identification is performed on the image to be repaired to determine the deformed image area where the deformed object exists in the image to be repaired, including: performing target detection on the image to be repaired to determine the rectangular area where the deformed object is located; performing pixel-level image segmentation on the rectangular area to segment out the deformed image area where the deformed object is located.

[0093] When determining the deformed image area, target detection can be performed on the image to be repaired to determine the rectangular area where the deformed object is located. Specifically, each object in the image to be repaired can be identified and it can be determined whether they are deformed. When identifying the object in the image to be repaired, it can be implemented based on target detection technology, and specifically, a deep learning model for target detection can be used to identify and locate the object in the image.

[0094] When performing target detection on the image to be repaired, the entire image to be repaired is detected to detect the approximate position of the target object. The position of the detected target object is often represented by a rectangular frame. Through target detection technology, each object in the image can be quickly and accurately identified and its position can be located.

[0095] After determining the rectangular area where the target object is located, the rectangular area can be segmented at the pixel level to segment the deformed image area where the deformed object is located. When performing image segmentation, each pixel point in the rectangular area can be assigned to one or more categories (for example, a deformed object category, a non-deformed object category), thereby dividing the image into different areas. By analyzing and classifying the pixels in the rectangular area, the area where the deformed object is located can be accurately segmented.

[0096] In the disclosed embodiment, target detection is performed on the image to be repaired to determine the rectangular area where the deformed object is located; then pixel-level image segmentation is performed on the rectangular area to segment the deformed image area where the deformed object is located. Since the approximate position of the deformed object is roughly determined by target detection first, and then pixel-level image segmentation is performed on the rectangular area to accurately determine the deformed area where the deformed object is located, the recognition speed can be improved while still ensuring the recognition accuracy.

[0097] In a possible implementation, the deformed object is a human body part, and the target detection is performed on the image to be repaired to determine the rectangular area where the deformed object is located, including: performing human body key point detection on the image to be repaired to obtain the area where the human body is located; based on the detected human body key points, determining the rectangular area where the human body part is located; performing part key point detection on the rectangular area where the human body part is located; when the ratio between the detected part key points exceeds the normal ratio range, determining the rectangular area where the part key points are located as the rectangular area where the deformed object is located.

[0098] In this implementation, the deformed object is a part of the human body, such as a deformed face, hand, etc. Then, when determining the rectangular area where the deformed object is located, the human body key point detection can be performed on the image to be repaired to obtain the area where the human body is located.

[0099] Human key point detection technology identifies the human body and determines the position and posture of the human body by identifying the key points of the human body's joints and bones (such as elbows, knees, spine, etc.). Based on the distribution and relative positions of the detected key points, the rectangular area where the human body is located can be determined, which can be an area that can cover the entire human body.

[0100] Through human key point detection, the human body in the image can be quickly and accurately identified, and the relative relationship and size ratio of each part of the human body to the human body are fixed. Therefore, the rectangular area where the human body part is located can be determined based on the detected human key points. Specifically, the position and size of each part can be determined based on the detected human key points and the relative relationship and size ratio of each part of the human body to the human body, and the rectangular area of ​​the human body part can be obtained.

[0101] After determining the rectangular area of ​​each human body part, the key point detection of the part can be performed on the rectangular area where the human body part is located, that is, a more detailed key point detection can be performed in the rectangular area where each part is located to identify the specific deformed part. Different key point detection methods can be used according to different parts. For example, in the rectangular area of ​​the head, the position and posture of key points such as eyebrows, eyes, and nose can be detected; in the rectangular area of ​​the arms or legs, the position and posture of the joints can be detected.

[0102] Through the detection results of these key points, it is possible to further determine whether each part has a deformity problem. Specifically, it is possible to determine whether each part has a deformity problem based on the normal proportion range of the key points. For example, if the position and posture of key points such as eyes and nose in the rectangular area of ​​the face do not conform to the normal proportion range, it can be determined that the head has a deformity problem. Accordingly, the rectangular area of ​​the head is determined as the rectangular area where the deformed object is located.

[0103] In the disclosed embodiment, the region where the human body is located is obtained by performing human body key point detection on the image to be repaired; based on the detected human body key points, the rectangular region where the human body parts are located is determined; the rectangular region where the human body parts are located is detected by part key point detection; when the ratio between the detected part key points exceeds the normal ratio range, the rectangular region where the part key points are located is determined as the rectangular region where the deformed object is located. Since the visual morphology of the deformed part is different from that of the normal part, and there are many types of deformities, it is impossible to directly determine the deformed object by target detection. Therefore, by first roughly determining the region of the human body, and then determining the rectangular region where each human body part is located according to the relationship between the human body parts, the region where the human body parts are located can be quickly detected. Furthermore, by judging whether the ratio of the part key points exceeds the normal ratio range, the deformed object can be quickly and accurately identified.

[0104] In a possible implementation, redrawing the deformed image region based on the text description using a diffusion model to generate a redrawn image includes:

[0105] The deformed image area is divided into blocks to obtain a plurality of image blocks; semantic recognition is performed on each image block to obtain the semantics of the image block; when the semantics of the image block is consistent with the text description, the image block is redrawn based on the text description and the non-deformed image area in the image to be repaired by using a diffusion model to generate a redrawn image block; and a redrawn image is obtained based on the redrawn image block.

[0106] By dividing the deformed image area into blocks, the deformed image area is divided into multiple smaller image blocks. For example, assume that there is a deformity problem in the face part of the image to be repaired. Then, semantic recognition is performed on each image block to obtain the semantics of the image block. Semantic recognition can be achieved through deep learning technology, such as convolutional neural network and recurrent neural network. Deep learning technology extracts the features of image blocks, analyzes and classifies them, and finally determines their semantic content. For example, for the head image block, the semantics of the eyes, nose, mouth and other parts can be identified; for the arm image block, the semantics of the elbow, wrist and other parts can be identified.

[0107] In addition, the deformed image area may also contain background parts, so the semantics of the image block may also be background, which is the part of the text description that does not match the subject. For example, the text description may be "generate a picture of a girl holding flowers", then, if the image block does not contain a girl and flowers, the image block can be identified as background.

[0108] When the semantics of the image block does not match the text description, there is no need to redraw the image block. For example, when it is desired to redraw a face, there is no need to redraw the background of the face, so that the deformed object can be redrawn without changing the overall style information of the original image.

[0109] When the semantics of the image block is consistent with the text description, the image block is redrawn using a diffusion model according to the text description and a non-deformed image region in the image to be repaired to generate a redrawn image block.

[0110] After obtaining the redrawn image blocks, all the redrawn image blocks can be combined with the non-redrawn image blocks to form a complete redrawn image. In the combination process, the redrawn image blocks can be appropriately adjusted and spliced ​​to ensure that they are consistent with the original image in style and structure.

[0111] In the disclosed embodiment, by dividing the deformed image area into blocks, the complex image structure can be better processed, and the efficiency and accuracy of subsequent processing can be improved. In addition, by dividing the blocks into blocks, it is not necessary to process the image blocks that do not match the text description, so that the deformed object can be redrawn without changing the overall style information of the original image.

[0112] In a possible implementation, the redrawing of the deformed image area based on the text description using a diffusion model to generate a redrawn image includes: redrawing the deformed image area based on the text description and a non-deformed image area in the image to be repaired using a diffusion model.

[0113] The non-deformed image region may be a region other than the deformed object in the image to be repaired. For example, if the mouth in a face image is deformed, then all regions other than the mouth may be the non-deformed image region.

[0114] The non-deformed image area provides important information about the overall style, color, texture, etc. of the image. In order to make the redrawn image consistent with the style of the image to be repaired and the transition natural, when redrawing the deformed image area, it can be drawn based on the non-deformed area.

[0115] When redrawing the deformed image area, a diffusion model may be used for drawing. In this implementation, the diffusion model is used to redraw the deformed image area according to the text description and the non-deformed image area in the image to be repaired to obtain a redrawn image. Specifically, the text description and the non-deformed image area in the image to be repaired may be input into the diffusion model to obtain a redrawn image.

[0116] In the disclosed embodiment, based on the text description and the non-deformed image area in the image to be repaired, the deformed image area is redrawn using a diffusion model, so that the redrawn image can be consistent with the style of the image to be repaired and the transition is natural, and the quality of the repaired image is better.

[0117] In a possible implementation, the redrawing of the deformed image area based on the text description and the non-deformed image area in the image to be repaired using a diffusion model includes: based on the text description and the non-deformed image area in the image to be repaired, iteratively adding noise to the deformed image area and gradually removing the noise until a redrawn image that conforms to the text description and naturally blends with the background of the non-deformed image area is generated.

[0118] In this implementation, the non-deformed area in the image to be repaired can be used as a reference to ensure that the newly generated image is consistent with the style and structure of the original image. The image is redrawn by iteratively adding noise in the deformed image area and gradually removing the noise. In an example, assuming that the face in the image to be repaired has a deformity problem, you can start with a simple geometric shape or random noise based on the text description, gradually add details and structure, and remove noise until the generated image block matches the text description and blends naturally with the background of the non-deformed area.

[0119] In the process of iteratively adding and removing noise, the newly generated image is continuously adjusted and optimized until it conforms to the text description and blends naturally with the background of the non-deformed area. In this process, noise is added and removed by comparing the newly generated image with the text description and the information of the non-deformed area.

[0120] In the embodiment of the present disclosure, by iteratively adding noise to the deformed image area and gradually removing the noise based on the text description and the non-deformed image area in the image to be repaired, until a redrawn image that conforms to the text description and naturally blends with the background of the non-deformed image area is generated, a redrawn image that conforms to the text description and naturally blends with the background of the non-deformed area can be generated, thereby improving the effect of the generated redrawn image.

[0121] In a possible implementation, based on the text description, the deformed image area is redrawn using a diffusion model to generate a redrawn image, including: performing resolution enlargement processing on the deformed image to obtain an enlarged image; based on the text description and a non-deformed image area in the image to be repaired, the diffusion model is used to redraw an area of ​​the enlarged image to generate an initial redrawn image; and the initial redrawn image is reduced in resolution to obtain a redrawn image.

[0122] Before redrawing the deformed image area, the deformed image may be subjected to resolution enlargement processing to obtain an enlarged image. Resolution enlargement may be achieved through an interpolation algorithm, such as bilinear interpolation, bicubic interpolation, etc., or through a deep learning method, such as a super-resolution network (SRCNN).

[0123] After the resolution of the deformed image is enlarged, the diffusion model can be used to redraw the enlarged deformed image area to generate an initial redrawn image. The specific operation can be to add and gradually remove noise from the enlarged image according to the text description and the information of the non-deformed area to generate new image content.

[0124] After the initial redrawn image is generated, the resolution of the generated initial redrawn image can be reduced back to the original size to facilitate subsequent fusion with the non-deformed image area. Resolution reduction can also be achieved through an interpolation algorithm or a deep learning method, which will not be described here.

[0125] In the disclosed embodiment, by performing resolution enlargement processing on the deformed image, the details and clarity of the image can be improved, so that the redrawn image has better details. Then, by performing resolution reduction processing on the initial redrawn image, the final redrawn image can be obtained. The image not only retains the style and structure of the original image, but also repairs the deformed part, thereby improving the accuracy and effect of image restoration.

[0126] In a possible implementation, the method of replacing the deformed image area in the image to be repaired with the redrawn image to obtain a repaired target image includes: fusing the redrawn image and the non-deformed image area to obtain an initial fused image, wherein there is a pixel-level jagged splicing between the redrawn image and the non-deformed image area; and performing color matching and lighting adjustment operations on the initial fused image to obtain a repaired target image.

[0127] When fusing the redrawn image and the non-deformed image area, the splicing method between the redrawn image and the non-deformed image area may be a toothed splicing, for example, a sawtooth splicing, or a concave-convex splicing. It should be noted that the toothed splicing is pixel-level, that is, the number of pixels between two adjacent teeth in the redrawn image and the non-deformed image area is in single digits, and a smooth splicing effect can be achieved by creating a pixel-level transition area between the redrawn image and the non-deformed area.

[0128] For example, assuming that the redrawn image is an image with clearer facial features, and the non-deformed area is the background part, the redrawn image and the non-deformed area can be fused by pixel-level jagged stitching to obtain an initial fused image. Then, a gradient transition area can be created between the face and the background to make the splicing look more natural and continuous.

[0129] In addition, color matching and lighting adjustment operations can be performed on the initial fused image to obtain the restored target image.

[0130] Color matching can achieve color uniformity by adjusting the color value of pixels; lighting adjustment can simulate real lighting effects by changing the brightness and contrast of the image. Example: Assuming that the initial fused image is different from the original image in terms of color and lighting, the restored target image can be further optimized through color matching and lighting adjustment techniques. For example, the color in the initial fused image can be adjusted by comparing the color distribution in the original image; at the same time, the real lighting effects can be simulated by increasing or decreasing the brightness and contrast, making the restored target image more natural and realistic.

[0131] In the disclosed embodiment, the redrawn image can be smoothly fused with the non-deformed area through the jagged splicing method at the pixel level, avoiding obvious splicing marks and improving the visual effect of the restoration. By performing color matching and lighting adjustment operations on the initial fused image, the restored target image can be further optimized to make it consistent with the original image in terms of color and lighting. This method can improve the accuracy and authenticity of the restoration and make the restored target image more natural and realistic.

[0132] The various possible implementations provided by the present disclosure can be combined to form new embodiments. For example, for an identified deformed image, a resolution enlargement process can be performed first to obtain an enlarged image; the enlarged image can then be divided into blocks to obtain multiple image blocks. When the semantics of the image block is consistent with the text description, the image block is redrawn based on the text description and the non-deformed image area using a diffusion model to generate a redrawn image block, and a redrawn image is obtained based on the redrawn image block. Alternatively, multiple implementations of generating a redrawn image can be executed in parallel to obtain a drawn image, and the present disclosure does not limit the execution order of each implementation.

[0133] In a possible implementation, the image restoration method can be executed by electronic devices such as terminal devices and servers. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be executed by a processor calling computer-readable instructions stored in a memory.

[0134] In addition, the present disclosure also provides an image restoration device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any image restoration method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0135] Figure 2 A block diagram of an image restoration device according to an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the device 30 comprises:

[0136] An identification module 31 is used to identify a deformed object in the image to be repaired, and determine a deformed image region where a deformed object exists in the image to be repaired, wherein the image to be repaired is an image generated based on a text description;

[0137] A redrawing module 32, configured to redraw the deformed image region based on the text description and using a diffusion model to generate a redrawn image;

[0138] The replacement module 33 is used to replace the deformed image area in the image to be repaired with the redrawn image to obtain a repaired target image.

[0139] In a possible implementation, the identification module 31 is used to:

[0140] Performing target detection on the image to be repaired to determine the rectangular area where the deformed object is located;

[0141] Perform pixel-level image segmentation on the rectangular area to segment out the deformed image area where the deformed object is located.

[0142] In a possible implementation, the deformed object is a human body part, and the recognition module 31 is used to:

[0143] Performing human body key point detection on the image to be repaired to obtain the area where the human body is located;

[0144] Based on the detected key points of the human body, determine the rectangular area where the human body parts are located;

[0145] Detect key points of human body parts in the rectangular area;

[0146] When the ratio between the detected key points of the parts exceeds the normal ratio range, the rectangular area where the key points of the parts are located is determined as the rectangular area where the deformed object is located.

[0147] In a possible implementation, the redrawing module 32 is used to:

[0148] Dividing the deformed image area into blocks to obtain a plurality of image blocks;

[0149] Perform semantic recognition on each image block to obtain the semantics of the image block;

[0150] When the semantics of the image block is consistent with the text description, the image block is redrawn based on the text description using a diffusion model to generate a redrawn image block;

[0151] A redrawn image is obtained based on the redrawn image block.

[0152] In a possible implementation, the redrawing module 32 is used to:

[0153] Based on the text description and the non-deformed image region in the image to be repaired, the deformed image region is redrawn using a diffusion model.

[0154] In a possible implementation, the redrawing module 32 is used to:

[0155] Based on the text description and the non-deformed image region in the image to be repaired, noise is iteratively added to the deformed image region and gradually removed until a redrawn image that conforms to the text description and blends naturally with the background of the non-deformed image region is generated.

[0156] In a possible implementation, the redrawing module 32 is used to:

[0157] Performing resolution enlargement processing on the deformed image to obtain an enlarged image;

[0158] Based on the text description and the non-deformed image region in the image to be repaired, the region of the enlarged image is redrawn using a diffusion model to generate an initial redrawn image;

[0159] The initial redrawn image is subjected to resolution reduction processing to obtain a redrawn image.

[0160] In a possible implementation, the replacement module 33 is used to:

[0161] Fusing the redrawn image and the non-deformed image region to obtain an initial fused image, wherein the redrawn image and the non-deformed image region are jaggedly spliced ​​at the pixel level;

[0162] Color matching and illumination adjustment operations are performed on the initial fused image to obtain a restored target image.

[0163] This method has a specific technical connection with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware computing efficiency or execution effect (including reducing the amount of data storage, reducing the amount of data transmission, increasing the hardware processing speed, etc.), thereby obtaining the technical effect of improving the internal performance of the computer system in accordance with the laws of nature.

[0164] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0165] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0166] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0167] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0168] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0169] Figure 3 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 3 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions that can be executed by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0170] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), Apple's graphical user interface operating system (Mac OS X TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0171] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0172] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0173] Computer-readable storage media can be a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive lists) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer-readable storage media used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0174] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0175] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of the computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0176] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0177] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0178] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0179] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0180] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0181] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0182] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0183] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0184] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image restoration method, characterized in that: include: Performing deformed object recognition on the image to be repaired, and determining a deformed image region where the deformed object exists in the image to be repaired, wherein the image to be repaired is an image generated based on a text description; Based on the text description, the deformed image region is redrawn using a diffusion model to generate a redrawn image, wherein the style of the redrawn image is consistent with the style of the image to be repaired; Using the redrawn image to replace the deformed image area in the image to be repaired, to obtain a repaired target image; Wherein, the deformed object is a human body part, and the deformed object identification is performed on the image to be repaired to determine the deformed image area where the deformed object exists in the image to be repaired, including: Performing human body key point detection on the image to be repaired to obtain the area where the human body is located; Based on the detected key points of the human body, as well as the relative relationship and size ratio of each part of the human body to the human body, determine the rectangular area where the human body part is located; Detect key points of human body parts in the rectangular area; When the ratio between the detected key points of the parts exceeds the normal ratio range, the rectangular area where the key points of the parts are located is determined as the rectangular area where the deformed object is located.

2. The method according to claim 1, characterized in that The method of identifying the deformed object in the image to be repaired and determining the deformed image area where the deformed object exists in the image to be repaired includes: Perform pixel-level image segmentation on the rectangular area where the deformed object is located to segment out the deformed image area where the deformed object is located.

3. The method according to claim 1, characterized in that The step of redrawing the deformed image region based on the text description and using a diffusion model to generate a redrawn image includes: Dividing the deformed image area into blocks to obtain a plurality of image blocks; Perform semantic recognition on each image block to obtain the semantics of the image block; When the semantics of the image block is consistent with the text description, the image block is redrawn based on the text description using a diffusion model to generate a redrawn image block; A redrawn image is obtained based on the redrawn image block.

4. The method according to claim 1, characterized in that The step of redrawing the deformed image region based on the text description and using a diffusion model to generate a redrawn image includes: Based on the text description and the non-deformed image region in the image to be repaired, the deformed image region is redrawn using a diffusion model.

5. The method according to claim 4, characterized in that The step of redrawing the deformed image region based on the text description and the non-deformed image region in the image to be repaired by using a diffusion model includes: Based on the text description and the non-deformed image region in the image to be repaired, noise is iteratively added to the deformed image region and gradually removed until a redrawn image that conforms to the text description and blends naturally with the background of the non-deformed image region is generated.

6. The method according to claim 1, characterized in that The step of redrawing the deformed image region based on the text description and using a diffusion model to generate a redrawn image includes: Performing resolution enlargement processing on the deformed image to obtain an enlarged image; Based on the text description and the non-deformed image region in the image to be repaired, the region of the enlarged image is redrawn using a diffusion model to generate an initial redrawn image; The initial redrawn image is subjected to resolution reduction processing to obtain a redrawn image.

7. The method according to claim 1, characterized in that The step of replacing the deformed image region in the image to be repaired with the redrawn image to obtain a repaired target image includes: Fusing the redrawn image and the non-deformed image region to obtain an initial fused image, wherein the redrawn image and the non-deformed image region are jaggedly spliced ​​at the pixel level; Color matching and illumination adjustment operations are performed on the initial fused image to obtain a restored target image.

8. An image restoration device, characterized in that: include: An identification module is used to identify deformed objects in the image to be repaired and determine a deformed image region where the deformed object exists in the image to be repaired, wherein the image to be repaired is an image generated based on a text description; A redrawing module, configured to redraw the deformed image region based on the text description and using a diffusion model to generate a redrawn image, wherein the style of the redrawn image is consistent with the style of the image to be repaired; A replacement module, used to replace the deformed image area in the image to be repaired with the redrawn image, so as to obtain a repaired target image; Wherein, the deformed object is a human body part, and the identification module is used to: Performing human body key point detection on the image to be repaired to obtain the area where the human body is located; Based on the detected key points of the human body, as well as the relative relationship and size ratio of each part of the human body to the human body, determine the rectangular area where the human body part is located; Detect key points of human body parts in the rectangular area; When the ratio between the detected key points of the parts exceeds the normal ratio range, the rectangular area where the key points of the parts are located is determined as the rectangular area where the deformed object is located.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images