Image repair method and device, and electronic device

Abstract

The present invention provides an image repair method, including: obtaining a paired condition-image data set and a large-scale image-text paired data set; obtaining a pre-trained latent variable diffusion generation model; constructing a conditional control network, the conditional control network being isomorphic to the pre-trained latent variable diffusion generation model; training the conditional control network based on the paired condition-image data set to obtain a trained conditional control network; fine-tuning the training to obtain a latent variable diffusion repair model based on the pre-trained latent variable diffusion generation model according to the large-scale image-text paired data set; generating an image of a region to be repaired based on the conditional control network and the latent variable diffusion repair model; constructing a smooth weight map based on a signed distance field SDF to fuse the image of the region to be repaired into the original image; and outputting a repaired image. The present invention also includes a corresponding image repair device and an electronic device. The solution of the present invention can output a high-fidelity repaired image with low computational overhead.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to an image repair method and device. Background Art

[0002] Image inpainting is a branch of the field of artificial intelligence image processing. It can be limited by natural language description to process the corresponding image, thereby achieving a certain degree of control over the content generated by the model.

[0003] In recent years, thanks to the development of computing power and the accumulation of big data, the effect of image repair through natural voice control is almost the same as that of painters. The output effect of some image processing models even exceeds the level of painters. On the other hand, the rapid development of image repair technology is also based on the breakthrough of the diffusion generation model technology.

[0004] However, there are still a lot of uncontrollable factors when only relying on natural language description to generate a complete image. In addition, in actual design and production work, it is often necessary to make multiple adjustments to the parts of the overall image in various layers, texts, special effects and their combinations before a complete image can be output. The process of adjusting image parameters is lengthy. Therefore, it is necessary to introduce more and more complete control factors in the generation process of the diffusion model to improve the image processing capabilities, which can further promote the application and implementation of the current diffusion generation model technology.

[0005] There are still some defects in the field of image repair: (1) The main input of the algorithm based on the diffusion generative model is still natural language text, which cannot achieve high-precision control of the picture layout, human posture, object contours, relative positions between objects, local redrawing, etc., limiting its practical application scenarios; (2) Images repaired based on the diffusion generative model or traditional algorithms, deep learning algorithms and other processing methods will have some transition and unnatural fusion defects, which often require some special post-processing operations to repair to a certain extent, but the problem still exists, and the image repair algorithm represented by the latent variable diffusion model will also cause distortion in the image area that does not need to be repaired.

[0006] Based on the above, this application provides a technical solution to solve the above technical problems. Summary of the invention

[0007] In view of the problems of complex training and poor fusion effect of image inpainting technology in the prior art, the present invention provides an image inpainting method, comprising the following steps:

[0008] Step S1, obtaining a pairing condition-image dataset and a large-scale image-text pairing dataset;

[0009] Step S2, obtaining a pre-trained latent variable diffusion generation model G1, where the pre-trained latent variable diffusion generation model G1 includes a one-stage autoencoder network VAE, a denoising network U-Net, and a text encoder Text Encoder;

[0010] Step S3, constructing a conditional control network C1, wherein the conditional control network C1 is isomorphic to the pre-trained latent variable diffusion generation model;

[0011] Step S4, training the conditional control network C1 based on the paired condition-image data set to obtain a trained conditional control network C;

[0012] Step S5: Based on the large-scale image-text pairing dataset, the latent variable diffusion generation model G1 is generated based on the pre-trained latent variable diffusion, and the latent variable diffusion patching model G is obtained by fine-tuning the training. inpaint ;

[0013] Step S6: patching model G based on conditional control network C and latent variable diffusion inpaint , generate the image of the area to be repaired;

[0014] Step S7, constructing a smoothing weight map based on the signed distance field SDF, and fusing the image of the area to be repaired into the original image;

[0015] Step S8: output the repaired image.

[0016] In a specific embodiment of the present invention, step S2 further includes:

[0017] Step S21, pre-training the text encoder Text Encoder of the pre-trained latent variable diffusion generation model based on a large-scale image-text pairing dataset;

[0018] Step S22: pre-training a text encoder Text Encoder using a comparative natural language-image pre-training framework according to the large-scale image-text pairing dataset;

[0019] Step S23: train the autoencoder network VAE and the image denoising network U-Net through a multi-stage training mode.

[0020] In a specific embodiment of the present invention, step S23 further includes:

[0021] The multi-stage training mode includes two training stages;

[0022] Step S231, using the image part in the data set to train the autoencoder network VAE;

[0023] Step S232: fix the autoencoder network VAE and the text encoder Text Encoder, add random noise to the original image based on a large-scale image-text pairing dataset, and train the denoising network U-Net.

[0024] In a specific embodiment of the present invention, step S4 also includes:

[0025] Step S41, initializing all parameters with zero;

[0026] Step S42, fixing the parameters of the pre-trained latent variable diffusion generation model G1, and using the input condition image in the paired condition-image data set as the input of the condition control network C1;

[0027] Step S43, outputting the intermediate states of different layers in the conditional control network C1 to the corresponding network layers of G1;

[0028] Step S44, optimizing the loss item in the conditional control network C1 corresponding to the denoising network U-Net to obtain a trained conditional control network C.

[0029] In a specific embodiment of the present invention, step S5 also includes:

[0030] Step S51: add a multi-layer network channel to the shallow input of the denoising network U-Net in the pre-trained latent variable diffusion generation model G1, and use it as the latent variable diffusion patching model G inpaint The initialization value of

[0031] Step S52, designing a variety of random mask generation strategies;

[0032] Step S53: Initialize G on a large-scale image and text dataset inpaint Perform fine-tuning training to obtain the trained latent variable diffusion patching model G inpaint .

[0033] In a specific embodiment of the present invention, step S6 also includes:

[0034] Step S61, taking the mask as the input of the denoising model and multiplying it with the conditional image point to obtain a product result;

[0035] Step S62, input the product result into the conditional control network C;

[0036] Step S63: Input the intermediate layer state of the conditional control network C into the latent variable diffusion patching model G inpaint The corresponding middle layer realizes the image generation control in the area to be repaired.

[0037] In a specific embodiment of the present invention, step S7 also includes:

[0038] Step S71: using a sine function as an activation function to compress the value range of the signed distance field SDF to [0, 1] as a smoothing weight map;

[0039] Step S72: multiply the smoothing weight map with the mask, the patched area image, and the original image to output a patched image.

[0040] In a specific embodiment of the present invention, step S7 also includes:

[0041] The signed distance field SDF is calculated by the following formula:

[0042]

[0043] The smoothed weight map is calculated as follows:

[0044]

[0045] Image fusion is calculated by the following formula:

[0046] I out =I inp *mask+I inp *(1-mask)*(1-weight)+I ori

[0047] *weight

[0048] Where d is the Euclidean distance function, and the distance from pixel x to the boundary of the two-dimensional figure contour is Ω. is the two-dimensional graphic outline, distance is the preset distance threshold to the two-dimensional graphic outline, mask is the mask image, I inp is the inpainted image, I ori is the original image, I out It is the high-fidelity image after fusion.

[0049] The present invention also provides an image repair device, comprising:

[0050] An acquisition module is used to acquire pairing conditions - image datasets, large-scale image-text pairing datasets, and pre-trained latent variable diffusion generation model G1, wherein the pre-trained latent variable diffusion generation model G1 includes a one-stage autoencoder network VAE, a denoising network U-Net, and a text encoder Text Encoder;

[0051] A conditional control network construction module, used to construct a conditional control network C1, wherein the conditional control network C1 is isomorphic to a pre-trained latent variable diffusion generation model;

[0052] The conditional control network training module trains the conditional control network C1 based on the paired condition-image data set to obtain the trained conditional control network C;

[0053] The latent variable diffusion patching model training module is used to generate the latent variable diffusion patching model G1 based on the pre-trained latent variable diffusion generation model according to the large-scale image-text pairing dataset, and fine-tune the training to obtain the latent variable diffusion patching model G inpaint ;

[0054] Inpainted image generation module, based on conditional control network C1 and latent variable diffusion inpainting model G inpaint , generate the image of the area to be repaired;

[0055] An image fusion module constructs a smooth weight map based on a signed distance field SDF, and fuses the image of the area to be repaired into the original image;

[0056] The output module is used to output the repaired image.

[0057] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned image repair method when executing the computer program.

[0058] The present invention can bring at least one of the following beneficial effects:

[0059] (1) A pre-trained latent variable diffusion generative model is used, and an isomorphic conditional generative network is constructed, which is trained on the paired condition-image dataset, so that the ability of conditional control can be achieved without losing the ability of the pre-trained generative model;

[0060] (2) The latent variable diffusion generation model is used to initialize the latent variable diffusion patching model and fine-tune it on a large-scale image and text dataset. The calculation process is simple and the training cost is reduced.

[0061] (3) A smooth weight map based on the signed distance field (SDF) is used to fuse the patched area and the original image area, making the transition between the patched area and the original image area smoother, the image patching effect more natural, and able to output a high-fidelity image that meets the requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The preferred implementation scheme will be described below in a clear and understandable manner with reference to the accompanying drawings to further illustrate the above-mentioned characteristics, technical features, advantages and their implementation methods.

[0063] Figure 1 Schematic diagram of the steps of the image repair method in an embodiment of the present invention;

[0064] Figure 2Schematic diagram of the training steps of the pre-trained latent variable diffusion generation model in an embodiment of the present invention;

[0065] Figure 3 Schematic diagram of the structure of an image repair device in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] Various aspects of the present invention are described in further detail below.

[0067] Unless otherwise defined or indicated, all professional and scientific terms used herein have the same meanings as those familiar to those skilled in the art. In addition, any methods and materials similar or equivalent to those described herein can be applied to the present invention.

[0068] The following is an explanation of the terms.

[0069] Unless otherwise clearly specified and limited, the "or" mentioned in the present invention includes the relationship of "and". The "and" is equivalent to the Boolean logic operator "AND", the "or" is equivalent to the Boolean logic operator "OR", and "AND" is a subset of "OR".

[0070] It is to be understood that although the terms "first", "second", etc. may be used herein to describe different elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. Therefore, the first element may be referred to as the second element without departing from the teachings of the present invention.

[0071] In the present invention, the terms "comprising", "including" or "comprising" indicate that various components can be applied together in the mixture or composition of the present invention. Therefore, the term "consisting mainly of..." is included in the terms "comprising", "including" or "comprising".

[0072] Unless otherwise clearly specified and limited, the terms "connected", "connected" and "connection" in the present invention should be understood in a broad sense, for example, it can be a fixed connection, or it can be connected through an intermediary medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0073] For example, if an element (or component) is referred to as being on, coupled to, or connected to another element, the element may be directly formed on, coupled to, or connected to the other element, or there may be one or more intervening elements between them. In contrast, if the expressions "directly on," "directly coupled to," and "directly connected to" are used herein, then no intervening elements are indicated. Other words used to describe the relationship between elements should be similarly interpreted, such as "between" and "directly between," "attached" and "directly attached," "adjacent" and "directly adjacent," and the like.

[0074] It should also be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the drawings. The words "inner" and "outer" used refer to directions toward or away from the geometric center of a particular component, respectively. It can be understood that these terms are used here to describe the relationship of one element, layer or region relative to another element, layer or region as shown in the drawings. In addition to the orientations described in the drawings, these terms should also include other orientations of the device.

[0075] Other aspects of the present invention will be apparent to those skilled in the art in view of the disclosure herein.

[0076] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0077] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The drawings only show the components related to the present application and are not drawn according to the number, shape and size of the components in actual implementation. The type, quantity and proportion of each component in actual implementation may be changed arbitrarily, and the component layout may also be more complicated. For example, the thickness of the components in the drawings may be exaggerated for the sake of clarity.

[0078] The following is an explanation of the terms involved in this disclosure:

[0079] Latent variable diffusion model: a diffusion model that generates in latent variable space.

[0080] Controllable image inpainting: The task of controllable generation of some local areas in the original image. The local areas are specified by masks. The controllable body allows the input of conditions other than natural language text, and the output result is consistent with the input conditions.

[0081] High fidelity: In image inpainting, the visual quality of the image in the inpainted area is high, the image outside the inpainted area is consistent with the input after processing, and the image in the inpainted area is visually integrated and coherent with the rest of the area.

[0082] Example

[0083] In view of the technical gap in the existing technology for image inpainting technology with complex training and poor fusion effect, as shown in the attached Figure 1 As shown, the present invention provides an image repair method, comprising the following steps:

[0084] Step S1: Obtain a pairing condition-image dataset and a large-scale image-text pairing dataset.

[0085] Specifically, the method for constructing a pairing condition-image dataset can be: using a pre-trained model to perform conditional extraction on the original image, using a human key point extraction network to extract the skeleton key points of the portrait as conditions, and using the original portrait as the pairing image; and / or, using a depth estimation network to extract the depth map of the natural scene image, and using the original scene image as the pairing image; and / or, using a traditional edge extraction algorithm to extract the image contour, and using the original image as the pairing image.

[0086] The present invention utilizes a pre-trained conditional extraction network to perform conditional extraction on an appropriate amount of images, has low computational complexity, and can construct a paired condition-image dataset at low cost, further reducing the computational overhead of the image inpainting method.

[0087] It should be understood that the above are only examples, including but not limited to the pairing condition-image data set construction method steps, and any form of addition or deletion should be included in the subject matter of the present invention as long as it does not limit the purpose of the present invention.

[0088] Step S2, obtaining a pre-trained latent variable diffusion generation model G1, the pre-trained latent variable diffusion generation model G1 includes: a one-stage autoencoder network VAE, a denoising network U-Net, and a text encoder Text Encoder.

[0089] In a preferred embodiment, as shown in the attached Figure 2 As shown, the pre-trained latent variable diffusion generation model G1 is trained, and the training steps include:

[0090] Step S21: pre-train the text encoder Text Encoder of the pre-trained latent variable diffusion generation model based on a large-scale image-text pairing dataset.

[0091] Step S22: pre-training a text encoder Text Encoder using a comparative natural language-image pre-training framework according to the large-scale image-text pairing dataset.

[0092] Step S23: train the autoencoder network VAE and the image denoising network U-Net through a multi-stage training mode.

[0093] In a preferred embodiment, step S23 further includes:

[0094] The multi-stage training mode includes two levels of training stages.

[0095] Step S231: Use the image part in the data set to train the autoencoder network VAE.

[0096] Step S232: fix the autoencoder network VAE and the text encoder Text Encoder, add random noise to the original image based on a large-scale image-text pairing dataset, and train the denoising network U-Net.

[0097] The multi-stage latent variable diffusion generation model training method proposed in the present invention takes into account both flexibility and operability, and has excellent training effects and stable results.

[0098] Step S3: construct a conditional control network C1, wherein the conditional control network C1 is isomorphic to the pre-trained latent variable diffusion generation model.

[0099] Step S4: training the conditional control network C1 based on the paired condition-image data set to obtain a trained conditional control network C.

[0100] In a preferred embodiment, step S4 further includes:

[0101] Step S41, initialize all parameters to zero.

[0102] Step S42, fix the parameters of the pre-trained latent variable diffusion generation model G1, and use the input condition image in the paired condition-image data set as the input of the condition control network C1.

[0103] Step S43: output the intermediate states of different layers in the conditional control network C1 to the corresponding network layers of G1.

[0104] Step S44, optimizing the loss item in the conditional control network C1 corresponding to the denoising network U-Net to obtain a trained conditional control network C.

[0105] In this step, all parameters are first initialized with zero to ensure the accuracy of the model, and then a pre-trained latent variable diffusion generative model is used to construct an isomorphic conditional generative network, which is trained on the paired condition-image dataset to achieve conditional control capabilities without losing the capabilities of the pre-trained generative model.

[0106] Step S5: Based on the large-scale image-text pairing dataset, the latent variable diffusion generation model G1 is generated based on the pre-trained latent variable diffusion, and the latent variable diffusion patching model G is obtained by fine-tuning the training. inpaint .

[0107] In a preferred embodiment, step S5 further includes:

[0108] Step S51: add a multi-layer network channel to the shallow input of the denoising model U-Net in the pre-trained latent variable diffusion generation model G1, and use it as the latent variable diffusion patching model G inpaint The initialization value of .

[0109] Step S52: design multiple random mask generation strategies.

[0110] Specifically, the random mask generation strategy includes random polygons, random ellipses, etc. It should be understood that the above shapes are only examples, including but not limited to other graphic shapes, and any form of addition or deletion should be included in the gist of the present invention as long as it does not limit the purpose of the invention.

[0111] Step S53: Initialize G on a large-scale image and text dataset inpaint Perform fine-tuning training to obtain the trained latent variable diffusion patching model G inpaint .

[0112] In this step, the latent variable diffusion generation model is used to initialize the latent variable diffusion patching model, and a variety of random mask generation strategies are designed. inpaint Fine-tuning not only makes the results more accurate, but also greatly reduces the training cost and saves computing resources.

[0113] Step S6: patching model G based on conditional control network C and latent variable diffusion inpaint , generate the image of the area to be repaired.

[0114] In a preferred embodiment, step S6 also includes:

[0115] Step S61: Use the mask as the input of the denoising model and multiply it with the conditional image point to obtain a product result.

[0116] Step S62: Input the product result into the conditional control network C.

[0117] Step S63: Input the intermediate layer state of the conditional control network C into the latent variable diffusion patching model G inpaint The corresponding middle layer realizes the image generation control in the area to be repaired.

[0118] Step S7: construct a smoothing weight map based on the signed distance field SDF, and fuse the image of the area to be repaired into the original image.

[0119] In a preferred embodiment, step S7 further includes:

[0120] Step S71: Use a sine function as an activation function to compress the value range of the signed distance field SDF to [0, 1] as a smoothing weight map.

[0121] Step S72: multiply the smoothing weight map with the mask, the patched area image, and the original image to output a patched image.

[0122] Wherein, step S7 also includes:

[0123] The signed distance field SDF is calculated by the following formula:

[0124]

[0125] The smoothed weight map is calculated as follows:

[0126]

[0127] Image fusion is calculated by the following formula:

[0128] I out =I inp *mask+I inp *(1-mask)*(1-weight)+I ori

[0129] *weight

[0130] Where d is the Euclidean distance function, and the distance from pixel x to the boundary of the two-dimensional figure contour is Ω. is the two-dimensional graphic outline, distance is the preset distance threshold to the two-dimensional graphic outline, mask is the mask image, I inp is the inpainted image, I ori is the original image, I out It is the high-fidelity image after fusion.

[0131] In this step, the repair area and the original image area are fused through a smooth weight map constructed based on the signed distance field SDF, thereby achieving a high-fidelity image repair result, so that the image of the area to be repaired and the original image are fused coherently and naturally with high image quality.

[0132] Step S8: output the repaired image.

[0133] The image inpainting method proposed in Example 1 realizes the ability to achieve conditional control without losing the ability of the pre-trained generation model. The output high-fidelity image that has been inpainted has natural fusion at the inpainted part, and the image inpainting method has low computational complexity and low computational overhead.

[0134] Embodiment 2

[0135] The present invention also provides an image repair device, comprising:

[0136] The acquisition module is used to obtain a paired condition-image dataset, a large-scale image-text paired dataset, and a pre-trained latent variable diffusion generation model G1, wherein the paired condition-image dataset includes a one-stage autoencoder network, a denoising network U-Net, and a text encoder Text Encoder.

[0137] The conditional control network construction module is used to construct a conditional control network C1, and the conditional control network C1 is isomorphic to the pre-trained latent variable diffusion generation model.

[0138] The conditional control network training module trains the conditional control network C1 based on the paired condition-image data set to obtain the trained conditional control network C.

[0139] The latent variable diffusion patching model training module is used to generate the latent variable diffusion patching model G1 based on the pre-trained latent variable diffusion generation model according to the large-scale image-text pairing dataset, and fine-tune the training to obtain the latent variable diffusion patching model G inpaint .

[0140] Inpainted image generation module, based on conditional control network C1 and latent variable diffusion inpainting model G inpaint , generate the image of the area to be repaired.

[0141] The image fusion module constructs a smooth weight map based on the signed distance field SDF, and fuses the image of the area to be repaired into the original image.

[0142] The output module is used to output the repaired image.

[0143] Embodiment 3

[0144] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, an image repair method described in Embodiment 1 is implemented.

[0145] It should be noted that a single or multiple technical features of each specific embodiment of the present invention may be combined with a single or multiple technical features of other embodiments, and these combinations all fall within the protection scope of the present invention.

[0146] In summary, the present invention has achieved the following effects:

[0147] The present invention proposes an image inpainting method, device and electronic device based on a latent variable diffusion model, which realize the ability to achieve conditional control without losing the ability of a pre-trained generative model. The output of the completed high-fidelity image has natural fusion of the inpainted parts, and the image inpainting method has low computational complexity and low computational overhead.

[0148] Based on this application, it should be understood by those skilled in the art that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0149] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0150] It should be noted that the above embodiments can be freely combined as needed. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

[0151] All documents mentioned in the present invention are cited as references in this application, just as each document is cited as reference separately. In addition, it should be understood that after reading the above content of the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the claims attached to this application.

Claims

1. An image repair method, characterized in that: The following steps are involved: Step S1, obtaining a pairing condition-image dataset and a large-scale image-text pairing dataset; wherein the pairing condition-image dataset is constructed by: using a pre-trained model to conditionally extract the original image, and the original image is used as the pairing image; Step S2, obtaining a pre-trained latent variable diffusion generation model G1, where the pre-trained latent variable diffusion generation model G1 includes a one-stage autoencoder network VAE, a denoising network U-Net, and a text encoder Text Encoder; Step S3, constructing a conditional control network C1, wherein the conditional control network C1 is isomorphic to the pre-trained latent variable diffusion generation model; Step S4, training the conditional control network C1 based on the paired condition-image data set to obtain a trained conditional control network C; Step S5: Based on the large-scale image-text pairing dataset, the latent variable diffusion generation model G1 is generated based on the pre-trained latent variable diffusion, and the latent variable diffusion patching model G is obtained by fine-tuning the training. inpaint ; Step S6: patching model G based on conditional control network C and latent variable diffusion inpaint , generate the image of the area to be repaired; Step S7, constructing a smooth weight map based on the signed distance field SDF, and fusing the image of the area to be repaired into the original image; Step S7 also includes: Step S71: using a sine function as an activation function to compress the value range of the signed distance field SDF to [0, 1] as a smoothing weight map; Step S72, multiplying the smoothing weight map with the mask, the patched area image, and the original image, and outputting a patched image; Step S8: output the repaired image.

2. The image repair method according to claim 1, characterized in that: Step S2 also includes: Step S21, pre-training the text encoder Text Encoder of the pre-trained latent variable diffusion generation model based on a large-scale image-text pairing dataset; Step S22: pre-training a text encoder Text Encoder using a comparative natural language-image pre-training framework according to the large-scale image-text pairing dataset; Step S23: train the autoencoder network VAE and the image denoising network U-Net through a multi-stage training mode.

3. The image repair method according to claim 2, characterized in that: Step S23 also includes: The multi-stage training mode includes two training stages; Step S231, using the image part in the data set to train the autoencoder network VAE; Step S232: fix the autoencoder network VAE and the text encoder Text Encoder, add random noise to the original image based on a large-scale image-text pairing dataset, and train the denoising network U-Net.

4. The image repair method according to claim 1, characterized in that: Step S4 also includes: Step S41, initializing all parameters with zero; Step S42, fixing the parameters of the pre-trained latent variable diffusion generation model G1, and using the input condition image in the paired condition-image data set as the input of the condition control network C1; Step S43, outputting the intermediate states of different layers in the conditional control network C1 to the corresponding network layers of G1; Step S44, optimizing the loss item in the conditional control network C1 corresponding to the denoising network U-Net to obtain a trained conditional control network C.

5. The image repair method according to claim 1, characterized in that: Step S5 also includes: Step S51: add a multi-layer network channel to the shallow input of the denoising network U-Net in the pre-trained latent variable diffusion generation model G1, and use it as the latent variable diffusion patching model G inpaint Initialization value of Step S52, designing a variety of random mask generation strategies; Step S53: Initialize G on a large-scale image and text dataset inpaint Perform fine-tuning training to obtain the trained latent variable diffusion patching model G inpaint .

6. The image repair method according to claim 5, characterized in that: Step S6 also includes: Step S61, taking the mask as the input of the denoising model and multiplying it with the conditional image point to obtain a product result; Step S62, input the product result into the conditional control network C; Step S63: Input the intermediate layer state of the conditional control network C into the latent variable diffusion patching model G inpaint The corresponding middle layer realizes the image generation control in the area to be repaired.

7. The image repair method according to claim 6, characterized in that: Step S7 also includes: The signed distance field SDF is calculated by the following formula: ； The smoothed weight map is calculated as follows: ； Image fusion is calculated by the following formula: ； Among them, d is the Euclidean distance function, and the distance from pixel x to the boundary of the two-dimensional figure contour is , is the two-dimensional graphic outline, distance is the preset distance threshold to the two-dimensional graphic outline, mask is the mask image, is the repaired image, For the original picture, It is the high-fidelity image after fusion.

8. An image repair device, characterized in that: include: The acquisition module is used to obtain a paired condition-image dataset, a large-scale image-text paired dataset, and a pre-trained latent variable diffusion generation model G1, wherein the paired condition-image dataset includes a one-stage autoencoder network VAE, a denoising network U-Net, and a text encoder Text Encoder; wherein the paired condition-image dataset is constructed by: using a pre-trained model to conditionally extract the original image, and the original image is used as the paired image; A conditional control network construction module, used to construct a conditional control network C1, wherein the conditional control network C1 is isomorphic to a pre-trained latent variable diffusion generation model; The conditional control network training module trains the conditional control network C1 based on the paired condition-image data set to obtain the trained conditional control network C; The latent variable diffusion patching model training module is used to generate the latent variable diffusion patching model G1 based on the pre-trained latent variable diffusion generation model according to the large-scale image-text pairing dataset, and fine-tune the training to obtain the latent variable diffusion patching model G inpaint ; Inpainted image generation module, based on conditional control network C1 and latent variable diffusion inpainting model G inpaint , generate the image of the area to be repaired; The image fusion module constructs a smooth weight map based on the signed distance field SDF, and fuses the image of the area to be repaired into the original image; uses a sine function as an activation function to compress the value range of the signed distance field SDF to [0,1] as a smooth weight map; multiplies the smooth weight map with the mask, the repair area image, and the original image, and outputs the repaired image; The output module is used to output the repaired image.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements an image repair method as claimed in any one of claims 1 to 7 when executing the computer program.

Images