A sketch-guided image editing method based on conditional diffusion model

Abstract

The present invention discloses a sketch-guided image editing method based on a conditional diffusion model. The method comprises the following steps: obtaining an image to be inpainted and a sketch, and randomly generating a mask image; using two encoders to perform dimensionality reduction encoding on the image to be inpainted and the sketch, respectively, to obtain two low-dimensional vectors, while simultaneously sampling the mask image to obtain a mask vector; using the two low-dimensional vectors and the mask vector as guiding conditions, and generating an inpainting vector based on the inverse diffusion of the guiding conditions using a conditional diffusion model; and using a decoder to decode the inpainting vector and reconstruct it back into the image pixel space to obtain the image editing result. Because the method introduces a sketch as a guide throughout the entire process, it achieves effective editing of the target area in the image while ensuring global image consistency.

Description

Technical Field

[0001] The present invention belongs to the technical field of image editing, and in particular relates to a sketch-guided image editing method based on a conditional diffusion model. Background Art

[0002] Image editing technology refers to the modification of the style and content of an image by adding guiding information such as text, audio, and visuals, or through user interactive operations. Common editing effects include deleting redundant objects in the image, modifying the shape or color of objects, adding additional objects, moving or rotating objects, etc.

[0003] Traditional image editing techniques primarily employ graphics methods, modifying and adjusting geometric features in image space to achieve editing effects such as object deformation, movement, addition, and deletion. Depending on whether the editing operation is real-time interactive, these techniques can be divided into two categories: non-real-time interactive image editing and real-time interactive image editing. For non-real-time interactive image editing, these techniques generally involve users drawing lines on the image to deform objects, or adding masks to delete objects. For real-time interactive image editing, these techniques typically rely on devices such as a mouse or touchpad to enable user interactive editing operations. Due to the high computational complexity required for these operations, they can generally only support the editing of simple animated images. However, these traditional image editing techniques are primarily based on low-level image features such as pixels and patches, and therefore often lack overall semantic information, making them applicable only to images with relatively simple scenes.

[0004] With the rapid development of AIGC technology for generating images from text, many powerful generative models have been proposed, such as variational autoencoder adversarial networks and diffusion models. However, in actual use, most models have the following problems:

[0005] (1) Local modifications are impossible. If the user is not satisfied with some details in the generated image, the text-generated image model cannot accurately modify the generated image. If the user adds additional descriptive information to the original input text and regenerates the image, it may cause modifications to parts of the image that the user does not want to change.

[0006] (2) Poor detail generation. Although the text-to-image model can generate images with the corresponding style and content based on the text information input by the user, the generated image details are often not ideal. There are problems such as face deformation, easy distortion of shapes in complex scenes, and incomplete object details.

[0007] (3) The expressive power of text information is limited. When users want to generate images that contain content with a specific shape or structure, using text to describe it may be too complex, and it may be difficult to accurately describe certain shapes or structures. In addition, even if complex descriptive text is provided, it may not be possible to accurately generate the image the user expects.

[0008] In summary, although the AIGC technology for generating images based on text meets the basic needs of users in terms of artistic style accuracy and production efficiency, it still has problems such as the inability to make local modifications, poor generation of details, and limited expression of text information. Summary of the Invention

[0009] In view of the above, an object of the present invention is to provide a sketch-guided image editing method based on a conditional diffusion model to achieve effective editing of target areas in an image and ensure the global consistency of the image.

[0010] To achieve the above-mentioned object of the invention, an embodiment further provides a sketch-guided image editing method based on a conditional diffusion model, comprising the following steps:

[0011] Get the image and sketch to be repaired and randomly generate a mask map;

[0012] Two encoders are used to reduce the dimension of the image to be repaired and the sketch to obtain two low-dimensional vectors, and the mask image is sampled to obtain a mask vector;

[0013] The two low-dimensional vectors and the mask vector are used as guiding conditions, and the conditional diffusion model is used to generate the repair vector based on the inverse diffusion of the guiding conditions;

[0014] The decoder is used to decode the repair vector and reconstruct it back to the image pixel space to obtain the image editing result.

[0015] In one embodiment, before the two encoders and decoders are applied, adversarial learning is used to optimize the encoder and decoder parameters by constructing a VQGAN framework including an encoder, a decoder, and a discriminator.

[0016] In one embodiment, the conditional diffusion model needs to be trained before being applied, and the training process includes:

[0017] Construct training samples: Generate an initial sketch corresponding to the color image, randomly generate a mask image, generate the image to be repaired based on the mask image and the color image, and generate a sketch based on the mask image and the initial sketch;

[0018] Constructing the guiding condition: Use two encoders to perform dimensionality reduction encoding on the image to be repaired and the sketch respectively to obtain two low-dimensional vectors. At the same time, sample the mask image to obtain a mask vector. The two low-dimensional vectors and the mask vector are concatenated as the guiding condition;

[0019] Constructing a conditional diffusion model: The conditional diffusion model includes a forward diffusion process and a reverse diffusion process based on a guided condition. In the forward diffusion process, a low-dimensional vector based on the color image is used as the initial forward diffusion vector. By adding actual noise at each diffusion step, forward diffusion is achieved and the forward diffusion vector of each diffusion step is obtained. In the reverse diffusion process, a vector randomly sampled from a Gaussian distribution is used as the initial reverse diffusion vector. For each diffusion step, the cumulative noise is calculated based on the guided condition and the reverse diffusion vector of the previous diffusion step, and the reverse diffusion vector of each diffusion step is calculated based on the cumulative noise.

[0020] Construct a loss function: The mean square error between the actual noise added by the forward diffusion process and the cumulative noise calculated by the reverse diffusion process in the same diffusion step is used as the loss function;

[0021] Model training: The conditional diffusion model is trained using a loss function to optimize the model parameters.

[0022] In one embodiment, the following formula is used to implement forward diffusion by adding actual noise in each diffusion step and obtain the forward diffusion vector of each diffusion step:

[0023]

[0024] Among them, z t represents the positive diffusion vector corresponding to the diffusion step t, Represents the noise parameter, the value range is (0,1), ε image (x) represents the color image after the encoder ε image The encoded low-dimensional vector, ∈ t represents the actual noise added by diffusion step t.

[0025] In one embodiment, the inverse diffusion vector of each diffusion step is calculated based on the accumulated noise using the following formula:

[0026]

[0027] in, and They represent the inverse diffusion vectors corresponding to diffusion steps t and t-1, respectively, c image ,c sketch ,c mask Represent the low-dimensional vector corresponding to the image to be repaired, the low-dimensional vector corresponding to the sketch, and the mask vector, respectively. Indicates that based on the guidance conditions and Calculated cumulative noise, σ t represents the adjustment parameter, z belongs to the standard normal distribution N(0,I), represents the noise parameter, the value range is (0,1), βt Indicates the standard deviation of Gaussian noise, the value range is (0,1), the variance Should be equal to The value range is (0,1) and the calculation formula is:

[0028]

[0029] In one embodiment, generating an initial sketch corresponding to the color image includes: performing edge contour extraction and binarization processing on the color image to obtain the initial sketch;

[0030] The randomly generated mask image includes: using a randomly generated rectangle to simulate a rectangular selection box to generate a corresponding first initial mask image, simulating an eraser by moving a circular area to generate a corresponding second initial mask image, and adding the first initial mask image and the second initial mask image to obtain the mask image.

[0031] In one embodiment, the method of generating the image to be repaired based on the mask image and the color image includes: performing an operation x⊙(1-mask) on the mask image mask and the color image x to obtain the image to be repaired x image ;

[0032] The step of generating a sketch based on the mask image and the initial sketch includes: performing an operation s⊙mask on the mask image mask and the initial sketch s to obtain a sketch s sketch , ⊙ represents the dot product.

[0033] In one embodiment, the generating of the repair vector by using the conditional diffusion model based on the reverse diffusion of the guided condition includes:

[0034] A vector randomly sampled from a Gaussian distribution is used as the initial inverse diffusion vector. For each inverse diffusion step, the cumulative noise is calculated based on the guidance condition and the inverse diffusion vector of the previous inverse diffusion step. The inverse diffusion vector of each diffusion step is calculated based on the cumulative noise, and the inverse diffusion vector obtained in the last inverse diffusion step is used as the repair vector.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] Based on the low-dimensional vectors corresponding to the image to be repaired, the sketch, and the mask image, a guiding condition is constructed. In the low-dimensional latent space, the inverse diffusion of the conditional diffusion model is used based on the guiding condition to generate the inverse diffusion vector of the final time step as the repair vector. On this basis, the decoder is used to decode the repair vector back to the image pixel space to obtain the image editing result. Since the sketch is introduced as a guide in the overall process, the image finally output by the model satisfies the editing effect guided by the shape and structure information provided by the sketch within the given editing area, that is, effective editing of the target area in the image is achieved and the global consistency of the image is guaranteed. This method reduces the inference cost, strengthens the guiding role of the sketch condition, and improves the quality of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 is a flowchart of a sketch-guided image editing method based on a conditional diffusion model provided by an embodiment;

[0039] Figure 2 is a binary image generated by the grayscale image provided in the embodiment at different thresholds;

[0040] Figure 3 is an example of a mask diagram provided in an embodiment;

[0041] Figure 4 It is a conditional diffusion model based on the latent space of the autoencoder provided in the embodiment;

[0042] Figure 5 is a flowchart of sketch-guided image editing provided by an embodiment;

[0043] Figure 6 This is the sketch editing effect on a multi-style self-built dataset provided by the embodiment;

[0044] Figure 7 2 is a schematic diagram of the VQGAN framework structure provided in the embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0046] Research has found that sketch information can provide more low-level image guidance information than other types of guidance information, and can more accurately express the user's expectations for detailed features such as object shape and structure during image editing. Therefore, the embodiment provides a sketch-guided image editing method based on the conditional diffusion model. Figure 1 As shown, the following steps are included:

[0047] Step 1: Construct training samples for the conditional diffusion model.

[0048] In the embodiment, a color image x of size H×W×3 is given, and an initial sketch S corresponding to the color image x is generated. Specifically, the edge contour extraction model LDC and binarization processing are used for the color image x to obtain an initial sketch S of corresponding size H×W×1. The value of S is 1, which represents sketch guiding information such as lines and shapes, and the value of 0 represents the background.

[0049] Specifically, the edge contour extraction model LDC and binarization processing are used to obtain the initial sketch s of the corresponding size H×W×1, including: first, using the LDC model to extract the edges of the image x to obtain the corresponding grayscale image; then, for the grayscale image, a threshold threshold threshold = 125 is selected for binarization and inversion operations, so that the initial sketch conforms to the user input in the actual application scenario. For the grayscale image, all pixel values ​​> threshold are set to 0, and all pixel values ​​≤ threshold are set to 1. When different thresholds are used, the resulting binary images are different, such as Figure 2 shown.

[0050] In the embodiment, during the training phase, a random mask image of size H×W×1 is used as the image region identifier to be repaired. A value of 1 in the mask indicates the region to be repaired, and a value of 0 indicates the region not to be repaired. Figure 3 As shown. On this basis, the image to be repaired x is generated by operating x⊙(1-mask) based on the mask map mask and the color image x image , based on the mask map mask and the initial sketch, the sketch s of the area to be repaired is generated by operating s⊙mask sketch , the sketch s sketch Provides sketch guidance information, and ⊙ represents dot product.

[0051] Specifically, a mask image with a random size of H×W×1 is generated, including: 1) using a randomly generated rectangle to simulate a rectangular selection box to generate a corresponding first initial mask image; 2) generating a corresponding second initial mask image by moving a circular area to simulate an eraser; 3) adding the first initial mask image and the second initial mask image to obtain a final mask image.

[0052] Among them, a rectangular selection box is simulated by randomly generating a rectangle to generate a corresponding first initial mask image, including: 1-1) initializing a zero matrix mask of size H×W; 1-2) setting maxMatrixWidth and maxMatrixHeight as the maximum width and maximum height of the simulated rectangular box, respectively, and setting maxDeltaWidth and maxDeltaHeight as the maximum floating amounts of the rectangular box in width and height, respectively, to achieve random adjustment of the height and width of the rectangular box within the floating amount range; 1-3) obtaining two random numbers pointX and pointY within the range of maxMatrixWidth and maxMatrixHeight respectively through the random.uniform function; 1-4) obtaining max respectively through the random.uniform function Two random numbers deltaWidth and deltaHeight in the range of DeltaWidth / 2 and maxMatrixHeight / 2; 1-5) Then calculate the four points of the rectangular box, which are: P1=pointY+deltaHeight, P2=pointY+maxMatrixHeight-deltaHeight, P3=pointX+deltaWidth, P4=pointY+maxMatrixWidth-deltaWidth; 1-6) Assign a value of 1 to the rectangular part enclosed by the four points in the mask matrix to obtain a mask image that simulates a rectangular selection box, that is, mask[P1:P2,P3:P4]=1; 1-7) The generated mask image will be further randomly flipped left and right and up and down to achieve data enhancement.

[0053] Among them, the corresponding second initial mask image is generated by moving the circular area to simulate the eraser, including: 2-1) initializing a zero matrix mask of size H×W; 2-2) setting maxNum and minNum as the maximum and minimum times of simulating the eraser sliding, respectively, and obtaining a random number num in the range of minNum to maxNum as the number of simulated eraser sliding through the random.uniform function; 2-3) obtaining two random numbers startX and startY in the range of W and H respectively through the random.uniform function; 2-4) setting maxBrushWidth and m inBrushWidth is the maximum and minimum width of the simulated eraser when sliding; 2-5) Use the random.uniform function to obtain the random number brushWidth in the range of minBrushWidth to maxBrushWidth as the width of the simulated eraser when sliding; 2-6) Set i = 0, iterate the operations of steps 2-7) to step 2-13) until i>num; 2-7) Set minAngle and maxAngle to the minimum and maximum angles of the eraser sliding adjustment, respectively, and use the random.uniform function to obtain random numbers in the range of minAngle and maxAngle as the angle angle of the random adjustment of the simulated eraser sliding; 2-8) Determine whether i is an even number. If so, perform the operation angle = 2*π-angle; if not, go to step 2-9); 2-9) Set maxLength to the maximum length of the simulated eraser sliding, and use the random.uniform function to obtain a random number in the range of maxLength as the length length of the simulated random eraser sliding; 2-10) Draw a line from the point (startX, startY) with an angle angle , the length of the line is length, and the width is brushWidth; 2-11) respectively assign startX = startX + length * sin(angle) and startY = startY + length * cos(angle); 2-12) Draw a circle with the point (startX, startY) as the circle, and the radius of the circle is brushWidth / 2; 2-13) assign i = i + 1; 2-14) The mask generated in steps 2-7) to 2-13) is further randomly flipped left and right and up and down to achieve data augmentation. The specific parameters for generating the mask are detailed in Table 1:

[0054] Table 1

[0055]

[0056] Based on the above, the color image, the image to be repaired, the sketch and the mask image constitute the training samples.

[0057] Step 2: Construct guidance conditions based on training samples.

[0058] In the embodiment, when constructing the guiding condition, two encoders are used to perform dimensionality reduction encoding on the image to be repaired and the sketch respectively to obtain two low-dimensional vectors, and the mask image is sampled to obtain a mask vector. The two low-dimensional vectors and the mask vector are spliced ​​as the guiding condition.

[0059] Step 3: Build a conditional diffusion model.

[0060] like Figure 4 As shown, a conditional diffusion model is constructed in the latent space of the autoencoder. The conditional diffusion model includes a forward diffusion process and a reverse diffusion process based on the guidance condition.

[0061] The forward diffusion process actually adds noise to the color image at each time step. Unlike DDPM, which directly adds noise to the color image x, the color image is first encoded into the latent space to obtain a low-dimensional vector z = ε image (x), and then add noise to z, that is, use the low-dimensional vector z based on the color image as the initial positive diffusion vector, and realize forward diffusion by adding actual noise in each diffusion step and obtain the positive diffusion vector of each diffusion step, which can be expressed as:

[0062]

[0063] Among them, z t represents the positive diffusion vector corresponding to the diffusion step t, Represents the noise parameter, the value range is (0,1), ε image (x) represents the color image after the encoder ε image The encoded low-dimensional vector, ε t represents the actual noise added by diffusion step t.

[0064] The reverse diffusion process actually denoises the initial inverse vector at each time step. Unlike DDPM, which directly denoises the initial inverse vector, it also considers the input of the sketch guidance condition, that is, a vector randomly sampled from the Gaussian distribution N(0,I) is used as the initial reverse diffusion vector. For each diffusion step, the cumulative noise is first calculated based on the guidance condition and the inverse diffusion vector of the previous diffusion step, which is expressed as:

[0065]

[0066] Among them, p θ represents the probability distribution of cumulative noise, and Respectively represent the inverse diffusion vector corresponding to diffusion step t and t-1, u θ (·) and ∑ θ (·) represents the mean and variance of the normal distribution N, which is obtained from The accumulated noise is obtained by sampling Among them, c image ,c sketch ,c mask Represent the low-dimensional vector corresponding to the image to be repaired, the low-dimensional vector corresponding to the sketch, and the mask vector respectively;

[0067] Then the inverse diffusion vector of each diffusion step is calculated based on the accumulated noise, which can be expressed as:

[0068]

[0069] Among them, σ t Represents the adjustment parameter, according to the formula variance Should be equal to The value range is (0,1), z belongs to the standard normal distribution N(0,I), represents the noise parameter, the value range is (0,1), β t Indicates the standard deviation of Gaussian noise, and its value range is (0,1).

[0070] Step 4: Construct a loss function and train the conditional diffusion model based on the training samples and guidance conditions.

[0071] In this embodiment, the actual noise ∈ added by the forward diffusion process of the same diffusion step t ~N(0,I) and the cumulative noise ∈ calculated by the reverse diffusion process θ The mean square error between is used as the loss function.

[0072] In the embodiment, the training goal is to transform the image to be repaired x in the training sample into image , sketches sketch , mask map mask as the input of the model, and the color image x is used as the target image of the sketch-guided image restoration. sketch and mask, treat the repaired image x image Repair is performed so that the repair result is as consistent as possible with the color image x. Since the conditional diffusion model is established in the latent space in this embodiment, the above loss function is used to achieve the training goal of c based on the input. sketch and c mask , for c image Repair is performed so that the repair result is as consistent as possible with z, and the parameters of the conditional diffusion model are continuously optimized based on the actual training goal.

[0073] Step 5: Use the trained conditional diffusion model to perform sketch-guided image editing.

[0074] In the embodiment, Figure 5 As shown in Figure 2, when using the trained conditional diffusion model for sketch-guided image editing, the following steps are taken:

[0075] 5-1) Obtain the image and sketch to be repaired, and randomly generate a mask map;

[0076] 5-2) Use two encoders to perform dimensionality reduction encoding on the image to be repaired and the sketch respectively to obtain two low-dimensional vectors, and at the same time sample the mask image to obtain a mask vector;

[0077] 5-3) Using the two low-dimensional vectors and the mask vector as guiding conditions, the conditional diffusion model is used to generate the repair vector based on the inverse diffusion of the guiding conditions, specifically including:

[0078] A vector randomly sampled from a Gaussian distribution is used as the initial reverse diffusion vector. For each reverse diffusion step, the cumulative noise is calculated based on the guidance condition and the reverse diffusion vector of the previous reverse diffusion step, and the reverse diffusion vector of each diffusion step is calculated based on the cumulative noise. The reverse diffusion vector obtained in the last reverse diffusion step is as a repair vector;

[0079] 5-4) Use the decoder to decode the repair vector and reconstruct it back to the image pixel space to obtain the image editing result like Figure 6 shown.

[0080] In the embodiment, the encoder for effectively reducing the dimensionality of the image to be repaired and the sketch and the latent space vector decoder need to be pre-trained before being used. Figure 7 The VQGAN framework shown in the figure consists of three parts: encoder ε (Encoder), decoder D (Decoder) and discriminator Ν (Discriminator), where Conv is the convolution layer, ResBlock is the residual block, Norm is the normalization layer, Upsample is the upsampling layer, and the input image x∈R H×W×d , after the encoder ε dimension reduction, we get z∈R H×W×d , f = W / w = H / h represents the scale factor of the encoder ε downsampling the image. The discriminator N is introduced to perform adversarial training, whose training goal is to distinguish the real image x from the reconstructed image as much as possible.

[0081] The input image x is encoded by the encoder ε to obtain z. Unlike the general autoencoder that directly inputs z into the discriminator D for reconstruction, VQGAN inputs z into the codebook (vector dictionary) for vector discretization and obtains the new Then Input into D. The role of Codebook is equivalent to clustering the latent vector, which can make the reconstruction result more stable. The process of vector discretization q(z) is to obtain the latent vector closest to z by querying Codebook The formula is as follows:

[0082]

[0083] The structure of decoder D is similar to that of encoder, and is responsible for the latent vector Upsampling is performed, and the feature dimension is increased by two interpolation upsampling operations in the decoder D. In addition, during training, the real image x and the reconstructed image Input into the discriminator N for adversarial training.

[0084] Based on the above VQGAN framework, the model is trained for the image to be repaired and the binary sketch respectively. image and model sketch Two autoencoders, where model image By encoder ε image and decoder D image Composition, model sketch By encoder ε sketch and decoder D sketch The parameters are shown in Table 2, and the encoder ε image The image to be repaired is reduced from the pixel space to the latent space, and at the same time, the encoder ε sketch Reduce the sketch from pixel space to latent space, then convert the above c image ,c sketch and c mask They are input as conditions into the conditional diffusion model based on latent space, thus realizing sketch-guided image content editing.

[0085] Table 2

[0086]

[0087] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A sketch-guided image editing method based on a conditional diffusion model, characterized in that: The following steps are involved: Get the image and sketch to be repaired and randomly generate a mask map; Two encoders are used to reduce the dimension of the image to be repaired and the sketch to obtain two low-dimensional vectors, and the mask image is sampled to obtain a mask vector; The two low-dimensional vectors and the mask vector are used as guiding conditions, and the conditional diffusion model is used to generate the repair vector based on the inverse diffusion of the guiding conditions; The decoder is used to decode the repair vector and reconstruct it back to the image pixel space to obtain the image editing result; Before the two encoders and decoders are applied, a VQGAN framework including an encoder, a decoder, and a discriminator is constructed, and adversarial learning is used to optimize the encoder and decoder parameters; The conditional diffusion model needs to be trained before being applied. The training process includes: Construct training samples: Generate an initial sketch corresponding to the color image, randomly generate a mask image, generate the image to be repaired based on the mask image and the color image, and generate a sketch based on the mask image and the initial sketch; Constructing the guiding condition: Use two encoders to perform dimensionality reduction encoding on the image to be repaired and the sketch respectively to obtain two low-dimensional vectors. At the same time, sample the mask image to obtain a mask vector. The two low-dimensional vectors and the mask vector are concatenated as the guiding condition; Constructing a conditional diffusion model: The conditional diffusion model includes a forward diffusion process and a reverse diffusion process based on a guided condition. In the forward diffusion process, a low-dimensional vector based on the color image is used as the initial forward diffusion vector. By adding actual noise at each diffusion step, forward diffusion is achieved and the forward diffusion vector of each diffusion step is obtained. In the reverse diffusion process, a vector randomly sampled from a Gaussian distribution is used as the initial reverse diffusion vector. For each diffusion step, the cumulative noise is calculated based on the guided condition and the reverse diffusion vector of the previous diffusion step, and the reverse diffusion vector of each diffusion step is calculated based on the cumulative noise. Construct a loss function: The mean square error between the actual noise added by the forward diffusion process and the cumulative noise calculated by the reverse diffusion process in the same diffusion step is used as the loss function; Model training: Use the loss function to train the conditional diffusion model to optimize the model parameters; The method of generating a repair vector by using a conditional diffusion model based on inverse diffusion of a guide condition includes: A vector randomly sampled from a Gaussian distribution is used as the initial inverse diffusion vector. For each inverse diffusion step, the cumulative noise is calculated based on the guidance condition and the inverse diffusion vector of the previous inverse diffusion step. The inverse diffusion vector of each diffusion step is calculated based on the cumulative noise, and the inverse diffusion vector obtained in the last inverse diffusion step is used as the repair vector.

2. The sketch-guided image editing method based on the conditional diffusion model according to claim 1, characterized in that: The following formula is used to implement forward diffusion by adding actual noise in each diffusion step and obtain the forward diffusion vector of each diffusion step: Among them, z t represents the positive diffusion vector corresponding to the diffusion step t, represents the noise parameter, ε image (x) represents the color image x after the encoder ε image The encoded low-dimensional vector, ∈ t represents the actual noise added by diffusion step t.

3. The sketch-guided image editing method based on the conditional diffusion model according to claim 1, characterized in that: The inverse diffusion vector for each diffusion step is calculated from the accumulated noise using the following formula: in, and They represent the inverse diffusion vectors corresponding to diffusion steps t and t-1, respectively, c image ,c sketch ,c mask Represent the low-dimensional vector corresponding to the image to be repaired, the low-dimensional vector corresponding to the sketch, and the mask vector, respectively. Indicates that based on the guidance conditions and Calculated cumulative noise, σ t represents the adjustment parameter, z belongs to the standard normal distribution N(0,I), represents the noise parameter, β t Indicates the standard deviation and variance of Gaussian noise Should be equal to The calculation formula is:

4. The sketch-guided image editing method based on the conditional diffusion model according to claim 1, characterized in that: Generating the initial sketch corresponding to the color image includes: extracting the edge contour and performing binarization processing on the color image to obtain the initial sketch; The randomly generated mask image includes: using a randomly generated rectangle to simulate a rectangular selection box to generate a corresponding first initial mask image, simulating an eraser by moving a circular area to generate a corresponding second initial mask image, and adding the first initial mask image and the second initial mask image to obtain the mask image.

5. The sketch-guided image editing method based on the conditional diffusion model according to claim 1, characterized in that: The method of generating an image to be repaired based on a mask image and a color image includes: performing an operation x⊙(1-mask) on the mask image mask and the color image x to obtain the image to be repaired x image ; The step of generating a sketch based on the mask image and the initial sketch includes: performing an operation s⊙mask on the mask image mask and the initial sketch s to obtain a sketch s sketch , ⊙ represents the dot product.

Images