Character image editing method and device for noise space inversion

Abstract

The invention discloses a figure image editing method and device for noise space inversion. According to the method, the diffusion model is used for performing inversion on the character image, so that accurate reconstruction of the character image is realized, and the problem of artifacts of the character in the editing process is solved; facial contour features of the figure image are extracted, convolution processing is carried out on the contour features by using a control network, and then the contour features are injected into a diffusion model, so that the consistency of figures in an editing process is ensured, and the problem of inconsistency of front and back figures caused by editing is solved; the character image is edited by using the text prompt, the feature image is edited by the text prompt through the cross attention layer in the diffusion model, and the problem that the editing mode of the user is single and difficult is solved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision information technology, and in particular relates to a method and device for editing character images using noise space inversion. Background Art

[0002] With the development of deep learning and computer vision, the editing of real-life portraits has attracted widespread attention. Real-life portrait editing aims to preserve the identity of the original image while also modifying the original image. This includes, but is not limited to, modifying certain features of the image, such as hair color, eye color, and facial expressions, adding accessories, and transforming from oil painting styles or ancient portraits to realistic styles.

[0003] In recent years, real-person image editing has been primarily categorized into two types based on the model framework: adversarial generative models and diffusion models. Adversarial generative model-based real-person image editing encompasses a range of techniques that manipulate the latent variable space to alter certain features of a person's image. For example, these techniques can extract specific parts of a reference person's image (eyes, nose, mouth) or modify a person's image attributes (age, expression). In contrast, diffusion model-based techniques focus more on the editing effects of textual cues on person images, significantly enhancing editability.

[0004] There are many different branches of methods based on adversarial generative models. The first type of methods uses pre-trained adversarial generative models to implement editing operations, explore the editing capabilities of the latent variable space, and apply and improve adaptive instance normalization, because the latent variable space of the adversarial generative model has good editability properties. They first extract a latent code for a given face and then operate on it to obtain the editing result. A major drawback of these methods is that they cannot obtain a perfect latent code for a given image, resulting in the edited image losing the rich details of the original face. The second type of methods uses image-to-image translation technology for face editing, where the original face image is also input into the network. However, due to the constraint of cycle consistency, the quality of the synthesis results of the second type of methods is lower than that of the first type of methods.

[0005] There are many different branches of diffusion model-based technology. The first type of method usually edits facial images by modifying text prompts or injecting additional information as conditional information into the cross-attention layer. The conditional information can be attribute labels, expression encoding or style descriptions, etc. The key to this type of method is to design an effective conditional encoding scheme and a mechanism for integrating conditional information. The second type of method adopts the mechanism of inversion and attention map replacement. First, the source image is inverted into the noise space to ensure that the inversion result can reconstruct the original image. During editing, the cross-attention layer of the editing branch is replaced to limit the scope of the editing area to prevent leakage and accurately control the editing effect. Whether it is inversion or conditional information encoding and injection, using them alone will cause the loss of original image information, thereby losing the rich details of the original face. Summary of the Invention

[0006] The purpose of the present invention is to address the shortcomings of current character image editing methods and to propose a character image editing method and device based on noise space inversion.

[0007] The object of the present invention is achieved through the following technical solution: a character image editing method based on noise spatial inversion, the method comprising:

[0008] S1. Preprocess the person image to be edited, encode the processed source image using a pre-trained diffusion model encoder, and encode the empty text, source text description, and edit text of the target image to obtain the empty text encoding, source text encoding, and edit text encoding;

[0009] S2. Use the DDIM inversion method to add noise to the source text code; use the diffusion model to reduce the noise of the added code, and obtain the latent code of the reconstructed image through DDIM sampling. Extract the features of the latent code and calculate the loss function. Use the stochastic gradient descent method to update the latent code.

[0010] S3. Extract the contour map of the source image, input the contour map into the control network, use the diffusion model with the control network, and use DDIM sampling to reduce noise on the updated latent code using the empty text code, the source text code, and the edit text code, where the DDIM sampling includes predicting noise using the source text code and predicting noise using the edit text code to obtain the latent code of the edited image;

[0011] S4. Decode the latent code of the edited image through the VAE decoder to obtain the final edited image.

[0012] Furthermore, the preprocessing includes: uniformly modifying the source image into a three-channel format, using a cropping transformation to modify the image resolution to an integer multiple of 64 pixels, and normalizing the RGB values; adjusting the dimensional information of the image to [number of channels, height, length] to obtain a processed source image.

[0013] Furthermore, the hyper parameters in the pre-trained diffusion model are a noise addition coefficient table using a standard diffusion model coefficient table, a denoising sampling method and number of times, a DDIM sampling number of times and a conditional guided diffusion coefficient.

[0014] Furthermore, the encoding process in S1 includes: encoding the processed source image using the VAE variational autoencoder provided by the diffusion model to obtain a distribution representation of the source image in the encoding space, and selecting the average value of the distribution as the latent code of the source image;

[0015] The CLIP text encoder provided by the diffusion model is used to encode the empty text, the text description of the source image, and the edit text description of the target image to obtain the empty text encoding, the source text encoding, and the edit text encoding.

[0016] Furthermore, in the DDIM inversion method, the noise addition formula is as follows:

[0017]

[0018] Where t is the timestamp, and the timestamp sequence is constructed according to the DDIM inversion steps; C src Encode the source text, the conditional guidance diffusion coefficient is 1, z t is the image code after adding noise at time t, z0 is the hidden code of the source image; α t is the hyperparameter of the diffusion model; ∈ θ (·,·,·) is the noise predicted by the diffusion model.

[0019] Furthermore, the loss function is to extract perceptual features from the latent codes of the source image and the reconstructed image, perform L1 loss between corresponding layers, and then sum the L1 losses of all layers to obtain the LPIPS value as the loss function.

[0020] Furthermore, the diffusion model with control network is specifically as follows: the control model processes the contour map information, receives the output of each encoder layer in the diffusion model Unet structure, and finally generates control information, and integrates the control information into the output of each decoder layer in the diffusion model Unet structure;

[0021] The control model uses stacked two-dimensional convolutional layers to process the contour map. The first two-dimensional convolutional layer accepts the contour map as input, and the remaining convolutional layers accept the output of the previous convolutional layer as input. The encoder block structure of the control network is exactly the same as the encoder block structure in the diffusion model Unet. The intermediate block structure of the control network is exactly the same as the intermediate block structure in the diffusion model Unet.

[0022] The encoder blocks are stacked. The first encoder block accepts the sum of the source image and the output of the stacked convolutional layers as input. The remaining encoder blocks accept the output of the previous encoder block as input, and the intermediate blocks accept the output of the last encoder block as input. The output of each encoder block and the intermediate layer of the control network are each passed through a two-dimensional convolutional layer to calculate the corresponding control conditions. The control conditions of the intermediate block are added to the output of the intermediate block of the diffusion model and then input to the first decoder block. The control conditions of each encoder block are summed with the output of the corresponding decoder block and then input to the subsequent network structure.

[0023] Furthermore, the DDIM sampling is used to perform noise reduction on the updated latent code using the empty text code, the source text code, and the edited text code. Specifically, the DDIM sampling formula is as follows:

[0024]

[0025] DDIM sampling consists of 20 steps. The first 4 steps of DDIM sampling use the source text encoding to predict noise. The formula is as follows:

[0026] ∈ θ+θ′ (z t ,t)=ω∈ θ+θ′ (z t ,t,C src )+(1-ω)∈ θ+θ′ (z t ,t,C φ )

[0027] The last 16 steps of DDIM sampling use edit text encoding to predict noise, the formula is as follows:

[0028] ∈ θ+θ′ (z t ,t)=ω∈ θ+θ′ (z t ,t,C edit )+(1-ω)∈ θ+θ′ (z t ,t,C φ )

[0029] Among them, the diffusion model of the control network is ∈ θ+θ′ (·,·,·); ω is the conditional guided diffusion coefficient, t is the timestamp, z t is the image code after adding noise at time t, α t is the hyperparameter of the diffusion model, C φ For empty text encoding, C src Encode the source text, C edit Encoding for editing text;

[0030] After the last DDIM sampling, the latent code of the edited image is obtained

[0031] On the other hand, the present invention also provides a device for editing character images with noise spatial inversion, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method for editing character images with noise spatial inversion.

[0032] On the other hand, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned method for editing character images using noise spatial inversion.

[0033] Beneficial effects of the present invention:

[0034] This invention enhances the flexibility of character image editing, allowing users to guide the editing process through simple text prompts, addressing the difficulty of expressing editing needs. After the text prompts are encoded by a text encoder, the resulting text embedding serves as a control condition to guide the model's denoising process.

[0035] 2. This invention solves the problem of maintaining consistency in person image editing. By inverting the image into a noise space through inversion technology, the coarse-grained information of the person image is preserved. Furthermore, the person's outline is extracted as additional information, processed through a control network, and injected into a diffusion model. Finally, it is merged with the original feature map through a convolutional layer, preserving the rich details of the person image.

[0036] 3. This invention improves the quality stability of character image editing results. Compared to adversarial generative models, diffusion models incorporate richer prior knowledge and can adapt to images from different domains. This provides greater robustness in editing and produces higher fidelity images.

[0037] This invention primarily addresses the problem of preserving character details during editing by using inversion techniques to embed character images into a noisy space. The character outline is then extracted and injected as an additional condition into the generation process of a diffusion model via a control network. The problem of editing character images is also addressed by encoding user-provided editing text prompts and then using the text embedding as a condition for interaction with the image during denoising via a cross-attention layer. By fully leveraging prior knowledge of the diffusion model, the reliability of character image editing can be effectively addressed. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 An image inversion flow chart provided by an embodiment of the present invention;

[0039] Figure 2An image editing flow chart provided by an embodiment of the present invention;

[0040] Figure 3 The diffusion model and control network data flow diagram provided by the embodiment of the present invention;

[0041] Figure 4 Schematic diagram of a person image editing device using noise spatial inversion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0043] like Figure 1 As shown, the present invention provides a method for editing character images by noise spatial inversion

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0045] First, perform inversion optimization on the source image. The specific steps are as follows:

[0046] Step 1: Encode the character image to be edited, including the following sub-steps:

[0047] (1) Prepare the character image to be edited and import it. Modify the source image to a three-channel format and use the cropping transformation to change the image resolution to an integer multiple of 64 pixels. Normalize the RGB values ​​to the range [-1, 1]. Adjust the image's dimensional information to [number of channels, height, length] to obtain the processed source image.

[0048] (2) Define the diffusion model network, import the pre-trained diffusion model parameters, and set the hyperparameters of the diffusion model. The hyperparameters used in the present invention are: the noise coefficient table adopts the standard diffusion model coefficient table; the denoising sampling method and number: DDIM sampling 20 times; the conditional guided diffusion coefficient ω is 7.5.

[0049] (3) Use the VAE variational autoencoder that comes with the diffusion model to encode the processed source image, obtain the distribution representation of the source image in the encoding space, and select the average value of the distribution as the latent code z0 of the source image.

[0050] (4) Use the CLIP text encoder that comes with the diffusion model to encode the empty text, the text description of the source image, and the edited text description of the target image to obtain the empty text encoding C φ , source text encoding C src and edit text encoding C edit .

[0051] Step 2: Noise the latent code of the source image to obtain the noisy code:

[0052] The DDIM inversion method is used to perform 20 steps of noise addition on the image code. The noise addition formula is as follows:

[0053]

[0054] Where t is the timestamp. The timestamp sequence can be constructed according to the DDIM inversion steps. In this invention, it is [1,51,101,151,201,251,301,351,401,451,501,551,601,651,701,751,801,851,901,951]. src Encode the source text, and the conditional guidance diffusion coefficient is 1. t is the image code after adding noise at time t, and z0 is the hidden code of the source image. t is the hyperparameter of the diffusion model. ∈ θ (·,·,·) is the noise predicted by the diffusion model.

[0055] The DDIM inversion process starts from the latent code z0 of the source image and adds noise to the image. The noise is obtained by the diffusion model prediction and added to the image code through the DDIM inversion formula to obtain z1, z2, and z3. 51 、z 101 ,…,z 951 .

[0056] Step 3: add noise code z 951 Perform gradient descent optimization:

[0057] The noise code z 951 Noise reduction is performed through the diffusion model, and after 20 steps of DDIM sampling, the latent code of the reconstructed image is obtained The hidden code With the LPIPS value of z0 as the loss function:

[0058] (1) Extraction using the VGG19 model The perceptual features of z0 are the output results of the 2nd, 7th, 12th, 21st, and 30th layers of the VGG19 model.

[0059] (2) Calculation of LPIPS value: The perceptual features of z0 are subjected to L1 loss between corresponding layers, and the L1 losses of all layers are summed to obtain the LPIPS value.

[0060] The loss function is calculated by the pytorch automatic derivation mechanism to calculate the loss function of the hidden code z 951Derivative, use stochastic gradient descent method to optimize and update hidden code, set the optimizer to Adam, the number of optimization steps to 300 steps, the learning rate to 1e-2, and get the optimized hidden code

[0061] After inversion optimization, edit the source image by controlling the editing text description, as follows:

[0062] Step 1: Use the cv2 library function cv2.Canny to extract the contour map of the source image.

[0063] Step 2: Define the control network and connect it to the diffusion model. Input the contour map into the control network, which processes the contour map information and accepts the outputs of the encoders at each layer of the diffusion model's Unet structure. Finally, it generates control information and incorporates it into the outputs of the decoders at each layer of the diffusion model's Unet structure. The substeps for the control network to process data are as follows:

[0064] (1) Use stacked two-dimensional convolutional layers to process the contour map. The first two-dimensional convolutional layer accepts the contour map as input, and the remaining convolutional layers accept the output of the previous convolutional layer as input. The stacked two-dimensional convolutional layer structure is as follows:

[0065] Convolutional layer 1: 3 input channels, 16 output channels, kernel size 3, padding 1, stride 1;

[0066] Convolutional layer 2: 16 input channels, 16 output channels, kernel size 3, padding 1, stride 1;

[0067] Convolutional layer 3: 16 input channels, 32 output channels, kernel size 3, padding 1, stride 2;

[0068] Convolutional layer 4: 32 input channels, 32 output channels, kernel size 3, padding 1, stride 1;

[0069] Convolutional layer 5: 32 input channels, 96 output channels, kernel size 3, padding 1, stride 2;

[0070] Convolutional layer 6: 96 input channels, 96 output channels, kernel size 3, padding 1, stride 1;

[0071] Convolutional layer 7: 96 input channels, 256 output channels, kernel size 3, padding 1, stride 2;

[0072] Convolutional layer 8: 256 input channels, 320 output channels, kernel size 3, padding 1, stride 1;

[0073] The source image is added to the output of the stacked convolutional layers as input to the first encoder block.

[0074] (2) The encoder block structure of the control network is exactly the same as that of the encoder block in the diffusion model Unet, and the intermediate block structure of the control network is exactly the same as that of the intermediate block in the diffusion model Unet. The encoder blocks are stacked, with the first encoder block accepting the sum of the source image and the output of the stacked convolutional layer as input, the remaining encoder blocks accepting the output of the previous encoder block as input, and the intermediate blocks accepting the output of the last encoder block as input.

[0075]

[0076]

[0077] The selection of text encoding is detailed in step three.

[0078] (3) The output of each encoder block and the intermediate layer of the control network passes through a two-dimensional convolutional layer to calculate the corresponding control conditions.

[0079]

[0080] (4) The control conditions of the intermediate block are added to the output of the diffusion model intermediate block, and the sum is input to the first decoder block. The control conditions of the last encoder block are added to the output of the first decoder block of the diffusion model, and the sum is input to the second decoder block, and so on. The control conditions of each encoder block are summed with the output of the corresponding decoder block, and then input to the subsequent network structure. The output of the diffusion network encoder block i is denoted as Ei (1 <= i <= 12), and the output of the intermediate block is denoted as E13.

[0081]

[0082]

[0083] Here, [E13+C13,E12+C12] means that E13 and C13 are added and summed, E12+C12 are added and summed, and then the two vectors are concatenated (torch.cat) in the channel dimension (dim=1), and the concatenated result is used as the input of the diffusion model encoder block 1.

[0084] Through the above steps, the control information C is injected into the diffusion model, and the diffusion model with the added control network is recorded as ∈ θ+θ′ (·,·,·).

[0085] Step 3: Use the diffusion model with control network to encode the optimized noise Perform noise reduction, with a total of 20 steps, using the DDIM sampling method. The DDIM sampling formula is as follows:

[0086]

[0087] The first four steps of DDIM sampling use the source text encoding to predict noise, and the formula is as follows:

[0088] ∈ θ+θ′ (z t ,t)=ω∈ θ+θ′ (z t ,t,C src )+(1-ω)∈ θ+θ′ (z t ,t,C φ )

[0089] The last 16 steps of DDIM sampling use edit text encoding to predict noise, the formula is as follows:

[0090] ∈ θ+θ′ (z t ,t)=ω∈ θ+θ′ (z t ,t,C edit )+(1-ω)∈ θ+θ′ (z t ,t,C φ )

[0091] After the last DDIM sampling, the latent code of the edited image is obtained

[0092] Step 4: Edit the hidden code of the image After decoding by the VAE decoder, the RGB value is linearly mapped from [-1, 1] to [0, 255] to obtain the final edited image.

[0093] Corresponding to the aforementioned embodiment of a person image editing method using noise spatial inversion, the present invention also provides an embodiment of a person image editing device using noise spatial inversion.

[0094] See also Figure 4 An embodiment of the present invention provides a device for editing character images using noise spatial inversion, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a method for editing character images using noise spatial inversion in the above embodiment.

[0095] The embodiment of the device for editing character images by noise spatial inversion provided by the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 4 As shown in the figure, a hardware structure diagram of a device with data processing capability for editing a person image by using noise spatial inversion provided by the present invention is provided. Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0096] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0097] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0098] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for editing a character image using noise spatial inversion in the above embodiment is implemented.

[0099] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0100] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for editing a character image by using noise spatial inversion is implemented.

[0101] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0102] It should be understood that the above general description and the detailed description that follows are exemplary and explanatory only and do not limit the present application. The present application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A method for editing a person image by noise spatial inversion, characterized in that: The method includes: S1. Preprocess the character image to be edited, encode the processed source image using the pre-trained diffusion model encoder, and encode the empty text, source text description and edit text of the target image to obtain the empty text encoding, source text encoding and edit text encoding; S2. Use the DDIM inversion method to add noise to the source text code; use the diffusion model to reduce the noise of the added code, and obtain the latent code of the reconstructed image after DDIM sampling. Extract the features of the latent code to calculate the loss function, and use the stochastic gradient descent method to update the latent code; S3, extracting the contour map of the source image, inputting the contour map into the control network, using the diffusion model with the control network, using the empty text encoding, the source text encoding and the edit text encoding to reduce the noise of the updated latent code by using DDIM sampling, wherein the DDIM sampling includes predicting the noise by using the source text encoding and predicting the noise by using the edit text encoding, and obtaining the latent code of the edited image; S4. Decode the latent code of the edited image through the VAE decoder to obtain the final edited image.

2. The method for editing a character image by noise spatial inversion according to claim 1, characterized in that: The preprocessing includes: modifying the source image uniformly into a three-channel format, using a cropping transformation to modify the image resolution to an integer multiple of 64 pixels, and normalizing the RGB values; adjusting the dimensional information of the image to [number of channels, height, length] to obtain a processed source image.

3. The method for editing a person image by noise spatial inversion according to claim 1, characterized in that: The hyperparameters in the pre-trained diffusion model are the noise addition coefficient table using a standard diffusion model coefficient table, the denoising sampling method and number of times, the DDIM sampling number of times and the conditional guided diffusion coefficient.

4. The method for editing a person image by noise spatial inversion according to claim 1, characterized in that: The encoding process in S1 includes: using the VAE variational autoencoder provided by the diffusion model to encode the processed source image, obtaining a distribution representation of the source image in the encoding space, and selecting the average value of the distribution as the latent code of the source image; The CLIP text encoder provided by the diffusion model is used to encode the empty text, the text description of the source image, and the edit text description of the target image to obtain the empty text encoding, the source text encoding, and the edit text encoding.

5. The method for editing a person image by noise spatial inversion according to claim 1, characterized in that: In the DDIM inversion method, the noise addition formula is as follows: Where t is the timestamp, and the timestamp sequence is constructed according to the DDIM inversion steps; C src Encode the source text, and the conditional guided diffusion coefficient is 1, z t is the image code after adding noise at time t, z0 is the hidden code of the source image; α t is the hyperparameter of the diffusion model; ∈ θ (·,·,·) is the noise predicted by the diffusion model.

6. The method for editing a person image by noise spatial inversion according to claim 1, characterized in that: The loss function is to extract perceptual features from the latent codes of the source image and the reconstructed image, perform L1 loss between corresponding layers, and then sum the L1 losses of all layers to obtain the LPIPS value as the loss function.

7. The method for editing a person image by noise spatial inversion according to claim 1, characterized in that: The diffusion model with control network is specifically as follows: the control model processes the contour map information, receives the output of each layer encoder in the diffusion model Unet structure, and finally generates control information, and integrates the control information into the output of each layer decoder in the diffusion model Unet structure; The control model uses stacked two-dimensional convolutional layers to process the contour map, the first two-dimensional convolutional layer accepts the contour map as input, and the remaining convolutional layers accept the output of the previous convolutional layer as input. The encoder block structure of the control network is completely consistent with the encoder block structure in the diffusion model Unet, and the intermediate block structure of the control network is completely consistent with the intermediate block structure in the diffusion model Unet; The encoder blocks are stacked. The first encoder block accepts the sum of the source image and the output of the stacked convolutional layer as input. The remaining encoder blocks accept the output of the previous encoder block as input. The middle block accepts the output of the last encoder block as input. The output of each encoder block and the middle layer of the control network passes through a two-dimensional convolutional layer to calculate the corresponding control condition. The control conditions of the intermediate block are added to the output of the diffusion model intermediate block and then input into the first decoder block; the control conditions of each encoder block are summed with the output of the corresponding decoder block and then input into the subsequent network structure.

8. The method for editing a person image by noise spatial inversion according to claim 1, characterized in that: The method of using the empty text encoding, the source text encoding and the edited text encoding to reduce noise by using DDIM sampling on the updated latent code is as follows: The DDIM sampling formula is as follows: DDIM sampling includes 20 steps in total. The first 4 steps of DDIM sampling use the source text encoding to predict noise. The formula is as follows: ∈ θ+θ′ (z t ,t)=ω∈ θ+θ′ (z t ,t,C src )+(1-ω)∈ θ+θ′ (z t ,t,C φ ) The last 16 steps of DDIM sampling use edit text encoding to predict noise, the formula is as follows: ∈ θ+θ′ (z t ,t)=ω∈ θ+θ′ (z t ,t,C edit )+(1-ω)∈ θ+θ′ (z t ,t,C φ ) Among them, the diffusion model of the control network is ∈ θ+θ′ (·,·,·); ω is the conditional guided diffusion coefficient, t is the timestamp, z t is the image code after adding noise at time t, α t is the hyperparameter of the diffusion model, C φ For empty text encoding, C src Encode the source text, C edit Encoding for editing text; After the last DDIM sampling, the hidden code of the edited image is obtained.

9. A person image editing device for noise spatial inversion, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a character image editing method of noise space inversion as described in any one of claims 1-8 is implemented.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a character image editing method of noise space inversion as described in any one of claims 1 to 8 is implemented.

Images