A method and apparatus for editing a portrait image using noise space inversion

Abstract

The application discloses a kind of noise space inversion figure image editing method and device.The application uses diffusion model to carry out inversion to figure image, realizes the accurate reconstruction of figure image, solves the problem that figure appears artifact in editing process;Extraction figure image face profile feature, after using control network to carry out convolution processing to profile feature, inject diffusion model, ensure the consistency of figure in editing process, solve the problem that figure is inconsistent before and after editing;Text prompt is used to edit figure image, and text prompt is edited to feature image by cross attention layer in diffusion model, solve the problem that user editing mode is single and difficult.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision information, and particularly relates to a noise space inversion human image editing method and device. BACKGROUND

[0002] With the development of deep learning and computer vision, editing of real human images has attracted extensive attention. Real human image editing aims to maintain the identity information of the human in the source image while editing the source image, including but not limited to modifying part of the features of the human image, such as hair color, eye color, expression modification, adding accessories to the human, oil painting style, ancient portrait to real style conversion, etc.

[0003] In recent years, real human image editing is mainly divided into two types according to the model skeleton: generative adversarial network and diffusion model. Real human image editing based on generative adversarial network covers a series of technologies, which can change certain features of human images by manipulating the latent variable space, such as extracting a certain part (eyes, nose, mouth) of a reference human image, or changing the attributes (age, expression) of the human image. In contrast, diffusion model-based technologies focus more on the editing effect of text prompts on human images, and the editing ability is greatly enhanced.

[0004] There are various different method branches based on generative adversarial network. The first type of method uses a pre-trained generative adversarial network to realize editing operation, explores the editing ability of the latent variable space, and uses and improves adaptive instance normalization, because the latent variable space of the generative adversarial network has good attribute editability. They first extract the latent code for a given face, and then manipulate it to obtain the editing result. A major drawback of these methods is that they cannot obtain a perfect latent code for a given image, resulting in the edited image losing the rich details of the original face. The second type of method uses image-to-image translation technology for face editing, in which the original face image is also input into the network. However, due to the constraint of cycle consistency, the synthesis result quality of the second type of method is lower compared with the first type of method.

[0005] There are various different method branches based on diffusion model technology. The first type of method usually modifies the text prompt or injects additional information as conditional information into the cross-attention layer for face image editing, and the conditional information can be attribute labels, expression encodings or style descriptions, etc. The key of this method lies in designing an effective conditional encoding scheme and a conditional information integration mechanism. The second type of method adopts the mechanism of inversion and attention map replacement. First, the source picture is inverted to a noise space to ensure that the inversion result can reconstruct the original picture. During editing, the cross-attention layer of the editing branch is replaced to limit the editing area range and accurately control the editing effect. Whether it is inversion or conditional information encoding and injection, separate use will cause the loss of original picture information, thereby losing the rich details of the original face. SUMMARY

[0006] The present application aims at the deficiencies of the current character image editing method, and proposes a noise space inversion character image editing method and device.

[0007] The purpose of the present application is achieved by the following technical scheme: a noise space inversion character image editing method, comprising:

[0008] S1, preprocessing the character image to be edited, using a pre-trained diffusion model encoder to encode the processed source image, and encoding the empty text, source text description and editing text of the target image to obtain empty text encoding, source text encoding and editing text encoding;

[0009] S2, using the DDIM inversion method to add noise to the source text encoding; denoising the noise-added encoding by a diffusion model, obtaining the hidden encoding of the reconstructed image through DDIM sampling, extracting the features of the hidden encoding to calculate the loss function, and updating the hidden encoding by using the stochastic gradient descent method;

[0010] S3, extracting the contour map of the source image, inputting the contour map into the control network, using the diffusion model with the control network, and using the empty text encoding, the source text encoding and the editing text encoding to denoise the updated hidden encoding using DDIM sampling, wherein the DDIM sampling includes predicting noise using the source text encoding and predicting noise using the editing text encoding, to obtain the hidden encoding of the edited image;

[0011] S4, decoding the hidden encoding of the edited image through the decoder of VAE to obtain the final edited image.

[0012] Further, the preprocessing includes: uniformly modifying the source image to a three-channel format, and using a cropping transform to modify the picture resolution to an integer multiple of 64 pixels, and normalizing the RGB values; adjusting the dimension information of the picture to [channel number, height, length] to obtain the processed source image.

[0013] Further, the hyperparameters in the pre-trained diffusion model are the noise coefficient table, the denoising sampling method and number of times, the DDIM sampling number of times, and the conditional guide diffusion coefficient.

[0014] Further, the encoding process in S1 includes using the VAE variational automatic encoder of the diffusion model to encode the processed source image to obtain the distribution representation of the source image in the encoding space, and selecting the average value of the distribution as the hidden encoding of the source image.

[0015] The CLIP text encoder of the diffusion model is used to encode the empty text, the text description of the source image, and the edited text description of the target image to obtain the empty text encoding, the source text encoding, and the edited text encoding.

[0016] Further, in the DDIM inversion method, the noise adding formula is as follows:

[0017]

[0018] Where t is a timestamp, and the timestamp sequence is constructed according to the DDIM inversion step number; C src is the source text encoding, and the conditional guide diffusion coefficient is 1, z t is the image encoding after adding noise at time t, and z0 is the hidden encoding of the source image; a t is a hyperparameter of the diffusion model; and θ (·,·,·) is the noise predicted by the diffusion model.

[0019] Further, the loss function is that after extracting the perceptual features from the hidden encoding of the source image and the hidden encoding of the reconstructed image, L1 loss is performed between the corresponding layers, and the L1 loss of all layers is summed to obtain the LPIPS value as the loss function.

[0020] Further, the diffusion model with a control network is specifically: the control model processes the contour graph information and accepts the output of each layer of the encoder in the diffusion model Unet structure, and finally generates control information, and the control information is integrated into the output of each layer of the decoder in the diffusion model Unet structure.

[0021] The control model processes the contour graph using stacked two-dimensional convolution layers, the first two-dimensional convolution layer accepts the contour graph as input, and the remaining convolution layers accept the output of the previous convolution layer as input. The encoder block structure of the control network is completely consistent with the encoder block structure in the diffusion model Unet, and the intermediate block structure of the control network is completely consistent with the intermediate block structure in the diffusion model Unet.

[0022] The encoder blocks are stacked, the first encoder block takes the sum of the source image and the output of the stacked convolutional layer as input, the remaining encoder blocks take the output of the previous encoder block as input, and the middle block takes the output of the last encoder block as input; the output of each encoder block and the intermediate layer of the control network is respectively calculated through a two-dimensional convolutional layer to obtain the corresponding control condition; the control condition of the middle block is added to the output of the diffusion model middle block, and then input to the first decoder block; the control condition of each encoder block is summed with the output of the corresponding decoder block, and then input to the subsequent network structure.

[0023] Further, the DDIM sampling is used for denoising of the updated latent code by using the empty text encoding, the source text encoding and the edited text encoding, and the DDIM sampling formula is as follows:

[0024]

[0025] The DDIM sampling includes 20 steps, and the first 4 steps of DDIM sampling use the source text encoding to predict the noise, and the formula is as follows:

[0026] ∈ θ+θ′ (z t ,t)=ω∈ θ+θ′ (z t ,t,C src )+(1-ω)∈ θ+θ′ (z t ,t,C φ )

[0027] The last 16 steps of DDIM sampling use the edited text encoding to predict the noise, and the formula is as follows:

[0028] ∈ θ+θ′ (z t ,t)=ω∈ θ+θ′ (z t ,t,C edit )+(1-ω)∈ θ+θ′ (z t ,t,C φ )

[0029] Wherein, the diffusion model of the control network is ∈ θ+θ′ (·,·,·);ω is a conditional guided diffusion coefficient, t is a timestamp, z t is an image code after adding noise at time t, α t is a hyperparameter of the diffusion model, C φ is an empty text encoding, C src is a source text encoding, and C edit is an edited text encoding.

[0030] After the last DDIM sampling, the hidden code of the edited image is obtained

[0031] In another aspect, the present application also provides a noise space inversion human image editing device, comprising a memory and one or more processors, wherein the memory stores executable code, and the processor executes the executable code to realize the noise space inversion human image editing method.

[0032] In another aspect, the present application also provides a computer readable storage medium having a program stored thereon, wherein the program is executed by a processor to realize the noise space inversion human image editing method.

[0033] The present application has the following advantages:

[0034] 1. The present application enhances the flexibility of human image editing, and users can guide the editing process through simple text prompts, solving the problem that users have difficulty expressing editing requirements. The text prompt is encoded by a text encoder to obtain a text embedding as a control condition to guide the model denoising process.

[0035] 2. The present application solves the problem of maintaining the consistency of the human image before and after editing. The image is inverted to the noise space through inversion technology, which can maintain the coarse-grained information of the human image. In addition, the human contour is extracted as additional information, which is processed by the control network and injected into the diffusion model, and finally merged with the original feature map through the convolution layer, which maintains the rich detail information of the human image.

[0036] 3. The present application improves the result quality stability of human image editing. Compared with the generative adversarial model, the diffusion model contains more prior knowledge, which can adapt to different domains of pictures and highlight stronger robustness in editing, and the generated image has better fidelity.

[0037] The present application mainly solves the problem of maintaining the details of the human image in the editing process by using inversion technology to embed the human image into the noise space, and extracting the human contour as additional information to inject into the generation process of the diffusion model through the control network. The following methods are used to solve the editing problem of the human image: the user-given editing text prompt is encoded, and then the text embedding is used as a condition to interact with the image in the denoising process through the cross-attention layer. In the case of fully utilizing the prior knowledge of the diffusion model, the reliability of the human image editing can be effectively solved. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The image inversion flowchart provided for the embodiments of the present application;

[0039] Figure 2An image editing flowchart provided by an embodiment of the present application;

[0040] Figure 3 A diffusion model and a control network data flowchart provided by an embodiment of the present application;

[0041] Figure 4 A schematic diagram of a noise space inversion human image editing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0042] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0043] As Figure 1 shown, the present application provides a noise space inversion human image editing method

[0044] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0045] First, the source image is optimized by inversion, and the specific steps are as follows:

[0046] Step one, encode the human image that needs to be edited, including the following sub-steps:

[0047] (1) Prepare the human image that needs to be edited and import it, uniformly modify the source image to three-channel format, and use cropping transformation to modify the picture resolution to an integer multiple of 64 pixels; normalize the RGB value to the interval [-1, 1]; adjust the dimension information of the picture to [channel number, height, length] to obtain the processed source image.

[0048] (2) Define the diffusion model network, import the pre-trained diffusion model parameters, and set the hyperparameters of the diffusion model. The hyperparameters used in the present application are as follows: the noise coefficient table uses the standard diffusion model coefficient table; the denoising sampling method and number of times: DDIM sampling 20 times; the conditional guide diffusion coefficient ω is 7.5.

[0049] (3) Use the VAE variational autoencoder provided by the diffusion model to encode the processed source image to obtain the distribution representation of the source image in the encoding space, and select the average value of the distribution as the hidden encoding z0 of the source image.

[0050] (4) Use the CLIP text encoder provided by the diffusion model to encode the empty text, the text description of the source image and the editing text description of the target image to obtain the empty text encoding C φ , the source text encoding C src and the editing text encoding C edit .

[0051] Step 2: Noise the latent code of the source image to obtain the noisy code:

[0052] The DDIM inversion method is used to perform 20 steps of noise addition on the image code. The noise addition formula is as follows:

[0053]

[0054] Where t is the timestamp. The timestamp sequence can be constructed according to the DDIM inversion steps. In this invention, it is [1,51,101,151,201,251,301,351,401,451,501,551,601,651,701,751,801,851,901,951]. src Encode the source text, and the conditional guidance diffusion coefficient is 1. t is the image code after adding noise at time t, and z0 is the hidden code of the source image. t is the hyperparameter of the diffusion model. ∈ θ (·,·,·) is the noise predicted by the diffusion model.

[0055] The DDIM inversion process starts from the latent code z0 of the source image and adds noise to the image. The noise is obtained by the diffusion model prediction and added to the image code through the DDIM inversion formula to obtain z1, z2, and z3. 51 、z 101 ,…,z 951 .

[0056] Step 3: add noise code z 951 Perform gradient descent optimization:

[0057] The noise code z 951 Noise reduction is performed through the diffusion model, and after 20 steps of DDIM sampling, the latent code of the reconstructed image is obtained The hidden code The LPIPS value of z0 is used as the loss function:

[0058] (1) Extraction using the VGG19 model The perceptual features of z0 are the output results of the 2nd, 7th, 12th, 21st, and 30th layers of the VGG19 model.

[0059] (2) Calculation of LPIPS value: The perceptual features of z0 are subjected to L1 loss between corresponding layers, and the L1 losses of all layers are summed to obtain the LPIPS value.

[0060] The loss function is calculated by the pytorch automatic derivation mechanism to calculate the loss function of the hidden code z 951Derivative, using stochastic gradient descent method optimization update hidden encoding, set the optimizer to Adam, optimization step number is 300 steps, learning rate is 1e-2, get the optimized hidden encoding

[0061] After inversion optimization, edit the source image by controlling the edited text description, as follows:

[0062] Step one, use the library function of cv2 to extract the contour map of the source image.

[0063] Step two, define the control network and connect the control network with the diffusion model. Input the contour map into the control network, the control network processes the contour map information, and accepts the output of each layer encoder in the diffusion model Unet structure, finally generates control information, and integrates the control information into the output of each layer decoder in the diffusion model Unet structure. The data processing substep of the control network is as follows:

[0064] (1) Use stacked two-dimensional convolutional layers to process the contour map, the first two-dimensional convolutional layer accepts the contour map as input, and the remaining convolutional layers accept the output of the previous convolutional layer as input. The stacked two-dimensional convolutional layer structure is as follows:

[0065] Convolutional layer 1: input channel number 3, output channel number 16, convolution kernel size 3, padding 1, step 1;

[0066] Convolutional layer 2: input channel number 16, output channel number 16, convolution kernel size 3, padding 1, step 1;

[0067] Convolutional layer 3: input channel number 16, output channel number 32, convolution kernel size 3, padding 1, step 2;

[0068] Convolutional layer 4: input channel number 32, output channel number 32, convolution kernel size 3, padding 1, step 1;

[0069] Convolutional layer 5: input channel number 32, output channel number 96, convolution kernel size 3, padding 1, step 2;

[0070] Convolutional layer 6: input channel number 96, output channel number 96, convolution kernel size 3, padding 1, step 1;

[0071] Convolutional layer 7: input channel number 96, output channel number 256, convolution kernel size 3, padding 1, step 2;

[0072] Convolutional layer 8: input channel number 256, output channel number 320, convolution kernel size 3, padding 1, step 1;

[0073] The source image is added to the output of the stacked convolutional layer as the input of the first encoder block.

[0074] (2) The encoder block structure of the control network is completely consistent with that of the diffusion model Unet, and the intermediate block structure of the control network is completely consistent with that of the diffusion model Unet. The encoder blocks are stacked, the first encoder block accepts the sum of the source image and the output of the stacked convolutional layer as the input, the remaining encoder blocks accept the output of the previous encoder block as the input, and the intermediate block accepts the output of the last encoder block as the input.

[0075]

[0076]

[0077] Wherein, the selection of text encoding is described in detail in step three.

[0078] (3) The output of each encoder block and intermediate layer of the control network is respectively passed through a two-dimensional convolutional layer to calculate the corresponding control condition.

[0079]

[0080] (4) The control condition of the intermediate block is added to the output of the diffusion model intermediate block, and then input to the first decoder block; the control condition of the last encoder block is added to the output of the first decoder block of the diffusion model, and then input to the second decoder block; and so on; the control condition of each encoder block is summed with the output of the corresponding decoder block, and then input to the subsequent network structure. The output of the diffusion network encoder block i is denoted as Ei (1<=i<=12), and the output of the intermediate block is denoted as E13.

[0081]

[0082]

[0083] Wherein, [E13+C13, E12+C12] represents the addition of E13 and C13, the addition of E12 and C12, and then the concatenation (torch.cat) of the two vectors in the channel dimension (dim=1), and the concatenation result is used as the input of the diffusion model encoder block 1.

[0084] The control information C is injected into the diffusion model through the above steps, and the diffusion model with the control network is denoted as θ+θ′ (·,·,·).

[0085] Step three, using the diffusion model with control network to optimize the noise-added code The noise is reduced, and the noise reduction is 20 steps in total, and the DDIM sampling method is used. The DDIM sampling formula is as follows:

[0086]

[0087] The first 4 steps of DDIM sampling use source text encoding to predict noise, and the formula is as follows:

[0088] ∈ θ+θ′ (z t ,t)=ω∈ θ+θ′ (z t ,t,C src )+(1-ω)∈ θ+θ′ (z t ,t,C φ )

[0089] The last 16 steps of DDIM sampling use edited text encoding to predict noise, and the formula is as follows:

[0090] ∈ θ+θ′ (z t ,t)=ω∈ θ+θ′ (z t ,t,C edit )+(1-ω)∈ θ+θ′ (z t ,t,C φ )

[0091] After the last DDIM sampling, the hidden encoding of the edited image is obtained

[0092] Step four, the hidden encoding of the edited image is decoded through the decoder of the VAE, and the RGB value is linearly mapped from [-1, 1] to [0, 255], to obtain the final edited image.

[0093] Corresponding to the foregoing embodiment of the noise space inversion character image editing method, the present application also provides an embodiment of a noise space inversion character image editing device.

[0094] Referring to Figure 4 , the embodiment of the present application provides a noise space inversion character image editing device, which comprises a memory and one or more processors, the memory stores executable code, and the processor executes the executable code to realize the noise space inversion character image editing method in the above-mentioned embodiment.

[0095] The embodiment of the device for editing a human image through noise spatial inversion provided by the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be realized by software, or by hardware or a combination of software and hardware. Taking software realization as an example, as a device in a logical sense, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor of the device with data processing capability. From the hardware level, as shown in Figure 4 the hardware structure diagram of the device for editing a human image through noise spatial inversion provided by the present application in any device with data processing capability, in addition to the processor, the memory, the network interface and the non-volatile memory shown in Figure 4 the device in the embodiment can also include other hardware according to the actual functions of the device with data processing capability, and details are not described herein.

[0096] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, and details are not described herein.

[0097] For the device embodiment, since it basically corresponds to the method embodiment, the related parts can be referred to the part of the method embodiment. The above described device embodiment is only schematic, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. According to the actual needs, part or all of the modules can be selected to achieve the purpose of the present application. Those skilled in the art can understand and implement it without creative labor.

[0098] The embodiment of the present application further provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the method for editing a human image through noise spatial inversion in the above embodiment.

[0099] The computer readable storage medium can be an internal storage unit of any of the aforementioned data processing capable devices, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device of any of the aforementioned data processing capable devices, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. Further, the computer readable storage medium can include both an internal storage unit and an external storage device of any of the aforementioned data processing capable devices. The computer readable storage medium is used to store the computer program and other programs and data required by the aforementioned data processing capable devices, and can also be used to temporarily store data that has been output or is about to be output.

[0100] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the method for editing a portrait image by noise space inversion.

[0101] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0102] It is to be understood that the above general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application. The application is not restricted to the precise construction and combinations of parts and steps described above and shown in the accompanying drawings, as such variations and modifications are intended to be included within the scope of the application. The scope of the application is limited only by the following claims.

Claims

1. A method of editing a portrait image using noise space inversion, characterized by, The method comprises: S1, preprocessing the image of the person to be edited, using a pre-trained diffusion model encoder to encode the processed source image, and encoding the empty text, source text description and edited text description of the target image to obtain the hidden encoding of the source image, the encoding of the empty text, the encoding of the source text and the encoding of the edited text; S2, the pre-trained diffusion model uses a DDIM inversion method to add noise to the source text encoding; the pre-trained diffusion model is used to denoise the noise-added encoding, and the hidden encoding of the reconstructed image is obtained through DDIM sampling; the features of the source image and the reconstructed image hidden encoding are extracted to calculate the loss function, and the random gradient descent method is used to update the hidden encoding; S3, extracting the contour map of the source image, defining a control network and connecting the control network with the diffusion model, inputting the contour map into the control network, using the diffusion model with the control network, and using the empty text encoding, the source text encoding and the edited text encoding to denoise the updated hidden encoding using DDIM sampling, wherein the DDIM sampling includes predicting noise using the source text encoding and predicting noise using the edited text encoding, to obtain the hidden encoding of the edited image; S4, decoding the hidden encoding of the edited image through the decoder of VAE to obtain the final edited image.

2. The noise space inversion method for portrait image editing according to claim 1, wherein, The preprocessing includes: uniformly modifying the source image to a three-channel format, and using a cropping transform to modify the picture resolution to an integer multiple of 64 pixels, and normalizing the RGB values; adjusting the dimension information of the picture to [channel number, height, length] to obtain the processed source image.

3. The method of claim 1, wherein the method further comprises: The hyperparameters in the pre-trained diffusion model include a noise coefficient table using a standard diffusion model coefficient table, a denoising sampling method and number of times, a DDIM sampling number and a conditional guide diffusion coefficient.

4. The method of claim 1, wherein the method further comprises: The encoding process in S1 includes using the VAE variational automatic encoder of the diffusion model to encode the processed source image to obtain the distribution representation of the source image in the encoding space, and selecting the average value of the distribution as the hidden encoding of the source image; using the CLIP text encoder of the diffusion model to encode the empty text, the text description of the source image and the edited text description of the target image to obtain the empty text encoding, the source text encoding and the edited text encoding.

5. The method of claim 1, wherein the method further comprises: In the DDIM inversion method, the latent encoding of the source image First, the image is added with noise, and the noise adding formula is as follows: ； where t is the timestamp, and the timestamp sequence is constructed according to the number of steps of DDIM inversion; is the encoding of the source image, and the conditional guidance diffusion coefficient is 1 at this time, is the encoding of the image after adding noise to the time t, is the latent encoding of the source image; is the hyperparameter of the diffusion model; is the noise predicted by the diffusion model.

6. The method of claim 1, wherein the method further comprises: The loss function is obtained by summing the L1 losses of all layers after extracting the perceptual features from the hidden encoding of the source image and the hidden encoding of the reconstructed image, and then obtaining the LPIPS value as the loss function.

7. The method of claim 1, wherein the method further comprises: The diffusion model with the control network specifically is that the control network processes the contour map information and accepts the output of each layer encoder in the diffusion model Unet structure, and finally generates control information which is integrated into the output of each layer decoder in the diffusion model Unet structure; The control network uses stacked two-dimensional convolution layers to process the contour map, the first two-dimensional convolution layer accepts the contour map as input, and the remaining convolution layers accept the output of the previous convolution layer as input; the encoder block structure of the control network is completely consistent with the encoder block structure in the diffusion model Unet, and the intermediate block structure of the control network is completely consistent with the intermediate block structure in the diffusion model Unet. The encoder blocks are stacked, the first encoder block takes the sum of the source image and the output of the stacked convolutional layer as input, the remaining encoder blocks take the output of the previous encoder block as input, and the middle block takes the output of the last encoder block as input; the output of each encoder block and the intermediate layer of the control network is respectively calculated through a two-dimensional convolutional layer to obtain the corresponding control condition; The control condition of the middle block is added to the output of the diffusion model middle block, and then input to the first decoder block; the control condition of each encoder block is summed with the output of the corresponding decoder block, and then input to the subsequent network structure.

8. The method of claim 1, wherein the method further comprises: The DDIM sampling is used for denoising of the updated latent code by using the empty text encoding, the source text encoding and the edited text encoding, and the DDIM sampling formula is as follows: ； The DDIM sampling includes 20 steps, the first 4 steps of the DDIM sampling use the source text encoding to predict the noise, and the formula is as follows: ； The last 16 steps of the DDIM sampling use the edited text encoding to predict the noise, and the formula is as follows: ； wherein the diffusion model of the control network is ; is a conditional directed diffusion coefficient, t is a time stamp, is an image encoding with noise added over time t, is a hyperparameter of the diffusion model, is a null text encoding, is a source text encoding, is an edited text encoding; After the last DDIM sampling, the latent encoding of the edited image is obtained .

9. An apparatus for editing a portrait image using noise space inversion, comprising a memory and one or more processors, wherein the memory stores executable code, and the executable code comprises the following steps of: The processor executes the executable code to implement the method for editing a human image by noise space inversion according to any one of claims 1-8.

10. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the method for editing a human image by noise space inversion according to any one of claims 1-8.

Images