Image editing method for rotating single target in image based on diffusion model

Abstract

The invention relates to an image editing method for rotating a single target in an image based on a diffusion model, and belongs to the field of image editing. According to the method, an image editing network based on a diffusion model is provided; adding a visual angle picture generated by an image-3D technology into the editing network as a reference picture; the noise sampled by the diffusion model is optimized through the comparison loss of the edited picture, the original picture and the reference picture; and finally, the target rotation effect is achieved. In order to keep the consistency of background contents, the region-based image filling is added into the sampling process of the diffusion model; in addition, super-resolution processing is carried out on the visual angle picture generated by the image-3D technology, and the editing quality is further improved. By means of the method, good performance can be effectively achieved on the object rotation editing task.

Description

Technical Field

[0001] The present invention belongs to the field of image editing, and specifically relates to an image editing method for completing the rotation of a single target in an image based on a diffusion model. An image editing network for completing the rotation of a single target in an image based on a diffusion model is provided. The network is based on the diffusion model, and uses an image-3D method to generate a multi-view image of a single target in an image, and performs super-resolution processing as a reference image. In the diffusion model sampling process, noise is optimized to complete the rotation task of the single target. Background Art

[0002] In recent years, the development of generative models has been remarkable, and various novel model architectures have emerged. Among them, the diffusion model [1] has attracted much attention due to its outstanding performance in image generation tasks. The original concept of the diffusion model originated from the diffusion process in physics, simulating the diffusion of particles from high-concentration areas to low-concentration areas. The diffusion model gradually generates realistic samples by simulating data distribution. Its core idea is to gradually degenerate data samples into pure noise under the perturbation of Gaussian noise, and then gradually denoise through an inverse process to restore the original data.

[0003] The main goal of image editing [2] is to modify the content of a given image in some way. Previous methods usually invert the image into the latent space of a GAN and then edit the image by manipulating the latent vector. Recently, DragGAN proposed a point-drag formulation for fine-grained image editing. However, limited by the capabilities of the GAN model, these methods have weaknesses in model generalization and image quality. Driven by the success of the text-to-image diffusion model, various text-guided image editing methods have been proposed [3]. Common editing strategies are (1) adding noise and then denoising it using the target text as a condition; (2) using cross-attention maps as the editing medium; and (3) using text as editing instructions. The text-to-image diffusion model (T2I) has become the mainstream of image editing, with high-quality and diverse editing capabilities. The pre-trained diffusion model can serve as a good prior knowledge to adapt to various downstream editing tasks such as object movement, resizing, content dragging, and object pasting. However, the generated results of the T2I model are usually very sensitive to the quality of the text [4]. Therefore, text-guided image editing cannot achieve fine-grained editing operations.

[0004] Recently, DiffEditor[5] provides a more novel guidance method that introduces image cues into the fine-grained image editing process and combines them with text cues to improve the editing quality through more detailed content descriptions. In addition, Dragdiff[6] and Dragondiff[7] achieve fine-grained image editing by optimizing the noise obtained by sampling in a pre-trained T2 I diffusion model and modifying the self-attention mechanism during the sampling process. However, these methods and most image editing methods based on diffusion models[8] cannot complete the task of rotating a single object in an image while keeping the background unchanged. Summary of the invention

[0005] The purpose of the present invention is to overcome the problem that the existing technology cannot complete the rotation editing effect of a single target in an image in a complex scene, and provide an image editing method for completing the rotation of a single target in an image based on a diffusion model, and add the image-3D technology [9] to the image editing process. The user only needs to provide a real picture or a generated picture, and the description text corresponding to the initial image and the title corresponding to the object to be rotated are used as initial conditions. Using the image-3D technology, the corresponding pictures of each view are generated for the single object in the image as reference pictures. The editing of the picture is completed by optimizing the noise in the sampling process of the diffusion model. In the sampling process, the initial picture and the reference picture are used as image prompts, and combined with the text prompts to add to the cross attention mechanism. In addition, region-based image filling and super-resolution processing

[10] are also proposed. The former can ensure that the background remains unchanged after rotation and fill the missing background part, and the latter can further improve the editing quality.

[0006] To achieve the above purpose, the technical solution of the present invention is: an image editing method for rotating a single target in an image based on a diffusion model, proposing an image editing network based on a diffusion model, adding a perspective picture generated by image-3D technology as a reference picture in the image editing network, and optimizing the noise sampled by the diffusion model by comparing the contrast loss between the editing picture and the original picture and the reference picture, so as to finally achieve the effect of target rotation.

[0007] In one embodiment of the present invention, in order to maintain the consistency of background content, region-based image filling is added to the sampling process of the diffusion model.

[0008] In one embodiment of the present invention, super-resolution processing is also performed on the viewing angle picture generated by the image-3D technology.

[0009] In one embodiment of the present invention, the method is specifically implemented as follows:

[0010] First, the user provides the initial image orl to be edited, and specifies the object to be rotated, the text title corresponding to the initial image, and the title corresponding to the object to be rotated;

[0011] In the preparation stage, the image editing network segments the object to be rotated through image segmentation, and inputs it into the image-3D network to obtain the corresponding view images of the object; the user selects the required view image, performs super-resolution processing, and obtains the reference image ref;

[0012] In the editing stage, the image editing network passes the initial image orl and the reference image ref through the VAE encoder to obtain the corresponding latent space (Z 0 orl , Z 0 ref ), and then DDIM encoding is performed simultaneously, and the noisy version of the encoding process is retained (Z t orl , Z t ref ) and K in the self-attention mechanism of the upsampling layer t orl 、V t orl , K t ref 、V t ref ; Next, the initial image orl and the reference image ref are encoded using the pre-trained image encoder, and fused through the cross attention mechanism of U-net as in the text encoding; The final DDIM encoding result of the initial image orl and the reference image ref (Z T orl , Z T ref ) is fused as the initial noise Z T ; The editing process uses DDIM sampling with a step length of 50 steps. In the first 30 steps, the sampled image Z t The intermediate feature F when passing through the 2nd and 3rd layers of the U-net upsampling layer t , and Z t orl The intermediate feature F corresponding to the number of steps t orl , and Z t ref The intermediate feature F corresponding to the number of steps t ref , compare the cosine similarity, get the loss, optimize Z t The noise ε predicted by U-net is iterated for 3 rounds at each step; the last 20 steps are normal DDIM sampling; and in each sampling process, the K in the self-attention mechanism of the U-net upsampling layer is added.t 、V t , replaced by K retained in the forward DDIM encoding t orl 、V t orl , K t ref 、V t ref , the final sampling result Z 0 , and the editing result is obtained through the decoder.

[0013] In one embodiment of the present invention, the noise version (Z) of the t-th step of DDIM encoding is t orl , Z t ref ), and K in the self-attention mechanism of the upsampling layer t orl 、V t orl , K t ref 、V t ref , stored through the memory bank.

[0014] In one embodiment of the present invention, during the editing process, in each sampling step, K in the self-attention mechanism of the U-net upsampling layer is t 、V t , replaced by K retained in the forward DDIM encoding t orl 、V t orl , K t ref 、V t ref , that is, using visual cross attention, connecting the initial image and the reference image in the length dimension K (K t orl , K t ref) and V(V t orl ,V t ref ); the visual cross-attention at each time step is defined as follows:

[0015]

[0016] Among them, the Attention function represents the self-attention mechanism in Unet, Q, K, and V are the query, key, and value generated by the image itself in the attention mechanism; the softmax function is a normal softmax operation, and d represents the number of dimensions.

[0017] In one embodiment of the present invention, the input image is passed through an image encoder to obtain feature vectors corresponding to 64 tags, which are then input into the cross-attention mechanism of the same U-net as the text tags; unconditional image prompts are formed by setting the image to zero during generation; finally, the image tags and text tags are processed separately with the query Q in the cross-attention mechanism of the U-net; and the results are fused together; the process is as follows:

[0018]

[0019] Among them, the Attention function is the cross attention mechanism in Unet, Q is the query generated by the image itself, K and V are the keys and values ​​generated by the text prompt or image prompt, (K', V') and (K", V") are used to represent K and V in the text prompt and image prompt respectively, λ is the balancing weight, the softmax function is the normal softmax operation, and d represents the number of dimensions.

[0020] In one embodiment of the present invention, F t 、F t orl 、F t ref , compare the cosine similarity, get the loss, optimize Z t The noise ε predicted by U-net is obtained by two binary masks m orl and m rep To locate the position that needs to be optimized, the edited position of the target on the original image and the position of the target on the reference image are represented respectively. The loss of the two regions is obtained by cosine similarity. The loss function loss edit as follows:

[0021]

[0022] Where sg(·) is the gradient clipping operation, α and β are two hyperparameters, and the S function is used to calculate the cosine similarity of two regions. The following S functions are all the same.

[0023] In one embodiment of the present invention, the region-based image filling is specifically implemented as follows:

[0024] Since the shape of the rotated object may be different from the original object, when editing, the uncovered part of the original object is filled with the background of the initial image based on the area. gud The binary mask represents the original location of the object, m orl The binary mask represents the position of the object after rotation, (m gud -m orl ) represents the part that needs to be filled, which is divided into different areas m by straight linesoverlap_0 、m overlap_1 Fill the respective background, m ref_0 、m ref_1 Indicates the location of the corresponding background; loss function loss ref as follows:

[0025]

[0026] Where ω is the weight parameter.

[0027] In one embodiment of the present invention, in order to maintain the consistency of the background content, m gud +m orl The complement of other To indicate the background part that needs to be consistent; loss function loss content as follows:

[0028]

[0029] Finally, the loss function loss is obtained by integration sum for:

[0030] loss sum =ω e ·loss edit +ω r ·loss ref +ω c ·loss content

[0031] where ω e ,ω r ,ω c is a hyperparameter that balances the losses;

[0032] The loss function loss is obtained by the final integration sum Optimize Z t Noise ε predicted by U-net.

[0033] Compared with the prior art, the present invention has the following beneficial effects: First, the present invention proposes an image editing network based on a diffusion model to complete the rotation of a single target in an image. The network only requires the user to provide an initial picture and a corresponding text title to complete the rotation of any object in the picture. Second, the present invention adds image-3D technology to image editing. Experimental results show that the diffusion model cannot exert its powerful generation capabilities in some scenes that require a lot of content imagination. 3D technology is required to generate corresponding perspective pictures. Third, the present invention proposes region-based image filling and super-resolution processing to further improve the editing quality. Fourth, the present invention experiments with the proposed network and achieves good performance in the editing task of object rotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 An image editing network framework that rotates a single object in an image based on a diffusion model;

[0035] Figure 2 Image - pictures from different perspectives generated by 3D technology;

[0036] Figure 3 Editing results without super-resolution processing;

[0037] Figure 4 The editing result without background filling;

[0038] Figure 5 The editing results in the complete framework. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0040] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0041] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0042] The present invention provides an image editing method for rotating a single target in an image based on a diffusion model, proposes an image editing network based on the diffusion model, adds a perspective picture generated by image-3D technology as a reference picture in the image editing network, and optimizes the noise sampled by the diffusion model by comparing the editing picture with the original picture and the reference picture, thereby finally achieving the effect of target rotation.

[0043] The following is the specific implementation process of the present invention.

[0044] like Figure 1As shown, the present invention proposes an image editing network based on a diffusion model to complete the rotation of a single target in an image, making full use of the generation ability of the diffusion model. First, the user provides the initial image orl to be edited, and specifies the object to be rotated, the text title corresponding to the initial image, and the title corresponding to the object to be rotated. In the preparation stage, the network segments the object to be rotated through image segmentation, and inputs it into the image-3D network to obtain each perspective image corresponding to the object. The user selects the required perspective image and performs super-resolution processing to obtain a reference image ref. In the editing stage, the network passes the initial image orl and the reference image ref through the VAE encoder to obtain the corresponding latent space (Z 0 orl , Z 0 ref ), and then DDIM encoding is performed simultaneously, and the noisy version of the encoding process is retained (Z t orl , Z t ref ) and K, V (K in the self-attention mechanism of the upsampling layer t orl , V t orl , K t ref , V t ref ). Next, the initial image orl and the reference image ref are encoded using the pre-trained image encoder, and fused through the cross attention mechanism of U-net as in the text encoding. The final DDIM encoding result of the initial image orl and the reference image ref (Z T orl , Z T ref ) is fused as the initial noise Z T DDIM sampling is used in the editing process with a step length of 50 steps. In the first 30 steps, the sampled image Z t The intermediate features F when passing through the 2nd and 3rd layers of the U-net upsampling layer t , and Z t orl The intermediate feature F corresponding to the number of steps t orl , and Z t ref The intermediate feature F corresponding to the number of steps t ref , compare the cosine similarity, get the loss, optimize Z t The noise ε predicted by U-net. Each step is iterated for 3 rounds. The last 20 steps are normal DDIM sampling. In addition, during each sampling step, the K in the self-attention mechanism of the previous sampling layer of U-net ist ,V t , replaced by K retained in the forward DDIM encoding memory t orl , V t orl , K t ref , V t ref The final sampling result Z 0 , and the editing result is obtained through the decoder.

[0045] Memory bank and visual cross-attention:

[0046] Memory library: In the image editing process, if only DDIM encoding is generated in the last step Z t As the initial noise, it is difficult to provide accurate generation guidance. Therefore, the potential noise z of each DDIM encoding step t can be stored in the memory bank t orl , z t ref , and the corresponding K in the self-attention module of the upsampling layer in UNet t orl , V t orl , K t ref , V t ref The information stored in the memory bank will provide more accurate guidance for the subsequent image editing process.

[0047] Visual cross attention: In order to better ensure the consistency between the editing result and the original image, during the editing process, the K and V in the self-attention module of the UNet upsampling layer are replaced with the K and V of the corresponding steps collected by the memory bank. The two groups K (K t orl , K t ref) and V(V t orl ,V t ref ). The visual criss-cross attention at each time step is defined as follows:

[0048]

[0049] The Attention function represents the self-attention mechanism in Unet. Q, K, and V are the query, key, and value generated by the image itself in the attention mechanism. The softmax function is a normal softmax operation, and d represents the number of dimensions.

[0050] (2) Image prompts

[0051] Compared with text prompts, image prompts can provide more detailed content descriptions, especially in some complex scenes, which can improve the quality of fine-grained image editing. The input image is passed through the image encoder to obtain feature vectors corresponding to 64 tags, which are then input into the same cross-attention module as the text tags. In order to build a classifier-free guidance like text conditions, unconditional image prompts are formed by setting the image to zero during generation. Finally, the image tags and text tags are processed separately with the query Q in the cross-attention module. And the results are fused together. The process is as follows:

[0052]

[0053] The Attention function is the cross attention mechanism in Unet, Q is the query generated by the image itself, K and V are the keys and values ​​generated by the text prompt or image prompt. (K', V') and (K", V") are used to represent K and V in the text prompt and image prompt respectively, and λ is the balance weight. The softmax function is the normal softmax operation, and d represents the number of dimensions.

[0054] Preparation stage:

[0055] Since the diffusion model still has generation difficulties in some scenes that require a lot of content imagination, when performing rotation tasks, the diffusion model cannot generate the content after the object is rotated.

[0056] Image-3D: Some studies believe that the reason why the content of the rotated object cannot be generated is that the diffusion model SD lacks 3D perception of a single object. Inspired by the 3D diffusion model, our work introduces the image-3D technology. Upload an RGBA image of the object only, and use the image-3D technology to generate the corresponding viewpoint images of the object as shown in the attached figure. Figure 2 . Use it as a reference image. This fills the gap of the diffusion model's lack of 3D perception.

[0057] Super-resolution processing: Since the initial image may be from a real image, the resolution is high. However, the reference images of each perspective generated by the image-3D technology have a lower resolution. If these two images are directly put into the editing stage and edited to the same resolution, the final editing result may cause the target object to be blurred, as shown in the following figure. Figure 3 Therefore, we introduced super-resolution technology to perform super-resolution processing on the required perspective reference image to obtain the final reference image.

[0058] Editing stage:

[0059] In the editing stage, we complete the editing through optimization, which mainly includes content editing after target rotation, background filling based on region, and background consistency.

[0060] Content editing after target rotation: At the tth time step, we transform the latent space Z of the current time step t , input to U-net, and obtain the intermediate feature F of the upsampling layer t . Using the same method, the Z of the corresponding step number is retrieved from the memory bank t orl ,Z t ref Extract feature F t orl ,F t ref We use two binary masks (m orl and m rep ) to locate the position that needs to be optimized. They represent the edited position of the target on the original image and the position of the target on the reference image. The loss of the two regions is obtained by cosine similarity, and this loss is used to optimize Z t The noise ε obtained by U-net. Loss function loss edit as follows:

[0061] S(F t ref ,m rep ,F t, m orl )=0.5·cos(F t ref [m rep ],sg(F t [m orl ]))+0.5

[0062]

[0063] Where sg(·) is the gradient clipping operation, where α and β are two hyperparameters, set to 1 and 4 respectively. The S function is used to calculate the cosine similarity of two regions, and the following S functions are all the same.

[0064] Background fill based on area: Since the shape of the rotated object may be different from the original object, when editing, some problems may occur if the uncovered part of the original object is not processed. Figure 4 To correct this problem, our method uses the background of the initial image to perform region-based filling on the uncovered parts. We use m gud The binary mask represents the original location of the object, m orlThe binary mask represents the position of the object after rotation, (m gud -m orl ) represents the part that needs to be filled, which is divided into different areas m by straight lines overlap_0 ,m overlap_1 Fill the respective background. ref_0 ,m ref_1 Indicates the location of the corresponding background. The background is also filled in through optimization. Loss function loss ref as follows:

[0065]

[0066] ω is the weight parameter.

[0067] Background consistency: In addition to editing the content, we hope that the unmodified parts remain consistent with the original image. This can also be achieved by using optimization methods. gud +m orl The complement of other To represent the background part that needs to remain consistent. Loss function loss content as follows:

[0068]

[0069] Finally, the loss function loss is obtained by integration sum for:

[0070] loss sum =ω e ·loss edit +ω r ·loss ref +ω c ·loss content

[0071] where ω e ,ω r ,ω c is a hyperparameter that balances these losses.

[0072] We use the loss function to optimize the noise ε predicted by U-net. And this operation is only performed in the first 30 steps, with 3 iterations per step. The last 20 steps are normal DDIM sampling.

[0073] The overall algorithm pseudo code is shown in Algorithm 1:

[0074]

[0075]

[0076] Figure 5It is the editing result under the complete framework of the network provided by the method of the present invention.

[0077] The present invention also provides an image editing system for completing the rotation of a single target in an image based on a diffusion model, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0078] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0079] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0080] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0081] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0083] References:

[0084] [1]Nichol AQ, Dhariwal P, Ramesh A, et al. GLIDE: Towards PhotorealisticImage Generation and Editing with Text-Guided Diffusion Models[C] / / International Conference on Machine Learning.PMLR, 2022:16784-16804.

[0085] [2] A, M, J,et al.Diffusion image analogies[C] / / ACMSIGGRAPH 2023Conference Proceedings.2023:1-10.

[0086] [3]Hertz A, Mokady R, Tenenbaum J, et al. Prompt-to-Prompt Image Editing with Cross-Attention Control[C] / / The Eleventh International Conference onLearning Representations.

[0087] [4]Segalis E,Valevski D,Lumen D,et al.A picture is worth a thousand words:Principled recaptioning improves image generation[J].arXiv preprintarXiv:2310.16656,2023.

[0088] [5]Mou C,Wang X,Song J,et al.Diffeditor:Boosting accuracy andflexibility on diffusion-based image editing[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2024:8488-8497.

[0089] [6]Shi Y,Xue C,Liew J H,et al.Dragdiffusion:Harnessing diffusionmodels for interactive point-based image editing[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2024:8839-8849.

[0090] [7]Mou C,Wang X,Song J,et al.DragonDiffusion:Enabling Drag-styleManipulation on Diffusion Models[C] / / The Twelfth International Conference onLearning Representations.

[0091] [8]Ju X,Zeng A,Bian Y,et al.Direct inversion:Boosting diffusion-basedediting with 3lines of code[J].arXiv preprint arXiv:2310.01506,2023.

[0092] [9]Shi R,Chen H,Zhang Z,et al.Zero123++:a single image to consistentmulti-view diffusion base model[J].arXiv preprint arXiv:2310.15110,2023.

[0093]

[10] Wang Y, Yang W, Chen

[0094] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. An image editing method for rotating a single object in an image based on a diffusion model, characterized in that: An image editing network based on the diffusion model is proposed. The perspective pictures generated by image-3D technology are added to the image editing network as reference pictures. The noise sampled by the diffusion model is optimized by comparing the contrast loss between the edited pictures and the original pictures and the reference pictures, and finally the effect of target rotation is achieved.

2. The image editing method for rotating a single target in an image based on a diffusion model according to claim 1, characterized in that: To maintain the consistency of background content, region-based image filling is added to the sampling process of the diffusion model.

3. The image editing method for rotating a single target in an image based on a diffusion model according to claim 2, characterized in that: The perspective pictures generated by the image-3D technology are also processed with super-resolution.

4. The image editing method for rotating a single object in an image based on a diffusion model according to claim 3, characterized in that: The method is specifically implemented as follows: First, the user provides the initial image orl to be edited, and specifies the object to be rotated, the text title corresponding to the initial image, and the title corresponding to the object to be rotated; In the preparation stage, the image editing network segments the object to be rotated through image segmentation, and inputs it into the image-3D network to obtain the corresponding pictures of each viewpoint of the object; The user selects the desired perspective image, performs super-resolution processing, and obtains the reference image ref; In the editing stage, the image editing network passes the initial image orl and the reference image ref through the VAE encoder to obtain the corresponding latent space (Z0 orl , Z0 ref ), and then DDIM encoding is performed simultaneously, and the noisy version of the encoding process is retained (Z t orl , Z t ref ) and K in the self-attention mechanism of the upsampling layer t orl 、V t orl , K t ref 、V t ref ; Next, the initial image orl and the reference image ref are encoded using the pre-trained image encoder, and fused through the cross attention mechanism of U-net, just like the text encoding; The final DDIM encoding result of the initial image orl and the reference image ref (Z T orl , Z T ref ) is fused as the initial noise Z T ; The editing process uses DDIM sampling with a step length of 50 steps. In the first 30 steps, the sampled image Z t The intermediate feature F when passing through the 2nd and 3rd layers of the U-net upsampling layer t , and Z t orl The intermediate feature F corresponding to the number of steps t orl , and Z t ref The intermediate feature F corresponding to the number of steps t ref , compare the cosine similarity, get the loss, optimize Z t The noise ε predicted by U-net is iterated for 3 rounds at each step; the last 20 steps are normal DDIM sampling; and in each sampling process, the K in the self-attention mechanism of the U-net upsampling layer is added. t 、V t , replaced by K retained in the forward DDIM encoding t orl 、V t orl , K t ref 、V t ref The final sampling result Z0 is obtained, and the editing result is obtained through the decoder.

5. The image editing method for rotating a single object in an image based on a diffusion model according to claim 4, characterized in that: The noisy version of the t-th step of DDIM encoding (Z t orl , Z t ref ), and K in the self-attention mechanism of the upsampling layer t orl 、V t orl , K t ref 、V t ref , stored through the memory bank.

6. The image editing method for rotating a single object in an image based on a diffusion model according to claim 4, characterized in that: During the editing process, in each sampling step, the K in the self-attention mechanism of the U-net upsampling layer is t 、V t , replaced by K retained in the forward DDIM encoding t orl 、V t orl , K t ref 、V t ref , that is, using visual cross attention, connecting the initial image and the reference image in the length dimension K (K t orl , K t ref) and V(V t orl ,V t ref ); the visual cross-attention at each time step is defined as follows: Among them, the Attention function represents the self-attention mechanism in Unet; Q, K, and V are the query, key, and value generated by the image itself in the attention mechanism; the softmax function is a normal softmax operation, and d represents the number of dimensions.

7. The image editing method for rotating a single object in an image based on a diffusion model according to claim 4, characterized in that: The input image is passed through the image encoder to obtain feature vectors corresponding to 64 tags, which are then fed into the cross-attention mechanism of the same U-net as the text tags; unconditional image prompts are formed by setting the image to zero during generation; finally, the image prompt and text prompt are processed separately with the query Q in the cross-attention mechanism of the U-net; and the results are fused together; the process is as follows: Among them, the Attention function is the cross attention mechanism in Unet, Q is the query generated by the image itself, K and V are the keys and values ​​generated by the text prompt or image prompt; (K', V') and (K", V") are used to represent K and V in the text prompt and image prompt respectively, λ is the balancing weight, the softmax function is the normal softmax operation, and d represents the number of dimensions.

8. The image editing method for rotating a single object in an image based on a diffusion model according to claim 4, characterized in that: F t 、F t orl 、F t ref , compare the cosine similarity, get the loss, optimize Z t The noise ε predicted by U-net is obtained by two binary masks m orl and m rep To locate the position that needs to be optimized, the edited position of the target on the original image and the position of the target on the reference image are represented respectively. The loss of the two regions is obtained by cosine similarity. The loss function loss edit as follows: where sg(·) is the gradient clipping operation, α and β are two hyperparameters, and the S function is used to calculate the cosine similarity of two regions.

9. The image editing method for rotating a single object in an image based on a diffusion model according to claim 8, characterized in that: The specific implementation of area-based image filling is as follows: Since the shape of the rotated object may be different from the original object, when editing, the uncovered part of the original object is filled with the background of the initial image based on the area. gud The binary mask represents the original location of the object, m orl The binary mask represents the position of the object after rotation, (m gud -m orl ) represents the part that needs to be filled, which is divided into different areas m by straight lines overlap_0 、m overlap_1 Fill the respective background, m ref_0 、m ref_1 Indicates the location of the corresponding background; loss function loss ref as follows: Where ω is the weight parameter.

10. The image editing method for rotating a single object in an image based on a diffusion model according to claim 9, characterized in that: To maintain the consistency of the background content, m gud +m orl The complement of other To indicate the background part that needs to be consistent; loss function loss content as follows: Finally, the loss function loss is obtained by integration sum for: loss sum =ω e ·loss edit +ω r ·loss ref +ω c ·loss content where ω e ,ω r ,ω c is a hyperparameter that balances the losses; The loss function loss is obtained by the final integration sum Optimize Z t Noise ε predicted by U-net.

Images