An enhanced object manipulation and background consistent image editing method

Abstract

The application discloses an enhanced object manipulation and background consistent image editing method, which adopts a diffusion model to perform image editing on an original image, and mainly comprises the following steps: gradually performing inversion updating on an original latent code within a preset number of time steps, transferring an editing object from a source region to a target region at a specific time step, repairing a background of the source region after the transfer of the editing object, maintaining the background of an unedited region after the transfer of the editing object, iteratively updating a specific latent code by using a final loss function at the specific time step in the inversion stage, and finally obtaining an updated latent code; and inputting the updated latent code and an instruction text of an editing object action into a UNet denoiser to perform sampling denoising and obtain an edited image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an enhanced object manipulation and background consistent image editing method. BACKGROUND

[0002] Diffusion models have significantly improved the image generation capability in the field of text-to-image generation. By utilizing a large amount of training data, these models can generate excellent images according to user prompts. Compared with generative adversarial networks (GAN), diffusion models provide excellent quality and diversity in image generation. Therefore, many studies have begun to study diffusion models for image editing tasks. These methods mainly involve text-conditioned editing, that is, the user provides text input to modify the image. Although some techniques can skillfully edit the image style, object appearance, and object category according to the text input, they do not change the scene layout or modify the object action in the image. However, in some cases, the user wants to edit the action of the object while keeping its appearance and background. This specific editing process is called consistent image editing.

[0003] Imagic introduced a consistent image editing method, but it requires time-consuming fine-tuning of text embedding and models. Subsequently, MasaCtrl provided a fine-tuning-free method that converts self-attention into mutual self-attention to achieve the same goal. Nevertheless, these methods are limited to editing the action of the object and do not allow control of the generated position of the object. DragonDiffusion can achieve object displacement within the image, but it cannot edit the action of the moved object. In practical applications, the user sometimes needs to edit both the action and the position of the object. In addition, the previous methods also have defects in background preservation. Since these methods only focus on editing the action of the object in the image and ignore the preservation of the background of the image, the background generated by these methods shows a certain loss in details. SUMMARY

[0004] The present application aims to provide an enhanced object manipulation and background consistent image editing method, which can solve the above technical problems, can edit the action and position of the object in the image at the same time, and keep the consistency of the background before and after image editing, has the advantage of improving the user experience.

[0005] In order to achieve the above purpose, the solution of the present application is:

[0006] An enhanced object manipulation and background consistent image editing method, which adopts a diffusion model to edit an original image, the diffusion model comprising a VAE encoder, a VAE decoder and a UNet denoiser; the UNet denoiser comprises a self-attention module and a cross-attention module, and is characterized by comprising the following steps:

[0007] Step S1, obtaining image editing information, inputting the original image into the VAE encoder to encode the original latent code:

[0008] The image editing information includes an instruction text of an editing object action and a target bounding box of an editing object generation position;

[0009] Suppose the original image is an editing object, the initial position of the editing object is a source region, the position of the target bounding box is a target region of the editing object, and the complement of the union of the target region and the source region is an unedited region;

[0010] Step S2, gradually updating the original latent code in a preset number of time steps, and performing iterative updating on the specific latent code of a specific time step in the inversion stage as follows:

[0011] Step S21, moving the editing object from the source region to the target region at the specific time step:

[0012] The cross-attention map of the cross-attention module in the UNet denoiser is used to update the specific latent code, so that the cross-attention map corresponding to the editing object obtains the highest attention score within the range of the target region, and the cross-attention map corresponding to the editing object obtains the lowest attention score outside the range of the target region, thereby moving the editing object from the source region to the target region;

[0013] Step S22, repairing the source region background after the editing object is moved:

[0014] After moving the editing object to the target region, the source region of the editing object is located by the high response region of the cross-attention map, and at the same time, the union of the source region and the target region is subtracted from the source region to obtain a new source region. The new source region is dilated to obtain the edge region around the new source region. The features of the specific latent code in the new source region and the features of the original latent code in the edge region are extracted, and the L1 loss between the features of the specific latent code in the new source region and the features of the original latent code in the edge region is calculated to repair the source region background;

[0015] Step S23, keeping the unedited region background intact after the editing object is moved:

[0016] The features of the original latent code and the specific latent code in the unedited region are extracted, and the L1 loss between the features of the original latent code in the unedited region and the features of the specific latent code in the unedited region is calculated to keep the integrity of the unedited region;

[0017] Step S24: At a specific time step in the inversion phase, the final loss function is used to iteratively update the specific potential code to inject the edit object into the target area while maintaining the integrity of the background;

[0018] Step S3: Input the updated latent code and the instruction text of the edited object's action into the UNet denoiser for sampling and denoising, and modify the action of the transferred edited object throughout the sampling phase to obtain the final latent code;

[0019] Step S4: Input the final potential code into the VAE decoder to decode and obtain the edited image.

[0020] Furthermore, the specific time step is the 35th time step, the preset number is 50, the specific potential code is iterated and updated 50 times; and the resolution of the cross-attention map is 16×16.

[0021] Furthermore, in step S21, the cross-attention map of the cross-attention module in the UNet denoiser is updated to ensure that the cross-attention map corresponding to the edited object obtains the highest attention score within the target region and the cross-attention map corresponding to the edited object obtains the lowest attention score outside the target region, thereby transferring the edited object from the source region to the target region. Specifically:

[0022] First, to ensure that the cross-attention map corresponding to the edited object obtains the highest attention score within the target region, the following loss function is proposed. :

[0023] ;

[0024] in, A mask representing the target area. Indicates corresponding to the first Cross-attention map for each time step, The operation is used to extract the largest value within the target area. Attention score, For time steps;

[0025] Secondly, by minimizing the attention score of the cross-attention map in regions outside the target region, thus preventing the edited object from being generated outside the target region, the following loss function is proposed. :

[0026] ;

[0027] in, A mask representing the target area. Indicates corresponding to the first cross attention map of a time step, N denotes the number of pixels of the non-target region, is a time step,

[0028] injecting the editing object into the target region from the source region, the editing object injection loss function is updated to inject the editing object into the target region from the source region, the editing object injection loss function is:

[0029] .

[0030] Further, in step S22, the UNet denoiser of the diffusion model is used to extract the feature of the specific latent code in the new source region and the feature of the original latent code in the edge region, and a source region repair loss function is used to calculate the L1 loss between the feature of the specific latent code in the new source region and the feature of the original latent code in the edge region;

[0031] The source region repair loss function aligns the features in the edge region and the new source region by repeating and truncating the feature of the edge region:

[0032]

[0033] wherein, and denote the new source region and the edge region, respectively; denotes the feature of the original latent code, denotes the feature of the specific latent code, is a repeating and truncating operation, is a time step.

[0034] Further, in step S23, the UNet denoiser of the diffusion model is used to extract the features of the original latent code and the specific latent code in the unedited region, and a background preservation loss function is used to calculate the L1 loss between the feature of the original latent code in the unedited region and the feature of the specific latent code in the unedited region, to preserve the integrity of the unedited region;

[0035] The background preservation loss function is:

[0036]

[0037] wherein, denotes the unedited region, denotes the feature of the original latent code, denotes the feature of the specific latent code, is a time step.

[0038] Further, in step S24, the final loss function is employed in a specific time step of the sampling phase to update the specific latent code to inject the edited object in the target region while keeping the background integrity, the final loss function is:

[0039] ;

[0040] wherein, , and are balance parameters; by using the final loss function , the specific latent code is updated with a step size of :

[0041] ;

[0042] As the number of update steps increases, the step size decays linearly, is the time step.

[0043] Further, in step S3, the updated latent code and the instruction text of the edited object action are input into the UNet denoiser for sampling denoising, and the action of the edited object is modified throughout the sampling phase to obtain the final latent code; specifically:

[0044] In the sampling phase, the UNet denoiser is used to gradually denoise the updated latent code in the same preset number of time steps as the inversion phase, and the cross-attention module of the UNet denoiser is used to integrate the instruction text of the edited object action into the sampling phase; and starting from the time step of the sampling phase, the key vector and the value vector of the self-attention module in the UNet denoiser are replaced by the key vector and the value vector in the corresponding time step of the inversion phase;

[0045] The process of replacing the key vector and the value vector of the self-attention module with the key vector and the value vector in the corresponding time step of the inversion phase is represented as:

[0046] ;

[0047] wherein, Q, K, V are the query vector, the key vector and the value vector in the self-attention module, denotes a certain self-attention layer in the UNet denoiser, and denote the key vector and the value vector in the corresponding time step of the inversion phase;

[0048] When the time step t of the sampling phase more than when, and self-attention layers are part of a predefined set , the key and value vectors K and V in the self-attention module are replaced by and .

[0049] Further, the decoder of the UNet denoiser consists of four blocks of different scales, wherein the third layer contains more geometric information; in step S22, the third layer is selected to extract features from the original latent code and the specific latent code.

[0050] The application also provides a computer system comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the enhanced object manipulation and background consistent image editing method.

[0051] The application also provides a computer readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by a processor to implement the enhanced object manipulation and background consistent image editing method.

[0052] After adopting the above technical solution, since the previous method only focuses on the action of editing an object in an image, and no specific method is proposed to maintain the consistency of the background before and after image editing, these methods also have defects in background maintenance. Compared with the previous method, the application can inject an editing object into a target region and maintain the integrity of the background in the inversion stage, and can ensure that the edited object produces the action specified by the instruction text in the sampling stage, thereby ensuring the consistency of the image content before and after editing. The results produced by the method of the application are more consistent with the user input editing text and the surrounding environment. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a method step flowchart of the application;

[0054] Figure 2 is a flowchart of the application;

[0055] Figure 3 is a schematic diagram of the application;

[0056] Figure 4 is a schematic diagram of positioning a source region through a cross-attention map and obtaining an edge region through an inflation operation;

[0057] Figure 5 is the editing result (I) of the present application;

[0058] Figure 6 is the editing result (II) of the present application;

[0059] Figure 7 is the comparison result (I) of the method of the present application and the previous method when only editing the action of the object;

[0060] Figure 8 is the comparison result (II) of the method of the present application and the previous method when only editing the action of the object;

[0061] Figure 9 is the comparison result (III) of the method of the present application and the previous method when only editing the action of the object. DETAILED DESCRIPTION

[0062] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments of the present application.

[0063] In the field of image editing, in actual application, a user sometimes needs to edit the action and position of an object in an image at the same time. In order to solve this problem, the present application proposes an enhanced object manipulation and background consistent image editing method, which is based on a diffusion model for image editing. The diffusion model is used for image editing of an original image, and the diffusion model includes a VAE encoder, a VAE decoder and a UNet denoiser. The UNet denoiser includes a self-attention module and a cross-attention module.

[0064] First, the object information is transferred from the source area to the target area and the background is repaired at a specific time step in the inversion stage, then the action of the object information is modified in the whole sampling stage, and finally the edited image is output.

[0065] Specifically, as shown in the figure, the image editing method includes the following steps: Figures 1 to 3

[0066] Step S1, obtaining image editing information, inputting an original image into a VAE encoder to obtain an original latent code:

[0067] The image editing information includes an editing object action indication text and an edited object generated position target bounding box.

[0068] In this embodiment, the action editing indication can be an editing text of natural language input by a user, for example Figure 2 ​The editing prompt shown is: A running bear. The editing instruction of the generated position can specify a target bounding box for the user, and the original image is referred to as the source region of the editing object, the target bounding box specified by the user is referred to as the target region of the editing object, and the complement of the union of the target region and the source region is referred to as the unedited region (equivalent to the background region).

[0069] Step S2, the original latent code is updated step by step within a preset number of time steps, that is, a noise is added to the original latent code, and the specific latent code at a specific time step in the inversion stage is updated iteratively as follows:

[0070] Step S21, the editing object is transferred from the source region to the target region at the specific time step:

[0071] The cross-attention map of the cross-attention module in the UNet denoiser is used to update the specific latent code, so that the cross-attention map corresponding to the editing object obtains the highest attention score within the range of the target region, and the cross-attention map corresponding to the editing object obtains the lowest attention score outside the range of the target region, thereby transferring the editing object from the source region to the target region;

[0072] Step S22, repairing the background of the source region after the transfer of the editing object:

[0073] After the editing object is transferred to the target region, the source region of the editing object is located by the high response region of the cross-attention map, and then an inflation operation is performed on the mask corresponding to the source region to obtain the edge region around the source region. The features of the specific latent code in the new source region and the features of the original latent code in the edge region are extracted, and the L1 loss between the features of the specific latent code in the new source region and the features of the original latent code in the edge region is calculated to repair the background of the source region.

[0074] Step S23, keeping the background of the unedited region after the transfer of the editing object intact:

[0075] The features of the original latent code and the specific latent code in the unedited region are extracted, and the L1 loss between the features of the original latent code in the unedited region and the features of the specific latent code in the unedited region is calculated to keep the unedited region intact.

[0076] Step S24, iteratively updating the specific latent code at the specific time step in the inversion stage using the final loss function to inject the editing object into the target region and keep the background intact.

[0077] Step S3, input the updated latent code and the instruction text of the editing object action into the UNet denoiser for sampling denoising, and modify the action of the transferred editing object in the whole sampling stage to obtain a final latent code;

[0078] Step S4, input the final latent code into the VAE decoder for decoding to obtain an edited image.

[0079] In this embodiment, the specific time step can be the 35th time step, the preset number can be 50, and the iteration number of updating the specific latent code can be 50 times; the resolution of the cross attention map can be 16x16.

[0080] The specific steps of the method of the present application are as follows:

[0081] (I) editing object transfer

[0082] In step S21, in order to generate an editing object in a target region, the present application updates a specific latent code by using a cross attention map of a cross attention module in a UNet denoiser at the 35th time step in the inversion stage, and adopts an editing object injection loss function to inject an editing object into the target region.

[0083] The present application uses a 16x16 resolution cross attention map to update the latent code, because the cross attention map of this specific resolution is rich in semantic information. In order to inject an editing object into a target region, the present application ensures that the cross attention map corresponding to the editing object obtains the highest attention score within the range of the target region, i.e. maximizes the attention score within the range of the target region, and proposes the following loss function :

[0084] (1)

[0085] wherein represents a mask of the target region, represents a cross attention map corresponding to the th time step, wherein . The operation is used to extract the maximum attention scores within the target region.

[0086] Correspondingly, in order to prevent the edited object from being generated outside the target region, the present application minimizes the attention scores of the regions outside the target region, ensures an accurate and controlled editing process, and proposes the following loss function :

[0087] (2)

[0088] wherein A mask representing the target area. Indicates corresponding to the first The cross-attention graph at each time step, where ;

[0089] In summary, the edit object injection loss function of the present invention can be obtained. for:

[0090] (3)

[0091] (II) Source Area Restoration

[0092] In step S22, after transferring the object to be edited to the target region, the source region needs to be repaired. This invention uses the high-response region of the cross-attention map to locate the source region of the object being edited, such as... Figure 3 As shown. Meanwhile, to avoid a conflict between source region background restoration and maximizing the attention score of the target region, this invention subtracts the union of the source and target regions from the original source region to obtain a new source region. This new source region is used as the actual source region for background restoration.

[0093] Typically, the background of the source region is similar to the background surrounding the source region. To achieve this, the present invention first performs a dilation operation on the mask corresponding to the new source region, thereby obtaining the region surrounding the new source region, such as... Figure 4 As shown. In this invention, the extended surrounding area is referred to as the edge region.

[0094] To reconcile the background of the new source region with the background of the edge region, this invention extracts features from these two regions of the latent code. Specifically, it extracts features of a specific latent code in the new source region and features of the original latent code in the edge region, then calculates the L1 loss between these two regions' features, and employs the source region insulation loss function. Background restoration of the source region is performed. Since the UNet denoiser in the diffusion model is an efficient feature extractor, this invention utilizes the UNet denoiser to extract features from the latent code. The UNet denoiser's decoder consists of four blocks at different scales, with the second layer containing more semantic information and the third layer containing more geometric information. In the method of this invention, the third layer is selected. Since the edge region and the source region are not necessarily of equal size, this invention repeats and truncates the features of the edge region to align the features in the two regions. Source region restoration loss function. for:

[0095] (4)

[0096] in and denote the new source region and the edge region, respectively. denote the features of the original latent code, denote the features of the specific latent code, where . are the repetition and truncation operations.

[0097] (Three) Background Preservation

[0098] In step S23, when relocating the editing object from the source region to the target region, the present application observes that the injection of object information and the subsequent repair of the source region can inadvertently cause damage to the image background. To solve this problem and preserve the integrity of the unedited region, the present application proposes a background preservation loss function .

[0099] With the rich image structure and texture information contained in the intermediate features of the UNet denoiser, the present application uses the UNet denoiser to extract features from the latent code. Subsequently, the present application calculates the L1 loss between the features of the original latent code in the unedited region and the features of the specific latent code in the unedited region, i.e., the background preservation loss function is:

[0100] (5)

[0101] where denotes the unedited region. The lack of may cause the loss of some background information in the edited image. By introducing , the present application ensures the consistency between the background region before and after editing. At the same time, the object information in the unedited region is suppressed, ensuring that the edited object more closely follows the user-specified target bounding box constraints.

[0102] (Four) Code Update

[0103] In step S24, the final loss function that injects the editing object into the target region and preserves the background integrity at the 35th time step of the inversion stage is:

[0104] (6)

[0105] where , and are balance parameters. By using the final loss function , the specific latent code is updated with a step size of :

[0106] (7)

[0107] With the increase of the update steps, the step size decays linearly, is the time step. This process can ensure a smooth and consistent image editing process, maintaining the background of the unedited area and generating the edited object to the target area.

[0108] (Five) consistent image editing

[0109] In step S3, the updated latent code is obtained by the inversion stage After that, the present application takes it as the starting point of the reverse denoising process, and inputs the editing instructions of the action in the sampling stage to modify and edit the action of the edited object, and finally outputs the edited image through the decoder.

[0110] In the sampling stage, the UNet denoiser gradually denoises the updated latent code in the same preset number of time steps as the inversion stage, and in order to ensure that the edited object produces the action specified by the instruction text of the action, the present application uses the cross-attention module of UNet to integrate the editing instructions of the action into the sampling stage. In addition, in order to ensure the consistency of the image content before and after editing, starting from the first time step of the sampling stage, the present application replaces the key vector and value vector in the self-attention module with the key vector and value vector in the corresponding time step of the inversion stage.

[0111] Unlike previous methods, the present application queries the key vector and value vector of the inversion stage, because the present application believes that the key vector and value vector of this stage contain richer object appearance and image background information compared to the sampling stage. In addition, using the key vector and value vector of the inversion stage reduces the image reconstruction cost.

[0112] Therefore, the optimization process of the present application changes the original three branches of inversion, reconstruction and editing into two branches: inversion and editing. The exchange process of the optimized key vector and value vector can be represented as:

[0113] (8)

[0114] where and represent the key and value of the corresponding time step of the inversion stage.

[0115] where Q, K, V are the query vector, key vector and value vector in the self-attention module, represents a self-attention layer in the UNet denoiser, and represent the key vector and value vector of the corresponding time step of the inversion stage.

[0116] When the time step t of the sampling stage exceeds and the self-attention layer is part of a predefined set , the key vector K and the value vector V in the self-attention module are replaced by and .

[0117] In summary, as shown in Figure 5 and Figure 6 , the left is the original image input, the middle image is the boundary box provided by the user, and the right is the edited image generated according to the boundary box and the editing text input by the user; the present application allows the user to input editing text to edit the action of the object, and also allows the user to provide a boundary box to specify the generation position of the edited object. Compared with the previous method, the present application provides more controllable and consistent image editing function.

[0118] In addition, since the previous Imagic, MasaCtrl, InstructPix2Pix and other methods only focus on the action of an object in the edited image, and do not propose a specific method to maintain the consistency of the background before and after image editing, these methods also have defects in background maintenance.

[0119] As shown in Figure 7 , Figure 8 and Figure 9 , by specifying the generation position of the object as its original position, the editing result generated by the method of the present application is compared with the previous method, wherein (a) is the original image, and (b)-(e) are edited images of the original image processed by InstructPix2Pix, PnP, MasaCtrl, and the method of the present application, respectively. The background generated by the previous method shows a certain loss in detail, while the background generated by the method of the present application is closer to the original image. At the same time, compared with the previous method, the result generated by the method of the present application is more consistent with the editing text input by the user, and also more consistent with the surrounding environment.

[0120] The present application provides a computer system, which comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to realize the enhanced object manipulation and background consistent image editing method.

[0121] The application further provides a computer readable storage medium, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to realize the enhanced object manipulation and background consistent image editing method.

Claims

1. An enhanced object manipulation and background consistent image editing method, using a diffusion model for image editing of an original image, the diffusion model comprising a VAE encoder, a VAE decoder and a UNet denoiser; the UNet denoiser comprising a self-attention module and a cross-attention module, characterized in that The method comprises the following steps: Step S1, obtaining image editing information, and inputting an original image into a VAE encoder to encode an original latent code: The image editing information comprises an instruction text of an editing object action and a target bounding box of an editing object generation position; Supposing an initial position of the editing object in the original image is a source region, a position of the target bounding box is a target region of the editing object, and a complement of a union of the target region and the source region is an unedited region; Step S2, performing iterative updating on a specific latent code at a specific time step in an inversion stage, and finally obtaining an updated latent code: Step S21, transferring the editing object from the source region to the target region at the specific time step: The cross-attention map of the cross-attention module in the UNet denoiser is used to update the specific latent code, so that the cross-attention map corresponding to the editing object obtains the highest attention score within the range of the target region and the lowest attention score outside the range of the target region, thereby transferring the editing object from the source region to the target region; Step S22, repairing a background of the source region after the editing object is transferred: After the editing object is transferred to the target region, the source region of the editing object is located through a high-response region of the cross-attention map, and a new source region is obtained by subtracting the union of the source region and the target region from the source region; an edge region around the new source region is obtained by performing an inflation operation on a mask corresponding to the new source region; the background of the source region is repaired by extracting features of the specific latent code in the new source region and features of the original latent code in the edge region, and calculating an L1 loss between the features of the specific latent code in the new source region and the features of the original latent code in the edge region; Step S23, keeping the background of the unedited region after the editing object is transferred intact: The features of the original latent code and the specific latent code in the unedited region are extracted, and an L1 loss between the features of the original latent code in the unedited region and the features of the specific latent code in the unedited region is calculated to keep the unedited region intact; Step S24, performing iterative updating on the specific latent code at the specific time step in the inversion stage by using a final loss function to inject the editing object into the target region and keep the background intact; Step S3, inputting the updated latent code and the instruction text of the editing object action into the UNet denoiser to perform sampling denoising, and modifying the action of the transferred editing object during the entire sampling stage to obtain a final latent code; Step S4, inputting the final latent code into a VAE decoder to decode an edited image.

2. The enhanced object manipulation and background consistent image editing method of claim 1, wherein: The specific time step is the 35th time step, the preset number is 50, the number of times of iterative updating of the specific latent code is 50, and the resolution of the cross-attention map is 16x16.

3. The enhanced object manipulation and background consistent image editing method of claim 1 or 2, wherein: In step S21, the cross-attention map of the cross-attention module in the UNet denoiser is used to update the specific latent code, so that the cross-attention map corresponding to the editing object obtains the highest attention score within the target region range and obtains the lowest attention score outside the target region range, thereby transferring the editing object from the source region to the target region, specifically: First, in order to ensure that the cross-attention map corresponding to the editing object obtains the highest attention score within the range of the target region, the following loss function is proposed : ； wherein, a mask representing a target region, a cross-attention map corresponding to the i-th time step, the i-th time step, operating to extract the largest K attention scores within the target region, K attention scores within the target region, is a time step; Second, by minimizing the attention score of the cross-attention map in the region outside the target region, the edited object is not generated outside the target region, and the following loss function is proposed : ； wherein, a mask representing a target region, a cross-attention map corresponding to the i-th time step, N is the number of pixels of non-target regions, is the time step; injecting an edit object into the loss function updating the particular latent code to inject the edit object from the source region into the target region, the edit object injection loss function is: 。 4. The enhanced object manipulation and background consistent image editing method of claim 3, wherein: In step S22, the UNet denoiser with diffusion model extracts the features of the specific latent code in the new source region and the features of the original latent code in the edge region, and calculates the L1 loss between the features of the specific latent code in the new source region and the features of the original latent code in the edge region by using a source region repair loss function ​ The source region repair loss function Aligning the features in the edge region and the new source region by repeating and truncating the features of the edge region: wherein, and denote the new source and edge regions, respectively; denote the features of the original latent code, denote the features of the specific latent code, are the repetition and truncation operations, is the time step.

5. The enhanced object manipulation and background consistent image editing method of claim 4, wherein: In step S23, the UNet denoiser with diffusion model extracts the features of the original latent code and the specific latent code in the unedited region, and adopts a background preservation loss function The L1 loss between the features of the original latent code in the unedited region and the features of the specific latent code in the unedited region is calculated to preserve the integrity of the unedited region; The background preserving loss function is: wherein, represents an unedited region, represents a feature of the original latent code, represents a feature of the particular latent code, is a time step.

6. The enhanced object manipulation and background consistent image editing method of claim 5, wherein: In step S24, the final loss function is employed in a particular time step of the inversion phase to update the particular latent code to inject the edit object in the target region while preserving the background integrity, the final loss function is: ； in, , and It is a balancing parameter; by using the final loss function Specific potential code by Update the step size: ； As the number of updates steps increases, the step size decays linearly, is the time step.

7. The enhanced object manipulation and background consistent image editing method of claim 6, wherein, In step S3, the updated latent code and the instruction text of the editing object action are input into the UNet denoiser for sampling denoising, and the action of the transferred editing object is modified during the entire sampling stage to obtain the final latent code. Specifically: In the sampling stage, the UNet denoiser is used to denoise the updated latent code step by step in the same preset number of time steps as in the inversion stage, while the cross-attention module of the UNet denoiser is used to integrate the instruction text of the editing object action into the sampling stage; and from the first time step of the sampling stage, the key vector and the value vector of the self-attention module in the UNet denoiser are replaced by the key vector and the value vector in the corresponding time step of the inversion stage. The process of replacing the key vector and value vector of the self-attention module with the key vector and value vector at the corresponding time step in the inversion stage is represented as: ； where Q, K, V are the query, key and value vectors in the self-attention module, respectively, denotes a certain self-attention layer in the UNet denoiser, and denotes the key and value vectors at the respective time step of the inversion phase. When the time step t of the sampling phase exceeds and the self-attention layer is part of a predefined set , the key vector K and the value vector V in the self-attention module are replaced by and .

8. The enhanced object manipulation and background consistent image editing method of claim 1 or 4, wherein: The decoder of the UNet denoiser is composed of four blocks of different scales, and the third layer contains more geometric information; in step S22, the third layer is selected to extract features from the original latent code and the specific latent code.

9. A computer system, characterized by The computer system includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to implement the enhanced object manipulation and background consistent image editing method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to implement the enhanced object manipulation and background consistent image editing method according to any one of claims 1 to 8.

Images