Image editing method and system based on text-to-image large model

Abstract

The application provides an image editing method and system based on a text-to-image large model. The method includes inputting text describing a source image and an edited image and a mask of a target editing area by a user, using the text-to-image large model to edit the image, and performing optimization processing on a latent space noise graph within a preset time step in a latent space noise graph denoising process. The optimization processing includes: first, inputting the latent space noise graph of the current step and the text prompt into a noise prediction network; then, extracting a cross-attention graph and a self-attention graph generated in the noise prediction network, and calculating cross-attention consistency constraints and self-attention reservation constraints; second, calculating gradients of the cross-attention consistency constraints and the self-attention reservation constraints on the latent space noise graph through back propagation, respectively, and updating the latent space noise graph using an adaptive time step scheduler. The application can balance the fidelity and editing effect of the target image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image editing in the field of artificial intelligence technology, more specifically, to an image editing method and system based on a text-to-image large model. BACKGROUND

[0002] Image editing has always been an important means of image processing and aesthetic expression for people, but traditional manual editing methods require a lot of time and effort. With the continuous development of computer technology, deep learning technology and large-scale data sets, text-to-image has become a research field that has attracted much attention due to its efficiency, flexibility and scalability, and has provided the possibility for automated image editing.

[0003] In text-based image editing (TIE), it is crucial to balance fidelity and editability, and the failure to balance the two often leads to over-editing or under-editing. Existing methods usually rely on injecting cross-attention maps or self-attention maps to maintain structure and using the inherent text alignment function of pre-trained text-to-image models to achieve editability, but they lack a clear and unified mechanism to appropriately balance the two goals.

[0004] Therefore, how to balance fidelity and editability in a unified framework is one of the research directions in the field of artificial intelligence image editing. SUMMARY

[0005] In view of the above problem that the existing image editing method cannot balance editability and fidelity in a unified framework, the purpose of the present application is to provide an image editing method and system, which uses an adaptive time scheduler to optimize the latent space noise map, so that the edited image can achieve the target effect while maintaining the overall image structure.

[0006] The image editing method based on a text-to-image large model provided by the present application comprises the following steps:

[0007] S110: obtaining a latent space noise map of a source image based on text-to-image input data; wherein the text-to-image input data comprises the source image, a mask of a target editing region, a text description of the source image, and a text description of an edited image;

[0008] S120: denoising the latent space noise map, and performing optimization processing of the latent space noise map within a preset time step of the denoising processing;

[0009] S130: decoding the latent space noise map obtained in the last step of the denoising processing to obtain an edited image;

[0010] The optimization processing of the latent space noise map comprises:

[0011] S121: input the latent space noise map of the current time step and the text embedding of the text description of the edited image into a preset noise prediction network for noise prediction;

[0012] S122: extract the cross attention map and the self attention map generated in the noise prediction network;

[0013] S123: calculate the mask-based cross attention consistency constraint and the self attention preservation constraint using the cross attention map and the self attention map, respectively;

[0014] S124: calculate the gradient of the latent space noise map with respect to the cross attention consistency constraint and the self attention preservation constraint by backpropagation, respectively, and update the latent space noise map using the adaptive time step scheduler and the gradient.

[0015] Optionally, the cross attention map includes a cross attention map corresponding to the edited token describing the edited image, and the self attention map includes all self attention maps of a preset resolution generated in the denoising network.

[0016] Optionally, the denoising processing of the latent space noise map includes a reconstruction branch and an editing branch; wherein,

[0017] the reconstruction branch is used to perform noise prediction using the text embedding of the text description of the source image and the latent space noise map at each time step of the denoising processing;

[0018] the editing branch is used to guide the latent space noise map at a preset time step of the denoising processing; wherein,

[0019] at a certain time step of the denoising processing, the latent space noise map of the reconstruction branch and the latent space noise map of the editing branch are mixed using a mask.

[0020] Optionally, the loss function of the self attention preservation constraint is as follows:

[0021] ,

[0022] wherein, is the self attention preservation constraint, is the current time step, is the self attention map generated in the reconstruction branch, is the self attention map generated in the editing branch; the mask is used to calculate the loss function of the local self attention preservation constraint to preserve high-frequency details:

[0023] ,

[0024] wherein, is a local self-attention preservation constraint, , is flattened resulting vector.

[0025] wherein, optionally, the loss function of the cross-attention consistency constraint is calculated as follows:

[0026] ,

[0027] wherein, is a cross-attention consistency constraint, represents the ratio of the cross-attention graph corresponding to the edit token inside and outside the mask, represents the total number of cross-attention graphs with a resolution of 16X16; wherein,

[0028] ,

[0029] wherein, represents the spatial coordinates of the cross-attention graph, represents the layer number where the cross-attention graph is located, represents the index corresponding to the edit token, represents the cross-attention graph generated in the edit branch.

[0030] wherein, optionally, the latent space noise map is updated using an adaptive time step scheduler and the gradient, comprising: based on the gradient of the latent space noise map with respect to the cross-attention consistency constraint and the gradient of the latent space noise map with respect to the self-attention preservation constraint, calculating a total gradient using the adaptive time step scheduler, updating the latent space noise map according to the total gradient.

[0031] wherein, optionally, the updating of the latent space noise map according to the total gradient is calculated as follows:

[0032] ,

[0033] wherein, is the updated latent space noise map, is the latent space noise map before updating, is the total gradient;

[0034] ,

[0035] wherein, and are adaptive time step schedulers respectively acting on the self-attention preservation constraint and the cross-attention consistency constraint, a gradient of the latent space noise map in the editing branch is reserved for the self-attention preservation constraint, a gradient of the latent space noise map is reserved for the cross-attention consistency constraint;

[0036]

[0037] wherein T is the total number of steps of the denoising process, is the current time step, is a hyperparameter that controls the adaptive time step scheduler.

[0038] wherein an optional solution is that the formula for mixing the latent space noise map of the reconstruction branch and the latent space noise map of the editing branch using a mask is as follows:

[0039]

[0040] wherein, is the latent space noise map after the mixing operation, is the latent space noise map of the editing branch, is the mask, is the latent space noise map of the reconstruction branch.

[0041] The application also provides an image editing method based on a text-to-image large model, which uses the image editing method based on the text-to-image large model as described above to edit an image, comprising:

[0042] a latent space noise map acquisition unit configured to acquire a latent space noise map of a source image based on text-to-image input data; wherein the text-to-image input data comprises the source image, a mask of a target editing region, a text description of the source image, and a text description of an edited image;

[0043] a denoising unit configured to perform denoising processing on the latent space noise map;

[0044] an optimization unit configured to perform optimization processing on the latent space noise map within a preset time step in the denoising processing;

[0045] a decoding unit configured to decode the latent space noise map obtained at the last step of the denoising processing to obtain an edited image;

[0046] wherein the optimization unit comprises:

[0047] a noise prediction unit configured to input a text embedding of a text description of the edited image and a latent space noise map at a current time step into a preset noise prediction network to perform noise prediction; ​​​​​

[0048] a cross-attention map and a self-attention map extraction unit configured to extract a cross-attention map and a self-attention map generated in the noise prediction network;

[0049] a constraint calculation unit configured to calculate a cross-attention consistency constraint and a self-attention preservation constraint based on the cross-attention map and the self-attention map, respectively;

[0050] an update unit configured to calculate gradients of the latent noise map with respect to the cross-attention consistency constraint and the self-attention preservation constraint, respectively, and update the latent noise map using an adaptive time scheduler and the gradients.

[0051] The present application also provides an electronic device, comprising:

[0052] at least one processor; and,

[0053] a memory connected in communication with the at least one processor; wherein,

[0054] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the steps in the image editing method based on the text-to-image large model as described above.

[0055] From the above technical solutions, the image editing method and system based on the text-to-image large model provided by the present application can edit an image by inputting text describing a source image and an edited image and a mask of a target editing region by a user and using a text-to-image large model; wherein, in the preset time step of the latent noise map denoising process, the present application first performs optimization processing of the latent noise map, which includes: first, inputting the latent noise map of the current step and the text prompt into the noise prediction network; second, extracting a cross-attention map and a self-attention map generated in the noise prediction network, and calculating a cross-attention consistency constraint and a self-attention preservation constraint based on the cross-attention map and the self-attention map, respectively;

[0056] Finally, the gradients of the latent noise map with respect to the cross-attention consistency constraint and the self-attention preservation constraint are calculated by backpropagation, and the latent noise map is updated using an adaptive time step scheduler and the gradients. In addition, the latent noise map of the reconstruction branch and the latent noise map of the editing branch can be mixed using a mask within a certain time step of the denoising process, so that the edited image achieves the target effect and maintains the image structure of the background. BRIEF DESCRIPTION OF DRAWINGS

[0057] Other objects and results of the present application will become more fully understood from the following detailed description, taken in conjunction with the accompanying drawings, wherein:

[0058] Figure 1 A flowchart of the image editing method based on the text-to-image large model according to an embodiment of the present application;

[0059] Figure 2 A schematic diagram of the principle of the image editing method based on the text-to-image large model according to an embodiment of the present application;

[0060] Figure 3 A schematic diagram of the framework of the system based on the text-to-image large model according to an embodiment of the present application;

[0061] Figure 4 A schematic diagram of the electronic device according to an embodiment of the present application;

[0062] Figure 5 A comparison diagram of the effects of applying the present application and other existing image editing schemes;

[0063] Figure 6 An effect example diagram of image editing using the image editing method and system based on the text-to-image large model according to the present application. DETAILED DESCRIPTION

[0064] In the following description, for the purpose of providing a thorough understanding of one or more embodiments, numerous specific details are set forth. It is apparent, however, to one skilled in the art that these embodiments can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate a description of one or more embodiments.

[0065] To solve the problem that fidelity and editability cannot be uniformly modeled in existing image editing schemes, the present application provides an image editing method and system based on a text-to-image large model, which.

[0066] In order to better illustrate the technical solutions of the present application, the following will first briefly explain some technical terms involved in the present application.

[0067] A text-to-image model is a model that generates corresponding pictures from simple text descriptions (English). Existing text-to-image models include the text-to-image large model DALL-E2 and Stable Diffusion, etc. Stable Diffusion is a generative large model in the field of computer vision, which can perform text-to-image (txt2img) and image-to-image (img2img) image generation tasks.

[0068] Null-text Inversion, the sequel of prompt-to-prompt, is used to solve the problem of how to apply prompt-to-prompt to real images.

[0069] The noise prediction network is used to process different modal inputs with a transformer structure backbone network to predict noise, such as using a U-net network to predict the noise of each step. U-Net is a semantic segmentation algorithm based on a fully convolutional network, which uses a symmetric U-shaped structure and mirror operation to process boundary pixels.

[0070] Cross-attention is a mechanism used in the noise prediction U-net network structure of Stable Diffusion, which allows one sequence to "focus" on another sequence.

[0071] The back propagation (BP) algorithm calculates the gradient of the loss function with respect to each parameter in the neural network using the derivative chain rule, and updates the parameters with an optimization method to reduce the loss function.

[0072] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0073] It should be noted that the description of the following exemplary embodiments is actually only illustrative, not as any limitation on the present application and its application or use. Techniques and equipment known to those skilled in the related art may not be discussed in detail, but in appropriate cases, the techniques and equipment should be considered part of the specification.

[0074] To illustrate the image editing method and system provided by the present application, Figure 1 , Figure 2 The flow and principle of the image editing method based on text-to-image large model according to the embodiments of the present application are exemplarily indicated respectively; Figure 3 The logical structure of the image editing system based on text-to-image large model according to the embodiments of the present application is exemplarily indicated.

[0075] As Figure 1 and Figure 2 indicated together, the image editing method based on text-to-image large model provided by the present application mainly includes the following steps:

[0076] S110: Obtain a latent space noise map of a source image based on text-to-image input data; wherein the text-to-image input data includes the source image, a mask of a target editing region, a text description of the source image, and a text description of an edited image;

[0077] S120: denoising the latent space noise map, and performing optimization processing on the latent space noise map in a preset time step in the denoising processing;

[0078] S130: decoding the latent space noise map obtained at the last step of the denoising processing to obtain an edited image.

[0079] Specifically, as an example, the text-to-image large model based on the present application can use the large model provided by Stable-Diffusion, such as stable-diffusion-v1-4. For the convenience of description, in the following embodiments, Stable-Diffusion is taken as an example of implementation of the text-to-image large model.

[0080] The above steps of the image editing method based on the text-to-image large model will be described in more detail in combination with specific embodiments.

[0081] For step S110, the specific method of obtaining the latent space noise map of the source image based on the text-to-image input data is as follows:

[0082] For the input source image, the mask of the target editing region , for the text description of the source image , for the text description of the edited image , first, the encoder of Stable-Diffusion can be used to perceptually compress the source image to obtain the latent space representation of the source image ; then, based on the obtained latent space representation of the source image, an image inversion method such as Null-text Inversion is used to obtain the latent space noise map of the source image .

[0083] After obtaining the latent space noise map of the source image, the U-net network of Stable-Diffusion can be used to predict the noise, so as to denoise the latent space noise map of the source image, and perform optimization processing on the latent space noise map in a preset time step in the denoising process. The U-net network is a neural network including a decoder, an encoder, and a bottleneck layer. The encoder reduces the feature map through convolution and pooling to extract features. The decoder is symmetrical with the encoder part, and the features are reserved through feature mapping cascade after upsampling and convolution with the encoding part. The bottleneck layer contains two 3 × 3 convolution layers. Finally, a 1 × 1 convolution layer is used to obtain the final output.

[0084] In an embodiment of the present application, the time step of the above denoising processing is 50 steps, and the preset time step of the optimization processing of the latent space noise map is 25 steps. That is, through the 50-step cycle denoising processing, and in the [50, 25] time step in the denoising processing, the optimization processing of the latent space noise map is added; the optimization processing is performed first, and then the denoising processing is performed after the optimization processing is completed, so as to better remove the noise in the latent space noise map and better edit the image reaching the target effect.

[0085] The optimization processing proposed in the present application is equivalent to saying that the t=[50, 0] denoising processing is a large framework of stable diffusion, and the optimization processing of the latent space noise map is performed before the t=[50, 25] denoising processing in the large framework. Specifically, as an example, the optimization processing of the latent space noise map includes:

[0086] S121: inputting the text embedding of the text description of the current time step of the latent space noise map and the edited image into a preset noise prediction network to perform noise prediction;

[0087] S122: extracting the cross-attention map and the self-attention map generated in the noise prediction network;

[0088] S123: calculating the mask-based cross-attention consistency constraint and the self-attention reservation constraint by using the cross-attention map and the self-attention map, respectively;

[0089] S124: calculating the gradient of the latent space noise map with respect to the cross-attention consistency constraint and the self-attention reservation constraint by back propagation, respectively, and updating the latent space noise map by using an adaptive time step scheduler.

[0090] The number of times of executing the above optimization processing is not specifically limited in the present application, which can be executed multiple times, or 1 time according to the optimization effect. In an embodiment of the present application, the above optimization processing is executed 1 time.

[0091] From the overall framework of the denoising processing, the entire denoising process of the present application includes two branches: a reconstruction branch and an editing branch. In the following embodiments, the latent space noise map of the reconstruction branch is denoted as z without an asterisk, and the latent space noise map in the editing branch is denoted as . Among them, for each step of the denoising processing process, the reconstruction branch is used to input the text embedding of the text description of the source image and the latent space noise map into the U-net network to predict the noise for noise prediction, and the editing branch is used to perform the optimization processing of the latent space noise map in a certain time step of the denoising process.

[0092] In other words, the reconstruction branch and the editing branch are two separate 50-step denoising processes, and the two branches are performed simultaneously. The optimization process performed in a loop is applied in the editing branch to make the latent noise map features in the mask more consistent with the edited text, and the structure of the latent noise map more similar to the structure of the reconstruction branch.

[0093] More specifically, as an example, the optimization process on the latent noise map includes the following operations:

[0094] In the reconstruction branch part, the latent noise map of the current step is input into the U-net network together with the text description of the edited image and the text embedding of the text description .

[0095] Five cross-attention maps with a resolution of generated in the U-net network, and self-attention maps with a resolution of , , , or self-attention maps with a resolution of , are extracted; since the resolution of can present the most significant semantic features, in the present embodiment, five cross-attention maps with a resolution of are extracted.

[0096] The cross-attention consistency constraint and the self-attention preservation constraint are calculated using the cross-attention maps and the self-attention maps;

[0097] The gradients of the cross-attention consistency constraint and the self-attention preservation constraint on the latent noise map are calculated respectively by backpropagation, and the adaptive time step scheduler is used to update the latent noise map .

[0098] After the denoising process, the decoder of Stable-Diffusion is used to decode the latent noise map obtained in the last step into the edited image I * .

[0099] In the embodiments shown in Figure 1 , Figure 2 , the cross-attention consistency constraint and the self-attention preservation constraint are integrated into the text-to-image model, and the optimization of the latent noise map is introduced in the denoising process without additional training.

[0100] At the time of introducing the attention adjustment constraint guide in the denoising process, at each time step t, first, the latent noise mask The input noise prediction network extracts all 16x16 resolution cross-attention maps. Then, the cross-attention consistency constraint is calculated using the cross-attention map corresponding to all editing words; the self-attention map of all 8x8, 16x16, 32x32, 64x64 resolution or 8x8, 16x16 resolution is extracted to calculate the self-attention preservation constraint.

[0101] Specifically, as an example, given the text description of the source image and the text description of the edited image , define the set of target editing tokens that appear in but do not appear in P as ; define the set of tokens that appear in both and P as S ={ s 1, s 2,... s J}. In order to alleviate the problem of inaccurate positioning of the cross-attention map, a mask is introduced in the invention to assist the editing process.

[0102] wherein the cross-attention map includes the cross-attention map corresponding to the editing tokens describing the edited image, and the self-attention map includes the self-attention map generated in the denoising network. Correspondingly, the cross-attention map is used to calculate the cross-attention consistency constraint, and the self-attention map is used to calculate the self-attention preservation constraint.

[0103] In one specific embodiment of the invention, the editing token corresponding to the cross-attention map at t is defined as , and the attention consistency constraint is used to adjust the value of the cross-attention map .

[0104] Specifically, the cross-attention map reflects the consistency of text features and image features. The larger the value, the higher the alignment of text and visual features, thereby improving the editability of the target region. Therefore, the cross-attention map aims to increase the value of the cross-attention map corresponding to the editing token within the mask to enhance the editing effect. The loss function of the cross-attention consistency constraint is as follows:

[0105] ,

[0106] wherein is the cross-attention consistency constraint, denotes the ratio of cross attention maps inside and outside the mask corresponding to the edit token, denotes the total number of cross attention maps with resolution 16X16; in this embodiment, take ;

[0107] ,

[0108] wherein, denotes the spatial coordinates of the cross attention map, denotes the layer number where the cross attention map is located, denotes the index corresponding to the edit token, denotes the cross attention map generated in the edit branch.

[0109] In order to maintain the structural fidelity, the present application also proposes a self-attention preservation constraint to reduce the difference between the self-attention map generated by the reconstruction branch and the self-attention map generated by the edit branch, and is more flexible than directly injecting the self-attention map. Specifically, as an example, the loss function of the self-attention preservation constraint is as follows:

[0110] ,

[0111] wherein, is the self-attention preservation constraint, is the current time step, is the self-attention map generated in the reconstruction branch, is the self-attention map generated in the edit branch.

[0112] In addition, the mask can also be used to further calculate the loss function of the local self-attention preservation constraint to preserve high-frequency details:

[0113]

[0114] wherein, is the local self-attention preservation constraint, , is the flattened vector. This scheme of preserving high-frequency details is particularly suitable for editing small objects in complex scenes.

[0115] Finally, in order to balance between editability and fidelity, the present application also proposes to calculate the total gradient by an adaptive time scheduler, and then the latent space noise map of the mask region content is updated using the total gradient. Wherein, the calculation of the adaptive time scheduler is as follows:

[0116] ,

[0117] The total gradient is calculated as follows:

[0118]

[0119] The latent noise map is updated using the following formula:

[0120]

[0121] In the above formula, the total gradient is subtracted from the mask region of the latent space noise map at a preset learning rate to obtain the updated latent space noise map of the mask region content. Wherein, is the latent space noise map of the editing branch, is the latent space noise map after gradient update in the editing branch, is the total gradient; and is an adaptive time step scheduler acting on the self-attention preservation constraint and the cross-attention consistency constraint, respectively, is the gradient of the latent space noise map in the editing branch for the self-attention preservation constraint, is the gradient of the latent space noise map for the cross-attention consistency constraint; T is the total number of steps of the denoising process, is the current time step, is a hyperparameter for controlling the adaptive time step scheduler.

[0122] The above optimization process controls the balance of the self-attention preservation constraint and the cross-attention consistency constraint through the adaptive time step scheduler, i.e., to make the editing effect balanced between fidelity and editability, and to obtain the optimal editing effect. And by adjusting the parameters, different editing types can be realized.

[0123] Wherein, the proportion coefficients β1 and β2 are usually set to 5. In one specific embodiment of the present application, k1 and k2 are set as follows: color change is 0.05, texture modification or background editing is 0.08, object replacement is 0.15, global style transfer is 0.1, and face attribute editing is 0.25.

[0124] In order to ensure that the image outside the editing area and the source image remain consistent, in the present embodiment, within a certain time step (e.g., 50 steps) of the denoising process, the latent space noise map of the reconstruction branch and the latent space noise map of the editing branch are further mixed (blend) as shown in the following formula using the mask:

[0125]

[0126] Wherein,​​​​​ a latent space noise map for a mixed operation, a latent space noise map for an editing branch, a mask, a latent space noise map for a reconstruction branch.

[0127] As shown in Figure 3 The present application also provides an image editing system based on a text-to-image large model, which uses the image editing method based on the text-to-image large model as described above to perform image editing. The image editing system based on the text-to-image large model 300 mainly includes three parts: a latent space noise map acquisition unit 310, an optimization unit 320, a denoising unit 330, and a decoding unit 340.

[0128] The latent space noise map acquisition unit 310 is configured to acquire a latent space noise map of a source image based on text-to-image input data. The text-to-image input data includes the source image, a mask of a target editing region, a text description of the source image, and a text description of an edited image.

[0129] The denoising unit 330 is configured to perform denoising processing on the latent space noise map.

[0130] The optimization unit 320 is configured to perform optimization processing on the latent space noise map within a preset time step during the denoising processing performed by the denoising unit.

[0131] The decoding unit 340 is configured to decode the latent space noise map obtained at the last step of the denoising processing to obtain an edited image.

[0132] The optimization unit 320 includes:

[0133] The noise prediction unit 321 is configured to input the latent space noise map at the current time step and the text embedding of the text description of the edited image into a preset noise prediction network to perform noise prediction.

[0134] The cross-attention map and self-attention map extraction unit 322 is configured to extract a cross-attention map and a self-attention map generated in the noise prediction network.

[0135] The constraint calculation unit 323 is configured to calculate a cross-attention consistency constraint and a self-attention preservation constraint based on the mask using the cross-attention map and the self-attention map, respectively.

[0136] The update unit 324 is configured to calculate the gradient of the latent space noise map with respect to the cross-attention consistency constraint and the self-attention preservation constraint by backpropagation, respectively, and update the latent space noise map using an adaptive time step scheduler and the gradient.

[0137] The image editing method described above is a corresponding implementation method of the image editing system described above, and the specific execution steps can refer to the specific embodiments of the image editing system described above, which will not be described again.

[0138] As can be seen from the above embodiments, the image editing method and system based on the text-to-image large model proposed by the present application edits an image by inputting the text describing the source image and the edited image and the mask of the target editing region by the user, and using the text-to-image large model for image editing. In the preset time step of the latent space noise map denoising process, the latent space noise map is first optimized. The process includes: first, input the latent space noise map and the text prompt of the current step into the noise prediction network; second, extract the cross-attention graph and self-attention graph generated in the noise prediction network, and calculate the cross-attention consistency constraint using the cross-attention graph and the self-attention reservation constraint using the self-attention graph; finally, calculate the gradient of the two constraints on the latent space noise map through back propagation respectively, and calculate the total gradient using the adaptive time step scheduler, and then update the latent space noise map using the total gradient. In addition, in a certain time step of the denoising process, the latent space noise map of the reconstruction branch and the latent space noise map of the editing branch are also mixed using the mask. In this way, the edited image achieves the target effect and maintains the overall image structure.

[0139] Figures 5 to 6 The image editing effects of the image editing method based on the text-to-image large model of the present application are shown respectively. Among them, Figure 5 is a comparison diagram of the effect of applying the present application and other existing image editing schemes, Figure 6 is an effect example of image editing by the image editing method and system based on the text-to-image large model of the present application.

[0140] As Figure 5 shown, the colored part of the text on the leftmost image is the edited token (only appears in the text describing the edited image), and the black part is the edited object (appears in both the text describing the edited image and the text describing the original image).

[0141] The first column of pictures is the original image, the last column of pictures "ours" represents the result of image editing by the image editing method and system based on the text-to-image large model of the present application, and the second column to the ninth column are the result images of other comparison schemes. Other comparison schemes are as follows:

[0142] DiffEdit: Diffusion-based semantic image editing with mask guidance;

[0143] P2P: Prompt-to-prompt image editing with cross attention control;

[0144] P2P+Bend: Combining blending operations and P2P;

[0145] PnP: Plug-and-play diffusion features for text-driven image-to-image translation;

[0146] PnP+Blend: Combining blending operations and PnP.

[0147] SPD Inv: Source Prompt Disentangled Inversion for Boosting Image Editability with Diffusion Models;

[0148] G-R: Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing;

[0149] MAG-Edit: MAG-Edit: Localized Image Editing in Complex Scenarios via Mask-Based Attention-Adjusted Guidance.

[0150] In Figure 6 , every 5 pictures are a group, the leftmost is the original picture, and the right five pictures are the editing effect pictures of the image editing method and system based on the text-to-image large model of the application. The text below each image on the right is a short phrase composed of an editing token (only appearing in the text describing the edited image) and an edited object (appearing in both the text describing the edited image and the text describing the original image).

[0151] As Figure 4 shown, the application also provides an electronic device, which comprises:

[0152] at least one processor; and,

[0153] a memory connected in communication with the at least one processor; wherein,

[0154] The memory stores a computer program executable by the at least one processor, which is executed by the at least one processor to enable the at least one processor to perform the steps of the aforementioned text-to-image model based image editing method.

[0155] It is understood by those skilled in the art that the structure illustrated does not constitute a limitation on the electronic device 1, which can include fewer or more components than those illustrated, or combine certain components, or different component arrangements.

[0156] For example, although not shown, the electronic device 1 can also include a power source (such as a battery) to power the various components. Preferably, the power source can be logically connected to the at least one processor 10 through a power management device, so that functions such as charge management, discharge management, and power consumption management can be achieved through the power management device. The power source can also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The electronic device 1 can also include various sensors, Bluetooth modules, Wi-Fi modules, and the like, which are not described here.

[0157] Further, the electronic device 1 can also include a network interface, which can optionally include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is typically used to establish a communication connection between the electronic device 1 and other electronic devices.

[0158] Optionally, the electronic device 1 can also include a user interface, which can be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch, etc. The display can also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visualized user interface.

[0159] It should be understood that the embodiments are for illustration only and are not limited in scope by the structure.

[0160] The text-to-image model based image editing program 12 stored in the memory 11 of the electronic device 1 is a combination of a plurality of instructions, which, when executed in the processor 10, can implement the following steps:

[0161] S110: obtain a latent space noise map of a source image based on text-to-image input data; wherein the text-to-image input data comprises the source image, a mask of a target editing region, a text description of the source image, and a text description of an edited image;

[0162] S120: perform denoising processing on the latent space noise map, and perform optimization processing on the latent space noise map in a preset time step in the denoising processing;

[0163] S130: decode the latent space noise map obtained in the last step of the denoising processing to obtain an edited image;

[0164] The optimization processing of the latent space noise map comprises:

[0165] S121: input the text embedding of the text description of the edited image and the latent space noise map of the current time step into a preset noise prediction network to perform noise prediction;

[0166] S122: extract a cross-attention map and a self-attention map generated in the noise prediction network;

[0167] S123: calculate a cross-attention consistency constraint based on the mask and a self-attention reservation constraint based on the cross-attention map and the self-attention map, respectively;

[0168] S124: calculate the gradient of the latent space noise map with respect to the cross-attention consistency constraint and the self-attention reservation constraint by back propagation, and update the latent space noise map using an adaptive time step scheduler and the gradient.

[0169] Specifically, the specific implementation method of the processor 10 on the above instructions can refer to Figure 4 The descriptions of related steps in the corresponding embodiments are not repeated here.

[0170] Further, the modules / units integrated in the electronic device 1 can be stored in a computer readable storage medium if they are realized in the form of software function units and sold or used as independent products. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM).

[0171] The image editing method and system based on the text-to-image large model according to the present application are described above with reference to the drawings in an exemplary manner. However, those skilled in the art should understand that various improvements can be made to the image editing method and system based on the text-to-image large model according to the present application described above without departing from the content of the present application. Therefore, the protection scope of the present application should be determined by the content of the appended claims.

Claims

1. An image editing method based on a large-scale text-based image model, characterized in that, Includes the following steps: S110: Obtain the latent space noise map of the source image based on the text image input data; wherein, the text image input data includes the source image, the mask of the target editing region, the text description of the source image, and the text description of the edited image; S120: Denoise the latent space noise graph and optimize the latent space noise adaptive time step scheduler graph within a preset time step of the denoising process. S130: Decode the latent space noise map obtained in the last step of the denoising process to obtain the edited image; The optimization processing of the latent space noise map includes: S121: Embed the latent space noise map of the current time step and the text description of the edited image into a preset noise prediction network for noise prediction; S122: Extract the cross-attention map and self-attention map generated in the noise prediction network; S123: Calculate the mask-based cross-attention consistency constraint and self-attention retention constraint using the cross-attention graph and the self-attention graph, respectively; S124: Calculate the gradients of the cross-attention consistency constraint and the self-attention retention constraint on the latent space noise map through backpropagation. Based on the gradients of the cross-attention consistency constraint and the self-attention retention constraint on the latent space noise map, calculate the total gradient using an adaptive time-step scheduler. Update the latent space noise map according to the total gradient as follows: in, This is the updated latent space noise map. This is the latent space noise map before the update. This represents the total gradient; in, and An adaptive time-step scheduler that acts on the self-attention retention constraint and the cross-attention consistency constraint, respectively. To preserve the gradient of the latent space noise map in the editing branch, the self-attention constraint is used. The gradient of the latent space noise map is given by the cross-attention consistency constraint; Where T is the total number of steps in the denoising process. For the current time step, To control the hyperparameters of the adaptive time-step scheduler.

2. The image editing method based on the large model of text-based images as described in claim 1, characterized in that, The cross-attention map includes cross-attention maps corresponding to the edit terms describing the edited image, and the self-attention map includes all self-attention maps at a preset resolution generated in the denoising network.

3. The image editing method based on a large text-based image model as described in claim 2, characterized in that, Denoising the latent space noise map includes reconstructing branches and editing branches; wherein, The reconstruction branch is used to perform noise prediction at each time step of the denoising process using text embeddings of the text description of the source image and latent space noise maps; The editing branch is used to guide the latent space noise map within a preset time step of the denoising process; wherein, Within a certain time step of the denoising process, the latent space noise map of the reconstructed branch and the latent space noise map of the edited branch are mixed using a mask.

4. The image editing method based on a large text-based image model as described in claim 3, characterized in that, The loss function for the self-attention preservation constraint is as follows: in, It is a self-attention retention constraint. For the current time step, It is the self-attention map generated in the reconstruction branch. Self-attention map generated in the edit branch; mask The loss function used to compute the local self-attention preservation constraint to preserve high-frequency details is as follows: in, It is a local self-attention preservation constraint. , for The vector obtained by flattening.

5. The image editing method based on a large text-based image model as described in claim 4, characterized in that, The loss function for the cross-attention consistency constraint is calculated as follows: in, For cross-attention consistency constraints, This represents the ratio of the cross-attention map corresponding to the edited word to that inside and outside the mask. This represents the total number of cross-attention maps with a resolution of 16x16; where, in, Represents the spatial coordinates of the cross-attention map. Indicates the layer number where the cross-attention map is located. This indicates the index corresponding to the edit term. This represents the cross-attention graph generated during the editing branch.

6. The image editing method based on a large text-based image model as described in claim 5, characterized in that, The formula for blending the latent space noise maps of the reconstructed branch and the edited branch using a mask is as follows: in, This is a latent space noise map after the mixing operation. To edit the latent space noise map of the branch, For the mask, To reconstruct the latent space noise map of the branch.

7. An image editing system based on a large-scale image model, comprising image editing based on the image editing method based on a large-scale image model as described in any one of claims 1-6, including: A latent space noise map acquisition unit is used to acquire a latent space noise map of a source image based on textual image input data; wherein, the textual image input data includes the source image, a mask of the target editing region, a text description of the source image, and a text description of the edited image; A denoising unit is used to denoise the latent space noise map; An optimization unit is used to perform optimization processing on the latent space noise map within a preset time step in the denoising process. The decoding unit is used to decode the latent space noise map obtained in the last step of the denoising process to obtain the edited image; The optimization unit includes: The noise prediction unit is used to embed the latent space noise map of the current time step and the text description of the edited image into a preset noise prediction network for noise prediction. A cross-attention map and self-attention map extraction unit is used to extract the cross-attention map and self-attention map generated in the noise prediction network; The constraint calculation unit is used to calculate the mask-based cross-attention consistency constraint and self-attention retention constraint using the cross-attention graph and the self-attention graph, respectively. The update unit is used to calculate the gradients of the cross-attention consistency constraint and the self-attention retention constraint on the latent space noise graph through backpropagation, and update the latent space noise graph using an adaptive time step scheduler and the gradients.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the steps in the image editing method based on the large model of the text image as described in any one of claims 1-6.

Images