Image editing method and device, storage medium and electronic device

Abstract

The invention discloses an image editing method and device, a storage medium and an electronic device. The method comprises the steps that an original image and an edited text are obtained, and the edited text is used for describing part of content to be replaced in image content of the original image; performing noise addition processing on the original image to obtain a real noise image; performing denoising processing on the real noise image according to the edited text to generate a first attention map; generating a target mask by using the editing text and the first attention map; and carrying out image editing on the original image based on the target mask to obtain an edited image. According to the method and the device, the technical problem of low semantic mask generation efficiency and image editing efficiency when a diffusion model is used for semantic image editing in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image editing method, device, storage medium and electronic device. Background Art

[0002] As a common task in the multimodal community, semantic image editing aims to modify the target instance of a given image based on the input text description, while the rest of the image information needs to be retained as much as possible. In the related art, when applying the diffusion model to the semantic image editing task, due to the lack of precise control over the diffusion step, it is necessary to introduce additional information to better control the image operation, such as reference images, attention maps, and semantic masks. Among them, using semantic masks to accurately limit the target image area and generating it through a text-to-image diffusion model is a relatively effective method for local image editing, but the acquisition of semantic masks often requires manual intervention, which greatly limits the generation efficiency of semantic masks and image editing efficiency.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] At least some embodiments of the present application provide an image editing method, apparatus, storage medium and electronic device to at least solve the technical problems of low semantic mask generation efficiency and low image editing efficiency when using a diffusion model for semantic image editing in the related art.

[0005] According to one embodiment of the present application, there is provided an image editing method, comprising: obtaining an original image and an editing text, wherein the editing text is used to describe a portion of the image content of the original image to be replaced; performing noise processing on the original image to obtain a real noise image; performing denoising processing on the real noise image according to the editing text to generate a first attention map; generating a target mask using the editing text and the first attention map; and performing image editing on the original image based on the target mask to obtain an edited image.

[0006] According to one embodiment of the present application, there is provided an image editing method, comprising: obtaining an original image and a target mask, wherein the target mask is generated using editing text and a first attention map, the editing text is used to describe a portion of the image content of the original image to be replaced, the first attention map is generated after denoising a real noise image based on the editing text, and the real noise image is obtained by denoising the original image; and image editing is performed on the original image based on the target mask to obtain an edited image.

[0007] According to one embodiment of the present application, an image editing device is also provided, including: an acquisition module, used to acquire an original image and an edited text, wherein the edited text is used to describe part of the image content of the original image to be replaced; a denoising module, used to perform noise processing on the original image to obtain a real noise image; a denoising module, used to perform denoising processing on the real noise image according to the edited text to generate a first attention map; a generation module, used to generate a target mask using the edited text and the first attention map; and an editing module, used to perform image editing on the original image based on the target mask to obtain an edited image.

[0008] According to one embodiment of the present application, a computer-readable storage medium is further provided, in which a computer program is stored, wherein the computer program is configured to execute any of the above-mentioned image editing methods when run.

[0009] According to one embodiment of the present application, there is also provided an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute any of the above-mentioned image editing methods.

[0010] In at least some embodiments of the present application, an original image and an edited text are obtained, and then the original image is denoised to obtain a real noise image. The real noise image is then denoised based on the edited text to generate a first attention map. The edited text and the first attention map are used to generate a target mask. Finally, the original image is edited based on the target mask to obtain an edited image, thereby achieving the purpose of efficiently performing semantic image editing on the original image using the edited text, thereby realizing the technical effect of improving the efficiency of semantic mask generation and image editing, and thereby solving the technical problem of low semantic mask generation efficiency and image editing efficiency when using a diffusion model for semantic image editing in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1 is a hardware structure block diagram of a mobile terminal of an image editing method according to one embodiment of the present application;

[0013] Figure 2 is a flow chart of an image editing method according to one embodiment of the present application;

[0014] Figure 3 is a flowchart of another image editing method according to one embodiment of the present application;

[0015] Figure 4 is a schematic diagram of an image editing method according to one embodiment of the present application;

[0016] Figure 5 is a structural block diagram of an image editing device according to one embodiment of the present application;

[0017] Figure 6 is a structural block diagram of another image editing device according to one embodiment of the present application;

[0018] Figure 7 is a schematic diagram of an electronic device according to one embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0020] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] In recent years, as the diffusion model has become more effective and influential, it has gradually replaced the Generative Adversarial Network (GAN) to become the mainstream paradigm for conditional image generation. Compared with GAN, the diffusion model proposes a completely different generation framework that can obtain more diverse and high-quality generation results. The successful application of the diffusion model in the field of text and image generation has also inspired researchers to apply it to semantic image editing tasks.

[0022] As a common task in the multimodal community, semantic image editing aims to modify the target instance of a given image based on the input text description, while the rest of the image information needs to be preserved as much as possible. In related technologies, semantic image editing can be implemented based on the Style Generative Adversarial Network (StyleGAN) model. By studying the entanglement of the latent space features of StyleGAN, the editing direction vector is extracted, and the image is edited by performing addition and subtraction operations in the latent space.

[0023] However, although the existing diffusion models have performed well in terms of generation quality and diversity of text-to-image generation, due to the lack of precise control over the diffusion steps, when applying the diffusion models to semantic image editing tasks, additional information needs to be introduced to better control image operations, such as reference images, attention maps, and semantic masks. Among them, using semantic masks to accurately limit the target image area and generating them through the text-to-image diffusion model is a more effective method for local image editing, but the acquisition of semantic masks often requires manual intervention, which greatly limits the efficiency of semantic mask generation and image editing.

[0024] The above method embodiments involved in the present disclosure can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, the mobile terminal can be a smart phone, a tablet computer, a handheld computer, a mobile Internet device, a PAD, a game console and other terminal devices. Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of an image editing method according to an embodiment of the present application. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP) chip, a microprocessor (MCU), a field-programmable logic device (FPGA), a neural network processor (NPU), a tensor processing unit (TPU), an artificial intelligence (AI) type processor, etc.) and a memory 104 for storing data. In one embodiment of the present application, it may also include: an input and output device 108 and a display device 110.

[0025] In some optional embodiments mainly based on game scenarios, the above-mentioned device can also provide a human-computer interaction interface with a touch-sensitive surface, which can sense finger contact and / or gestures to perform human-computer interaction with a graphical user interface (GUI). The human-computer interaction function may include the following interactions: creating web pages, drawing, word processing, making electronic documents, games, video conferencing, instant messaging, sending and receiving emails, call interface, playing digital videos, playing digital music and / or web browsing, etc. The executable instructions for executing the above-mentioned human-computer interaction functions are configured / stored in a computer program product or readable storage medium executable by one or more processors.

[0026] Those skilled in the art will understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0027] According to one embodiment of the present application, an embodiment of an image editing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] Figure 2 is a flowchart of an image editing method according to one embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0029] Step S21, obtaining an original image and an edit text, wherein the edit text is used to describe a portion of the image content of the original image to be replaced;

[0030] Step S22, performing noise processing on the original image to obtain a real noise image;

[0031] Step S23, denoising the real noise image according to the edited text to generate a first attention map;

[0032] Step S24, generating a target mask using the edited text and the first attention map;

[0033] Step S25, editing the original image based on the target mask to obtain an edited image.

[0034] The above original image is the object to be processed in the semantic image editing task, which can be specifically an image containing semantic information, in which each pixel is labeled as a specific semantic category, such as road, car, pedestrian, etc.

[0035] The above-mentioned editing text is used to describe the part of the image content of the original image to be replaced, that is, for a given original image, the specific attributes or content of the original image can be modified according to the description of the editing text. The editing text can be an instruction or requirement described in natural language to guide the operation of image editing, and its corresponding languages ​​include but are not limited to Chinese and English.

[0036] The above editing text can be customized according to specific tasks and requirements to guide specific editing operations on the image. For example, assuming there is an original image containing a cat, the editing text can be the following: "Change the color of the cat to brown", this editing text describes the operation of changing the color of the cat in the original image from the original color to black; "Double the size of the cat", this editing text describes the operation of doubling the size of the cat in the original image, making the cat larger in the image; "Add a red ball to the image", this editing text describes the operation of adding a red ball to the original image, so that there is a ball in addition to the cat in the image; "Change the background in the image to a beach scene", this editing text describes the operation of changing the background in the original image to a beach scene, so that the background of the image becomes a beach scene.

[0037] Furthermore, by adding noise to the original image, the latent space representation of the noisy version of the image can be obtained, where the noisy version of the image refers to the real noisy image after the original image is interfered with by noise. The latent space representation of the noisy version of the image refers to the conversion of the original image into a vector representation in the latent space by encoding and decoding the original image under a certain noise model. This vector representation can be used to analyze and process images, such as noise reduction, image restoration and other tasks. The latent space representation of the noisy version of the image can be obtained through deep learning models such as autoencoders and variational autoencoders (VAE).

[0038] The real noisy image is denoised according to the edited text to generate the first attention map, where the first attention map is the attention map of the cross attention layer. The cross attention layer is often used in image processing tasks to adaptively learn the degree of association between different feature maps in order to better fuse feature information of different scales or levels. The attention map of the cross attention layer refers to the weight map generated in the cross attention mechanism by calculating the relationship between the input feature maps to guide feature fusion.

[0039] A target mask is generated using the edited text and the first attention map. The target mask can be used to locate or highlight a target area related to the edited text in the image, such as an object or area described by the edited text in the image.

[0040] Based on the above steps S21 to S25, by obtaining the original image and the edited text, the original image is denoised to obtain a real noise image, and then the real noise image is denoised according to the edited text to generate a first attention map, and the edited text and the first attention map are used to generate a target mask. Finally, the original image is edited based on the target mask to obtain an edited image, thereby achieving the purpose of efficiently performing semantic image editing on the original image using the edited text, thereby achieving the technical effect of improving the efficiency of semantic mask generation and image editing, and thus solving the technical problems of low semantic mask generation efficiency and image editing efficiency when using diffusion models for semantic image editing in related technologies.

[0041] The image editing method in the embodiment of the present application is further introduced below.

[0042] Optionally, in step S22, performing noise processing on the original image to obtain a real noise image includes:

[0043] Step S221, performing image coding processing on the original image to obtain a coded image;

[0044] Step S222, performing noise processing on the encoded image to obtain a real noise image.

[0045] Specifically, for the original image I, it first passes through a VAE-based image encoder E I Obtain the latent space representation x0 of the original image, and then add noise through the forward propagation step of the diffusion model to obtain the latent space representation x of the noisy version of the image t The specific implementation process is shown in the following formula (1) and formula (2):

[0046] x0=E I (I)(1)

[0047]

[0048] in, is the parameter for the fusion of the original image and the noise, which is specifically a function related to the time step t, for example, It can be a function of y=x, where the independent variable is the time step t, t belongs to [0,1000], and when t is 1, The true noise image for a certain moment in the surface is obtained by fusion of the noise and the original image.

[0049] Based on the above optional embodiment, by performing image encoding processing on the original image to obtain a coded image, and then performing noise adding processing on the coded image, a real noise image can be quickly obtained.

[0050] Optionally, in step S23, denoising the real noise image according to the edited text to generate the first attention map includes:

[0051] Step S231, performing text editing processing on the edit text to obtain a text feature vector;

[0052] Step S232, denoising the real noise image according to the text feature vector to generate a first attention map.

[0053] Specifically, for the edited text S, the text encoder CLIP based on the vision transformer (ViT) extracts each word vector c in the edited text S i =CLIP t (S i ), and combined as the text feature vector c∈{c1,c2,…,c N}, where N is the length of the word vector of the edited text S plus the special character representing the beginning of the sentence. Furthermore, the real noise image is denoised based on the text feature vector to generate the first attention map.

[0054] Based on the above optional embodiment, by performing text editing processing on the edited text to obtain a text feature vector, and then performing denoising processing on the real noise image based on the text feature vector, the first attention map can be quickly generated.

[0055] Optionally, in step S232, denoising the real noise image according to the text feature vector to generate a first attention map includes:

[0056] Step S2321, performing projection transformation on the text feature vector to obtain a key matrix;

[0057] Step S2322, performing a projection transformation on the real noise image to obtain a query matrix;

[0058] Step S2323, perform attention calculation on the feature dimensions of the key matrix, query matrix and text feature vector to obtain a first attention map.

[0059] In the attention mechanism, the key matrix, value matrix, and query matrix are three important matrices. The key matrix is ​​a matrix used to represent the importance of each element in the text feature vector, which can be calculated by multiplying each element in the text feature vector by a weight vector. The key matrix is ​​used to calculate the attention weight to determine the target content in the text feature vector that the model should focus on. The value matrix is ​​a matrix used to represent the information of each element in the text feature vector, which can be calculated by multiplying each element in the text feature vector by a weight vector. The value matrix is ​​used to calculate the weighted sum according to the attention weight to obtain the encoded representation of the text feature vector. The query matrix is ​​a matrix used to represent the feature information of the real noise image, which can be calculated by the hidden state of the model or the output of the previous moment. The query matrix is ​​used to calculate the attention weight to determine the target content in the text feature vector that the model should focus on.

[0060] Specifically, the text feature vector c is used as the guiding condition and the diffusion model is used to represent the latent space x of the noisy version image. t Denoising. Among them, the text feature vector c and the latent space representation x of the noisy version image t The projection transformation is performed by three trainable linear layers, represented as the first trainable linear layer Second trainable linear layer and a third trainable linear layer The text feature vector c passes through the second trainable linear layer The third trainable linear layer Perform projection transformation to obtain the key matrix K t Sum value matrix V t , the latent space representation of the noisy version image x t Then through the first trainable linear layer Get the query matrix Q by performing projection transformation t , and then the first attention map can be obtained through the attention calculation formula, that is, the attention map A of the cross attention layer can be obtained through formula (3) and formula (4) t .

[0061]

[0062]

[0063] Among them, d k Indicates that the feature dimension of the text feature vector c is k, and then the first attention map A t and value matrix V t Combined, we get the final output A of the cross attention layer t ·Vt , Denotes the key matrix K t , so as to be used for subsequent calculations.

[0064] Based on the above optional embodiment, by performing a projection transformation on the text feature vector to obtain a key matrix, and then performing a projection transformation on the real noise image to obtain a query matrix, and finally performing attention calculation on the feature dimensions of the key matrix, query matrix and text feature vector, the first attention map can be quickly obtained.

[0065] Optionally, in step S24, generating a target mask using the edited text and the first attention map includes:

[0066] Step S241, using the edited text, obtaining a position vector corresponding to the first attention map;

[0067] Step S242, performing weighted calculation on the first attention map and the position vector to obtain a second attention map;

[0068] Step S243, generating a target mask based on the second attention map.

[0069] In the attention mechanism, the position vector refers to the vector used to represent each position in the first attention map, which can be used to calculate the attention weight. The position vector can be fixed or a learnable parameter. The attention weight is a weight used to measure the degree of attention of the model to each position in the first attention map. It can be obtained by calculating the similarity between each position in the first attention map and the current query vector. The attention weight is a probability distribution that indicates the degree of attention of the model to different positions and can be used to perform weighted aggregation of information at different positions in the first attention map.

[0070] Based on the above optional embodiment, by using the edited text, the position vector corresponding to the first attention map is obtained, and then the first attention map and the position vector are weighted to obtain the second attention map. Finally, the target mask can be quickly generated based on the second attention map, further improving the efficiency of semantic mask generation. Specifically, in the τth time step of the stable diffusion model denoising, by τ (i.e., the average attention map) is weighted with the position vector P to obtain the second attention map (i.e., the refined attention map).

[0071] Optionally, in step S241, using the edited text, obtaining the position vector corresponding to the first attention map includes:

[0072] Step S2411, extracting a preset word vector from the text feature vector corresponding to the edited text, wherein the preset word vector is a word vector corresponding to a special character indicating the starting position of the text;

[0073] Step S2412, obtaining a third attention map corresponding to the preset word vector;

[0074] Step S2413, obtain the position vector based on the third attention map.

[0075] Specifically, the preset word vector corresponding to the special character (Start Token) representing the starting position of the text is extracted from the text feature vector corresponding to the edited text, and then the third attention map corresponding to the preset word vector is obtained. And use it as the information source for Mask generation, and then based on the third attention map Get the position vector.

[0076] Based on the above optional embodiment, by extracting a preset word vector from the text feature vector corresponding to the edited text, and then obtaining a third attention map corresponding to the preset word vector, the position vector can be quickly obtained based on the third attention map.

[0077] Optionally, in step S2413, obtaining a position vector based on the third attention map includes: locating a fourth attention map from the edited text based on the third attention map, wherein the fourth attention map has the highest similarity to the third attention map; performing similarity calculation between the fourth attention map and the attention map corresponding to the word vector to be compared to obtain a similarity vector, wherein the word vector to be compared is the remaining word vectors in the text feature vector except for the preset word vectors; and filtering the similarity vector to obtain a position vector.

[0078] Specifically, formula (5) is used to locate the third attention map from the edited text The fourth attention map with the highest similarity

[0079]

[0080] In order to obtain a more accurate source of information for mask generation, it is necessary to further aggregate information related to the concept and eliminate irrelevant information. Specifically, the fourth attention map is obtained using formula (6): Attention map corresponding to other tokens of edit text The similarity between them is used to obtain the similarity vector S∈R 1×N :

[0081]

[0082] Among them, the similarity of the attention map at each marker is closely related to the semantic similarity of the sentence. The more relevant the attention map is to the core semantics, the greater the similarity will be, and vice versa.

[0083] Furthermore, the similarity vector is filtered by the first threshold and the second threshold, so that the position vector P can be obtained. The position vector P is used to filter the first attention map A. t Weighted calculation is performed to make full use of the attention map information to refine the information source of Mask generation.

[0084] Based on the above optional embodiment, the fourth attention map is obtained by locating the fourth attention map from the edited text based on the third attention map, and then the similarity between the fourth attention map and the attention map corresponding to the word vector to be compared is calculated to obtain a similarity vector, and finally the similarity vector is filtered to quickly obtain a position vector.

[0085] Optionally, filtering the similarity vector to obtain the position vector includes: in response to the similarity vector being in a first value range, determining a first value, wherein the first value range is that the similarity vector is greater than a first threshold; in response to the similarity vector being in a second value range, determining a second value, wherein the second value range is that the similarity vector is less than a second threshold; in response to the similarity vector being in a third value range, determining a third value, wherein the third value range is the remaining value range excluding the first value range and the second value range; determining the position vector based on the first value, the second value and the third value.

[0086] Specifically, the similarity vector is filtered by the first threshold γ1 and the second threshold γ2, so as to obtain the position vector P. i is greater than the first threshold γ1, the first value is determined to be 1; if the similarity vector S i is less than the second threshold γ2, the second value is determined to be -1; if the similarity vector S i When it is in the remaining value ranges except the first value range and the second value range, the third value is determined to be 0, as shown in formula (7), and the position vector P is determined based on the first value, the second value and the third value.

[0087]

[0088] Based on the above optional embodiments, by determining a first value in response to the similarity vector being in a first value range, determining a second value in response to the similarity vector being in a second value range, and determining a third value in response to the similarity vector being in a third value range, the position vector can be quickly determined based on the first value, the second value and the third value.

[0089] Optionally, in step S243, generating a target mask based on the second attention map includes:

[0090] Step S2431, binarizing the second attention map corresponding to the first time step to obtain an initial mask;

[0091] Step S2432, generating a third latent space representation based on the initial mask, the first latent space representation, and the second latent space representation, wherein the first latent space representation is the latent space representation of the predicted noise image output by the diffusion model at the first time step, the second latent space representation is the latent space representation of the real noise image at the second time step, and the third latent space representation is the latent space representation of the predicted noise image output by the diffusion model at the second time step, and the second time step is the previous time step adjacent to the first time step;

[0092] Step S2433, determining a second attention map corresponding to the second time step using the third latent space representation;

[0093] Step S2434, binarize the second attention map corresponding to the second time step to obtain a target mask.

[0094] Specifically, in the τth time step of the stable diffusion model denoising, as shown in formula (8), by τ The weighted calculation is performed with the position vector P to obtain the refined second attention map

[0095]

[0096] Furthermore, as shown in formula (9), the second attention map corresponding to the first time step is binarized to obtain the initial mask M τ :

[0097]

[0098] Then based on the initial mask, the first latent space representation x τ ′ and the second latent space represents y τ , generate the third latent space representation x τ :

[0099] x τ =M τ ·x′ τ +(1-M τ )·y τ (10)

[0100] Among them, x′ τ represents the latent space representation of the predicted noise image output by the diffusion model at the first time step, y τrepresents the latent space representation of the predicted noise image output by the diffusion model at the second time step, x τ Represents the latent space representation of the predicted noise image output by the diffusion model at the second time step.

[0101] After obtaining the third latent space representation, the second attention map corresponding to the second time step is determined using the third latent space representation, and the second attention map corresponding to the second time step is binarized to obtain the target mask.

[0102] Based on the above optional embodiment, the second attention map corresponding to the first time step is binarized to obtain an initial mask, and then a third latent space representation is generated based on the initial mask, the first latent space representation and the second latent space representation. The second attention map corresponding to the second time step is then determined using the third latent space representation, and the second attention map corresponding to the second time step is binarized, so that the target mask can be quickly generated.

[0103] Optionally, in step S25, the original image is edited based on the target mask to obtain the edited image, including: inputting the target mask and the original image into an image restoration model, and performing image restoration on the original image based on the target mask to obtain the edited image.

[0104] Specifically, the target mask and the original image are input into an image restoration model, and the original image is restored based on the target mask to obtain an edited image.

[0105] Exemplarily, the original image and the target mask are preprocessed to meet the input requirements of the image restoration model, which may include but is not limited to adjusting the image size, normalizing the pixel value range, etc. Further, the preprocessed original image and the target mask are input into the image restoration model, and the image restoration model may specifically use a deep learning model, such as a generative adversarial network or a convolutional neural network, etc. The image restoration model is then used to fill in the missing parts of the original image based on the target mask to generate an edited image.

[0106] Figure 3 is a flowchart of another image editing method according to one embodiment of the present application. Figure 3 As shown, the method comprises the following steps:

[0107] Step S31, obtaining an original image and a target mask, wherein the target mask is generated using the edit text and the first attention map, the edit text is used to describe the part of the image content of the original image to be replaced, the first attention map is generated by denoising the real noise image according to the edit text, and the real noise image is obtained by denoising the original image;

[0108] Step S32, editing the original image based on the target mask to obtain an edited image.

[0109] The above original image is the object to be processed in the semantic image editing task, which can be specifically an image containing semantic information, in which each pixel is labeled as a specific semantic category, such as road, car, pedestrian, etc.

[0110] The above-mentioned editing text is used to describe the part of the image content of the original image to be replaced, that is, for a given original image, the specific attributes or content of the original image can be modified according to the description of the editing text. The editing text can be an instruction or requirement described in natural language, which is used to guide the operation of image editing, and its corresponding languages ​​include but are not limited to Chinese and English. The above-mentioned editing text can be customized according to specific tasks and needs, and is used to guide specific editing operations on the image.

[0111] Furthermore, by adding noise to the original image, the latent space representation of the noisy version of the image can be obtained, where the noisy version of the image refers to the real noisy image after the original image is interfered with by noise. The latent space representation of the noisy version of the image refers to the conversion of the original image into a vector representation in the latent space by encoding and decoding the original image under a certain noise model. This vector representation can be used to analyze and process images, such as noise reduction, image restoration and other tasks. The latent space representation of the noisy version of the image can be obtained through deep learning models such as autoencoders and variational autoencoders VAE.

[0112] The real noisy image is denoised according to the edited text to generate the first attention map, where the first attention map is the attention map of the cross attention layer. The cross attention layer is often used in image processing tasks to adaptively learn the degree of association between different feature maps in order to better fuse feature information of different scales or levels. The attention map of the cross attention layer refers to the weight map generated in the cross attention mechanism by calculating the relationship between the input feature maps to guide feature fusion.

[0113] A target mask is generated using the edited text and the first attention map. The target mask can be used to locate or highlight a target area related to the edited text in the image, such as an object or area described by the edited text in the image.

[0114] Based on the above steps S31 to S32, by acquiring the original image and the target mask, and then performing image editing on the original image based on the target mask, the edited image is obtained, thereby achieving the purpose of efficiently performing semantic image editing on the original image using the editing text, thereby achieving the technical effect of improving the semantic mask generation efficiency and image editing efficiency, and thus solving the technical problem of low semantic mask generation efficiency and image editing efficiency when using diffusion models for semantic image editing in related technologies.

[0115] Figure 4 is a schematic diagram of an image editing method according to one embodiment of the present application, such as Figure 4 As shown in the figure, the original image and the edited text are obtained, where the original image is a photo of a panda and the edited text is "a photo of a Persian cat". The original image is encoded to obtain an encoded image, and the encoded image is denoised to obtain a real noise image. Then, the edited text is edited to obtain a text feature vector, and then the real noise image is denoised based on the text feature vector to generate the first attention map. Next, the edited text is used to obtain the position vector corresponding to the first attention map, and then the first attention map A is denoised. t The second attention map A1 is obtained by weighted calculation with the position vector, and the target mask is generated based on the second attention map. Finally, the target mask and the original image are input into the image restoration model, and the original image is restored based on the target mask to obtain the edited image, which is a photo of a Persian cat.

[0116] In an embodiment of the present application, a diffusion model image editing method is proposed, called Instant Diffusion Editing (InstDiffEdit), which is a novel and efficient diffusion-based image editing (Diffusion-based Image Editing, DIE) method for solving the Mask automation acquisition task in image editing tasks. DIE tasks usually apply Masks to control the target area of ​​diffusion-based editing, thereby realizing local editing operations. However, most of the solutions in the related art obtain these Masks through manual operation or offline processing, which greatly reduces the editing efficiency. Taking into account the lack of model automation capabilities, an instant attention mask generation module that does not require further training or fine-tuning is proposed. By utilizing the cross-modal attention capability in the diffusion step of the existing diffusion model, a training-free refinement scheme is provided to achieve automatic and accurate mask generation, and instant mask guidance is realized in the diffusion step.

[0117] In advanced diffusion models, such as the Stable Diffusion model, effective multimodal spaces and excellent cross-attention mapping have been well established by learning a large number of image-text pairs. Therefore, InstDiffEdit fully explores and properly utilizes this excellent cross-modal alignment capability, that is, by extracting the hidden attention map in the diffusion step as the information source for Mask generation, thereby performing instant Mask generation. Considering that these hidden attention maps are difficult to use directly and are usually full of noise, the embodiment of the present application also equips InstDiffEdit with a learning-free mask positioning and refinement scheme, which can adaptively aggregate attention distribution according to editing instructions. It is worth noting that InstDiffEdit is a plug-and-play component that can seamlessly connect most diffusion models and implement semantic editing functions at a small cost of additional reasoning time without further training and fine-tuning, thereby improving the efficiency of semantic mask generation and image editing.

[0118] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0119] In this embodiment, an image editing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0120] Figure 5 is a structural block diagram of an image editing device according to one embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0121] The acquisition module 501 is used to acquire the original image and the edit text, wherein the edit text is used to describe the part of the image content of the original image to be replaced;

[0122] The noise adding module 502 is used to perform noise adding processing on the original image to obtain a real noise image;

[0123] A denoising module 503 is used to denoise the real noise image according to the edited text to generate a first attention map;

[0124] A generating module 504, configured to generate a target mask using the edited text and the first attention map;

[0125] The editing module 505 is used to edit the original image based on the target mask to obtain an edited image.

[0126] Optionally, the noise adding module 502 is further used to: perform image encoding processing on the original image to obtain a coded image; and perform noise adding processing on the coded image to obtain a real noise image.

[0127] Optionally, the denoising module 503 is further used to: perform text editing processing on the edited text to obtain a text feature vector; and perform denoising processing on the real noise image according to the text feature vector to generate a first attention map.

[0128] Optionally, the denoising module 503 is also used to: perform a projection transformation on the text feature vector to obtain a key matrix; perform a projection transformation on the real noise image to obtain a query matrix; perform attention calculation on the feature dimensions of the key matrix, the query matrix and the text feature vector to obtain a first attention map.

[0129] Optionally, the generation module 504 is further used to: use the edited text to obtain a position vector corresponding to the first attention map; perform weighted calculation on the first attention map and the position vector to obtain a second attention map; and generate a target mask based on the second attention map.

[0130] Optionally, the generation module 504 is also used to: extract a preset word vector from the text feature vector corresponding to the edited text, wherein the preset word vector is a word vector corresponding to a special character representing the starting position of the text; obtain a third attention map corresponding to the preset word vector; and obtain a position vector based on the third attention map.

[0131] Optionally, the generation module 504 is also used to: locate a fourth attention map from the edited text based on the third attention map, wherein the fourth attention map has the highest similarity to the third attention map; perform similarity calculation on the fourth attention map and the attention map corresponding to the word vector to be compared to obtain a similarity vector, wherein the word vector to be compared is the remaining word vectors in the text feature vector except the preset word vector; and perform filtering processing on the similarity vector to obtain a position vector.

[0132] Optionally, the generation module 504 is also used to: in response to the similarity vector being in a first value range, determine a first value, wherein the first value range is that the similarity vector is greater than a first threshold; in response to the similarity vector being in a second value range, determine a second value, wherein the second value range is that the similarity vector is less than a second threshold; in response to the similarity vector being in a third value range, determine a third value, wherein the third value range is the remaining value range excluding the first value range and the second value range; determine a position vector based on the first value, the second value and the third value.

[0133] Optionally, the generation module 504 is also used to: binarize the second attention map corresponding to the first time step to obtain an initial mask; generate a third latent space representation based on the initial mask, the first latent space representation and the second latent space representation, wherein the first latent space representation is the latent space representation of the predicted noise image output by the diffusion model at the first time step, the second latent space representation is the latent space representation of the real noise image at the second time step, the third latent space representation is the latent space representation of the predicted noise image output by the diffusion model at the second time step, and the second time step is the previous time step adjacent to the first time step; determine the second attention map corresponding to the second time step using the third latent space representation; binarize the second attention map corresponding to the second time step to obtain a target mask.

[0134] Optionally, the editing module 505 is further used to: input the target mask and the original image into an image restoration model, perform image restoration on the original image based on the target mask, and obtain an edited image.

[0135] Figure 6 is a structural block diagram of another image editing device according to one embodiment of the present application, such as Figure 6 As shown, the device comprises:

[0136] An acquisition module 601 is used to acquire an original image and a target mask, wherein the target mask is generated using the edit text and the first attention map, the edit text is used to describe the part of the image content of the original image to be replaced, the first attention map is generated by denoising the real noise image according to the edit text, and the real noise image is obtained by denoising the original image;

[0137] The editing module 602 is used to edit the original image based on the target mask to obtain an edited image.

[0138] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0139] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0140] Optionally, in this embodiment, the above-mentioned computer-readable storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0141] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0142] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0143] S1, obtaining an original image and an edit text, wherein the edit text is used to describe a portion of the image content of the original image to be replaced;

[0144] S2, adding noise to the original image to obtain a real noise image;

[0145] S3, denoising the real noise image according to the edited text to generate the first attention map;

[0146] S4, generates the target mask using the edited text and the first attention map;

[0147] S5, performing image editing on the original image based on the target mask to obtain an edited image.

[0148] Optionally, the computer-readable storage medium is further configured to store program codes for executing the following steps: performing image encoding processing on the original image to obtain an encoded image; and performing noise addition processing on the encoded image to obtain a real noise image.

[0149] Optionally, the computer-readable storage medium is also configured to store program codes for executing the following steps: performing text editing processing on the edited text to obtain a text feature vector; performing denoising processing on the real noise image based on the text feature vector to generate a first attention map.

[0150] Optionally, the above-mentioned computer-readable storage medium is also configured to store program codes for executing the following steps: performing a projection transformation on the text feature vector to obtain a key matrix; performing a projection transformation on the real noise image to obtain a query matrix; performing attention calculation on the feature dimensions of the key matrix, the query matrix and the text feature vector to obtain a first attention map.

[0151] Optionally, the above-mentioned computer-readable storage medium is also configured to store program code for executing the following steps: using the edited text, obtaining the position vector corresponding to the first attention map; performing weighted calculation on the first attention map and the position vector to obtain a second attention map; and generating a target mask based on the second attention map.

[0152] Optionally, the above-mentioned computer-readable storage medium is also configured to store program code for executing the following steps: extracting a preset word vector from a text feature vector corresponding to the edited text, wherein the preset word vector is a word vector corresponding to a special character representing the starting position of the text; obtaining a third attention map corresponding to the preset word vector; and obtaining a position vector based on the third attention map.

[0153] Optionally, the computer-readable storage medium is also configured to store program codes for executing the following steps: locating a fourth attention map from the edited text based on the third attention map, wherein the fourth attention map has the highest similarity to the third attention map; performing similarity calculation between the fourth attention map and the attention map corresponding to the word vector to be compared to obtain a similarity vector, wherein the word vector to be compared is the remaining word vectors in the text feature vector except for the preset word vectors; and filtering the similarity vector to obtain a position vector.

[0154] Optionally, the computer-readable storage medium is further configured to store program codes for executing the following steps: in response to the similarity vector being in a first value range, determining a first value, wherein the first value range is that the similarity vector is greater than a first threshold; in response to the similarity vector being in a second value range, determining a second value, wherein the second value range is that the similarity vector is less than a second threshold; in response to the similarity vector being in a third value range, determining a third value, wherein the third value range is the remaining value range excluding the first value range and the second value range; determining a position vector based on the first value, the second value and the third value.

[0155] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: binarizing the second attention map corresponding to the first time step to obtain an initial mask; generating a third latent space representation based on the initial mask, the first latent space representation and the second latent space representation, wherein the first latent space representation is the latent space representation of the predicted noise image output by the diffusion model at the first time step, the second latent space representation is the latent space representation of the real noise image at the second time step, the third latent space representation is the latent space representation of the predicted noise image output by the diffusion model at the second time step, and the second time step is the previous time step adjacent to the first time step; determining the second attention map corresponding to the second time step using the third latent space representation; binarizing the second attention map corresponding to the second time step to obtain a target mask.

[0156] Optionally, the computer-readable storage medium is further configured to store program codes for executing the following steps: inputting the target mask and the original image into an image restoration model, performing image restoration on the original image based on the target mask, and obtaining an edited image.

[0157] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0158] S1, obtaining an original image and a target mask, wherein the target mask is generated using the edit text and the first attention map, the edit text is used to describe the part of the image content of the original image to be replaced, the first attention map is generated by denoising the real noise image according to the edit text, and the real noise image is obtained by denoising the original image;

[0159] S2, performing image editing on the original image based on the target mask to obtain an edited image.

[0160] In the computer-readable storage medium of this embodiment, an original image and an edited text are obtained, and then the original image is denoised to obtain a real noise image. Then, the real noise image is denoised according to the edited text to generate a first attention map. The edited text and the first attention map are used to generate a target mask. Finally, the original image is edited based on the target mask to obtain an edited image, thereby achieving the purpose of efficiently performing semantic image editing on the original image using the edited text, thereby realizing the technical effect of improving the efficiency of semantic mask generation and image editing, and thus solving the technical problem of low semantic mask generation efficiency and image editing efficiency when using a diffusion model for semantic image editing in the related art.

[0161] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0162] In an exemplary embodiment of the present application, a program product capable of implementing the above method of the present embodiment is stored on a computer-readable storage medium. In some possible implementations, various aspects of the embodiments of the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present application described in the above "Exemplary Method" section of the present embodiment.

[0163] According to the program product for implementing the above method in the embodiment of the present application, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the embodiment of the present application is not limited thereto. In the embodiment of the present application, the computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0164] The program product may be in any combination of one or more computer-readable media. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive) of computer-readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0165] It should be noted that the program code contained in the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0166] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0167] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0168] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0169] S1, obtaining an original image and an edit text, wherein the edit text is used to describe a portion of the image content of the original image to be replaced;

[0170] S2, adding noise to the original image to obtain a real noise image;

[0171] S3, denoising the real noise image according to the edited text to generate the first attention map;

[0172] S4, generates the target mask using the edited text and the first attention map;

[0173] S5, performing image editing on the original image based on the target mask to obtain an edited image.

[0174] Optionally, the processor may also be configured to execute the following steps through a computer program: performing image encoding processing on the original image to obtain an encoded image; and performing noise addition processing on the encoded image to obtain a real noise image.

[0175] Optionally, the processor may also be configured to execute the following steps through a computer program: performing text editing processing on the edited text to obtain a text feature vector; performing denoising processing on the real noise image based on the text feature vector to generate a first attention map.

[0176] Optionally, the above-mentioned processor can also be configured to perform the following steps through a computer program: perform a projection transformation on the text feature vector to obtain a key matrix; perform a projection transformation on the real noise image to obtain a query matrix; perform attention calculation on the feature dimensions of the key matrix, query matrix and text feature vector to obtain a first attention map.

[0177] Optionally, the processor can also be configured to perform the following steps through a computer program: using the edited text to obtain a position vector corresponding to the first attention map; performing weighted calculation on the first attention map and the position vector to obtain a second attention map; and generating a target mask based on the second attention map.

[0178] Optionally, the above-mentioned processor can also be configured to perform the following steps through a computer program: extracting a preset word vector from a text feature vector corresponding to the edited text, wherein the preset word vector is a word vector corresponding to a special character representing the starting position of the text; obtaining a third attention map corresponding to the preset word vector; and obtaining a position vector based on the third attention map.

[0179] Optionally, the processor can also be configured to perform the following steps through a computer program: locating a fourth attention map from the edited text based on the third attention map, wherein the fourth attention map has the highest similarity to the third attention map; performing similarity calculation between the fourth attention map and the attention map corresponding to the word vector to be compared to obtain a similarity vector, wherein the word vector to be compared is the remaining word vectors in the text feature vector except for the preset word vectors; and filtering the similarity vector to obtain a position vector.

[0180] Optionally, the processor may also be configured to perform the following steps through a computer program: in response to the similarity vector being in a first value range, determining a first value, wherein the first value range is that the similarity vector is greater than a first threshold; in response to the similarity vector being in a second value range, determining a second value, wherein the second value range is that the similarity vector is less than a second threshold; in response to the similarity vector being in a third value range, determining a third value, wherein the third value range is the remaining value range excluding the first value range and the second value range; determining a position vector based on the first value, the second value and the third value.

[0181] Optionally, the processor can also be configured to perform the following steps through a computer program: binarizing the second attention map corresponding to the first time step to obtain an initial mask; generating a third latent space representation based on the initial mask, the first latent space representation and the second latent space representation, wherein the first latent space representation is the latent space representation of the predicted noise image output by the diffusion model at the first time step, the second latent space representation is the latent space representation of the real noise image at the second time step, the third latent space representation is the latent space representation of the predicted noise image output by the diffusion model at the second time step, and the second time step is the previous time step adjacent to the first time step; determining the second attention map corresponding to the second time step using the third latent space representation; binarizing the second attention map corresponding to the second time step to obtain a target mask.

[0182] Optionally, the processor may also be configured to perform the following steps through a computer program: inputting the target mask and the original image into an image restoration model, performing image restoration on the original image based on the target mask, and obtaining an edited image.

[0183] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0184] S1, obtaining an original image and a target mask, wherein the target mask is generated using the edit text and the first attention map, the edit text is used to describe the part of the image content of the original image to be replaced, the first attention map is generated by denoising the real noise image according to the edit text, and the real noise image is obtained by denoising the original image;

[0185] S2, performing image editing on the original image based on the target mask to obtain an edited image.

[0186] In the electronic device of this embodiment, an original image and an edited text are obtained, and then the original image is denoised to obtain a real noise image. Then, the real noise image is denoised according to the edited text to generate a first attention map. The edited text and the first attention map are used to generate a target mask. Finally, the original image is edited based on the target mask to obtain an edited image, thereby achieving the purpose of efficiently performing semantic image editing on the original image using the edited text, thereby realizing the technical effect of improving the efficiency of semantic mask generation and image editing, and thereby solving the technical problems of low semantic mask generation efficiency and image editing efficiency when using a diffusion model for semantic image editing in related technologies.

[0187] Figure 7 is a schematic diagram of an electronic device according to an embodiment of the present application. Figure 7 As shown, the electronic device 700 is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0188] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include but are not limited to: at least one processor 710, at least one memory 720, a bus 730 connecting different system components (including the memory 720 and the processor 710), and a display 740.

[0189] The memory 720 stores program codes, which can be executed by the processor 710, so that the processor 710 executes the steps described in the method part of the embodiment of the present application according to various exemplary implementations of the present application.

[0190] The memory 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 7201 and / or a cache memory unit 7202, and may further include a read-only memory unit (ROM) 7203, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory.

[0191] In some examples, the memory 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof may include the implementation of a network environment. The memory 720 may further include a memory remotely disposed relative to the processor 710, and these remote memories may be connected to the electronic device 700 via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0192] Bus 730 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a local bus of processor 710, or any of a variety of bus architectures.

[0193] The display 740 may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 700 .

[0194] Optionally, the electronic device 700 may also communicate with one or more external devices 800 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 700, and / or any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 760. Figure 7 As shown, the network adapter 760 communicates with other modules of the electronic device 700 via the bus 730. It should be understood that although Figure 7 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0195] The electronic device 700 may further include: a keyboard, a cursor control device (such as a mouse), an input / output interface (I / O interface), a network interface, a power supply and / or a camera.

[0196] It can be understood by those skilled in the art that Figure 7The structure shown is for illustration only and does not limit the structure of the above electronic device. Figure 7 More or fewer components as shown, or with Figure 1 The memory 720 may be used to store computer programs and corresponding data, such as the computer programs and corresponding data corresponding to the image editing method in the embodiment of the present application. The processor 710 executes various functional applications and data processing by running the computer programs stored in the memory 720, that is, implementing the above-mentioned image editing method.

[0197] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0198] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0199] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0200] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0201] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0202] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0203] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image editing method, characterized in that: include: Acquire an original image and an edit text, wherein the edit text is used to describe a portion of the image content of the original image to be replaced; Performing noise processing on the original image to obtain a real noise image; De-noising the real noise image according to the edited text to generate a first attention map; Generating a target mask using the edited text and the first attention map; The original image is edited based on the target mask to obtain an edited image.

2. The image editing method according to claim 1, characterized in that: Performing noise processing on the original image to obtain the real noise image includes: Performing image coding processing on the original image to obtain a coded image; The coded image is subjected to noise adding processing to obtain the real noise image.

3. The image editing method according to claim 1, characterized in that: Performing denoising on the real noise image according to the edited text to generate the first attention map includes: Performing text editing processing on the edit text to obtain a text feature vector; The real noise image is denoised according to the text feature vector to generate the first attention map.

4. The image editing method according to claim 3, characterized in that: Performing denoising on the real noise image according to the text feature vector to generate the first attention map includes: Performing a projection transformation on the text feature vector to obtain a key matrix; Performing a projection transformation on the real noise image to obtain a query matrix; Attention calculation is performed on the feature dimensions of the key matrix, the query matrix, and the text feature vector to obtain the first attention map.

5. The image editing method according to claim 1, characterized in that: Generating the target mask using the edit text and the first attention map includes: Using the edited text, obtaining a position vector corresponding to the first attention map; Performing weighted calculation on the first attention map and the position vector to obtain a second attention map; The object mask is generated based on the second attention map.

6. The image editing method according to claim 5, characterized in that: Using the edited text, obtaining the position vector corresponding to the first attention map includes: Extracting a preset word vector from the text feature vector corresponding to the edited text, wherein the preset word vector is a word vector corresponding to a special character indicating a starting position of the text; Obtain a third attention map corresponding to the preset word vector; The position vector is obtained based on the third attention map.

7. The image editing method according to claim 6, characterized in that: Acquiring the position vector based on the third attention map includes: Based on the third attention map, a fourth attention map is located from the edited text, wherein the fourth attention map has the highest similarity with the third attention map; Performing similarity calculation on the fourth attention map and the attention map corresponding to the word vector to be compared to obtain a similarity vector, wherein the word vector to be compared is the remaining word vectors in the text feature vector except the preset word vector; The similarity vector is filtered to obtain the position vector.

8. The image editing method according to claim 7, characterized in that: The similarity vector is filtered to obtain the position vector, which includes: In response to the similarity vector being in a first value range, determining a first value, wherein the first value range is that the similarity vector is greater than a first threshold; In response to the similarity vector being in a second value range, determining a second value, wherein the second value range is that the similarity vector is less than a second threshold; In response to the similarity vector being in a third value range, determining a third value, wherein the third value range is a remaining value range except the first value range and the second value range; The position vector is determined based on the first value, the second value, and the third value.

9. The image editing method according to claim 5, characterized in that: Generating the target mask based on the second attention map includes: Binarize the second attention map corresponding to the first time step to obtain an initial mask; Based on the initial mask, the first latent space representation and the second latent space representation, a third latent space representation is generated, wherein the first latent space representation is a latent space representation of a predicted noise image output by a diffusion model at a first time step, the second latent space representation is a latent space representation of the real noise image at a second time step, and the third latent space representation is a latent space representation of a predicted noise image output by the diffusion model at a second time step, and the second time step is a previous time step adjacent to the first time step; Determine the second attention map corresponding to the second time step using the third latent space representation; Binarize the second attention map corresponding to the second time step to obtain the target mask.

10. The image editing method according to claim 1, characterized in that: Performing image editing on the original image based on the target mask to obtain the edited image comprises: The target mask and the original image are input into an image restoration model, and the original image is restored based on the target mask to obtain the edited image.

11. An image editing method, characterized in that: include: Obtaining an original image and a target mask, wherein the target mask is generated using an edit text and a first attention map, the edit text is used to describe a portion of the image content of the original image to be replaced, the first attention map is generated by denoising a real noise image according to the edit text, and the real noise image is obtained by denoising the original image; The original image is edited based on the target mask to obtain an edited image.

12. An image editing device, characterized in that: include: An acquisition module, used for acquiring an original image and an edit text, wherein the edit text is used for describing a part of the image content of the original image to be replaced; A noise adding module, used for performing noise adding processing on the original image to obtain a real noise image; A denoising module, configured to perform denoising processing on the real noise image according to the edited text to generate a first attention map; A generating module, configured to generate a target mask using the edited text and the first attention map; The editing module is used to edit the original image based on the target mask to obtain an edited image.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the image editing method according to any one of claims 1 to 11 when executed by a processor.

14. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the image editing method according to any one of claims 1 to 11.

Images