Image editing method and device based on text guidance, equipment and medium

Abstract

The invention provides an image editing method and device based on text guidance, equipment and a medium. The method comprises the following steps: inputting an input picture, a source cue word and a target cue word into a model; an input picture is subjected to noise addition through an inversion process to form a noise picture, then the noise picture enters a reconstruction branch and an editing branch, and a cross attention graph of the reconstruction process of the reconstruction branch is injected into the editing branch to maintain the structure and content of the picture; a contrast attention focusing strategy is adopted in the inversion process, and attention is adaptively and accurately guided to an editing area according to attention leakage conditions of different degrees; then, a mask-guided hybrid sampling technology is used in an editing branch to provide different conditional guidance for an editing area and a non-editing area so as to ensure that change and retention in the image are balanced and integrated, only the area needing to be edited is modified in the editing process, unnecessary editing of the non-editing area is prevented, and the editing efficiency is improved. And the phenomenon of excessive editing is prevented.

Description

Technical Field

[0001] The present invention relates to the field of image editing technology and deep learning, and in particular to a text-guided image editing method, device, equipment and medium. Background Art

[0002] Large-scale text-image diffusion models trained on a large number of datasets have made great progress in generating high-fidelity and diverse images. Leveraging these models' ability to understand text, many methods for image editing using text have emerged. Users can clarify their editing needs solely through text descriptions without the need for any additional annotations.

[0003] P2P replaces the attention map of the target image with the source image to achieve image editing. The spatial features of the reconstruction phase are injected into specific layers of the editing process to maintain the structure of the input image. Masactrl implements mutual self-attention to achieve consistent image generation and non-rigid image editing. Pix2Pix-Zero identifies the editing direction by calculating the difference between text embeddings. InstructPix2Pix introduces an instruction-based image editing method. Hive incorporates human preferences and improves the effect of instruction-based image editing methods by designing a reward function.

[0004] Since image editing relies solely on text guidance, existing technologies often suffer from over-editing, which means that unnecessary modifications are made to non-edited areas of the input image. The reason for over-editing is that previous image editing methods only rely on coarse-grained text descriptions to guide the editing process, making it difficult to accurately locate the target editing area. The deeper reason is that attention leakage occurs during the editing process, that is, the cross-attention map related to editing shows high response values ​​in both the target area and the non-target area. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an image editing method, device, equipment and medium based on text guidance, which only modifies the areas that need to be edited during the editing process, prevents unnecessary editing of non-editing areas, and prevents the occurrence of over-editing.

[0006] In a first aspect, the present invention provides an image editing method based on text guidance, wherein an input image, a source prompt word and a target prompt word are input into a text image diffusion model; the source prompt word is used to describe the content of the original image, and the target prompt word is used to describe the content of the original image after editing;

[0007] The input image is firstly denoised into a noise image through an inversion process, and then enters a reconstruction branch and an editing branch respectively. The reconstruction branch is used to reconstruct the input image, and the editing branch is used to edit the input image. The cross attention map of the reconstruction process of the reconstruction branch is injected into the editing branch to maintain the structure and content of the image.

[0008] The inversion process adopts a contrastive attention focusing strategy to adaptively guide attention to the editing area accurately for different degrees of attention leakage; then the mask-guided hybrid sampling technique is used in the editing branch to provide different conditional guidance for the editing area and the non-editing area to ensure a balanced integration of changes and retention in the image.

[0009] In a second aspect, the present invention provides an image editing device based on text guidance, comprising:

[0010] An input module, used to input an input image, a source prompt word and a target prompt word into the text image diffusion model; the source prompt word is used to describe the content of the original image, and the target prompt word is used to describe the content of the original image after editing;

[0011] An Inversion module, used to perform an inversion process on the input image, adding noise to obtain a noise image;

[0012] The reconstruction branch module is used to reconstruct the input image;

[0013] Editing branch module, used to edit the input image;

[0014] An injection module, used for injecting the cross attention map of the reconstruction process of the reconstruction branch into the editing branch to maintain the structure and content of the picture;

[0015] A contrastive attention focusing strategy module, used to adopt a contrastive attention focusing strategy in the inversion process, and adaptively guide attention to the editing area accurately according to different degrees of attention leakage;

[0016] A hybrid sampling module is used to use a mask-guided hybrid sampling technique in the editing branch to provide different conditional guidance for the editing area and the non-editing area to ensure a balanced integration of changes and reservations in the image.

[0017] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0019] One or more technical solutions provided by the present invention have at least the following technical effects or advantages: when the input image of the present invention is noisy and converted into a noise image after the inversion process, a contrasting attention focusing strategy is adopted to adaptively guide the attention to the editing area accurately for different degrees of attention leakage. When the noise image enters the reconstruction branch and the editing branch, the cross-attention map of the reconstruction branch will be injected into the editing branch to maintain the structure and content of the image. Thereby, accurate text-guided image editing can be achieved, and only the areas that need to be edited are modified during the editing process, preventing unnecessary editing of non-editing areas. The experimental comparison results show that the present invention surpasses the current technology both quantitatively and qualitatively, especially in maintaining image structure and non-editing areas, which is much better than the current technology.

[0020] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0022] Figure 1 It is the system framework and process diagram of the present invention;

[0023] Figure 2 It is a schematic diagram of qualitative comparison effect between the embodiment of the present invention and other technologies;

[0024] Figure 3 A quantitative comparison data table of the embodiment of the present invention using direct inversion and other technologies;

[0025] Figure 4 A quantitative comparison data table of the embodiments of the present invention using null-text inversion and negative-promptinversion with other technologies;

[0026] Figure 5 It is a structural schematic diagram of the device in the second embodiment of the present invention; DETAILED DESCRIPTION

[0027] The embodiment of the present application is to provide an image editing method based on text guidance, which only modifies the areas that need to be edited during the editing process, prevents unnecessary editing of non-editing areas, and prevents the occurrence of over-editing.

[0028] The technical solution in the embodiment of the present application has the following general idea: when the input image of the text image diffusion model is noisy and converted into a noise image after the inversion process, a contrast attention focusing strategy is first adopted to adaptively guide the attention to the editing area accurately for different degrees of attention leakage. When the noise image enters the reconstruction branch and the editing branch, the cross-attention map of the reconstruction branch is injected into the editing branch to maintain the structure and content of the image. This enables accurate text-guided image editing, and only the areas that need to be edited are modified during the editing process, preventing unnecessary editing of non-editing areas.

[0029] Embodiment 1

[0030] like Figure 1 As shown, this embodiment provides an image editing method based on text guidance, inputting an input image, a source prompt word and a target prompt word into a text image diffusion model; the source prompt word is used to describe the content of the original image, and the target prompt word is used to describe the content of the original image after editing;

[0031] The input image is firstly denoised into a noise image through an inversion process, and then enters a reconstruction branch and an editing branch respectively. The reconstruction branch is used to reconstruct the input image, and the editing branch is used to edit the input image. The cross attention map of the reconstruction process of the reconstruction branch is injected into the editing branch to maintain the structure and content of the image.

[0032] The inversion process adopts a contrastive attention focusing strategy to adaptively guide attention to the editing area accurately for different degrees of attention leakage; then the mask-guided hybrid sampling technique is used in the editing branch to provide different conditional guidance for the editing area and the non-editing area to ensure a balanced integration of changes and retention in the image.

[0033] Figure 1 In the example, the source prompt is “sourcepromptP:“an orange van with surfboards on top”; the target prompt is “targetpromptP:“an orange van flowers on top”. The noise image is Z T .

[0034] The contrasting attention focusing strategy is used to adaptively guide attention to the editing area accurately according to different degrees of attention leakage. Specifically:

[0035] In the field of local image editing, an edit pair is extracted from the source cue and the target cue, denoted as<key1,key2> The formats of the source prompt and the target prompt are respectively “P=··· <key1>···" and "P * =··· <key2>···";For example, Figure 1 The edit pair in consists of <surfboard, flower>.

[0036] Since the content described in the source prompt comes directly from the input image, "key1" in the source prompt provides more accurate editing area localization compared to "key2" mentioned in the target prompt. Therefore, we will focus on the cross-attention map related to "key1" to generate a new cross-attention map The cross attention map High response values ​​only in the edited area;

[0037] In the editing branch, replace the original cross-attention map corresponding to "key2" with To solve the problem of attention leakage.

[0038] In the inversion process, the required cross attention map is obtained by averaging the attention maps with a resolution of 16*16 for all heads and time steps in the inversion process. and self-attention map Enhanced Cross-Attention Map Contrast between the edited area and the area where attention leakage occurred:

[0039] To avoid To divert attention to non-editable areas, we enhance The contrast between the edited area and the area where attention leakage occurs. First, the existing cross attention map Apply min-max normalization to generate a scaling map s, where each element in the scaling map s represents a cross attention map The magnitude of the corresponding elements in :

[0040]

[0041] Formula (1) is used to convert the cross attention map Mapping to the interval [s l ,s r ];s l Cross-Attention Map Corresponding to the left endpoint of the matrix, s r Cross-Attention Map Corresponding to the right endpoint of the matrix, by crossing the attention map Multiplying with the scaling map s amplifies the contrast between the response values ​​of the cross-attention map; this contrast enhancement effectively filters out irrelevant attention for non-editing areas while maintaining a high level of attention on the areas to be edited;

[0042] Perform N contrast enhancement operations, while performing a normalization operation after each operation to restore the original value size:

[0043]

[0044] Wherein, j refers to the number of contrast enhancement operations currently performed, j = 1 to N, N is determined according to the degree of attention leakage. The higher the degree of attention leakage, the larger N is. To determine the required number of contrast enhancement operations, we consider the cosine similarity between text embeddings:

[0045]

[0046] where e is the reference text embedding, and e i represents other text embeddings, L is the total number of text tokens, and α adjusts the sensitivity of N to similarity;

[0047] Cross-Attention Map After performing N times of contrast enhancement operation, it is recorded as It effectively filters unnecessary attention to the non-editing area and highlights the response value of the editing area;

[0048] Using self-attention map characteristics to restore The shape of the high response area in is consistent with the shape of the editing area; and self-attention map Perform matrix multiplication and get

[0049]

[0050] After executing the operation of formula (4), The high response area covers the editing area, but also slightly increases the focus on the non-editing area;

[0051] Then, in order to eliminate Pay attention to the non-editing area and further enhance its attention to the editing area. Perform attention-enhancing operations:

[0052]

[0053] Among them, k represents each element in the iterative cross attention map; τ1 represents the threshold, which can be set according to experience;

[0054] right After the attention enhancement operation, the attention values ​​below τ1 are set to zero, and the attention values ​​above τ1 are further enhanced.

[0055] At the same time, the present invention also proposes a new sampling technology, namely mask-guided hybrid sampling, which limits the editing effect to the target editing area.

[0056] The hybrid sampling technique guided by the mask provides different conditional guidance for the editing area and the non-editing area. Specifically:

[0057] Replace the non-editable regions of the edit branch's potential code with the corresponding non-editable regions of the rebuild branch's potential code:

[0058]

[0059] in, represents the potential code obtained by denoising each step of the editing branch, m represents the mask of the editing area, represents the latent code obtained by denoising at each step of the reconstruction branch;

[0060] Given that The high response area of ​​​​the edited area accurately contains the edited area, and the threshold τ2 is used to Extract the mask m from:

[0061]

[0062] Among them, τ2 is set according to experience;

[0063] The mask-guided hybrid sampling of the present invention provides different conditional guidance tailored for the edited and non-edited regions; ensuring that the edited regions follow the target hints while keeping the non-edited regions in their original state:

[0064]

[0065] Among them, c s and c t Represent the text embeddings of the source cue word and the target cue word respectively; g s and g t denote the strength of the conditional guidance applied to the non-edited region and the edited region, respectively, and m denotes the mask of the edited region.

[0066] like Figure 2 As shown, four examples are implemented using the method of the present invention, the first column is the input picture, the second column is the Pix2Pix-Zero method in the prior art, the third column uses the MasaCtrl method in the prior art, the fourth column uses the PnP method in the prior art, the fifth column uses the P2P method in the prior art, and the sixth column uses the method of the present invention. It can be seen that compared with various methods in the prior art, the present invention has the best effect in ensuring that the editing area follows the target prompt while keeping the non-editing area in the original state.

[0067] like Figure 3 and Figure 4 As shown, for the quantitative comparison data of the present invention with other technologies when the direct inversion is used in the embodiment of the present invention, see Figure 3 ; For quantitative comparison data of the present invention using null-text inversion and negative-prompt inversion with other technologies, see Figure 4 .

[0068] Embodiment 2

[0069] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0070] like Figure 5 As shown, in this embodiment, a text-guided image editing device is provided, comprising:

[0071] An input module, used to input an input image, a source prompt word and a target prompt word into the text image diffusion model; the source prompt word is used to describe the content of the original image, and the target prompt word is used to describe the content of the original image after editing;

[0072] An Inversion module, used to perform an inversion process on the input image, adding noise to obtain a noise image;

[0073] The reconstruction branch module is used to reconstruct the input image;

[0074] Editing branch module, used to edit the input image;

[0075] An injection module, used for injecting the cross attention map of the reconstruction process of the reconstruction branch into the editing branch to maintain the structure and content of the picture;

[0076] A contrastive attention focusing strategy module, used to adopt a contrastive attention focusing strategy in the inversion process, and adaptively guide attention to the editing area accurately according to different degrees of attention leakage;

[0077] A hybrid sampling module is used to use a mask-guided hybrid sampling technique in the editing branch to provide different conditional guidance for the editing area and the non-editing area to ensure a balanced integration of changes and reservations in the image.

[0078] The contrastive attention focus strategy module extracts an edit pair from the source prompt and the target prompt in the local image editing domain, denoted as<key1,key2> , the formats of the source prompt and the target prompt are respectively "P=··· <key1>···" and "P*=··· <key2>···";

[0079] Focus on the cross attention map related to "key1" to generate a new cross attention map The cross attention map High response values ​​only in the edited area;

[0080] The hybrid sampling module replaces the original cross attention map corresponding to "key2" in the editing branch with To solve the problem of attention leakage.

[0081] The contrast attention focus strategy module specifically performs the following process:

[0082] In the inversion process, the required cross attention map is obtained by averaging the attention maps with a resolution of 16*16 for all heads and time steps in the inversion process. and self-attention map Enhanced Cross-Attention Map Contrast between the edited area and the area where attention leakage occurred:

[0083] First, the existing cross attention map Apply min-max normalization to generate a scaling map s, where each element in the scaling map s represents a cross attention map The magnitude of the corresponding elements in :

[0084]

[0085] Formula (1) is used to convert the cross attention map Mapping to the interval [s l ,s r ];s l Cross-Attention Map Corresponding to the left endpoint of the matrix, s r Cross-Attention Map Corresponding to the right endpoint of the matrix, by crossing the attention map Multiply it with the scaling map s to amplify the contrast between the response values ​​of the cross-attention map;

[0086] Perform N contrast enhancement operations, while performing a normalization operation after each operation to restore the original value size:

[0087]

[0088] Wherein, j refers to the number of contrast enhancement operations currently performed, j = 1 to N, N is determined by the cosine similarity between text embeddings:

[0089]

[0090] where e is the reference text embedding, and e i represents other text embeddings, L is the total number of text tokens, and α adjusts the sensitivity of N to similarity;

[0091] Cross-Attention Map After performing N times of contrast enhancement operation, it is recorded as

[0092] Using self-attention map characteristics to restore The shape of the high response area in is consistent with the shape of the editing area; and self-attention map Perform matrix multiplication and get

[0093]

[0094] After executing the operation of formula (4), The high response area covers the editing area, but also slightly increases the focus on the non-editing area;

[0095] Afterwards, Perform attention-enhancing operations:

[0096]

[0097] Where k represents each element in the iterative cross attention map; τ1 represents the threshold, which is set based on experience;

[0098] right After the attention enhancement operation, the attention values ​​below τ1 are set to zero, and the attention values ​​above τ1 are further enhanced.

[0099] The mixed sampling module specifically performs the following process:

[0100] Replace the non-editable regions of the edit branch's potential code with the corresponding non-editable regions of the rebuild branch's potential code:

[0101]

[0102] in, represents the potential code obtained by denoising each step of the editing branch, m represents the mask of the editing area, represents the latent code obtained by denoising at each step of the reconstruction branch;

[0103] Given that The high response area of ​​​​the edited area accurately contains the edited area, and the threshold τ2 is used to Extract the mask m from:

[0104]

[0105] Among them, τ2 is set according to experience;

[0106] Mask-guided hybrid sampling provides different conditional guidance tailored for edited and non-edited regions; ensuring that edited regions follow target cues while keeping non-edited regions pristine:

[0107]

[0108] Among them, c s and c t Represent the text embeddings of the source cue word and the target cue word respectively; g s and g t denote the strength of the conditional guidance applied to the non-edited region and the edited region, respectively, and m denotes the mask of the edited region.

[0109] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0110] Embodiment 3

[0111] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to Embodiment 1, see Embodiment 3 for details. This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any implementation method in Embodiment 1 can be implemented.

[0112] Since the electronic device introduced in this embodiment is a device used to implement the method in the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements the device used by the method in the embodiment of the present application, it belongs to the scope of protection of the present application.

[0113] Embodiment 4

[0114] Based on the same inventive concept, this application provides a storage medium corresponding to Embodiment 1, see Embodiment 4 for details. This embodiment provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, any implementation method in Embodiment 1 can be implemented.

[0115] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0116] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0118] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0119] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0120] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A text-guided image editing method, characterized in that: Inputting an input image, a source prompt word and a target prompt word into the text image diffusion model; the source prompt word is used to describe the content of the original image, and the target prompt word is used to describe the content of the original image after editing; The input image is firstly denoised into a noise image through an inversion process, and then enters a reconstruction branch and an editing branch respectively. The reconstruction branch is used to reconstruct the input image, and the editing branch is used to edit the input image. The cross attention map of the reconstruction process of the reconstruction branch is injected into the editing branch to maintain the structure and content of the image. The inversion process adopts a contrastive attention focusing strategy to adaptively guide attention to the editing area precisely according to different degrees of attention leakage; Then, a mask-guided hybrid sampling technique is used in the editing branch to provide different conditional guidance for the editing area and the non-editing area to ensure a balanced integration of changes and preservation in the image.

2. The text-guided image editing method according to claim 1, characterized in that: The contrasting attention focusing strategy is used to adaptively guide attention to the editing area accurately according to different degrees of attention leakage. Specifically: In the field of local image editing, an edit pair is extracted from the source cue and the target cue, denoted as<key1,key2> , the formats of the source prompt and the target prompt are respectively "P=··· <key1>···" and "P * =··· <key2> ···”；< / key2> Focus on the cross attention map related to "key1" to generate a new cross attention map The cross attention map High response values ​​only in the edited area; In the editing branch, replace the original cross-attention map corresponding to "key2" with To solve the problem of attention leakage.

3. The text-guided image editing method according to claim 2, characterized in that: In the inversion process, the required cross attention map is obtained by averaging the attention maps with a resolution of 16*16 for all heads and time steps in the inversion process. and self-attention map Enhanced Cross-Attention Map Contrast between the edited area and the area where attention leakage occurred: First, the existing cross attention map Apply min-max normalization to generate a scaling map s, where each element in the scaling map s represents a cross attention map The magnitude of the corresponding elements in : Formula (1) is used to convert the cross attention map Mapping to the interval [s l ,s r ];s l Cross-Attention Map Corresponding to the left endpoint of the matrix, s r Cross-Attention Map Corresponding to the right endpoint of the matrix, by crossing the attention map Multiply it with the scaling map s to amplify the contrast between the response values ​​of the cross-attention map; Perform N contrast enhancement operations, while performing a normalization operation after each operation to restore the original value size: Wherein, j refers to the number of contrast enhancement operations currently performed, j = 1 to N, N is determined by the cosine similarity between text embeddings: where e is the reference text embedding, and e i represents other text embeddings, L is the total number of text tokens, and α adjusts the sensitivity of N to similarity; Cross-Attention Map After performing N times of contrast enhancement operation, it is recorded as Using self-attention map characteristics to restore The shape of the high response area in is consistent with the shape of the editing area; and self-attention map Perform matrix multiplication and get After executing the operation of formula (4), The high response area covers the editing area, but also slightly increases the focus on the non-editing area; Afterwards, Perform attention-enhancing operations: Where k represents each element in the iterative cross attention map; τ1 represents the threshold, which is set based on experience; right After the attention enhancement operation, the attention values ​​below τ1 are set to zero, and the attention values ​​above τ1 are further enhanced.

4. The text-guided image editing method according to claim 3, characterized in that: The hybrid sampling technique guided by the mask provides different conditional guidance for the editing area and the non-editing area. Specifically: Replace the non-editable regions of the edit branch's potential code with the corresponding non-editable regions of the rebuild branch's potential code: in, represents the potential code obtained by denoising each step of the editing branch, m represents the mask of the editing area, represents the latent code obtained by denoising at each step of the reconstruction branch; Given that The high response area of ​​​​the edited area accurately contains the edited area, and the threshold τ2 is used to Extract the mask m from: Among them, τ2 is set according to experience; Mask-guided hybrid sampling provides different conditional guidance tailored for edited and non-edited regions; ensuring that edited regions follow target cues while keeping non-edited regions pristine: Among them, c s and c t Represent the text embeddings of the source cue word and the target cue word respectively; g s and g t denote the strength of the conditional guidance applied to the non-edited region and the edited region, respectively, and m denotes the mask of the edited region.

5. An image editing device based on text guidance, characterized in that: include: An input module, used for inputting an input picture, a source cue word, and a target cue word into the Wensheng graph diffusion model; The source prompt word is used to describe the content of the original image, and the target prompt word is used to describe the content of the original image after editing; An Inversion module, used to perform an inversion process on the input image, adding noise to obtain a noise image; The reconstruction branch module is used to reconstruct the input image; Editing branch module, used to edit the input image; An injection module, used for injecting the cross attention map of the reconstruction process of the reconstruction branch into the editing branch to maintain the structure and content of the picture; A contrastive attention focusing strategy module, used to adopt a contrastive attention focusing strategy in the inversion process, and adaptively guide attention to the editing area accurately according to different degrees of attention leakage; A hybrid sampling module is used to use a mask-guided hybrid sampling technique in the editing branch to provide different conditional guidance for the editing area and the non-editing area to ensure a balanced integration of changes and reservations in the image.

6. The text-guided image editing device according to claim 5, characterized in that: The contrastive attention focus strategy module extracts an edit pair from the source prompt and the target prompt in the local image editing domain, denoted as<key1,key2> , the formats of the source prompt and the target prompt are respectively "P=··· <key1>···" and "P * =··· <key2> ···”；< / key2> Focus on the cross attention map related to "key1" to generate a new cross attention map The cross attention map High response values ​​only in the edited area; The hybrid sampling module replaces the original cross attention map corresponding to "key2" in the editing branch with To solve the problem of attention leakage.

7. The text-guided image editing device according to claim 6, characterized in that: The contrast attention focus strategy module specifically performs the following process: In the inversion process, the required cross attention map is obtained by averaging the attention maps with a resolution of 16*16 for all heads and time steps in the inversion process. and self-attention map Enhanced Cross-Attention Map Contrast between the edited area and the area where attention leakage occurred: First, the existing cross attention map Apply min-max normalization to generate a scaling map s, where each element in the scaling map s represents a cross attention map The magnitude of the corresponding elements in : Formula (1) is used to convert the cross attention map Mapping to the interval [s l ,s r ];s l Cross-Attention Map Corresponding to the left endpoint of the matrix, s r Cross-Attention Map Corresponding to the right endpoint of the matrix, by crossing the attention map Multiply it with the scaling map s to amplify the contrast between the response values ​​of the cross-attention map; Perform N contrast enhancement operations, while performing a normalization operation after each operation to restore the original value size: Wherein, j refers to the number of contrast enhancement operations currently performed, j = 1 to N, N is determined by the cosine similarity between text embeddings: where e is the reference text embedding, and e i represents other text embeddings, L is the total number of text tokens, and α adjusts the sensitivity of N to similarity; Cross-Attention Map After performing N times of contrast enhancement operation, it is recorded as Using self-attention map characteristics to restore The shape of the high response area in is consistent with the shape of the editing area; and self-attention map Perform matrix multiplication and get After executing the operation of formula (4), The high response area covers the editing area, but also slightly increases the focus on the non-editing area; Afterwards, Perform attention-enhancing operations: Where k represents each element in the iterative cross attention map; τ1 represents the threshold, which is set based on experience; right After the attention enhancement operation, the attention values ​​below τ1 are set to zero, and the attention values ​​above τ1 are further enhanced.

8. The text-guided image editing device according to claim 7, characterized in that: The mixed sampling module specifically performs the following process: Replace the non-editable regions of the edit branch's potential code with the corresponding non-editable regions of the rebuild branch's potential code: in, represents the potential code obtained by denoising each step of the editing branch, m represents the mask of the editing area, represents the latent code obtained by denoising at each step of the reconstruction branch; Given that The high response area of ​​​​the edited area accurately contains the edited area, and the threshold τ2 is used to Extract the mask m from: Among them, τ2 is set according to experience; Mask-guided hybrid sampling provides different conditional guidance tailored for edited and non-edited regions; ensuring that edited regions follow target cues while keeping non-edited regions pristine: Among them, c s and c t Represent the text embeddings of the source cue word and the target cue word respectively; g s and g t denote the strength of the conditional guidance applied to the non-edited region and the edited region, respectively, and m denotes the mask of the edited region.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images