Text-guided image editing method and device, medium and equipment

Abstract

The invention discloses a text-guided image editing method and device, a medium and equipment, and relates to the technical field of image editing. Aiming at the limitation of an existing method in understanding an editing instruction, global vision and local entity depth perception are firstly carried out on an original image, and the original image is converted into priori knowledge in a natural language form, so that the knowledge of non-explicit modeling of a diffusion model in image editing is made up, the cognitive ability of an image editing scene is improved, and the image editing efficiency is improved. Therefore, priori knowledge is combined with the editing instruction, multi-angle deep editing reasoning is performed on the incomplete or fuzzy editing instruction, a clearer editing scheme with constraint is generated, the interpretability and the constraint of the editing instruction are enhanced, the adaptability of the model in a complex editing task is improved, and the editing efficiency is improved. And the quality of an image editing result is improved.

Description

Technical Field

[0001] This invention relates to the field of image editing technology, and in particular to a text-guided image editing method, apparatus, medium, and device. Background Technology

[0002] In recent years, driven by generative models, Artificial Intelligence Generated Content (AIGC) technology has developed rapidly and has become a major research hotspot in the technology field. AIGC refers to the process of generating various forms of content such as text, images, audio, and video using artificial intelligence (AI) technology, encompassing multiple fields such as natural language processing, computer vision, and audio synthesis. As an important application area of ​​AIGC, image generation is profoundly changing the way images are created and edited. Especially in the field of image editing, the introduction of AI technology has not only significantly improved creative efficiency but also provided new possibilities for more precise and intelligent editing methods.

[0003] Image editing tasks typically include region selection (such as cropping and filling), color adjustment (such as hue, saturation, and contrast), image compositing (such as combining and merging multiple images), style transfer (such as converting an image to a specific artistic style), and object editing (such as removing unwanted objects or replacing specific elements). In AI-driven image editing, image editing not only relies on simple pixel-level processing but also focuses on understanding and generating image content, striving to achieve natural and context-appropriate editing effects without destroying the overall structure of the image.

[0004] In current technologies, with the advancement of deep learning and computer vision, deep learning-based image editing techniques have gradually emerged. In particular, the introduction of Convolutional Neural Networks (CNNs) has enabled computers to automatically learn image features, thereby achieving more precise image editing. Building upon this foundation, generative models such as Generative Adversarial Networks (GANs), Transformer models, and Diffusion Models (DMs) have appeared in recent years. These technologies not only enable computers to analyze and process images but also generate new image content, greatly expanding the application scenarios of image editing. The introduction of diffusion models, in particular, has further improved the accuracy and flexibility of image editing. Diffusion models learn image information from training data through progressive denoising, generating high-quality images and providing a theoretical basis for detailed and diverse image editing.

[0005] With the advent of these deep learning models, image editing has become more convenient and easier to operate. Users no longer need to master complex technical details; they can guide AI to modify images through various means such as natural language, images, or voice. For example, users only need to input a simple text description, such as "change the sky to a sunset tone," and the AI ​​can generate an image effect that meets the requirements based on that description.

[0006] Despite significant advancements in diffusion-based image editing techniques, demonstrating strong advantages in enhancing editing flexibility and automation, existing text-guided image editing methods primarily rely on attention mechanisms for global style modeling and text-image alignment. However, in practical editing tasks, these mechanisms often struggle to accurately interpret editing intent. Furthermore, the editing capabilities of diffusion models are highly dependent on user-provided textual prompts, which are often vague or lack crucial information, making it difficult for the model to precisely understand editing requirements. This impacts the accuracy and controllability of the generated results. In summary, current text-guided image editing methods struggle to accurately understand editing needs, resulting in poor-quality editing outcomes. Summary of the Invention

[0007] Therefore, it is necessary to provide a text-guided image editing method, apparatus, medium, and device to address the aforementioned technical problems.

[0008] The present invention adopts the following technical solution: This invention provides a text-guided image editing method, comprising: Obtain the original image to be edited and the editing instructions to guide the image editing process; Extract global visual features from the original image, and generate a scene semantic description of the original image based on the global visual features and a preset global structured extraction template. The original image is segmented to obtain image regions corresponding to multiple entities. Local visual features of each entity's image region are extracted. Based on the local visual features and the preset entity structure extraction template, an entity semantic description is generated. The interaction relationship between entities is extracted to obtain a semantic relationship map. Based on scene semantic description, entity semantic description, semantic relationship graph, and editing instructions, an editing constraint text description is obtained through multi-angle editing reasoning; the multi-angle editing reasoning includes at least one of instruction analysis, entity analysis, description analysis, and positional relationship analysis; Guided by the textual description of editing constraints, image editing is performed through a diffusion generation process based on the original image to be edited.

[0009] Optionally, the global visual features and local visual features are extracted by a visual feature extractor, which includes a visual encoder, a Q-Former, and a visual feature projection layer. Scene semantic descriptions and entity semantic descriptions are generated through a scene-aware adapter, which includes a large language model. The large language model is used to generate scene semantic descriptions of the original image based on global visual features and a preset global structured extraction template, and to generate entity semantic descriptions based on local visual features and a preset entity structured extraction template.

[0010] Optionally, the step of generating a scene semantic description of the original image based on global visual features and a preset global structured extraction template specifically includes: The scene semantic description of the original image is generated using the following formula based on global visual features and a preset global structured extraction template: , ; in, For the scene semantic description of the original image, This is a natural language representation corresponding to the global description. Natural language expression corresponding to the image style For the corresponding scenario type of natural language expression, For global visual features, This is a global structured extraction template. This is the template content corresponding to the global description. Template content corresponding to the image style, Template content for the corresponding scenario type, For large language model decoders, For text encoders.

[0011] Optionally, the step of generating entity semantic descriptions based on local visual features and a preset entity structure extraction template specifically includes: The following formula is used to generate entity semantic descriptions based on local visual features and a pre-defined entity structure extraction template: , ; in, For entity semantic description, For the natural language representation of the corresponding entity description, For the natural language representation of the corresponding entity type, For the first Local visual features of the image region corresponding to each entity For entity structure extraction template, The template content for the corresponding entity description, Template content for the corresponding entity type, For text encoders, It is a decoder for large language models.

[0012] Optionally, the extraction of interaction relationships between entities to obtain a semantic relationship graph specifically includes: The semantic relationship map is obtained by extracting the interaction relationships between entities from the original image using the OpenPSG framework.

[0013] Optionally, the step of obtaining the editing constraint text description through multi-angle editing reasoning based on scene semantic description, entity semantic description, semantic relationship graph, and editing instructions specifically includes: Based on scene semantic description, entity semantic description, semantic relationship graph and editing instructions, the large language model guides reasoning based on the preset thought chain reasoning question template corresponding to the task type of the editing instructions, and generates editing constraint text description based on the preset task example corresponding to the task type of the editing instructions. The task types include: deletion, addition, and replacement.

[0014] This invention provides a text-guided image editing device, comprising: The acquisition module is used to acquire the original image to be edited and the editing instructions used to guide the image editing process; The global extraction module is used to extract global visual features from the original image, and generate a scene semantic description of the original image based on the global visual features and a preset global structured extraction template. The local extraction module is used to perform instance segmentation on the original image to obtain image regions corresponding to multiple entities, extract local visual features of the image regions corresponding to each entity, generate entity semantic descriptions based on local visual features and preset entity structure extraction templates, and extract the interaction relationships between entities to obtain a semantic relationship graph. The editing constraint module is used to obtain an editing constraint text description through multi-angle editing reasoning based on scene semantic description, entity semantic description, semantic relationship graph and editing instructions; the multi-angle editing reasoning includes at least one of instruction analysis, entity analysis, description analysis and positional relationship analysis; The Guided Editing module is used to guide image editing through a diffusion generation process based on the original image to be edited, guided by an edit constraint text description.

[0015] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described text-guided image editing method.

[0016] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described text-guided image editing method.

[0017] The above-mentioned at least one technical solution adopted in this invention can achieve the following beneficial effects: This invention addresses the limitations of existing methods in understanding editing instructions by first performing global visual and local entity depth perception on the original image and converting it into prior knowledge in natural language form. This compensates for the lack of explicit modeling of knowledge in the diffusion model during image editing, improving the cognitive ability of the image editing scenario. By combining prior knowledge with editing instructions, incomplete or ambiguous editing instructions can be subjected to multi-angle deep editing reasoning, thereby generating clearer and more binding editing schemes. This not only enhances the interpretability and binding nature of editing instructions but also improves the model's adaptability to complex editing tasks, ultimately improving the quality of image editing results. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0019] Figure 1 This invention provides a schematic flowchart of a text-guided image editing method. Figure 2 A schematic diagram illustrating the framework of the proposed KRG-Edit method provided by this invention; Figure 3 A schematic diagram of a scene perception pre-training process provided by the present invention; Figure 4 A schematic diagram of the overall framework for the knowledge reasoning stage provided by this invention; Figure 5 A schematic diagram illustrating a qualitative comparison between the method of this invention and the masked hybrid baseline method provided by this invention; Figure 6 A schematic diagram illustrating a qualitative comparison between the method of this invention and a baseline method based on an attention mechanism, provided for the purposes of this invention; Figure 7 A schematic diagram of a text-guided image editing device provided by the present invention; Figure 8 A schematic diagram of a computer device for implementing a text-guided image editing method provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0021] Currently, image editing based on mask blending is one of the important applications of diffusion models in the field of image processing. Its core idea is to accurately locate the region to be modified using a user-provided editing mask or region of interest, and then utilize the generative capabilities of the diffusion model to complete local content editing while maintaining the semantic consistency of the non-edited regions.

[0022] Pioneering methods such as Blended Diffusion employ CLIP-based noise fusion strategies to perform operations such as adding, removing, and replacing local content. To improve local editing results in complex scenes, MAG-Edit utilizes a mask-based attention cues to adjust latent features through an inference-stage strategy, thereby improving cue alignment and structural consistency, making it suitable for various complex compositions. However, their reliance on manual mask annotations highlights the limitations of their application on geometrically complex objects.

[0023] Driven by the increasing demand for high-quality editing, high-resolution optimization methods such as HD-Painter and PowerPaint have emerged. For example, PowerPaint introduces a cue-aware mechanism to enhance detail control, thereby enabling high-resolution image editing. Meanwhile, automatic mask generation methods such as Inpaint Anything and Learnable Regions combine Segment Anything (SAM) models with cross-modal attention to improve editing efficiency, although they still face challenges in small object detection and complex scene understanding. Other multimodal fusion techniques, including SmartBrush and ZONE, attempt to enhance user control by integrating text, geometry, and other contextual cues. However, due to the limitations of semantic parsing, they sometimes fail to accurately understand user intent (e.g., with a cue of "red circular balloon," the shape mask may over-constrain color generation, leading to a final result that is not entirely consistent with the text description; or when handling the ambiguous instruction "make the picture more vivid," mask extraction may deviate from the user's true intent), resulting in unexpected generated results.

[0024] Image editing techniques based on attention mechanisms differ from mask blending methods. They adjust specific regions of an image by directly manipulating attention weights (such as self-attention and cross-attention) within the diffusion model. The advantage of this approach is that it eliminates the need for explicit user-provided masks; the model can adaptively identify and edit target regions, making image editing more flexible and efficient. Essentially, this technique relies on a deep understanding and control of the diffusion process, achieving precise image modification through manipulation of feature layers.

[0025] The groundbreaking prompt-to-prompt (P2P) editing approach enables text-driven local editing by replacing or mixing cross-attention maps, but its fine-grained adjustment is limited.

[0026] Subsequent improvements, such as the MasaCtrl method, utilize attention mechanisms to query local content and entity textures across images, supporting multi-view compositing and non-rigid editing of the same subject. However, they still struggle with complex structural editing. To address direct manipulation of attention mechanisms, some methods employ inversion techniques to enhance edit controllability. For example, Null-text Inversion and LEDITS++ optimize implicit constraints, improving edit fidelity by ensuring the texture of modified regions remains consistent with the overall image. To advance natural language understanding in graphics editing, researchers have developed multimodal fusion methods, including InstructPix2Pix and BrushEdit. These methods introduce large language models to enhance the interpretation of text editing instructions, enabling more flexible user interaction and supporting complex editing commands. Simultaneously, parallel development of general editing frameworks, represented by InstructDiffusion and EmuEdit, aims to integrate various editing tasks into a unified architecture, significantly broadening their potential applications. However, these methods still face ongoing challenges, exhibiting inconsistent performance across different editing scenarios, limiting their overall generalization capabilities.

[0027] Despite significant progress in diffusion-based image editing techniques, which have demonstrated strong advantages in enhancing editing flexibility and automation, current methods still face numerous challenges in terms of editing quality and consistency.

[0028] (1) Insufficient image understanding Insufficient image understanding is a key issue affecting editing quality. Existing diffusion models primarily rely on self-attention and cross-attention mechanisms for image content modeling, but they still have limitations in parsing editing targets, maintaining overall image style consistency, and reconstructing complex structures. Common problems include inconsistencies between the edited area and background style, loss of local details, and distorted structural relationships, affecting the realism and visual effect of the edit. Specifically, while self-attention mechanisms can capture global features to maintain overall style consistency, the feature learning of local areas may conflict with the overall style after the introduction of editing constraints, resulting in a mismatch between the edited area and the original image. Furthermore, mask-based editing methods rely entirely on diffusion models for reconstruction after removing the original content. Due to the lack of detailed features of the original entities and the lack of appropriate constraints on those details, the generated content often differs from the background in terms of shape, attributes, lighting, or texture, lacking coherence and further reducing the quality of the edit.

[0029] (2) Lack of editing knowledge On the other hand, the lack of editing knowledge is also a major bottleneck for current diffusion model editing methods. The editing capabilities of diffusion models not only rely on their own visual prior knowledge but also need to be combined with external constraints to ensure the rationality of the editing results. However, existing methods still have significant shortcomings in editing instruction parsing, region segmentation, and model knowledge utilization. For example, current methods heavily rely on text prompts provided by users, but user descriptions often lack key information, making it difficult for the model to accurately understand the editing requirements, thus affecting the accuracy and controllability of the generated results. For example, the editing instruction "add a bear to the bedside" does not specify details such as material, size, or style, which may lead to some methods generating content that does not match the scene (e.g., generating a real bear instead of the expected teddy bear).

[0030] Furthermore, mask-based editing methods typically perform edits only within a local area. When the task involves large-scale content replacement, the constraints of local masks can lead to distorted proportions or abnormal shapes in the generated content, weakening the naturalness and consistency of the edit. For example, after adjusting the position of an object, the model fails to maintain reasonable lighting and perspective, disrupting the overall visual harmony, or different entity proportions appear inconsistent, severely affecting visual consistency. This may be because the editing task shatters the common-sense knowledge of diffusion models (such as the size relationships between generated entities).

[0031] In summary, existing methods still have significant shortcomings in image understanding and knowledge incorporation, severely impacting editing quality and application scope. Therefore, improving the image understanding capabilities of diffusion models and effectively incorporating editing knowledge has become an important research direction for enhancing editing quality.

[0032] To address the aforementioned challenges, this invention proposes a knowledge reasoning and guided image editing method (KRG-Edit). This method designs a scene-aware knowledge reasoning module that combines a large language model (LLM) with a visual language model (such as the BLIP-2 model) for global and entity perception, integrating global and local semantic information to improve the understanding of editing instructions. Furthermore, it utilizes image segmentation and other models to construct supplementary knowledge representations of entity location information and interaction relationships, thus forming multi-layered scene knowledge. This compensates for the lack of explicit knowledge encoding in the pre-training of diffusion models and enhances the model's robustness in handling complex editing scenarios.

[0033] To enhance the reasoning capabilities required for parsing editing instructions, chain-like reasoning and contextual learning are further employed based on scene knowledge to parse the instructions, infer the implicit editing intent, and fill in missing details. This generates clear and binding editing knowledge, improving editing accuracy. This enables the large language model to infer missing or ambiguous details based on common-sense knowledge, generating explicit and interpretable editing plans. This significantly improves the reliability and interpretability of image editing tasks, ensuring more accurate and predictable results.

[0034] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0035] Figure 1 This is a schematic diagram of a text-guided image editing method according to the present invention, which specifically includes the following steps: S101: Obtain the original image to be edited and the editing instructions used to guide the image editing.

[0036] S102: Extract global visual features from the original image to generate a scene semantic description of the original image based on the global visual features and a preset global structured extraction template.

[0037] S103: Perform instance segmentation on the original image to obtain image regions corresponding to multiple entities, extract local visual features of the image regions corresponding to each entity, generate entity semantic descriptions based on local visual features and preset entity structure extraction templates, and extract the interaction relationships between entities to obtain a semantic relationship map.

[0038] S104: Based on the scene semantic description, entity semantic description, semantic relationship graph and editing instructions, obtain the editing constraint parameter text description through multi-angle editing reasoning; the multi-angle editing reasoning includes at least one of instruction analysis, entity analysis, description analysis and positional relationship analysis.

[0039] S105: Guided by the edit constraint text description, image editing is performed through a diffusion generation process based on the original image to be edited.

[0040] For ease of explanation, the following description focuses solely on the server as the executing entity. The server mentioned in this invention can be a server set up on a business platform, or a device such as a desktop computer or laptop computer capable of executing the solution of this invention.

[0041] To overcome the key limitations of current diffusion-based editing methods, especially the limited understanding of editing objectives and heavy reliance on vague text prompts, this invention proposes KRG-Edit, which significantly improves editing quality and controllability by utilizing scene-aware knowledge reasoning.

[0042] Figure 2 This is a schematic diagram of the framework of the proposed KRG-Edit method in this invention. Figure 2 It can be seen that it mainly includes three stages: (1) Scene perception pre-training, fine-tuning the visual understanding of LLM to achieve a deeper image understanding; (2) Constructing a hierarchical scene representation, integrating global and local visual information; (3) Knowledge-guided reasoning, using chain thinking prompts and context learning, generating precise editing constraints by jointly analyzing scene knowledge and editing goals.

[0043] (1) Scene perception pre-training To achieve a deep understanding and comprehensive perception of edited images, this invention designs a novel scene-aware pre-training task. This task integrates the visual representation extraction capabilities of the BLIP-2 model with the text generation capabilities of a large language model, enabling multi-level perception of the global scene and individual instances within an image. The perceived content includes multi-dimensional information such as visual descriptions, attribute features, and style features. The core of this method lies in enhancing the complementarity of visual and semantic features, thereby constructing fine-grained scene prior knowledge and laying the foundation for subsequent knowledge reasoning tasks.

[0044] Figure 3 This is a schematic diagram of a scene perception pre-training process in this invention, as shown below. Figure 3 As shown, the scene-aware pre-training task mainly consists of two core components: a visual feature extractor and a scene-aware adapter. The visual feature extractor, based on the BLIP-2 model, includes a visual encoder, a Q-Former, and a visual feature projection layer, responsible for extracting high-quality visual feature sequences from the input image. The scene-aware adapter is a task adaptation module designed specifically for large language models, optimizing the model's adaptability to scene-aware tasks and enhancing its cross-modal understanding capabilities.

[0045] The specific process is as follows: First, the original image to be edited is processed by a visual feature extractor to generate a visual feature sequence. These feature sequences are then input into a large language model to enhance its ability to perceive image region features. Subsequently, the large language model uses an autoregressive generation method to gradually convert visual information into a coherent text description. The autoregressive generation method ensures the coherence and consistency of the text, thereby achieving a deep fusion of visual and linguistic information. This process ensures that the model can accurately understand the global structure, entity features, and semantic relationships of the image, thereby improving scene recognition capabilities.

[0046] 1) Visual Feature Extractor The visual feature extractor is a key component based on the BLIP-2 model, primarily responsible for extracting rich visual feature information from the input image. This module consists of three parts: a visual encoder, a Q-Former, and a visual feature projection layer.

[0047] In one or more embodiments of the present invention, the visual encoder can be implemented based on a pre-trained Vision Transformer (ViT) encoder. Although the visual encoder can capture rich visual features, direct input to the language model may lead to semantic mismatch due to the modal differences between the visual features and the language model input. To solve this problem, the Q-Former module is introduced for modal alignment, drawing inspiration from the BLIP-2 design. As the core component of the visual feature extractor, the Q-Former employs a multi-head attention mechanism and a feedforward network to effectively align visual features with linguistic semantic information, thereby establishing a connection between the two.

[0048] Furthermore, in one or more embodiments of the present invention, to optimize the representation of visual features, the visual feature extractor introduces a trainable visual feature projection layer. On one hand, this achieves dimensionality mapping, mapping the visual feature vectors output by the Q-Former to the language model input space, ensuring dimensionality matching. On the other hand, it performs feature fusion, enhancing the interaction between visual and linguistic features, enabling visual information to effectively participate in subsequent knowledge reasoning tasks. Through the collaborative work of the visual encoder, Q-Former, and visual feature projection layer, the visual feature extractor can output high-quality, semantically rich visual representations, providing strong support for subsequent scene perception and knowledge reasoning.

[0049] 2) Scene-aware adapter During scene-aware pre-training, the parameters of the large language model remain frozen, meaning its built-in common-sense knowledge and reasoning abilities remain unchanged. Therefore, the trainable parameters are primarily concentrated on the Q-Former and visual feature projection layers. Although the visual feature extractor is optimized to extract high-quality global visual features and local entity features, ensuring these features are understandable by the language model, the large language model's knowledge and reasoning abilities remain at the original training stage because it hasn't been fine-tuned. This can lead to insufficient understanding in complex scenes. Furthermore, computational resource constraints and limited training data size are also key challenges in the pre-training process.

[0050] To address the aforementioned challenges, this invention introduces a scene-aware adapter based on Low-Rank Adaptation (LoRA) for adaptive adjustment. The core idea of ​​this adapter is to introduce a low-rank matrix into the weight matrix of a pre-trained large language model, thereby efficiently fine-tuning parameters. Compared to directly fine-tuning the entire model, this method optimizes only the low-rank portion of a specific weight matrix, significantly reducing computational resource consumption and storage overhead. Furthermore, through incremental learning, LoRA can enhance the adaptability of large language models in specific tasks while preserving the original knowledge and reasoning capabilities to the greatest extent possible.

[0051] In terms of implementation details, the query weights in the Transformer self-attention module ( ) and value weight matrix ( Applying LoRA to [the target area] yields the best results. Therefore, in [the context of LoRA implementation]... and A LoRA adapter is added to the weight matrix to optimize the understanding of visual features by the large language model.

[0052] 3) Two-stage training process The scene perception pre-training process references the two-stage training strategy of BLIP-2, which is divided into the Q-Former vision-language joint pre-training stage and the scene perception adapter fine-tuning stage.

[0053] Phase 1: Q-Former joint pre-training for vision and language The goal of the first phase is to train Q-Former so that the learnable query vectors can extract the most informative visual representation from the text. Specifically, this involves establishing a pre-training task for image-text pairs. Q-Former receives a sequence of visual features generated by a visual encoder, text embeddings generated by a language model encoder, and a set of learnable query vectors. By setting three main tasks—image-text contrastive learning, image-based text generation, and image-text matching—the parameters of Q-Former are optimized to ensure that the learnable query vectors establish effective alignment between the visual and linguistic modalities.

[0054] Phase Two: Fine-tuning of the Scene Awareness Adapter Building upon the first stage, the second stage introduces a scene-aware adapter to fine-tune the model in a lightweight manner to adapt it to specific scene-aware tasks. In this stage, the scene-aware adapter receives visual features processed by the Q-Former and visual feature projection layers. Then, the visual feature sequence, along with the question template, is used as input to a large language model to train the adapter's ability to generate image descriptions. The adapter then progressively outputs descriptions in an autoregressive manner, transforming visual information into coherent text.

[0055] Due to the architecture and attention mechanism of the large language model, the training loss function for scene-aware adapter fine-tuning adopts cross-entropy loss, which measures the difference between the image description generated by the model and the actual image description in each decoding step. Specifically, the training objective of the model is to maximize the log-likelihood of each word in the image description based on the given image features and prompt text. Therefore, the loss function for scene-aware adapter fine-tuning is defined as Equation (1):

[0056] (1) In the above formula , indicating the first in the sequence The subsequence consisting of all words preceding the given word Indicates the given prompt template Visual feature sequences and subsequences Under the given conditions, the model predicts the next word as The probability. The algorithm optimizes the model parameters by maximizing the log-likelihood of the entire sequence. .

[0057] (2) Constructing a hierarchical scene representation Based on scene-aware pre-training, this invention further constructs a multi-layered scene knowledge representation to support knowledge reasoning for editing tasks, including global scene knowledge, local entity knowledge, and entity interaction knowledge. Each part of the knowledge provides semantic information at different granular levels.

[0058] 1) Global scenario knowledge Global scene knowledge extraction aims to extract global features such as global description and style information from an image through a holistic understanding. Specifically, based on scene-aware methods, global scene knowledge extraction uses cue templates to enable the model to extract global semantic features from the original image, including scene description, overall style, and other scene-related information. The specific process is as follows:

[0059] Enter the original image Global visual features are obtained through a visual feature extractor. As shown in equation (2): (2) in, It is a scene-aware pre-trained visual feature extractor. Subsequently, a large language model, also pre-trained in scene awareness, extracts global visual features... and structured prompt templates Generate a scene semantic description, as shown in equation (3):

[0060] (3) in, It is a text encoder. A large language model decoder pre-trained with scene awareness. This indicates a feature concatenation operation, outputting triples. These correspond to natural language expressions for global description, image style, and scene type, respectively.

[0061] 2) Local entity knowledge Local entity knowledge is a crucial component of scene knowledge fusion, improving the quality of new entity generation through precise semantic constraints and spatial layout guidance. This invention constructs a local entity knowledge representation that integrates entity semantic features and spatial layout information, meticulously capturing the features of image entities from two dimensions: attribute description and geometric location. In terms of technical implementation, an image segmentation model is used to obtain basic data such as entity category labels and instance masks. Simultaneously, based on a scene-aware paradigm, a structured prompt template is used, combined with entity attribute information, to obtain the semantic description of the entity. The specific implementation is as follows:

[0062] Entity location information extraction primarily involves accurately parsing the original image using an image segmentation model. This process precisely segments each entity within the original image and outputs its spatial location, width, height, and other geometric information. Given an original image... The structured geometric representation is obtained after processing by the segmentation network, as shown in equation (4):

[0063] (4) in, For instance segmentation mask tensors, The bounding box coordinate matrix, To determine the number of entities detected. For each detected entity... Its spatial positioning information can be expressed as equations (5) and (6):

[0064] (5) (6) In the formula, These represent the width and height of the entity, respectively. These are the coordinates of the entity's center. This geometric encoding provides spatial constraints for subsequent layout reasoning.

[0065] The extraction of entity description information is similar to the global scene knowledge extraction method. It mainly achieves fine-grained joint parsing of attributes, descriptions, and entity categories through the synergy of a pre-trained scene-aware model and structured prompt templates. The specific process is as follows: Given the first Image regions corresponding to each entity (in (For instance segmentation mask), local visual features are extracted through an entity-aware visual encoder, as shown in equation (7): (7) in, This is a visual feature extractor pre-trained on an entity perception task, which is then combined with an entity-level structured cue template. The large language model, pre-trained for scene awareness, generates detailed semantic descriptions of entities, as shown in equation (8): (8) in, It is a text encoder. It is a large language model decoder that has been pre-trained for scene awareness. This indicates a feature concatenation operation. The output is... These correspond to natural language expressions for entity descriptions and entity types, respectively.

[0066] 3) Entity interaction knowledge To further enhance the realism and rationality of generated editing instructions, this invention extracts the interaction relationships between entities in images and constructs medium-granularity scene knowledge. This medium-level semantic information can better capture the complex relationships between entities in images, providing support for refined editing reasoning. Specifically, it uses a scene graph model to analyze entities in images and their semantic relationships, constructing a structured relation representation to support refined editing reasoning. The implementation method is as follows:

[0067] For the input image, entities are first extracted using an image segmentation model, and the semantic relationships between entities are parsed to generate a semantic relationship scene graph of the image. The scene graph model can automatically extract and model the interaction relationships between entities in the image and transform these relationships into a structured graph representation. Specifically, this invention is based on the OpenPSG framework to construct a semantic relationship graph. ,in, Represents all entities in the image, each entity Represents a specific object in the image. Represents a set of relationships between entities, where each relationship... Connect two entities and describe the semantic relationship between them.

[0068] Furthermore, the format of semantic relation triples can be defined as equation (9): (9) in, Represents the set of all relation triples. These are semantic descriptions representing interactive relationships, such as "holding hands," "located," and "lying down." Through this structured representation of relationships, the semantic information in the image is systematically expressed, providing a convenient interface for subsequent reasoning and manipulation.

[0069] (3) Knowledge-guided reasoning In image editing tasks, accurately encoding the geometric constraints and interactive semantics of the target entity based on scene knowledge of the original image and user editing instructions is a core problem that urgently needs to be solved in image understanding. To address this challenge, a knowledge reasoning pre-training task based on fine-tuning of a large language model is proposed, and the reasoning ability and knowledge utilization efficiency of the large language model are enhanced through context learning and thought chain reasoning. In the reasoning stage, structured scene knowledge is transformed into explicit constraints that can be executed by the generative model, thereby achieving joint modeling of geometric and semantic constraints. Figure 4 The diagram illustrates the overall framework of the knowledge reasoning stage. This invention utilizes a large language model to guide reasoning based on scene semantic descriptions, entity semantic descriptions, semantic relationship graphs, and editing instructions. It uses preset thought chain reasoning question templates corresponding to the task types of the editing instructions to generate editing constraint text descriptions based on preset task examples corresponding to the task types of the editing instructions. The task types include: deletion, addition, and replacement.

[0070] This module first receives scene knowledge from the image perception module and editing instructions input by the user, integrating this information into a prompt template. To improve the reasoning ability of the large language model, example prompts are added during context learning to help the model better understand task requirements. Simultaneously, a thought chain reasoning method is introduced to further enhance the large language model's ability to understand and apply complex semantic relationships. Finally, the processed context examples and prompt templates are input into the large language model for knowledge reasoning fine-tuning, thereby generating explicit constraints on the editable entity, including key information such as location, size, and detailed description.

[0071] 1) Modeling of reasoning tasks This invention formalizes the knowledge reasoning pre-training task into a sequence-to-sequence conditional generation task, and defines the knowledge generation process as a conditional probability modeling problem. Specifically, the knowledge generation task is set as the following conditional probability expression (10):

[0072] (10) in, To incorporate prior knowledge of the scenario and user instructions into the prompt template text, This involves editing the constraint sequence (position, size, relationships, etc.) of entities, i.e., the model's input and output. Based on the language understanding and generation capabilities of the large language model, the following input prompt template text is set. and the output edit entity constraint sequence .

[0073] Input prompt template To enable the large language model to effectively understand and process scene information from different dimensions, the input data is designed as a structured natural language description. Prior knowledge extracted by the image perception module and user prompts are integrated into the prompt template, aiming to enable the model to infer the required editable entity information from structured scene knowledge and user-provided text instructions. Figure 4 The input suggestion template section (top left corner) displays the input suggestion template. An example. Specifically, it includes the following parts:

[0074] Global scene knowledge: Provides a global description of the image, including scene type, style, and overall layout. For example: "There is a picture description as follows: [A few sheep grazing near a villa by the sea]". This information helps the model understand the overall background and context of the image.

[0075] Local entity knowledge: This involves providing detailed descriptions of each entity in the image and its attributes, including location, size, and visual features. For example, "Each entity is described as follows: [sky], [building], [sheep], [grasses], [sea]". This information enables the model to recognize and understand the specific features and location information of each entity in the image.

[0076] Entity interaction relationships: This describes the interaction relationships between entities in the image, including spatial and semantic relationships. For example, "The relationship between each entity is as follows: [building] under [sky], [sheep] on [grasses], [sea] behind [building]." This information helps the model understand the interactions and associations between entities.

[0077] Instruction prompts: Clearly state the user's editing needs, instructing the model to generate specific information. For example: "[villa] needs to be replaced with [building]. Based on the above information, please infer its reasonable location / detailed description?" This information guides the model to generate the necessary editing constraint text description based on the context.

[0078] Output editing constraints To enable output editing constraints To better align with the generative characteristics of large language models and effectively express the geometric constraints, positional relationships, and interaction relationships of entities, in one or more embodiments of this invention, HTML format can be selected as the output format. HTML format not only clearly expresses the positional information, size, and other editing constraints of entities, but also displays the relationships between entities through intuitive, structured tags. The definition is as shown in equation (11):

[0079] (11) in, The language format must be HTML. It is the location information of the edited entity. It is the descriptive information of the edited entity.

[0080] 2) Setting up questions in the thinking chain To effectively guide large language models to deeply understand prior knowledge and perform complex knowledge reasoning, this invention designs a thought-chain reasoning questioning mechanism based on prompt templates. This mechanism guides the model step-by-step to analyze details in images, helping it capture deeper semantic information and thus improving the accuracy and rationality of its reasoning. The core of thought-chain questioning lies in unlocking key information in complex scenes from multiple angles through a series of step-by-step questions, enabling the model not only to understand individual entities but also to correctly grasp the relationships between entities.

[0081] This mechanism is inspired by human cognitive processes. When analyzing complex problems, humans typically do so through step-by-step analysis, analogical reasoning, and inductive summarization. Based on this, the Mind Chain Questioning mechanism simulates this cognitive process, enabling large language models to understand and reason about complex scenarios step-by-step, just like humans do. This allows it to handle multi-layered relationships and complex scenarios that traditional reasoning methods cannot address.

[0082] Specifically, various types of thought chain question templates were designed based on different editing task requirements. Each template sets different questioning strategies for different task objectives, including instruction analysis, entity analysis, description analysis, and positional relationship analysis, systematically guiding the model to conduct in-depth knowledge reasoning. Table 1 lists the designed thought chain question templates and their specific content illustrations.

[0083] Table 1. Content of the Mind Chain Question Template In practical applications, the thought chain questioning mechanism is integrated into the knowledge reasoning process. Based on the specific task type, the system randomly selects suitable thought chain question templates to better adapt to different editing needs. Guided by these question templates, the large language model can gradually analyze and understand key information in images, enhancing its reasoning ability for complex scenes and providing stronger cognitive support for subsequent knowledge reasoning and editing tasks.

[0084] 3) Contextual learning To enable large language models to better understand image-based prior knowledge editing tasks, this invention employs a context-based learning approach. The core idea of ​​context learning is to provide multiple example tasks as contextual cues to help the model understand the semantic background and reasoning logic of the task. These example tasks not only provide the model with clear task objectives but also guide the model on how to use structured input data (such as cue templates and inference chains) to generate reasonable outputs.

[0085] Specifically, the context learning in the knowledge reasoning task consists of multiple high-quality example tasks, each comprising three key parts: a prompt template, a thought chain reasoning, and a reasoning result. These tasks cover different image editing scenarios, such as adding, deleting, and replacing entities. Each task details the input / output format, reasoning process, and specific steps for result generation, ensuring that the large language model can quickly adapt and generate reasonable editing constraints when faced with new tasks.

[0086] Through this structured example, large language models can learn how to extract key information from prompt templates, analyze task requirements through thought chain reasoning, and ultimately generate logically consistent text descriptions of editing constraints. This contextual learning approach can significantly improve the model's performance in editing reasoning tasks.

[0087] Finally, the constraint text description can be edited as a guide, and image editing is performed based on the original image to be edited through a diffusion generation process. The core objective of this invention is to enhance the editing instructions used to guide image editing. The enhanced constraint text description is still a text format, but compared to the initial editing instructions, it contains more prior knowledge, is more complete and clear, and has better editing constraints. Therefore, this invention does not limit how the enhanced constraint text description is applied to image editing. For example, existing mature text-guided image editing methods can be used, replacing the application of the initial editing instructions with the constraint text description obtained by this invention.

[0088] Furthermore, in order to improve the semantic alignment of the edited area and the background consistency of the non-edited area, in one or more embodiments of the present invention, image editing can be performed using editing constraint instructions guided by knowledge based on attention modulation.

[0089] Specifically, all editing operations can be decomposed into two basic processes: entity deletion and addition. The control over the editing region is enhanced or weakened by adjusting the cross-attention weights corresponding to the target entity tokens during sampling. This mechanism ensures that newly generated entities accurately fall within the target region. Simultaneously, the deletion operation relies on a region suppression mechanism to effectively prevent deleted entities from reappearing in the background. Furthermore, to better maintain the consistency of the background region, this invention proposes a two-stage background preservation method. This method divides the denoising process into two stages: entity generation and background optimization. In the entity generation stage, the focus is on the editing region, ensuring that the target entities are generated reasonably according to preset constraints. In the background optimization stage, fine-grained background information fusion ensures that non-editing regions are not disturbed during the editing process, avoiding unnecessary changes. This two-stage generation strategy not only effectively improves the quality of edited entities but also ensures the coherence of background content, making the final generated image more natural.

[0090] Entity editing for attention modulation: (1) Review of attention mechanisms In the diffusion model, the attention mechanism plays a central role in guiding image editing, primarily regulating entity generation and deletion through self-attention and cross-attention. In the denoising process... During the step, convolutional layer Extracted depth image features First, the image passes through a self-attention layer to ensure the rationality of the global structure. Then, these features pass through a cross-attention layer for semantic alignment with the editing constraint text description, ensuring that the generated image content conforms to the user's editing intent. The calculation of the attention mechanism is shown in Equation (12):

[0091] (12) in, Query features as the feature dimension Key features Sum value characteristics The source of attention determines the specific function of the attention mechanism: In the self-attention layer , and All data originates from image features. This layer establishes long-range dependencies within the image by calculating the similarity between different pixel regions, ensuring the rationality of object layout, perspective relationships, and overall composition.

[0092] In the cross-attention layer, It still originates from image features, and and This comes from the edit constraint text description. This layer maps the concepts in the edit constraint text description to the visual feature space by calculating the semantic similarity between the text and the image, directly affecting the generation location, shape, and attributes of the target entity. By adjusting the attention weight of a specific text token, the generation strength of the corresponding entity can be enhanced or weakened, achieving precise text-guided editing.

[0093] (2) Detailed description of the method Based on the above analysis, a method for entity editing by dynamically adjusting attention weights is proposed. This method does not require additional training, but directly intervenes in the weight distribution in the attention mechanism to precisely control the existence, position and attributes of the target entity in the image.

[0094] Self-attention analyzes the correlation between regions within an image to determine the logic for generating the image structure. For deletion tasks, it reduces the self-attention weight of the region to be deleted (e.g., by setting the weight to a minimum value) to block information interaction between that region and surrounding features. For adding entities, it increases the self-attention weight within the target region to strengthen the correlation strength of local features, thereby ensuring the correctness of the entity editing region.

[0095] Cross-attention establishes a mapping relationship between text descriptions and image features, controlling the intensity of semantic content generation. Therefore, by enhancing or suppressing the cross-attention weights corresponding to entity cues, alignment of entity semantics within the editing region with the image space can be achieved.

[0096] This method mainly includes three key steps: feature parsing and noise initialization, self-attention dynamic control, and cross-attention guidance. It ensures more precise control over the target entity during the editing process while maintaining the overall consistency of the background.

[0097] Step 1: Feature Analysis and Noise Initialization In traditional diffusion model editing methods, directly using inverted noise for editing often results in edits that are constrained by the inherent structure of the original image, making it difficult to freely adjust the editing effect. To address this, a two-stage feature processing mechanism is designed to break the structural inertia of the original content and provide greater freedom for subsequent editing. First, DDIM inversion is used to transform the original image... Encoded as a time-series noise sequence To restore each time step in the diffusion process The potential characteristics are calculated as shown in equation (13):

[0098] (13) During this process, each time step is extracted simultaneously. With network layer Attention key features , This is used to establish a structural feature library of the original image. To further enhance the plasticity of the editing area, a region erasure mechanism is introduced during the noise initialization stage to erase the region to be deleted. Replace with random noise This removes the original information from the area, providing greater freedom for subsequent editing. The process is calculated as shown in equation (14):

[0099] (14) in, This represents element-wise multiplication. This is random noise. This strategy can effectively remove the inherent constraints of the region to be edited, allowing the model to fully rely on editing guidance information to generate new content during subsequent denoising processes.

[0100] Step Two: Dynamic Regulation of Self-Attention During the denoising process, to ensure that the target entity can be generated according to the editing instructions, and to prevent other areas from being interfered with, at each time step... and attention layer Self-attention weight adjustment will be implemented. Since the processing of attention features differs between deleting and adding entities, adjustments will be made to these two tasks separately, but the core idea remains the same: to change the structural information interaction of the target region through self-attention adjustment in order to precisely control the editing effect.

[0101] During the entity deletion process, based on the deletion region mask By adjusting the key features of the self-attention module Sum value characteristics This prevents information exchange between this area and other areas. The process is as shown in equations (15) and (16):

[0102] (15) (16) in, This is a very small value, which will cause the attention weight of the deleted region to approach zero in the SoftMax calculation, ensuring that the region no longer interacts with surrounding features during the editing process, thereby avoiding the occurrence of residual artifacts. The adjusted self-attention calculation formula is as shown in equation (17):

[0103] (17) During the process of adding entities, we want to enhance the target area. Internal self-attention weights ensure that newly generated content remains consistent with its surroundings. Simultaneously, to prevent interference from external regions, attention to non-target regions is appropriately attenuated. The adjustment formulas are shown in equations (18) and (19):

[0104] (18) (19) in, Used to enhance target areas The internal relationships between them allow newly added entities to be embedded more naturally into the image, while This is used to weaken the influence of non-editable regions, thereby reducing the interference of irrelevant content on the generation of new entities. The adjusted self-attention mechanism calculation formula is the same as equation (17).

[0105] This self-attention dynamic control strategy ensures complete suppression of deleted regions while enhancing the spatial consistency of added entities.

[0106] Step 3: Guiding Cross-Attention When achieving precise editing of entities, relying solely on self-attention adjustments is insufficient. This invention further utilizes cross-attention control to ensure semantic alignment between the edit constraint text description and the target region, enhancing the semantic consistency of the edited region while preventing the deleted region from being affected by residual effects of the original editing instructions.

[0107] In the cross-attention mechanism, query features Derived from image features, while key-value features and From text embedding. To prevent entities from deleted regions from reappearing in the image, the text token corresponding to the deleted entity (indexed as...) is reduced. The cross-attention weights are used to spatially reduce the deletion regions. The attention effect is as shown in Equation (20). This adjustment ensures that the deleted entities are not regenerated in the image.

[0108] (20) When adding an entity, we want to enhance the entity addition token (index representation). In the editing target area The attention weight within the token is adjusted, while the attention distribution of the token in other areas is reduced to avoid generating new entities in irrelevant areas. The specific adjustment formula is shown in equation (21):

[0109] (twenty one) in, Used to enhance the generation of new entities in the target region, This reduces its tendency to generate in non-target areas.

[0110] This strategy ensures that the text description accurately guides the editing process, strictly aligning the attributes and form of newly added entities with the text information, thus achieving highly controllable image editing. The current time step can be obtained based on cross-attention. Momentary noise .

[0111] Background optimization for two-stage generation: To ensure the integrity of the original image background during new entity generation, this invention also proposes a two-stage background optimization strategy. This strategy achieves precise control over the edited and unedited regions by separating object generation from the background fusion process. Its main objective is to avoid interference from target entity generation on the background region while ensuring a natural transition between the edited and unedited regions. The method comprises the following two core stages:

[0112] Phase 1: Object Initialization In the early stages of noise reduction (i.e., time step) (At this point, the noise level is high), and the main task is to generate the target entity without affecting the integrity of the background. To this end, this invention, based on the attention modulation entity editing method proposed above, uses a single object description prompt to guide the generation of the editing area, thereby avoiding interference from other object information. This generation process can be defined by equation (22):

[0113] (twenty two) in, It is a sampler for the diffusion model. Provide hints for generating entity descriptions.

[0114] Meanwhile, in order to preserve the original information of the background area and prevent it from being disturbed during the generation process, a noise preservation mechanism is introduced. This mechanism ensures that the noise in the background area is not modified during the denoising process, so that it always remains in its original state, and the generated edited area object is fused with the background area noise. The specific process is shown in Equation (23):

[0115] (twenty three) in, Represents time step Latent variables after time fusion This involves adding a region mask. At this point, background noise is preserved, while the target object for the editing region is initially generated.

[0116] Phase Two: Background Blending Optimization After the target object is generated, the editing task enters the later stage of noise reduction. At this point, the focus shifts to optimizing the details of the target object to improve its quality and realism; ensuring a natural blend between the generated object and the background area, avoiding harsh stitching marks. The specific steps are as follows:

[0117] At time step (in Pause denoising and remove latent variables. Decode into preview image Next, based on the preset bounding box, a high-precision entity mask is generated using SAM. To accurately determine the shape and position of the target object. In the remaining time steps ( Separately optimize the editing area and the background area: continue denoising optimization within the editing area to refine the details of the object; preserve the original image outside the editing area. The latent characteristics of the background region are used to keep the background content unchanged and avoid unnecessary background modifications. During this process, the latent variables of the background region maintain consistency, and the optimization process can be expressed as equation (24):

[0118] (twenty four) Ultimately, by introducing a high-precision mask... This ensures a natural blend between the target entity and the background area, not only optimizing the details of the editing area but also avoiding unnecessary changes to the background, thereby achieving higher quality entity editing.

[0119] based on Figure 1The text-guided image editing method presented in this invention addresses the limitations of existing methods in understanding editing instructions. It first performs global visual and local entity depth perception on the original image and transforms this into prior knowledge in natural language form. This compensates for the lack of explicit modeling of knowledge in the diffusion model during image editing, improving the cognitive ability of the image editing scenario. By combining prior knowledge with editing instructions, it performs multi-angle deep editing reasoning on incomplete or ambiguous instructions, generating clearer and more binding editing schemes. This not only enhances the interpretability and binding nature of editing instructions but also improves the model's adaptability to complex editing tasks, ultimately improving the quality of image editing results.

[0120] The knowledge reasoning module aims to enhance image understanding capabilities and strengthen knowledge support for editing tasks. To this end, it proposes scene-aware tasks, knowledge fusion methods, and knowledge reasoning tasks, and elaborates on their implementation mechanisms. These methods collectively construct high-quality editing constraint knowledge, laying the foundation for precise image editing.

[0121] The knowledge-guided module aims to fully utilize these editing constraints to improve image editing quality. An attention-based knowledge-guided method is proposed, which dynamically adjusts self-attention and cross-attention to make region entity editing more precise and natural. Furthermore, to further optimize background consistency in non-edited areas, a two-stage generation algorithm is proposed, effectively improving the fusion of newly generated content with the original background.

[0122] When applying the text-guided image editing method provided by this invention, it is not necessary to follow the instructions provided. Figure 1 The steps shown are executed in sequence. The specific execution order of each step can be determined as needed, and this invention does not impose any restrictions on it.

[0123] Furthermore, this invention also provides embodiments for applying this invention, as follows: In local image editing tasks, although existing general datasets (such as ImageNet, MSCOCO, Open Images, LSUN-Bedroom, etc.) cover a wide range of image scenes, they are not entirely suitable for knowledge-based reasoning and guided diffusion model editing tasks. The main shortcomings are as follows: On the one hand, these datasets usually focus on specific categories (such as faces or indoor scenes) or specific detail regions, lacking a complete semantic understanding of complex scenes; on the other hand, the datasets contain a large number of low-quality, cluttered real-life scene images, which are difficult to meet the requirements of knowledge-based reasoning image editing for high quality and clear structure.

[0124] This embodiment constructs a dataset specifically designed for knowledge reasoning and guided image editing tasks based on diffusion models—the Structured Scene Understanding Dataset (SSUD). Building upon existing open datasets, SSUD optimizes for issues such as single-scene representation, low image resolution, and lack of entity interaction. Through high-quality data collection and a large-model-based annotation process, it provides a more challenging and knowledge-rich benchmark dataset to support KRG-Edit training and evaluation. The SSUD dataset ultimately contains 98,578 high-quality original images, covering multiple scenes including food, animals, indoor scenes, street scenes, and sports.

[0125] Evaluation Metrics: This embodiment constructs an evaluation system based on three core dimensions: editing accuracy, consistency of non-edited areas, and realism of the edited image. It combines automated evaluation metrics with manual evaluation methods to systematically measure the image editing effect, ensuring the comprehensiveness and reliability of the evaluation. To measure the semantic matching degree and editing directionality of the edited area, this embodiment uses CLIP-related metrics for quantitative evaluation, including CLIP semantic similarity (CLIP-Sim) and CLIP directional consistency (CLIP-Dir). To ensure the integrity and consistency of non-edited areas during image editing, this embodiment uses Learned Perceptual Image Patch Similarity (LPIPS) and Peak Signal-to-Noise Ratio (PSNR) metrics for evaluation. Since automated evaluation cannot comprehensively measure the realism of the image, this embodiment introduces human evaluation (HE) to further verify the model performance from a subjective visual perception perspective. The final score uses the mean and standard deviation of the scores for each dimension to reflect the stability of the model's overall editing quality.

[0126] Implementation Details: In the pre-training phase of the knowledge perception task, this embodiment uses an LLaMA-2 model (7B or 13B parameter size) as the backbone network. A comparative experiment on parameter size is conducted to explore the impact of model capacity on the ability to understand complex scenes. During training, 90% of the parameters of the original pre-trained model are frozen, and only the LoRA adapter and output layer are trained, thereby significantly reducing memory usage and improving training efficiency. In the knowledge reasoning task, five sets of context learning examples are provided, including two sets of entity addition reasoning examples, one set of entity deletion reasoning examples, and two sets of entity modification reasoning examples.

[0127] During the image editing stage, an editing backbone was constructed based on Stable Diffusion v2-1-base, combined with the editing constraint text descriptions output by the knowledge reasoning module, and a DDIM scheduling strategy was adopted. A 50-step denoising process is performed. The guidance mechanism uses a classifier-free guide with a guidance scale set to 7.5.

[0128] To verify the effectiveness of the KRG-Edit algorithm, this embodiment selects two mainstream image editing models for comparative analysis: one is an image editing model based on mask mixing, which relies on the editing area provided by the user for local modification; the other is an image editing model based on attention mechanism, which can automatically locate the area that needs to be modified and reduce user intervention.

[0129] The image editing models compared are based on mask blending and include the following four: BLD (2022), Stable 2.0 Inpainting (SD-2.0, 2022), HD-Painter (2023), and PowerPaint (2024). The image editing models compared are based on attention mechanisms and include the following four: InstructPix2Pix (iP2P, 2022), LEDITS++ (2023), LearnableRegions (LR, 2024), and BrushEdit (2024). Table 2 shows the quantitative comparison results between the method of this invention and other image editing models.

[0130] Table 2. Quantitative comparison results of the method of the present invention with other image editing models. In terms of semantic alignment, the proposed method outperforms all baseline models, demonstrating its ability to integrate contextual knowledge. Among maskless models, BrushEdit performs well but lags behind the proposed method in directional consistency, highlighting the advantages of knowledge-based reasoning. Regarding background consistency, the proposed method is the best among maskless methods, even surpassing the top mask-based model PowerPaint, proving that knowledge-guided attention modulation can reduce unintended distortion. In terms of realism, the proposed method outperforms all baseline models, and these results underscore its effectiveness in accurate, context-aware, and stable image editing.

[0131] Figure 5 This diagram illustrates a qualitative comparison between KRG-Edit and a mask-based hybrid baseline method, as described in this invention. Ours represents the method of this invention. Experimental results show that, without requiring the user to provide a mask, KRG-Edit's editing performance is comparable to mask-dependent image editing algorithms, and even outperforms them in terms of semantic consistency and scene plausibility.

[0132] In the entity adding task (e.g.) Figure 5 In the first "Add Bear" task and the second "Add Poster" task, KRG-Edit relies on knowledge reasoning to accurately determine the size and position of the newly added entity, allowing it to blend naturally into the scene. In contrast, BLD and SD-2.0, due to blurry mask boundaries, often cause misaligned or distorted positions of newly added objects, affecting the overall editing quality. In entity deletion tasks (such as...) Figure 5 The third row, "Remove People," and the fourth row, "Remove Rainbow," are tasks that require KRG-Edit to use an attention modulation mechanism to completely preserve background details (such as the texture of ocean waves, the direction of mountains and rivers) while eliminating the target, making the image more natural and coherent. Its editing quality is comparable to PowerPaint's negative cue-driven method, but it does not rely on high-precision masks, thus avoiding the influence of mask quality on the final result.

[0133] In entity replacement tasks (such as) Figure 5 In the fifth row ("Replace Orange with Apple") and the sixth row ("Replace Lighthouse with Castle"), KRG-Edit further demonstrates its deep understanding of the original image. In the "Replace Orange with Apple" task, the model not only recognizes the morphological features of the original object (such as a cut orange) but also reasonably infers the shape of the replacement object, ensuring the generated apple remains cut, thus maintaining visual consistency. In the "Replace Lighthouse with Castle" task, the model accurately understands the environmental features of the original image, clearly defining the time and lighting conditions of the editing scene (such as a seaside atmosphere at sunset), resulting in a more natural and harmonious castle in terms of lighting effects and color matching. In contrast, PowerPaint and HDP, lacking a knowledge reasoning module, are prone to scene mismatch issues in replacement tasks, such as castle size misalignment or lighting direction inconsistencies with the original scene, affecting the overall consistency of the editing results.

[0134] Figure 6 This is a schematic diagram showing a qualitative comparison between KRG-Edit and a baseline method based on attention mechanisms in this invention. Experiments show that KRG-Edit generates explicit constraints through knowledge reasoning, which significantly improves the accuracy and realism of editing in open scenarios.

[0135] In entity addition tasks (such as "add a customer to the chair" or "add flowers to the bookshelf"), KRG-Edit relies on a knowledge reasoning model to understand the geometric relationships of the scene (such as the size of the chair and the space of the bookshelf), accurately determining the position of the new entity and allowing it to blend naturally into the environment. In contrast, iP2P and LEDITS++, lacking structured knowledge guidance, often exhibit abnormal object positioning or perspective errors, such as a floating customer or distorted perspective between the flowers and the bookshelf. Figure 6 The first and second rows). In entity removal tasks (such as the "remove glass bottle" and "remove pushcart puppy" tasks), both KRG-Edit and BrushEdit effectively remove the target and perform similarly in background restoration quality. This further verifies the effectiveness of the attention modulation-based entity removal method, resulting in a more natural-looking image after removal. Figure 6 (Third and fourth lines).

[0136] In entity replacement tasks (such as the "cat → dog" and "car → bus" tasks), KRG-Edit fully leverages the advantages of its knowledge reasoning module, demonstrating deep understanding capabilities in semantic association and scene consistency. For example, in the "cat → dog" task, the replaced dog inherits the original cat's posture, maintaining a reasonable pose and interaction relationship. Figure 6 (Fifth line) In the "Car → Bus" task, the model was precisely adjusted to match the bus's size to the road proportions, ensuring overall visual harmony. Figure 6 (Sixth line). LR only relies on the region generation network to locate the editing range, but does not model the relationship of entity attributes, resulting in the inconsistency between the pose of the newly added dog and the original cat; although BrushEdit can determine the editing position with the help of a multimodal large model, it does not fully understand the physical size relationship of scene entities, resulting in the size ratio of the replaced bus being unbalanced and not matching the road scene.

[0137] The above describes a text-guided image editing method provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding text-guided image editing device, such as... Figure 7 As shown.

[0138] Figure 7 A schematic diagram of a text-guided image editing device provided by the present invention includes: The acquisition module 201 is used to acquire the original image to be edited and the editing instructions used to guide the image editing; The global extraction module 202 is used to extract global visual features of the original image, so as to generate a scene semantic description of the original image based on the global visual features and a preset global structured extraction template. The local extraction module 203 is used to perform instance segmentation on the original image to obtain image regions corresponding to multiple entities, extract local visual features of the image regions corresponding to each entity, generate entity semantic descriptions based on local visual features and preset entity structure extraction templates, and extract the interaction relationships between entities to obtain a semantic relationship map. The editing constraint module 204 is used to obtain editing constraint parameters through multi-angle editing reasoning based on scene semantic description, entity semantic description, semantic relationship graph and editing instructions; the multi-angle editing reasoning includes at least one of instruction analysis, entity analysis, description analysis and positional relationship analysis; The guided editing module 205 is used to guide image editing based on the original image to be edited through a diffusion generation process, guided by an editing constraint text description.

[0139] Specific limitations regarding the text-guided image editing device can be found in the limitations of the text-guided image editing method described above, and will not be repeated here. Each module in the aforementioned text-guided image editing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0140] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 Provides text-guided image editing methods.

[0141] The present invention also provides Figure 8 The schematic diagram of the computer device shown is as follows: Figure 8 As shown, at the hardware level, this computer device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above. Figure 1 Provides text-guided image editing methods.

[0142] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0143] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

Claims

1. A text-guided image editing method, characterized in that, include: Obtain the original image to be edited and the editing instructions to guide the image editing process; Extract global visual features from the original image, and generate a scene semantic description of the original image based on the global visual features and a preset global structured extraction template. The original image is segmented to obtain image regions corresponding to multiple entities. Local visual features of each entity's image region are extracted. Based on the local visual features and the preset entity structure extraction template, an entity semantic description is generated. The interaction relationship between entities is extracted to obtain a semantic relationship map. Based on scene semantic description, entity semantic description, semantic relationship graph, and editing instructions, an editing constraint text description is obtained through multi-angle editing reasoning; the multi-angle editing reasoning includes at least one of instruction analysis, entity analysis, description analysis, and positional relationship analysis; Guided by the textual description of editing constraints, image editing is performed through a diffusion generation process based on the original image to be edited.

2. The text-guided image editing method as described in claim 1, characterized in that, The global and local visual features are extracted by a visual feature extractor, which includes a visual encoder, a Q-Former and a visual feature projection layer connected in sequence. Scene semantic descriptions and entity semantic descriptions are generated through a scene-aware adapter, wherein the scene-aware adapter includes a large language model; the large language model is a large language model that has been pre-trained by adding a LoRA adapter to the query weight matrix and value weight matrix of the self-attention module. The large language model is used to generate scene semantic descriptions of the original image based on global visual features and a preset global structured extraction template, and to generate entity semantic descriptions based on local visual features and a preset entity structured extraction template.

3. The text-guided image editing method as described in claim 1, characterized in that, The step of generating a scene semantic description of the original image based on global visual features and a preset global structured extraction template specifically includes: The scene semantic description of the original image is generated using the following formula based on global visual features and a preset global structured extraction template: ， ； in, For the scene semantic description of the original image, This is a natural language representation corresponding to the global description. Natural language expression corresponding to the image style For the corresponding scenario type of natural language expression, For global visual features, This is a global structured extraction template. This is the template content corresponding to the global description. Template content corresponding to the image style, Template content for the corresponding scenario type, For large language model decoders, For text encoders.

4. The text-guided image editing method as described in claim 1, characterized in that, The step of generating entity semantic descriptions based on local visual features and a preset entity structure extraction template specifically includes: The following formula is used to generate entity semantic descriptions based on local visual features and a pre-defined entity structure extraction template: ， ； in, For entity semantic description, For the natural language representation of the corresponding entity description, For the natural language representation of the corresponding entity type, For the first Local visual features of the image region corresponding to each entity For entity structure extraction template, The template content for the corresponding entity description, Template content for the corresponding entity type, For text encoders, It is a decoder for large language models.

5. The text-guided image editing method as described in claim 1, characterized in that, The extraction of interaction relationships between entities to obtain a semantic relationship graph specifically includes: The semantic relationship map is obtained by extracting the interaction relationships between entities from the original image using the OpenPSG framework.

6. The text-guided image editing method as described in claim 1, characterized in that, The process of obtaining editing constraint text descriptions through multi-faceted editing reasoning based on scene semantic descriptions, entity semantic descriptions, semantic relationship graphs, and editing instructions specifically includes: Based on scene semantic description, entity semantic description, semantic relationship graph and editing instructions, the large language model guides reasoning based on the preset thought chain reasoning question template corresponding to the task type of the editing instructions, and generates editing constraint text description based on the preset task example corresponding to the task type of the editing instructions. The task types include: deletion, addition, and replacement.

7. A text-guided image editing device, characterized in that, include: The acquisition module is used to acquire the original image to be edited and the editing instructions used to guide the image editing process; The global extraction module is used to extract global visual features from the original image, and generate a scene semantic description of the original image based on the global visual features and a preset global structured extraction template. The local extraction module is used to perform instance segmentation on the original image to obtain image regions corresponding to multiple entities, extract local visual features of the image regions corresponding to each entity, generate entity semantic descriptions based on local visual features and preset entity structure extraction templates, and extract the interaction relationships between entities to obtain a semantic relationship graph. The editing constraint module is used to obtain an editing constraint text description through multi-angle editing reasoning based on scene semantic description, entity semantic description, semantic relationship graph and editing instructions; the multi-angle editing reasoning includes at least one of instruction analysis, entity analysis, description analysis and positional relationship analysis; The Guided Editing module is used to guide image editing through a diffusion generation process based on the original image to be edited, guided by an edit constraint text description.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method as described in any one of claims 1 to 6.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1 to 6.

Images