Fine-grained image editing method based on multi-modal thinking chain reasoning

Abstract

The invention discloses a fine-grained image editing method based on multi-modal thinking chain reasoning, and belongs to the technical field of computer vision and image processing. The invention aims to solve the problem that the existing image editing method cannot meet the requirements of controllability and refined editing at the same time in a complex editing scene. The method comprises the following steps: firstly, generating text chain thinking reasoning according to an editing instruction and an input image by utilizing a multi-modal generation-understanding unified model so as to determine a target object referred by a user; and on this basis, a pixel-level visual positioning image corresponding to the target is generated. Secondly, the model generates semantic reasoning of an editing description and an editing result according to a multi-modal positioning clue, and executes local region editing to generate an accurate edited image; in the training process, positioning enhancement is realized through a multi-modal thinking chain alignment mechanism and auxiliary mask supervision, so that semantic consistency and positioning accuracy between an inference chain and an actual editing area are ensured. According to the method, the editing capability with high interpretability, accurate space alignment and interactivity is realized.

Description

Technical Field

[0001] This invention relates to a fine-grained image editing method based on multimodal thought chain reasoning, belonging to the field of computer vision and image processing technology. Background Technology

[0002] With the development of image generation and processing technologies, instruction-based image editing has gradually become an important interactive method. Existing image editing technologies can be mainly divided into three categories:

[0003] The first category is editing methods based on Generative Adversarial Networks (GANs). These methods typically invert the image to be edited into the latent space of a pre-trained GAN model, adjusting latent variables to achieve the editing operations corresponding to user commands. While this type of method is relatively straightforward, its latent space representation is limited, often failing to adequately express complex or fine-grained editing commands.

[0004] The second category is editing methods based on diffusion model inversion. These methods reconstruct the inversion trajectory of the input image within the diffusion model and introduce textual instructions during the inversion or de-diffusion process to regulate the generation process, thereby achieving image modification and reconstruction. While diffusion models possess strong generative and expressive capabilities, the inversion process is computationally intensive, and when handling local editing or scenarios requiring high spatial accuracy, issues such as positional shifts or semantic inconsistencies can easily arise.

[0005] The third type of method relies on joint training of large-scale image and text datasets and large diffusion models. These methods train the model on massive amounts of labeled or synthetic editing samples, enabling the model to directly perform target editing operations on the input image based on the user's natural language instructions without additional inversion. This type of method has strong instruction understanding capabilities and good versatility, but its performance typically depends on the coverage of the training data; for editing tasks lacking explicit spatial indications, its localization accuracy may be limited.

[0006] In summary, existing instruction-based image editing methods still have limitations in terms of expressive power, computational cost, spatial positioning accuracy, and editing consistency, making it difficult to simultaneously meet the needs of controllability and fine-grained editing in various complex editing scenarios.

[0007] 1) Difficulty in reference disambiguation: When user instructions contain descriptions of multiple entities or composite references (such as "the clothes of the person facing to the left"), existing text or visual models have difficulty in performing accurate reference resolution and spatial localization.

[0008] 2) Disconnect between reasoning and execution: Although some works that use chain thinking can provide an explainable reasoning process at the text level, there is often an inconsistency between this reasoning process and the actual visual positioning and editing results;

[0009] 3) Lack of multimodal interwoven positioning mechanism: Existing visual-language alignment methods are mostly one-time alignment (text-to-region or vision-to-text), lacking interwoven, multi-level positioning guidance, making it difficult to achieve point-to-point, pixel-level precise editing;

[0010] 4) Limited dataset coverage: Existing editing datasets are mostly oriented towards salient objects or single salient regions, lacking large-scale, interwoven localization editing samples that emphasize referential and spatial reasoning, which limits the model's generalization ability in complex situations. Summary of the Invention

[0011] To address the problem that existing image editing methods cannot simultaneously meet the requirements of controllability and fine-grained editing in complex editing scenarios, this invention provides a fine-grained image editing method based on multimodal thought chain reasoning.

[0012] The present invention provides a fine-grained image editing method based on multimodal thought chain reasoning, comprising:

[0013] A multi-entity complex scene image is acquired, multi-target detection and segmentation are performed, editing targets requiring fine spatial reasoning are selected, and a visual positioning thought chain corresponding to the editing target is generated; based on the visual positioning thought chain, the editing target is determined in the multi-entity complex scene image, and editing instructions and text thought chains for the editing target are generated; the editing target is cropped in the multi-entity complex scene image and modified according to the editing instructions to obtain the modified editing target; the modified editing target is fused into the multi-entity complex scene image to obtain the modified complex scene image.

[0014] A multimodal interleaved reasoning model consisting of an understanding module and a generation module is constructed. Multi-entity complex scene images with configuration modification instructions are used as training samples, visual positioning thought chains and text thought chains are used as intermediate reasoning, and modified complex scene images are used as the final editing target. The model parameters are adjusted by combining a mask supervision mechanism and a multimodal thought chain alignment mechanism to train the multimodal interleaved reasoning model.

[0015] After training, the multimodal interleaved inference model performs fine-grained image editing on the input image to be processed according to the input modification instructions, and outputs the edited image.

[0016] According to the fine-grained image editing method based on multimodal thinking chain reasoning of the present invention, the multi-entity complex scene image is subjected to multi-target recognition and detection segmentation by RAM model and Grounded-SAM model, and then the editing target requiring fine spatial reasoning is obtained by Qwen2.5-VL-7B module.

[0017] The visual positioning mind chain includes a target box and a semi-transparent mask.

[0018] According to the fine-grained image editing method based on multimodal thought chain reasoning of the present invention, the text thought chain of the editing target is a four-segment text thought chain;

[0019] The Qwen2.5-VL-72B module is used to generate editing instructions and text thought chains based on the target box, and then the Qwen3 module is used to rewrite the editing instructions and enhance language diversity.

[0020] The textual thought chain includes scene description, location reasoning, editing intent, and result interpretation.

[0021] The fine-grained image editing method based on multimodal thought chain reasoning according to the present invention provides a method for obtaining modified complex scene images, comprising:

[0022] The Bagel module is used to modify the editing target according to the editing instructions; the modification includes editing processing and removal; the modified editing target after editing processing is seamlessly integrated with the multi-entity complex scene image to obtain the modified complex scene image; the removed modified editing target is used with the LaMa model and the editing target mask to perform consistent completion of the multi-entity complex scene to obtain the modified complex scene image.

[0023] The modified complex scene images are reviewed for semantic and visual consistency using the Qwen2.5-VL-7B module, and the modified complex scene images that meet the review threshold are retained.

[0024] According to the fine-grained image editing method based on multimodal thought chain reasoning of the present invention, during the training of the multimodal intertwined reasoning model, the understanding module performs text reasoning parsing on the modification instructions to obtain the text thought chain of the editing target; the generation module obtains the visual positioning thought chain of the editing target based on the text thought chain, and processes the editing target to obtain the edited image.

[0025] According to the fine-grained image editing method based on multimodal thought chain reasoning of the present invention, the mask supervision mechanism includes:

[0026] A text projector is used to project the text thought chain output by the comprehension module to obtain the text projection result;

[0027] A visual projector is used to project the visual positioning thought chain output by the generation module and the edited image onto the image respectively, so as to obtain the visual positioning visual projection result and the edited image visual projection result.

[0028] An image encoder is used to encode images of complex scenes with multiple entities to obtain the encoded image features;

[0029] The mask encoder predicts the text prediction mask based on the features of the encoded image and the text projection results; it predicts the location prediction mask based on the features of the encoded image and the visual positioning projection results; and it predicts the editing prediction mask based on the features of the encoded image and the visual projection results of the edited image.

[0030] The mask supervision loss function is calculated based on the comparison results between the text prediction mask, the location prediction mask, and the edit prediction mask and the real mask, and the model parameters are adjusted according to the loss function.

[0031] According to the fine-grained image editing method based on multimodal thought chain reasoning of the present invention, the text projection result is represented as... The visual positioning and visual projection results are represented as follows: The edited image visual projection result is represented as ;but:

[0032] ,

[0033] ,

[0034] ,

[0035] In the formula For text prediction mask, Indicates a mask encoder. For encoded image features, To locate the predicted mask, To edit the predictive mask;

[0036] Masked Supervision Loss Function for:

[0037] ,

[0038] In the formula It is the binary cross-entropy. This is the real mask.

[0039] According to the fine-grained image editing method based on multimodal thought chain reasoning of the present invention, the multimodal thought chain alignment mechanism includes a text thought chain alignment mechanism:

[0040] The mean semantic feature is obtained by averaging the features of all the smallest semantic units corresponding to the editing target in the text thinking chain. The mean noise feature is obtained by averaging all the noise features of the editing target. ;

[0041] noise characteristic mean Through the fully connected layer Perform projection and calculate the mean of the projected features and semantic features. The cosine similarity is used to obtain the text thought chain alignment loss. :

[0042] .

[0043] According to the fine-grained image editing method based on multimodal thought chain reasoning of the present invention, the multimodal thought chain alignment mechanism further includes a visual positioning thought chain alignment mechanism:

[0044] The visual feature units of the editing target corresponding to the visual positioning mind chain are represented as: ;

[0045] Generate noisy VAE editing features and perform interpolation adjustments to obtain the noise features. and make noise characteristics With visual feature units They have the same spatial resolution;

[0046] noise characteristics Through the fully connected layer Projection is performed, and the noise characteristics and visual feature units after projection are calculated. The cosine similarity is used to adjust the model parameters based on the calculation results to improve the noise characteristics. With visual feature units Align in spatial position.

[0047] According to the fine-grained image editing method based on multimodal thought chain reasoning of the present invention, visual localization thought chain alignment loss for:

[0048] .

[0049] The beneficial effects of this invention are as follows: First, the method utilizes a multimodal generation-understanding unified model to generate textual chain-like reasoning based on editing instructions and the input image to determine the target object referred to by the user. Based on this, a pixel-level visual localization image of the corresponding target is further generated. Second, the model generates semantic reasoning of the editing description and editing result based on multimodal localization cues, and performs local region editing to generate a precise edited image. During training, localization enhancement is achieved through a multimodal thought chain alignment mechanism and auxiliary mask supervision, thereby ensuring semantic consistency and localization accuracy between the inference chain and the actual editing region. This invention proposes a corresponding dataset construction method that can automatically generate training samples containing the original image, textual chain-like reasoning, visual chain-like reasoning, target mask, bounding box, and edited image. This invention enables high-precision target localization, interpretable inference chains, and semantically consistent fine-grained region editing in complex scenes, and is suitable for applications such as open-world image editing, multimodal reasoning, and visual generation.

[0050] The method of this invention constructs and utilizes a specialized training dataset and designs a multimodal alignment and auxiliary supervision mechanism, thereby achieving highly interpretable, spatially accurate, and interactive editing capabilities.

[0051] The method of this invention has precise referential parsing and localization capabilities: through an interwoven process of textual reasoning followed by visual localization, it can parse editing targets in complex references and multi-entity scenes, achieving consistent semantic and spatial localization.

[0052] Reasoning-Execution Consistency: The CoT alignment mechanism explicitly constrains the consistency between reasoning features and editing features, reduces errors caused by reasoning shortcuts, and improves the interpretability and reliability of editing results;

[0053] Fine-grained spatial awareness: Enhanced localization representation through auxiliary mask supervision (shared mask decoder) enables the model to have stronger region discrimination ability at the pixel or patch level, which is beneficial for high-precision editing tasks;

[0054] High-quality training data support: The constructed interwoven localization and editing dataset provides text-visual hybrid training samples, which significantly improves the model's generalization ability in complex reference scenarios;

[0055] Interactivity and controllability: The explicit location clues (bounding boxes, masks, text CoT) output by the system facilitate user intervention and revision, and support interactive high-quality editing. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the reasoning framework of the multimodal interleaved reasoning model described in this invention;

[0057] Figure 2 This is a diagram illustrating the text-based thought chain alignment mechanism;

[0058] Figure 3 This is a diagram illustrating the visual positioning mind chain alignment mechanism;

[0059] Figure 4 This is a schematic diagram of the training data construction framework for a multimodal interleaved inference model;

[0060] Figure 5 This is a diagram showing the output results of the multimodal interleaved inference model described in this invention;

[0061] Figure 6 This is a visual comparison chart of image editing using the method of this invention and other existing methods. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] Specific Implementation Method 1: Combination Figures 1 to 4 As shown, this invention provides a fine-grained image editing method based on multimodal thought chain reasoning, including:

[0064] A multi-entity complex scene image is acquired, multi-target detection and segmentation are performed, editing targets requiring fine spatial reasoning are selected, and a visual positioning thought chain corresponding to the editing target is generated; based on the visual positioning thought chain, the editing target is determined in the multi-entity complex scene image, and editing instructions and text thought chains for the editing target are generated; the editing target is cropped in the multi-entity complex scene image and modified according to the editing instructions to obtain the modified editing target; the modified editing target is fused into the multi-entity complex scene image to obtain the modified complex scene image.

[0065] A multimodal interleaved reasoning model consisting of an understanding module and a generation module is constructed. Multi-entity complex scene images with configuration modification instructions are used as training samples, visual positioning thought chains and text thought chains are used as intermediate reasoning, and modified complex scene images are used as the final editing target. The model parameters are adjusted by combining a mask supervision mechanism and a multimodal thought chain alignment mechanism to train the multimodal interleaved reasoning model.

[0066] After training, the multimodal interleaved inference model performs fine-grained image editing on the input image to be processed according to the input modification instructions, and outputs the edited image.

[0067] Combination Figure 1 As shown, this embodiment constructs a multimodal interleaved reasoning framework, which parses the editing target referred to by the user through text reasoning; generates an image-format mask and bounding box based on the semantic information obtained from text reasoning; and accurately locates the editing area in the image by interleaving and fusing multimodal localization cues, thereby achieving image editing operations with accurate spatial location.

[0068] A multimodal interwoven thought chain alignment mechanism is constructed to explicitly constrain the consistency between editing features and text-visual location cues, thereby enhancing the coherence of the location reasoning process and making the resulting reasoning process more interpretable and spatially aligned.

[0069] An auxiliary mask supervision mechanism is constructed to guide the learning of localization features and editing features simultaneously through the mask decoding process, enabling it to have a finer-grained spatial perception capability and improve the accuracy of target localization.

[0070] Construct a multimodal interleaved inference dataset and train a multimodal interleaved inference model according to the training mechanism described above.

[0071] In practical applications, the multimodal interleaved inference model first parses the original image and user instructions to infer the user's intended editing target, thereby forming a text-level scene understanding and localization description. Then, guided by the inference results, it performs visual localization by generating an image with a red box and a semi-transparent mask, achieving accurate perception and localization of the target area. It then generates a fine-grained editing description and combines it with the inference explanation of the editing results to guide subsequent editing operations. The interleaved inference chain can guide the model to perform fine-grained and accurate image editing, achieving point-to-point editing effects of the target area.

[0072] This implementation method performs fine-grained image editing based on multimodal chain-of-thought (CoT) reasoning, including image understanding, visual localization, and controllable image editing.

[0073] Furthermore, the multi-entity complex scene image is subjected to full-image multi-target recognition and detection segmentation using RAM model and Grounded-SAM model, and then the Qwen2.5-VL-7B module is used to filter out significant targets that do not require reasoning, thereby obtaining editable targets that require fine spatial reasoning and generating visual positioning CoT (red box + semi-transparent red mask).

[0074] The visual positioning mind chain includes a target box and a semi-transparent mask.

[0075] The textual thought chain for editing the target is a four-segment structure.

[0076] The Qwen2.5-VL-72B module is used to generate editing instructions and text thought chains based on the target box, and then the Qwen3 module is used to rewrite the editing instructions and enhance language diversity.

[0077] The textual thought chain includes scene description, location reasoning, editing intent, and result interpretation.

[0078] Furthermore, methods for obtaining modified images of complex scenes include:

[0079] The Bagel module is used to modify the editing target according to the editing instructions; the modification includes editing processing and removal; the modified editing target after editing processing is seamlessly integrated with the multi-entity complex scene image to obtain the modified complex scene image; the removed modified editing target is used with the LaMa model in conjunction with the editing target mask to perform consistent completion of the multi-entity complex scene to obtain the modified complex scene image; ensuring the semantics and visual naturalness of the final image;

[0080] The modified complex scene images were reviewed for semantic and visual consistency using the Qwen2.5-VL-7B module, retaining only those that met the review thresholds. Failed cases were filtered out, ultimately constructing a refined localization editing data set containing high-quality text-visual interleaved localization (CoT).

[0081] Specifically as follows:

[0082] Data source and target selection: SAM-1B (or an equivalent large-scale multi-entity scene set) is used as the original image source; Recognize Anything Model (RAM) and Grounded-SAM are applied to perform multi-object detection and segmentation on the image to obtain the bounding boxes and segmentation masks of candidate entities; results with low detection confidence are filtered out; for each image, targets that need to be inferred are randomly or strategically selected from the candidate entity set to avoid selecting targets that are too salient or do not need inference (difficulty filtering and screening are performed using models such as Qwen2.5-VL-7B), and visual annotations of the visual localization CoT (red border + semi-transparent red mask) are generated for the selected targets to facilitate subsequent supervision.

[0083] Text CoT and Instruction Generation: Based on the bounding box and visual cues of the selected target, a powerful multimodal language model (such as Qwen2.5-VL series or higher) is used to generate a text chain-of-thought annotation containing four segments: scene description, localization inference, edit description, and post-edit explanation. Subsequently, the generated instructions are rewritten through a paraphrasing model (such as Qwen3) to enrich the language style and expressive diversity, thereby improving the language generalization ability of the training data.

[0084] Local editing and compositing: Local editing tools (such as Bagel) are used to perform specified types of editing on the selected target region (color replacement, local enhancement, attribute changes, etc.). For removal-type scenes, inpainting algorithms such as LaMa are used to maintain content consistency. The edited local patches are seamlessly integrated back into the original image to generate the edited result image, which serves as part of the supervised samples.

[0085] Data quality control: Using a multimodal evaluation model (e.g., Qwen2.5-VL-7B), the generated instructions, inference CoT, and editing results are scored for semantic and visual consistency. Samples that do not meet the accuracy or visual consistency requirements are removed, and finally, high-quality interwoven positioning editing data is collected.

[0086] During the training of the multimodal interleaved reasoning model, the understanding module performs text reasoning parsing on the modification instructions to obtain the textual thought chain of the editing target; the generation module obtains the visual positioning thought chain of the editing target based on the textual thought chain, processes the editing target, and obtains the edited image.

[0087] In this embodiment, the mask monitoring mechanism includes:

[0088] A text projector is used to project the text thought chain output by the comprehension module to obtain the text projection result;

[0089] A visual projector is used to project the visual positioning thought chain output by the generation module and the edited image onto the image respectively, so as to obtain the visual positioning visual projection result and the edited image visual projection result.

[0090] An image encoder is used to encode images of complex scenes with multiple entities to obtain the encoded image features;

[0091] The mask encoder predicts the text prediction mask based on the features of the encoded image and the text projection results; it predicts the location prediction mask based on the features of the encoded image and the visual positioning projection results; and it predicts the editing prediction mask based on the features of the encoded image and the visual projection results of the edited image.

[0092] The mask supervision loss function is calculated based on the comparison results between the text prediction mask, the location prediction mask, and the edit prediction mask and the real mask, and the model parameters are adjusted according to the loss function.

[0093] To further enhance the model's localization capabilities during chained reasoning and editing processes, assisted localization supervision is introduced. The model's region awareness capability is explicitly improved through mask reconstruction tasks, and unified alignment of multimodal localization representations is achieved.

[0094] Construct a mask reconstruction target, and reconstruct the target region mask through a shared mask decoder, thereby forcing the model to learn more discriminative localization features and achieve more accurate editing behavior related to the target.

[0095] Three types of hidden states are used as sources of localization conditional features:

[0096] Text location CoT related tokens;

[0097] Noisy VAE token for visual positioning CoT;

[0098] The VAE token for adding noise to the editing area.

[0099] Text projectors using three layers of MLP and visual projector get As conditional features in a mask decoder. Text features. Spatial features are introduced through cross-modal attention after averaging, and... and Then directly with the image features encoded by SAM Blending to maintain spatial consistency. The text projection result is represented as... The visual positioning and visual projection results are represented as follows: The edited image visual projection result is represented as ;but:

[0100] ,

[0101] ,

[0102] ,

[0103] In the formula For text prediction mask, Indicates a mask encoder. For encoded image features, To locate the predicted mask, To edit the predictive mask;

[0104] Prediction masks from three modalities , , With the real mask Applying binary cross-entropy (BCE) supervision helps the model establish a consistent region understanding across text-based reasoning localization, visual localization, and edited regions.

[0105] Masked Supervision Loss Function for:

[0106] ,

[0107] In the formula It is the binary cross-entropy. This is the real mask.

[0108] During the inference phase, the image to be edited, I, and the user's text command, T, are input into a trained multimodal interleaved inference model. The model outputs the text-level inference chain, visual localization mask / bounding box, and the final editing command and result. The specific process includes: text parsing. Text CoT generation Text-guided visual positioning (output) (with bbox) CoT alignment check Pixel-level editing guided by a mask Output the edited image.

[0109] The multimodal thought chain alignment mechanism includes a text thought chain alignment mechanism:

[0110] By applying consistency constraints to the editing features and the multimodal localization thought chain representation, precise region editing based on inference results is achieved, ensuring spatial consistency and inference consistency.

[0111] The mean semantic feature is obtained by averaging the features of all the smallest semantic units (tokens) corresponding to the editing target in the text thinking chain. The mean noise feature is obtained by averaging all the noise features of the editing target. ;

[0112] noise characteristic mean Through learnable fully connected layers Perform projection and calculate the mean of the projected features and semantic features. The cosine similarity is used to obtain the text thought chain alignment loss. :

[0113] .

[0114] By maximizing cosine similarity, the semantic localization of text reasoning is forced to be consistent with visual editing features.

[0115] The multimodal thinking chain alignment mechanism also includes a visual positioning thinking chain alignment mechanism:

[0116] The visual feature unit (ViT token) of the editing target corresponding to the visual positioning mind chain is represented as: ;

[0117] Generate noisy VAE editing features and perform interpolation adjustments to obtain the noise features. and make noise characteristics With visual feature units They have the same spatial resolution;

[0118] noise characteristics Through learnable fully connected layers Projection is performed, and the noise characteristics and visual feature units after projection are calculated. The cosine similarity is used to adjust the model parameters based on the calculation results to improve the noise characteristics. With visual feature units Align in spatial position.

[0119] make Alignment is performed between the patch and the projected noise features using cosine similarity, thereby achieving spatial consistency between visual reasoning localization and edited features.

[0120] Visual positioning mind chain alignment loss for:

[0121] .

[0122] The CoT alignment loss is applied to the intermediate Transformer layer of the multimodal interleaved inference model to provide clear and semantically consistent localization guidance before the deep generation stage, so that subsequent layers can focus on appearance refinement and content generation, thereby achieving spatially consistent and inference-consistent image editing.

[0123] In summary, the method of this invention significantly improves the accuracy of target localization and the controllability of editing in image editing tasks. It fully utilizes the understanding and generation capabilities of large-scale multimodal models, employing a chain-like reasoning mechanism interwoven with text and vision during the inference stage to automatically parse the editing intent from the user's natural language commands and infer the precise location of the target object. Based on visual chain-like localization technology, high-quality target segmentation masks and bounding boxes can be generated, providing reliable spatial constraints for subsequent pixel-level editing. Users only need to input simple natural language commands to complete complex operations such as target modification, replacement, removal, or attribute adjustment. With its highly automated command parsing capabilities, precise visual localization mechanism, and flexible and controllable editing framework, it demonstrates significant practical value and broad application prospects in fields such as intelligent image processing, content creation, advertising design, film and television production, cultural relic restoration, and virtual reality. The invention will be further described below with reference to specific embodiments.

[0124] Example 1:

[0125] like Figure 1As shown, the specific process of the multimodal thinking chain inference framework is as follows: First, the user inputs the original image I and the editing text instruction T. The model performs semantic parsing on the input natural language content. This step, through the unified model's understanding ability, automatically identifies the target object in the instruction, forming a preliminary scene understanding and task description from the text level. After completing text inference, the framework further generates visual localization results based on the localization cues in the text, including target region images with red bounding boxes and semi-transparent masks. This process combines a visual encoding model, using cross-modal association capabilities to map the semantic location implied in the language to the image space, generating accurate visual localization cues. This localization result is not only used to guide subsequent image editing but also serves as an explicit supervision signal to participate in the optimization of the inference chain, making the localization more accurate and reliable. After the inference chain and localization mask are determined, the framework enters the final editing stage. The model performs pixel-level editing within the specified area based on the semantic inference and spatial localization results, ensuring that the generated content is consistent with the lighting, texture, and semantics of the surrounding image. The final output image not only conforms to the user's instructions but also has a natural and realistic visual effect. The multimodal thinking chain reasoning framework in this embodiment effectively solves the problems of inaccurate target positioning and weak editing control in traditional methods, and realizes truly fine-grained, interpretable, and diversified intelligent image editing capabilities. Figure 1 In the context of EOS, the conversation ends; in the context of BOS, the conversation begins.

[0126] like Figure 2 and Figure 3 As shown, the specific process of the multimodal intertwined thought chain alignment mechanism is as follows: This mechanism aims to solve the problems of semantic offset and spatial inconsistency that may occur between the "text inference path" and the "visual localization path" in complex editing tasks. The mechanism introduces an explicit alignment loss in the intermediate layer of the model to ensure that the semantic position of language inference is consistent with the spatial distribution of visual features, thereby ensuring that the final edited features have coherence and spatial consistency throughout the inference process. The mechanism first extracts the corresponding latent vector representations of key tokens (such as nouns referring to the target object, locative words, etc.) in the text localization CoT inference process. Then, average pooling is performed on it to obtain the global semantic localization vector. Meanwhile, the visual latent space features of the edited region with added noise are obtained through VAE. And similarly, take its average pooling to obtain Subsequently, through a learnable multi-layer fully connected projector Project visual features into the text semantic space and maximize their correlation with the semantic space. The cosine similarity is used to ensure that the visual editing features are semantically consistent with the text location.

[0127] At the visual level, the alignment mechanism further utilizes the ViT token generated by the visual reasoning chain as a representation. The noisy VAE editing feature N is interpolated to its same spatial resolution and passed through another learnable MLP projector. Alignment is performed to maximize cosine similarity at each patch level. This patch-by-patch alignment method maintains high-precision localization consistency in the model's spatial structure, thereby preventing the model from shifting or covering non-target regions during the generation process.

[0128] To ensure that alignment operations truly play a role in inference, the aforementioned CoT alignment loss is applied to the intermediate layers of the Transformer. This allows the model to obtain clear and reliable localization constraints before deep layer generation, enabling deeper layers to focus on appearance refinement, texture rendering, and other aspects without compromising localization accuracy. This mechanism significantly enhances the interpretability of the inference chain, avoiding the common problem of "text describing A but visual editing showing B" found in traditional methods, and truly achieving semantically and spatially consistent image editing capabilities.

[0129] like Figure 1 As shown, the specific process of the auxiliary mask supervision mechanism is as follows: Figure 1 As shown, the auxiliary mask supervision mechanism provided in this embodiment aims to further enhance the model's localization ability during chained inference and editing processes, enabling it to possess stronger region awareness and cross-modal consistency. Traditional image editing models often lack explicit localization supervision, resulting in unstable target masks, blurred boundaries, or deviations from user requirements. This invention constructs a mask reconstruction task and utilizes explicit supervision signals to constrain the model's localization behavior during multimodal inference.

[0130] The mechanism first extracts latent vectors from three different sources: text localization CoT, visual localization CoT, and noisy VAE features of the edit region. These represent the features of text semantic localization, visual reasoning localization, and editing latent space localization, respectively. Then, a text projector is used... and visual projector Mapping these three types of features to a unified localization feature space yields... .

[0131] Text features After average pooling, the text is fed into the mask decoder via cross-modal attention, allowing semantic instructions from the text to participate in localization and prediction; while visual features... and This is then spatially concatenated or fused with image feature I, thus preserving the fine spatial structure. Finally, the mask decoder... Generate three types of prediction masks and predict text location masks respectively. Visual positioning mask Edit the positioning mask .

[0132] This mechanism applies a binary cross-entropy loss (BCE) to the three inference paths (text, vision, and editing) and the true mask M, thereby unifying the model's localization across the three different inference chains. This joint supervision not only improves localization accuracy but also significantly enhances cross-modal alignment in the inference chains, making the representations of the text, vision, and editing paths more consistent, thus providing reliable and stable region constraints for the final pixel-level editing.

[0133] like Figure 4 As shown, the specific method for constructing the multimodal intertwined inference dataset is as follows: First, complex scene images with multiple entities and rich interaction structures are selected from SAM-1B, and multi-object detection and segmentation are completed using RAM and Grounded-SAM. To ensure that the task has inference difficulty, Qwen2.5-VL-7B is used to filter out those salient and simple targets that do not require inference, retaining only the target regions that require "text-visual" interactive inference. Then, based on the filtered target boxes, the image input large model Qwen2.5-VL-72B generates a four-segment text thought chain, including scene description, localization inference, editing intent, and explanation of editing results, giving the text a causal logical chain. To increase language diversity and data quality, Qwen3 is used to rewrite, enhance, and generate styles diversify these instructions. Finally, a complete and high-quality text inference chain is obtained. In the process of constructing the visual editing results, Bagel is used to perform local editing of the target regions and achieve seamless integration to ensure that the generated editing results are natural and realistic. For removal tasks, the LaMa model is used in conjunction with the target mask for image completion, making the background coherent and avoiding unnatural texture breaks. Finally, Qwen2.5-VL-7B was used to perform semantic and visual consistency checks on the generated data, automatically filtering out samples with inference errors, positioning offsets, or editing failures to ensure the reliability of the final dataset. The data constructed in this embodiment not only includes the original image, editing instructions, inference chain, visual positioning masks / bounding boxes, and real editing results, but also various intermediate modalities used to supervise model inference, which are key supports for training the multimodal interleaved inference model.

[0134] This invention significantly enhances the accuracy of target recognition and spatial localization capabilities through a unified text-visual reasoning framework. Leveraging the semantic understanding and visual perception advantages of large-scale multimodal models, this method automatically parses the editing intent in user commands and generates localization masks and bounding boxes consistent with the reasoning process, thus providing stable and accurate spatial constraints for subsequent pixel-level editing. Based on this, the method can perform various editing tasks, such as modifying, replacing, removing, or adjusting attributes of targets, even in complex scenarios. Experimental results show that this method has significant advantages in localization accuracy, editing consistency, and user controllability, and can be widely applied in various scenarios such as intelligent image processing, content generation, film and television post-production, cultural relic restoration, and virtual reality, demonstrating good practical value and promising prospects for wider application.

[0135] Figure 5 This is the multimodal thought chain output by the method of the present invention during prediction. It can be seen that the method of the present invention can, through multimodal reasoning, first parse user instructions through text, then locate the editing target through visual positioning, and then precisely edit the specific target based on the editing description without affecting other content and targets in the image.

[0136] Figure 5 The description of scene A1 is as follows: The image shows a historical site with carved rock formations and stairs leading to them. Several people are scattered around; some are taking pictures, while others are observing their surroundings. Trees and a cobblestone path complete the scene.

[0137] B1 target positioning: The target is a person wearing a white hoodie and black pants, standing with their back to the camera. She is located near the center left of the image, close to the stone wall, slightly behind another person also with their back to the camera.

[0138] Scene A2 is described as follows: The image shows an ancient ruin with tall pillars lining a path. Visitors walk along the path, exploring the historical site under a clear blue sky. The ruins consist of weathered stone structures and scattered fragments.

[0139] Target B2 is defined as the tallest pillar on the right side of the image, with a smooth surface and no visible capital letters. It is striking because it lacks any ornate features compared to the surrounding pillars. It is located near the edge of the frame and is taller than any other pillar without capital letters.

[0140] The C1 editing instructions state: A person wearing a white hoodie and black pants will be removed from the image without affecting other people in the surrounding environment or scene.

[0141] The edited D1 description is as follows: After editing, the area where the person wearing a white hoodie and black pants is standing will be empty, maintaining the integrity of the stone wall and surrounding elements. The rest of the scene, including other people and the background, will remain unchanged.

[0142] The C2 editing instructions state: The highest column with a smooth surface and no visible capital letters will turn blue. This particular column is identified by its unique characteristics and position in the scene, ensuring that other objects are not affected.

[0143] The edited version of D2 is described as follows: After editing, the tallest column, with its smooth surface and no visible capitals, will appear blue, contrasting sharply with the surrounding reddish-brown ruins. All other elements in the scene, including visitors and other structures, will remain unchanged.

[0144] Table 1 shows the quantitative comparison results between the method of the present invention and the prior art. It can be found that the method of the present invention achieves higher accuracy in editing target positioning and editing score than other methods, and has higher value in practical applications.

[0145] Table 1

[0146]

[0147] Figure 6 The visual comparison between the method of the present invention and the prior art shows that the method of the present invention can understand the user's editing instructions more accurately than other methods, and find the correct target in the image to perform more granular editing. Figure 6 Ours in this invention refers to the method of the present invention.

[0148] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0149] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A fine-grained image editing method based on multimodal thought chain reasoning, characterized in that... include, Acquire images of complex scenes with multiple entities, perform multi-target detection and segmentation, filter editing targets that require fine spatial reasoning, and generate visual localization thought chains corresponding to the editing targets; Based on the visual positioning thinking chain, the editing target is determined in the image of a complex scene with multiple entities, and the editing instructions and text thinking chain of the editing target are generated. The target is cropped and edited in a multi-entity complex scene image and modified according to the editing instructions to obtain the modified edited target. The modified edited target is then merged into the multi-entity complex scene image to obtain the modified complex scene image. A multimodal interleaved reasoning model consisting of an understanding module and a generation module is constructed. Multi-entity complex scene images with configuration modification instructions are used as training samples, visual positioning thought chains and text thought chains are used as intermediate reasoning, and modified complex scene images are used as the final editing target. The model parameters are adjusted by combining a mask supervision mechanism and a multimodal thought chain alignment mechanism to train the multimodal interleaved reasoning model. After training, the multimodal interleaved inference model performs fine-grained image editing on the input image to be processed according to the input modification instructions, and outputs the edited image.

2. The fine-grained image editing method based on multimodal thought chain reasoning according to claim 1, characterized in that, The multi-entity complex scene image is subjected to multi-target recognition, detection and segmentation using RAM model and Grounded-SAM model, and then the Qwen2.5-VL-7B module is used to filter and obtain the editing targets that require fine spatial reasoning; The visual positioning mind chain includes a target box and a semi-transparent mask.

3. The fine-grained image editing method based on multimodal thought chain reasoning according to claim 2, characterized in that, The textual thought chain for editing the target is a four-segment structure. The Qwen2.5-VL-72B module is used to generate editing instructions and text thought chains based on the target box, and then the Qwen3 module is used to rewrite the editing instructions and enhance language diversity. The textual thought chain includes scene description, location reasoning, editing intent, and result interpretation.

4. The fine-grained image editing method based on multimodal thought chain reasoning according to claim 3, characterized in that, Methods for obtaining modified images of complex scenes include: The Bagel module is used to modify the editing target according to the editing instructions; the modification includes editing processing and removal; the modified editing target after editing processing is seamlessly integrated with the multi-entity complex scene image to obtain the modified complex scene image; the removed modified editing target is used with the LaMa model and the editing target mask to perform consistent completion of the multi-entity complex scene to obtain the modified complex scene image. The modified complex scene images are reviewed for semantic and visual consistency using the Qwen2.5-VL-7B module, and the modified complex scene images that meet the review threshold are retained.

5. The fine-grained image editing method based on multimodal thought chain reasoning according to claim 4, characterized in that, During the training of the multimodal intertwined reasoning model, the understanding module performs text reasoning parsing on the modification instructions to obtain the textual thought chain of the editing target; The generation module obtains the visual positioning thought chain of the editing target based on the text thought chain, processes the editing target, and obtains the edited image.

6. The fine-grained image editing method based on multimodal thought chain reasoning according to claim 5, characterized in that, The mask monitoring mechanism includes: A text projector is used to project the text thought chain output by the comprehension module to obtain the text projection result; A visual projector is used to project the visual positioning thought chain output by the generation module and the edited image onto the image respectively, so as to obtain the visual positioning visual projection result and the edited image visual projection result. An image encoder is used to encode images of complex scenes with multiple entities to obtain the encoded image features; The mask encoder predicts the text prediction mask based on the features of the encoded image and the text projection results; it predicts the location prediction mask based on the features of the encoded image and the visual positioning projection results; and it predicts the editing prediction mask based on the features of the encoded image and the visual projection results of the edited image. The mask supervision loss function is calculated based on the comparison results between the text prediction mask, the location prediction mask, and the edit prediction mask and the real mask, and the model parameters are adjusted according to the loss function.

7. The fine-grained image editing method based on multimodal thought chain reasoning according to claim 6, characterized in that, The text projection result is represented as The visual positioning and visual projection results are represented as follows: The edited image visual projection result is represented as ;but: ， ， ， In the formula For text prediction mask, Indicates a mask encoder. For encoded image features, To locate the predicted mask, To edit the predictive mask; Masked Supervision Loss Function for: ， In the formula It is the binary cross-entropy. This is the real mask.

8. The fine-grained image editing method based on multimodal thought chain reasoning according to claim 7, characterized in that, The multimodal thought chain alignment mechanism includes a text thought chain alignment mechanism: The mean semantic feature is obtained by averaging the features of all the smallest semantic units corresponding to the editing target in the text thinking chain. The mean noise feature is obtained by averaging all the noise features of the editing target. ; noise characteristic mean Through the fully connected layer Perform projection and calculate the mean of the projected features and semantic features. The cosine similarity is used to obtain the text thought chain alignment loss. : 。 9. The fine-grained image editing method based on multimodal thought chain reasoning according to claim 8, characterized in that, The multimodal thinking chain alignment mechanism also includes a visual positioning thinking chain alignment mechanism: The visual feature units of the editing target corresponding to the visual positioning mind chain are represented as: ; Generate noisy VAE editing features and perform interpolation adjustments to obtain the noise features. and make noise characteristics With visual feature units They have the same spatial resolution; noise characteristics Through the fully connected layer Projection is performed, and the noise characteristics and visual feature units after projection are calculated. The cosine similarity is used to adjust the model parameters based on the calculation results to improve the noise characteristics. With visual feature units Align in spatial position.

10. The fine-grained image editing method based on multimodal thought chain reasoning according to claim 9, characterized in that, Visual positioning mind chain alignment loss for: 。

Images