Image completion method and related equipment based on multimodal reasoning and iterative feedback

Abstract

This application provides an image completion method and related equipment based on multimodal reasoning and iterative feedback, relating to the field of computer vision technology. The method includes: acquiring a masked image, the mask, and text instructions; extracting image features and predicting feature representations of missing regions; using a multimodal large language model to perform chain-like reasoning on the input, generating multi-step semantic descriptions as intermediate reasoning chains; fusing the original instructions and reasoning chains to form enhanced prompts, and using these to guide a conditional generation model to complete the masked region; again using the multimodal large language model to evaluate the semantic consistency of the generated results and generate text feedback; if the score does not meet a threshold, automatically updating the generated prompts based on feedback and starting the next round of iterative generation until the result meets the requirements or reaches the iteration limit. This application can significantly improve the semantic accuracy, logical rationality, and user intent consistency of image completion, and achieves automated iterative optimization of the generation process.

Description

Technical Field

[0001] This application relates to the fields of computer vision and multimodal artificial intelligence, and in particular to an image completion method and related equipment based on multimodal reasoning and iterative feedback. Background Technology

[0002] Image completion is an important research area in computer vision, aiming to reconstruct missing parts that are occluded or damaged by masks using information from known regions of an image. This technology is widely used in photo editing, security monitoring, medical image analysis, industrial inspection, and other scenarios.

[0003] In recent years, with the development of generative models, especially latent diffusion models, text-guided image completion techniques have made significant progress. These methods allow users to guide the completion process with natural language descriptions, improving controllability and flexibility. However, existing techniques still have the following limitations: 1) Insufficient semantic understanding and logical consistency: Existing text-guided methods typically use short user instructions directly as generation conditions, lacking the ability to deeply reason about complex semantics and long-distance contextual logic. When faced with instructions that require understanding spatial relationships between objects, physical constraints (such as support and occlusion), or complex scene descriptions, the model is prone to semantic contradictions, object structural errors, or results that violate physical laws (such as object suspension or limb misalignment).

[0004] 2) Lack of self-evaluation and correction capabilities: Existing methods mostly generate results in a one-time process, lacking an automatic evaluation mechanism for the quality of the generated results. Once the generated results do not meet the user's intentions or have obvious defects, adjustments can only be made manually by re-entering instructions, failing to form an automated optimization loop.

[0005] 3) Insufficient multimodal information fusion: Most methods only perform shallow fusion of visual features and text prompts, failing to fully utilize the powerful capabilities of multimodal large language models in visual-language joint understanding, reasoning, and planning to guide the generation process. Summary of the Invention

[0006] The main objective of this application is to propose an image completion method, apparatus, electronic device, storage medium, and program product based on multimodal reasoning and iterative feedback, aiming to solve the problems of poor semantic consistency, weak complex logic processing capability, and lack of automated iterative optimization mechanism in existing image completion methods.

[0007] To achieve the above objectives, one aspect of this application proposes an image completion method based on multimodal reasoning and iterative feedback, the method comprising: Obtain the original image with mask to be completed. , corresponding mask and text instructions ; For the masked original image With mask Feature extraction is performed to obtain the principal scale features. ; Based on the principal scale features With mask In the feature domain, the semantic and texture representations of the missing regions are predicted, resulting in... Aligned Predictive Features ; The masked original image Text instructions Input a multimodal large language model and generate intermediate inference chain text with multi-step semantic constraints describing the completed content. ; Integrate the text instructions With the intermediate inference chain text This forms extended hints to guide the generation process. ; With the extended prompt As a constraint, the mask The specified region is used to generate the conditional generation, resulting in a completed image. ; The completed image Original image with mask Text instructions and intermediate inference chain text Input the multimodal large language model to obtain a score that evaluates the consistency between the generated result and the instructions and inference chain. and text feedback information ; Determine the consistency score Is it below the preset threshold? ; If so, then based on the text feedback information Update the extended prompt The process of generating the conditions and subsequent steps is repeated based on the updated prompts until the consistency score is achieved. Not lower than the threshold Or the number of iterations has reached its limit; If not, or if the maximum number of iterations is reached, output the current completed image as the final result.

[0008] In some embodiments, the generation of intermediate inference chain text In the steps, the information input to the multimodal large language model also includes information based on the predicted features. The generated prompt message.

[0009] In some embodiments, in the formation of extended prompts Following that, it also includes: The predicted features Encode it as a conditional vector, and combine it with the extended hint. Together, they serve as the joint constraint conditions generated for the conditions applied to the mask region.

[0010] In some embodiments, the intermediate inference chain text This includes a text sequence that describes step-by-step at least one of the following: the attributes of objects in the completed area, the spatial relationships between objects, the background lighting style, and the detailed texture.

[0011] In some embodiments, the step of relying on the text feedback information Update the extended prompt ,include: Based on the text feedback information The identified defects, for the extended hints Or the intermediate inference chain text Make corrections, additions, or rewrites to generate a new round of extended hints. .

[0012] In some embodiments, a dynamic sampling strategy is adopted when repeatedly performing condition generation: the representation of the completed image generated in the previous iteration in the latent space is used as the initial state of the next denoising process, and some noise is superimposed for local redrawing and optimization.

[0013] In some embodiments, the method further includes a step of training an image completion model, wherein the loss function used in the training step includes at least: Pixel reconstruction loss: based on the completed image Compared with the reference truth image Calculation of pixel-level differences between them; Perceptual Consistency Loss: Based on Completed Images Compared with the reference truth image Distance calculation in the feature space of a pre-trained deep network; Structural similarity loss: based on incomplete images Compared with the reference truth image Calculation of structural similarity index; Semantic consistency loss: based on a multimodal large model for the completed image With the text instructions The semantic matching score is calculated.

[0014] In some embodiments, the loss function used in the training step further includes a step-aware attention decoupling loss, which is used to constrain the generative model during the denoising process so that the cross-attention modules corresponding to different inference chain steps within it focus on different spatial regions or semantic attributes of the image.

[0015] In some embodiments, the training step employs a reinforcement learning strategy to incorporate the consistency score. As a reward signal, the parameters of the conditional generation model are optimized using the policy gradient method to maximize the expected consistency score of the generated results.

[0016] To achieve the above objectives, another aspect of this application proposes an image completion device based on multimodal reasoning and iterative feedback, the device comprising: The feature extraction module is configured to extract features from the input masked raw image. With mask Perform feature extraction and output main scale features. ; The missing feature prediction module is configured to receive the main scale features. With mask Predict the features of the missing region and output the predicted features. ; The chain-based reasoning module integrates a multimodal large language model and is configured to receive... and and output the intermediate inference chain text. ; Prompt to construct modules and configure them for fusion. and Generate extended hints ; The condition generation module is configured to use... As constraints, content is generated in the mask area, and the completed image is output. ; The evaluation and feedback module integrates the aforementioned multimodal large language model and is configured to provide feedback on... Conduct a consistency assessment and output a score. With text feedback ; The iterative control module is configured to compare the scores. With threshold and in According to The prompt construction module is controlled to update the prompt and triggers the condition generation module to perform a new round of generation iterations until the stopping condition is met.

[0017] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0018] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0019] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0020] Compared with the prior art, the technical solution provided in this application has at least the following beneficial effects: 1) Significantly improve semantic and logical consistency: By introducing a multimodal large language model for "chain thinking reasoning", ambiguous user instructions are transformed into structured multi-step semantic planning, providing clear and detailed guidance for the generation process, effectively avoiding generation deviations caused by instruction ambiguity, and significantly improving the consistency between the completed content and the context and physical logic.

[0021] 2) Achieving automated iterative optimization: A closed-loop system of "generation-evaluation-feedback-correction" has been creatively constructed. By using the same multimodal large language model to evaluate the consistency of the generated results and generate text feedback, the system can automatically optimize the generated prompts based on the feedback and start the next round of iteration, achieving self-improvement without human intervention, which greatly improves the reliability of the final result and the consistency with user intent.

[0022] 3) Enhanced adaptability to complex scenarios: The chain reasoning mechanism can decompose complex instructions and gradually reason out details such as object attributes and spatial relationships, enabling the model to better handle complex completion tasks involving multiple object interactions and specific scenario constraints.

[0023] 4) High integration depth and flexibility: The framework proposed in this application achieves deep coupling of visual feature extraction, semantic reasoning, conditional generation, and quality assessment. At the same time, each module (such as the feature extraction network and the generative model) can be implemented with different architectures and efficiently integrated through a lightweight control head, thus combining performance and deployment flexibility. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall framework of an image completion method provided in one embodiment of this application.

[0025] Figure 2This is a structural block diagram of an image completion device in one embodiment of this application.

[0026] Figure 3 This is a schematic diagram of the loss function during the training phase in one embodiment of this application.

[0027] Figure 4 This is a flowchart of an image completion method based on multimodal reasoning and iterative feedback provided in one embodiment of this application.

[0028] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0031] Image completion is an important area of ​​low-level vision and imaging reconstruction. Its core objective is to reconstruct content from missing, damaged, or occluded parts of an image using information from known regions. Image completion technology is widely used in consumer photography, security monitoring, remote sensing surveying, medical imaging, and industrial inspection, and has a direct impact on the reliability of subsequent recognition, measurement, and decision-making.

[0032] With the development of deep learning technology, especially the rise of latent diffusion models, techniques represented by stable diffusion models have achieved significant breakthroughs in text-guided image completion. This technological paradigm allows users to guide the image completion process through natural language descriptions, improving the flexibility of image editing. However, existing technologies still have shortcomings in practical applications. Purely visual-driven or simple text-guided completion lacks deep semantic consistency. When faced with complex semantic logic, intricate spatial relationships, and long-distance contextual dependencies, the lack of physical constraints on entity poses, limb positions, and contact relationships between objects easily leads to anatomical errors or objects appearing to float, violating physical laws. Furthermore, existing models have limited multimodal understanding capabilities. Most image completion models rely solely on visual cues or simply treat textual conditions as external constraints, failing to fully understand the semantic details in user instructions and their relationship with the image context.

[0033] Therefore, there is an urgent need for an image completion method that can deeply understand multimodal instructions, perform chain-like logical reasoning, and iteratively optimize through automatic evaluation feedback, so as to improve the semantic accuracy, logical rationality, and user intent consistency of the completion results.

[0034] In view of this, this application provides an image completion method, apparatus, electronic device, storage medium, and program product based on multimodal reasoning and iterative feedback. This scheme predicts the missing representation caused by pixel missingness in the feature domain, uses a multimodal large language model to generate chain-inference text to construct an enhanced prompt, the enhanced prompt is used to guide the generation of content in the mask missing area, and after each round of generation, the same multimodal large language model performs consistency evaluation and text feedback output on the generation result, thereby automatically updating the enhanced prompt and triggering the next round of generation, forming a fully automatic closed-loop process of "generation-evaluation-correction".

[0035] The image completion method provided in this application relates to the fields of computer vision and multimodal artificial intelligence technology. The image completion method provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the image completion method, but is not limited to the above forms.

[0036] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0037] like Figure 4 As shown, this embodiment provides an image completion method based on multimodal reasoning and iterative feedback, including the following steps: S1. Obtain the original masked image to be completed. , corresponding mask and text instructions ; S2, the original image with mask With mask Feature extraction is performed to obtain the principal scale features. ; S3. Based on the aforementioned main scale features With mask In the feature domain, the semantic and texture representations of the missing regions are predicted, resulting in... Aligned Predictive Features ; S4. The original image with mask Text instructions Input a multimodal large language model and generate intermediate inference chain text with multi-step semantic constraints describing the completed content. ; S5. Integrate the text instructions. With the intermediate inference chain text This forms extended hints to guide the generation process. ; S6, with the extended prompts As a constraint, the mask The specified region is used to generate the conditional generation, resulting in a completed image. ; S7. Complete the image. Original image with mask Text instructions and intermediate inference chain text Input the multimodal large language model to obtain a score that evaluates the consistency between the generated result and the instructions and inference chain. and text feedback information ; S8. Determine the consistency score. Is it below the preset threshold? ; S9. If so, then based on the text feedback information... Update the extended prompt The process of generating the conditions and subsequent steps is repeated based on the updated prompts until the consistency score is achieved. Not lower than the threshold Or the number of iterations has reached its limit; S10. If not, or if the maximum number of iterations is reached, output the current completed image as the final result.

[0038] The solutions of the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific application examples.

[0039] This embodiment provides an image completion method based on multimodal chain-like reasoning and a large model scoring feedback loop, including the following steps: Step 1: Obtain the original image to be completed with the mask, the corresponding mask, and the text instructions, and construct the training set, validation set, and test set.

[0040] For example, a clear image is obtained from a public dataset or a self-built dataset as a reference ground truth image, and the first... The reference true value image is denoted as A mask is applied to the reference ground truth image to generate a masked original image, and the corresponding mask is denoted as... ,in And the original image with the mask is denoted as Optionally, at least one degradation method is applied to the reference ground truth image or the original image with a mask to construct a more complex completed scene. The degradation methods include Gaussian noise, Poisson noise, motion blur, defocus blur, and JPEG compression artifacts. Color domain and frequency domain data augmentation is performed on the samples to expand the sample distribution; the image size is uniformly adjusted to... ,in These represent the number of image channels, height, and width, respectively; and the pixel values ​​are normalized.

[0041] Step 2: Build an image completion model and train it using the training set; the input of the model is the degraded image and the output is the reconstructed image.

[0042] Specifically, an image completion model based on missing feature prediction and a multimodal large model scoring feedback loop is constructed, and the image completion model is trained using a training set. The input of the image completion model is the original image with a mask, the mask, and text instructions. The image completion model sequentially performs feature extraction, missing information prediction, chain-like reasoning prompt construction, condition generation, and large model evaluation feedback iterative update, and outputs a completed image. The completed image is the restoration result after the content of the masked region of the original image with a mask is completed.

[0043] In one embodiment, the image completion model is composed of the following functional units connected in sequence: Feature extraction unit, used for processing the input masked raw image With mask Extract the original feature set at multiple scales, and align it to the principal scale through cascading or projection. The feature extraction unit employs a convolutional structure, a Transformer structure, or a hybrid of both, and the number of feature channels... Set in Interval, number of layers Set in The interval can optionally include a degradation estimation branch and share early layer features with the trunk; The missing feature prediction unit is used to predict the semantic and texture representations corresponding to the missing regions in the feature domain. The missing feature prediction unit receives... With mask Output and Predictive features aligned with spatial and channel dimensions ; Chain-like reasoning units are used to process masked original images. Text instructions And optional missing feature hints are mapped to intermediate inference chain text for multi-step semantic description. The chain-like reasoning unit is implemented using a multimodal large language model; it can be combined with INT8 / FP16 quantization and distillation to meet latency and power consumption constraints. Prompt construction unit, used to merge original text instructions With intermediate inference chain text Form extended prompts and optionally Encode it as a conditional vector to be used together with extended hints as generation conditions; Condition generation unit, used to receive extended prompts In addition to optional missing feature conditions, conditional sampling is performed on the mask region to generate a completed image. The condition generation unit adopts a latent diffusion model, which includes a variational autoencoder and a denoising network, and injects extended cue conditions through a cross-attention mechanism. Evaluation feedback unit, used to complete the image Original image with mask Text instructions With intermediate inference chain text Input a multimodal large language model and output a consistency score. and text feedback information ; The update and iteration control unit is prompted to update the consistency score. Below the preset threshold At that time, based on the text feedback information Update extension tips get It controls the condition generation unit to execute the next round of generation; when the consistency score reaches the threshold or the number of iterations reaches the upper limit, it outputs the final completed image.

[0044] Step 3: Input the test set into the trained model for testing and evaluation.

[0045] The method flow in this embodiment is divided into two parts: a training phase and a testing phase. In the training phase, an image completion model is first constructed, encompassing feature extraction, missing feature prediction, chain-like reasoning, prompt construction, condition generation, evaluation feedback, and prompt update and iteration control. By modeling some degradation types and intensities, the model learns the representation for completing "missing pixels / styles" in the feature domain and its dynamic fusion strategy with the original features. In the testing phase, the image completion model performs reasoning on degradation samples that have not appeared before or whose distribution has changed. Based on the completion process and fusion gating learned in the training phase, it restores the input image (completing mask shapes, missing region locations, or scene combinations not appearing in the training phase, and restoring the structure, semantics, and texture details of the missing regions under the constraints of the multimodal large model scoring feedback loop), and outputs a reconstruction result with the same resolution as the input.

[0046] After the model is trained, it can be deployed and enter the inference process, as follows: Obtain the original image of the mask mask and text instructions As input, the feature extraction unit extracts a multi-scale original feature set, which is then aligned to the principal scale via cascaded or linear projection. Subsequently, the missing feature prediction unit predicts the semantic and texture representations corresponding to the missing regions in the feature domain. This unit receives... With mask The output in the spatial and channel dimensions is the same as Aligned Predictive Features .

[0047] Based on this, the chain-like reasoning unit will transform the masked original image. Text instructions And optional missing feature hints are mapped to intermediate inference chain text for multi-step semantic description. This reasoning chain is implemented by a multimodal large language model and can be represented as:

[0048] Among them, the reasoning chain This is used to explicitly define multi-step constraints such as object attributes, spatial relationships, background lighting / style, and detail textures, thereby reducing ambiguity in short prompts and providing executable semantic planning for subsequent generation. The prompt construction unit then integrates the original text instructions. With intermediate inference chain text Form extended prompts and optionally The encoding is used as a conditional vector, which, together with the extended hints, serves as the generation condition. , recorded as .

[0049] Extended prompts are received by the condition generation unit. and optional missing feature conditions Conditional sampling is performed on the masked area to generate and output a completed image. The condition generation unit employs a latent diffusion model, comprising a variational autoencoder and a denoising network, and injects extended cue conditions through a cross-attention mechanism; it can be represented as:

[0050] Once generated, the evaluation feedback unit will complete the image. Original image with mask Text instructions With intermediate inference chain text Input a multimodal large language model and output a consistency score. and text feedback information , can be represented as:

[0051] Among them, consistency score Used to quantify the degree of matching between the generated results and the constraints of the inference chain and the user's intent; text feedback information. This is used to indicate object attributes, spatial relationships, or detailed defects that need correction. Finally, the prompt update and iteration control unit determines whether the consistency score falls below a preset threshold. Update extended prompts based on text feedback information. get It also controls the condition generation unit to execute the next round of generation; when the consistency score reaches the threshold or the number of iterations reaches the upper limit. The final completed image is output in real time. The above closed loop can be uniformly represented in iterative form:

[0052]

[0053] In the judgment conditions or Time-terminated iteration output To improve iteration efficiency, a dynamic sampling strategy can be adopted, which uses the results generated in the previous round as the starting point for the next round of sampling and adds some noise for local redrawing, which can be expressed as:

[0054] in This reduces the computational overhead of initializing from pure noise in each round and enhances convergence stability.

[0055] Through the above process, degenerate completion and detail restoration can be completed without external masks and explicit priors, and stable reconstruction quality and inference overhead can be maintained in various degenerate combination scenarios.

[0056] (1) Data preparation and preprocessing See Figure 1 The training data is obtained by sampling reference ground truth images from public or self-built high-quality image libraries. For each Generate mask ,in Indicates missing regions, Represent the known region and construct the original image with a mask. Images are standardized to the same size. Pixel normalization to or During the training phase, enhancements such as random cropping, flipping, and color perturbation can be applied. (Text instructions) The objects / attributes / styles or constraints that should be filled in to describe missing regions or to maintain consistency with the context can be generated by manual annotation or templates; the dataset is divided into training, validation and test sets according to the proportions for training and evaluation.

[0057] Data augmentation employs strategies including random pruning, horizontal / vertical flipping, color perturbation, and frequency domain perturbation to enhance the model's robustness and generalization ability. The dataset is divided into training, validation, and test sets in a proportional ratio (e.g., 8:1:1), where the test set should cover degradation intensities or types outside the training distribution to evaluate the model's generalization performance. When text prompts are introduced... At the same time, style / content guidance prompts can be built (such as "enhance texture details", "soften background noise", "sharpen edge structure", etc.) and matched with the corresponding degradation type.

[0058] (2) Construct and train an image restoration model that combines missing feature prediction and large model feedback. See Figure 2 and Figure 3 In this embodiment, the model is composed of functional units connected sequentially. First, the feature extraction unit processes the input masked original image. With mask Extract multi-scale features and align them to the principal scale to obtain the original features. The feature extraction unit can employ a convolutional structure, a Transformer structure, or a hybrid of both, with a certain number of feature channels. number of floors Subsequently, the missing feature prediction unit predicts the semantic and texture representations corresponding to the missing regions in the feature domain, and outputs the result. Aligned Predictive Features Based on this, the chain-like reasoning unit will transform the masked original image. Text instructions And optional missing feature hints mapped to intermediate inference chain text This reasoning chain, implemented using a multimodal large language model, explicitly structures "what to supplement, how to supplement, and what consistency to maintain," thereby reducing ambiguity in short prompts and improving generation controllability. The prompt construction unit integrates the original text instructions. With inference chain text Get extended hints The condition is then used as the input to the conditional generation unit; the conditional generation unit uses a latent diffusion model to perform conditional sampling on the mask region to generate the completed image. Its core generation process can be summarized by the following formula:

[0059] Once generated, the evaluation feedback unit will complete the image. Original image with mask Text instructions With inference chain text Input a multimodal large language model and output a consistency score. and text feedback information This is used to evaluate whether the generated result satisfies the constraints of the instruction and inference chain. Finally, the prompt update and iteration control unit determines the result based on the scoring threshold. Decide whether to continue iterating: when When, based on the feedback information, update the extended prompts and trigger the next round of generation; when Or the number of iterations has reached the limit. The final completed result is output at that time. This closed-loop process can be briefly expressed by the following formula:

[0060] and with or Output as a stop condition To improve iteration efficiency, this embodiment may employ a dynamic sampling strategy, which uses the result generated in the previous round as the starting point for the next round and adds a small amount of noise for local redrawing, thereby reducing the overhead of starting with pure noise in each round; the initialization of its latent space can be expressed as:

[0061] (3) Loss function Defining and using appropriate loss functions during model training is crucial for ensuring comprehensive improvement in model performance. The objective function during the training phase is a weighted combination of multiple loss functions.

[0062] Reconstruction losses Including L1 and L2 losses:

[0063] Perceived loss Feature distance based on VGG network or LPIPS:

[0064] Structural similarity loss Enhance local structural fidelity:

[0065] Semantic consistency loss Calculate image-text semantic alignment based on a pre-trained multimodal model:

[0066] This is a quality evaluation score based on a multimodal large model.

[0067] Regularization loss Constraints are imposed on the control parameters:

[0068] The total loss is:

[0069] Optionally, a step-aware attention decoupling loss is constructed to constrain the correspondence between inference chain steps and cross-attention responses, prompting the generative model to focus on different spatial regions or attribute dimensions at different inference steps. Optionally, a reinforcement learning objective based on consistency scores is constructed, using the consistency scores as reward signals, and the parameters of the conditional generation unit are optimized through the policy gradient method to maximize the consistency expectation of the generation results.

[0070] Repeat the training steps multiple times to obtain multiple loss values; find the minimum total loss value among these values, and use the weights and biases corresponding to the minimum total loss value as the optimal weight vector and optimal bias term for the model, denoted as follows: and ; Complete the training of an image restoration model based on missing feature prediction and large model scoring feedback.

[0071] The step of inputting the test set into the trained image restoration model for testing includes: The test set is input into the trained image restoration model, utilizing... and Prediction is performed to obtain the restored images corresponding to the test set, and subjective and objective evaluation metrics are used to evaluate the image restoration effect.

[0072] The optimizer uses either Adam or AdamW, and the initial learning rate is set to... The training employs either cosine decay or stepwise descent strategies; the total number of training epochs is 300–600, with batch size determined by hardware specifications; mixed precision (AMP) and gradient clipping are enabled during training. The model parameters are stabilized using the exponential moving average (EMA); the model with the lowest total loss is selected for inference on the validation set. The training steps are repeated Q times to obtain the image restoration training model, resulting in a total of Q*N loss values.

[0073] (4) Model testing The test set is input into the trained model for testing. The preprocessed data is input into the trained image reconstruction model, utilizing... and Perform prediction to obtain the completed image corresponding to the test set. This enables an image completion method based on missing feature prediction and large model feedback. Optionally, subjective and objective evaluation methods such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and image-text matching score are used to calculate the image... With tips The semantic similarity is used as a semantic consistency reference to verify the image restoration effect based on missing feature prediction and large model feedback.

[0074] In summary, the method of the present invention has at least the following advantages and beneficial effects compared with the prior art: 1) Explicit semantic reasoning eliminates ambiguity. Enhanced prompts generated through chained reasoning effectively reduce the risk of generated content deviating from the user's intent.

[0075] 2) Multimodal joint understanding improves consistency. The multimodal large language model undertakes semantic planning and consistency evaluation, while the diffusion model undertakes perceptual generation execution, ensuring the unity of global semantics and local pixels.

[0076] 3) Feedback loop enables adaptive optimization. The model automatically identifies and corrects unreasonable aspects in the output, achieving a leap from single-step generation to adaptive iterative optimization.

[0077] 4) Consistency constraint training improves reliability. Through training strategies such as step-level attention guidance and semantic rewards, the model achieves both high visual fidelity and logical correctness.

[0078] 5) Modularity and scalability. The encoder, missing feature prediction module, multimodal modulation module and decoder of this invention are all pluggable, which can be easily replaced with implementations of different scales or architectures (convolution, Transformer or hybrid), meeting various needs from high performance to lightweight.

[0079] This application also provides an image completion device based on multimodal reasoning and iterative feedback, which can implement the above method. The device includes: The feature extraction module is configured to extract features from the input masked raw image. With mask Perform feature extraction and output main scale features. ; The missing feature prediction module is configured to receive the main scale features. With mask Predict the features of the missing region and output the predicted features. ; The chain-based reasoning module integrates a multimodal large language model and is configured to receive... and and output the intermediate inference chain text. ; Prompt to construct modules and configure them for fusion. and Generate extended hints ; The condition generation module is configured to use... As constraints, content is generated in the mask area, and the completed image is output. ; The evaluation and feedback module integrates the aforementioned multimodal large language model and is configured to provide feedback on... Conduct a consistency assessment and output a score. With text feedback ; The iterative control module is configured to compare the scores. With threshold and in According to The prompt construction module is controlled to update the prompt and triggers the condition generation module to perform a new round of generation iterations until the stopping condition is met.

[0080] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0081] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0082] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0083] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 502 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 using the methods described in the embodiments of this application. The input / output interface 503 is used to implement information input and output; The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504); The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0084] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0085] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0086] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0087] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0088] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented in the embodiments of this program product are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages ​​such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0089] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0090] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0093] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0094] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0096] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0097] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0098] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An image completion method based on multimodal reasoning and iterative feedback, characterized in that, The method includes the following steps: Obtain the original image with mask to be completed. , corresponding mask and text instructions ; For the masked original image With mask Feature extraction is performed to obtain the principal scale features. ; Based on the principal scale features With mask In the feature domain, the semantic and texture representations of the missing regions are predicted, resulting in... Aligned Predictive Features ; The masked original image Text instructions Input a multimodal large language model and generate intermediate inference chain text with multi-step semantic constraints describing the completed content. ; Integrate the text instructions With the intermediate inference chain text This forms extended hints to guide the generation process. ; With the extended prompt As a constraint, the mask The specified region is used to generate the conditional generation, resulting in a completed image. ; The completed image Original image with mask Text instructions and intermediate inference chain text Input the multimodal large language model to obtain a score that evaluates the consistency between the generated result and the instructions and inference chain. and text feedback information ; Determine the consistency score Is it below the preset threshold? ; If so, then based on the text feedback information Update the extended prompt The process of generating the conditions and subsequent steps is repeated based on the updated prompts until the consistency score is achieved. Not lower than the threshold Or the number of iterations has reached its limit; If not, or if the maximum number of iterations is reached, output the current completed image as the final result.

2. The method according to claim 1, characterized in that, In the generation of intermediate inference chain text In the steps, the information input to the multimodal large language model also includes information based on the predicted features. The generated prompt message.

3. The method according to claim 1, characterized in that, In the formation of extended prompts Following that, it also includes: The predicted features Encode it as a conditional vector, and combine it with the extended hint. Together, they serve as the joint constraint conditions generated for the conditions applied to the mask region.

4. The method according to claim 1, characterized in that, The intermediate inference chain text This includes a text sequence that describes step-by-step at least one of the following: the attributes of objects in the completed area, the spatial relationships between objects, the background lighting style, and the detailed texture.

5. The method according to claim 1, characterized in that, Based on the text feedback information Update the extended prompt ,include: Based on the text feedback information The identified defects, for the extended hints Or the intermediate inference chain text Make corrections, additions, or rewrites to generate a new round of extended hints. .

6. The method according to claim 1, characterized in that, When repeatedly generating conditions, a dynamic sampling strategy is adopted: the representation of the completed image generated in the previous iteration in the latent space is used as the initial state of the next denoising process, and some noise is superimposed for local redrawing and optimization.

7. The method according to claim 1, characterized in that, The method further includes a step of training an image completion model, wherein the loss function used in the training step includes at least: Pixel reconstruction loss: based on the completed image Compared with the reference truth image Calculation of pixel-level differences between them; Perceptual Consistency Loss: Based on Completed Images Compared with the reference truth image Distance calculation in the feature space of a pre-trained deep network; Structural similarity loss: based on incomplete images Compared with the reference truth image Calculation of structural similarity index; Semantic consistency loss: based on a multimodal large model for the completed image With the text instructions The semantic matching score is calculated.

8. The method according to claim 7, characterized in that, The loss function used in the training step also includes step-aware attention decoupling loss, which is used to constrain the generative model during the denoising process so that the cross-attention modules corresponding to different inference chain steps within it focus on different spatial regions or semantic attributes of the image.

9. The method according to claim 7, characterized in that, The training step employs a reinforcement learning strategy, which incorporates the consistency score. As a reward signal, the parameters of the conditional generation model are optimized using the policy gradient method to maximize the expected consistency score of the generated results.

10. An image completion device based on multimodal reasoning and iterative feedback, characterized in that, The device includes: The feature extraction module is configured to extract features from the input masked raw image. With mask Perform feature extraction and output main scale features. ; The missing feature prediction module is configured to receive the main scale features. With mask Predict the features of the missing region and output the predicted features. ; The chain-based reasoning module integrates a multimodal large language model and is configured to receive... and and output the intermediate inference chain text. ; Prompt to construct modules and configure them for fusion. and Generate extended hints ; The condition generation module is configured to use... As constraints, content is generated in the mask area, and the completed image is output. ; The evaluation and feedback module integrates the aforementioned multimodal large language model and is configured to provide feedback on... Conduct a consistency assessment and output a score. With text feedback ; The iterative control module is configured to compare the scores. With threshold and in According to The prompt construction module is controlled to update the prompt and triggers the condition generation module to perform a new round of generation iterations until the stopping condition is met.

Images