Training method of image editing model, image editing method and corresponding device

Abstract

Embodiments of the present application disclose a training method of an image editing model, an image editing method and corresponding devices. The training method comprises: obtaining training samples composed of source image samples, editing instruction samples and target image samples, and obtaining source image features, target image features and editing instruction features after encoding the samples; generating a region indication map containing editing region and reserved region indication information based on the feature difference between the source image and the target image, obtaining a noisy feature after adding noise to the target image feature, using a diffusion transform network to denoise the noisy feature with the source image feature and the editing instruction feature as conditions, and collecting generated field information predicted in the noise process; using a loss function with regional constraints to iteratively optimize diffusion transform network parameters, and using the trained diffusion transform network parameters to construct a final image editing model. The present application can realize high-fidelity and high-harmony scene text editing in a scene without manual annotation of masks.

Description

Technical Field

[0001] This application relates to the fields of generative artificial intelligence and computer vision technology, and in particular to a training method for an image editing model, an image editing method, and a corresponding device. Background Technology

[0002] In the field of computer vision technology, the core objective of image editing technology is to replace, modify, or adjust specified content in real-world or generated images using only natural language editing instructions. For example, STE (SceneText Editing) technology replaces, modifies, or adjusts text in images using natural language editing instructions. As a core branch of generative visual content editing, this technology has demonstrated significant application value in recent years in scenarios such as cultural and creative design, media content production, and autonomous driving data augmentation.

[0003] Current mainstream image editing techniques heavily rely on explicit spatial prior information to complete the editing process. These methods typically require users to provide a precise binary editing region mask in addition to the image to be edited and the editing instructions. This mask constrains the model's editing scope and prevents accidental modification of the background area. However, this type of existing technology, which relies on explicit masks, has revealed several inherent and difficult-to-solve flaws in practical applications. First, the interaction is costly and lacks flexibility. Users need to manually annotate precise editing area masks, which not only raises the technical barrier to entry, but also makes it difficult for non-professional users to draw high-precision masks that fit the text edges.

[0004] Second, the blending effect of the editing results is stiff. Editing constraints based on hard binary masks often result in a rigid layout of the final edited content, or an unnatural splicing effect at the boundaries of the edited content, leading to a significant decrease in the overall visual harmony of the final edited result. Summary of the Invention

[0005] In view of this, this application provides a training method for an image editing model, an image editing method, and a corresponding device to achieve high-fidelity and high-harmony image editing effects in scenarios without manually labeled masks.

[0006] This application provides the following solution: According to the first aspect, a method for training an image editing model is provided, the method comprising: Acquire training data comprising multiple training samples, wherein the training samples are triplets consisting of source image samples, editing instruction samples, and target image samples; The training samples are encoded to obtain source image features, target image features, and editing instruction features; Based on the differences between the source image features and the target image features, a region indication map is generated, which includes information indicating the edited region and information indicating the retained region. The target image features are subjected to noise processing to obtain the noisy features of the target image; Using the source image features and the editing instruction features as the first condition, a diffusion transform network is used to denoise the noise features of the target image based on the first condition, and the predicted generation field information is collected during the denoising process. Using the loss function corresponding to the first training objective, the parameters of the diffusion transform network are iteratively optimized. The first training objective includes: minimizing the difference between the predicted generated field information and the generated field information corresponding to the target image features in the edit region, and minimizing the difference between the predicted generated field information and the generated field information corresponding to the source image features in the preserved region. The diffusion transform network is used to construct the first image editing model after training.

[0007] Optionally, the training samples are obtained in the following manner: Obtain source image samples and editing instruction samples; The source image samples are edited using a second image editing model to obtain candidate target images; The candidate target images are subjected to quality assessment, and candidate target images that meet the preset quality requirements are selected as target image samples. The triple consisting of the source image sample, the editing instruction sample, and the target image sample is used as a training sample.

[0008] Optionally, obtaining the editing instruction sample includes: Editing instruction samples are generated for the source image samples using a visual language model.

[0009] Optionally, the second image editing model is trained in the following manner: The source image sample is edited using the basic image editing model based on the editing instruction sample to obtain the initial target image; Obtain the reverse editing instructions from the sample editing instructions; The initial target image and the reverse editing instructions are used as inputs to the basic image editing model, and the source image samples are used as the target outputs of the basic image editing model. The parameters of the basic image editing model are iteratively optimized. The base image editing model after iterative optimization is used as the second image editing model.

[0010] Optionally, the quality assessment of the candidate target image includes at least one of the following: The candidate target image is subjected to target object recognition, and a consistency comparison is performed between the editing instruction sample and the target object recognition result. The quality assessment result is determined based on the consistency comparison result. Determine at least one of the peak signal-to-noise ratio (PSNR) and structural similarity between the candidate target image and the source image sample in the reserved region, and determine the quality assessment result based on at least one of the PSNR and structural similarity; The color distribution divergence between the candidate target image and the source image sample is determined, and the quality assessment result is determined based on the color distribution divergence.

[0011] Optionally, generating a region indication map based on the difference between the source image features and the target image features includes: Determine the pixel-level distance between the source image features and the target image features in the channel dimension to obtain a difference map; The difference map is mapped to edit region weights and retain region weights using an exponential weighting algorithm, and the edit region weights and retain region weights constitute the region indicator map.

[0012] Optionally, the loss function corresponding to the first training objective includes: a first loss term and a second loss term; The first loss term is obtained by weighting the difference between the predicted generated field information and the generated field information corresponding to the target image features using the edit region weights; The second loss term is obtained by weighting the difference between the predicted generated field information and the generated field information corresponding to the source image features using the preserved region weights.

[0013] Optionally, wavelet transforms at at least one scale level are performed on the predicted generation field information, the generation field information corresponding to the target image features, and the generation field information corresponding to the source image features in at least one direction to obtain high-frequency subbands of the predicted generation field information, the generation field information corresponding to the target image features, and the generation field information corresponding to the source image features in at least one direction and at at least one level. The loss function corresponding to the first training objective includes: a third loss term and a fourth loss term; The third loss term is obtained by using the edit region weights to weight the differences between the predicted generated field information and the generated field information corresponding to the target image features in the same direction and at the same level, and then summing the results of the weighting in the at least one direction and at the at least one level. The fourth loss term is obtained by using the preserved region weights to weight the differences between the predicted generated field information and the generated field information corresponding to the source image features in the same direction and at the same level in the high-frequency sub-bands, and then summing the results of the weighting in the at least one direction and at the at least one level.

[0014] Optionally, if the diffusion transformation network is a flow matching model, then the generated field information is velocity field information; If the diffusion transformation network is a denoising diffusion probability model, then the generated field information is noise field information.

[0015] According to a second aspect, an image editing method is provided, the image editing method comprising: Obtain the image to be edited and the editing instructions; The first image editing model is used to edit the image to be edited based on the image to be edited and the editing instructions to obtain the edited image; The first image editing model is trained based on any of the aforementioned training methods.

[0016] Optionally, the edited image includes multiple candidate edited images, and the image editing method further includes: Based on the multiple candidate edited images, the harmony score of each candidate edited image is evaluated using a harmony evaluation model. The final edited image is selected based on the harmony score of each candidate edited image.

[0017] According to a third aspect, a training apparatus for an image editing model is provided, the apparatus comprising: The first acquisition unit is used to acquire training data including multiple training samples, wherein the training samples are triplets consisting of source image samples, editing instruction samples and target image samples; The encoding unit is used to encode the training samples to obtain source image features, target image features, and editing instruction features; A generation unit is configured to generate a region indication map based on the difference between the source image features and the target image features, the region indication map including information indicating the edited region and information indicating the retained region; A noise-adding unit is used to add noise to the target image features to obtain the noise-adding features of the target image; The denoising unit is used to take the source image features and the editing instruction features as first conditions, and use a diffusion transform network to denoise the noise features of the target image based on the first conditions, and collect the predicted generation field information during the denoising process; An optimization unit is used to iteratively optimize the parameters of the diffusion transform network using the loss function corresponding to the first training objective. The first training objective includes minimizing the difference between the predicted generated field information and the generated field information corresponding to the target image features in the edit region, and minimizing the difference between the predicted generated field information and the generated field information corresponding to the source image features in the retained region. The diffusion transform network is used to construct the first image editing model after training.

[0018] According to a fourth aspect, an image editing apparatus is provided, the image editing apparatus comprising: The second acquisition unit is used to acquire the image to be edited and the editing instructions; An editing unit is used to edit the image to be edited based on the image to be edited and the editing instructions using a first image editing model, to obtain an edited image; The first image editing model is trained based on any of the aforementioned training methods.

[0019] According to a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the training method for the image editing model described in any one of the first aspects, or the steps of the image editing method described in any one of the second aspects.

[0020] According to a sixth aspect, an electronic device is provided, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, perform the steps of the training method of the image editing model described in any one of the first aspects, or perform the steps of the image editing method described in any one of the second aspects.

[0021] According to a seventh aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the training method for the image editing model described in any one of the first aspects, or the steps of the image editing method described in any one of the second aspects.

[0022] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application achieves decoupling and separation of the editing region and the retention region by adaptively generating a region indicator map based on the potential spatial difference between the source image and the target image. During the training phase of the diffusion transform network, independent gradient optimization objectives are assigned to the two types of regions: the editing region is required to align with the features of the target image, while the retention region retains the features of the source image. This ensures that the background region is highly consistent with the source image while achieving text editing, avoiding background degradation and artifacts. It enables high-fidelity and high-harmony scene text editing in scenarios without manual masking.

[0023] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a system architecture diagram applicable to the embodiments of this application.

[0026] Figure 2 A flowchart illustrating the training method for the image editing model provided in this application embodiment.

[0027] Figure 3 This is a schematic diagram of the training framework for the image editing model provided in the embodiments of this application.

[0028] Figure 4 This is a flowchart of the image editing process provided for an embodiment of this application.

[0029] Figure 5 This is a schematic diagram of the reasoning framework of the image editing model provided in the embodiments of this application.

[0030] Figure 6 This is a schematic diagram of the structure of the training device for the image editing model provided in the embodiments of this application.

[0031] Figure 7 This is a schematic diagram of the structure of the image editing device provided in the embodiments of this application.

[0032] Figure 8 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0034] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0035] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0036] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0037] Traditional image editing methods mostly rely on manually annotated editing area masks, requiring users to pre-define the location of the content to be modified. This has a high barrier to entry and is difficult to adapt to the automated editing needs of open scenarios. To solve this problem, the industry has recently been shifting towards maskless image editing solutions guided by natural language commands. These solutions allow users to complete editing simply by inputting editing commands in natural language, eliminating the need for pre-annotated masks and significantly lowering the barrier to entry.

[0038] Currently, the widely adopted approach is based on instruction-guided maskless editing models, such as those based on denoising diffusion probability models or flow matching models. These methods do not rely on mask annotation; instead, they utilize a basic editing model to edit the source image according to editing instructions to obtain a pseudo-target image. This pseudo-target image is then used as a supervision signal to train the image editing model across the entire image using a mean squared error loss function, enabling it to learn the mapping from the source image and editing instructions to the target image. The initial design goal of this approach is to eliminate manual annotation of the editing region, thereby achieving high editing flexibility and automation.

[0039] However, when application scenarios require that the edited image not only accurately modify the edited area but also maintain strict consistency with the source image in the background area, avoiding color shifts or texture degradation, the above-mentioned solutions do not perform ideally. There is a fundamental contradiction here: the target image used in this solution is not a perfect reference; it often exhibits subtle variations in non-edited areas that differ from the source image. The uniform mean squared error loss function across the entire image indiscriminately treats these variations as supervision targets, forcing the model to fit the differences in non-edited areas during training. This inevitably compromises background fidelity while optimizing the edited area modification. For example, in the task of replacing the text on a sign from "tea" to "wine," the pseudo-target image generated by the basic editing model may have slight shifts in local texture, lighting, or color in the background. During training, the model will learn these shifts as correct transformations, resulting in artifacts or blurring in the background area of ​​the final edited image, failing to maintain consistency with the source image. Furthermore, the mean squared error loss in the spatial domain has low-pass characteristics, tending to suppress high-frequency information, easily causing excessive smoothing of text edges and texture details, presenting a sticker-like visual effect.

[0040] In view of this, this application provides a new approach. To facilitate understanding of this application, the system architecture on which this application is based will first be described. Figure 1 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 1 As shown, the system architecture may include: user terminals, server components, and data storage.

[0041] The server and user terminal can be the two main components of an application service. The server can use a server, server cluster, or cloud server as its hardware infrastructure and deploy an image editing model, which may include an encoder, a diffusion transform network, and a decoder. The server-side training device can train or update the image editing model using collected training samples. The server-side image editing device can receive data such as the image to be edited and editing instructions sent by the user terminal, and invoke the image editing model to perform image editing and other processing.

[0042] User terminals can include, but are not limited to, smart mobile terminals, wearable devices, PCs, in-vehicle terminals, IoT terminals, virtual reality devices, augmented reality devices, or mixed reality devices. User terminals can receive user-inputted images and editing instructions, and send data to be processed or processing requests to the server via a network. User terminals can also display descriptive text and show generated images or videos.

[0043] Data storage can be used to store training data, model parameters, and intermediate processing results. For example, the data storage terminal can store training samples, encoder and decoder parameters, parameters of the diffusion transform network, and latent variable representations generated during training.

[0044] In one deployment method, the encoder, diffusion transform network, and decoder are deployed uniformly on the server side as components of the image editing model, and the user terminal is only responsible for uploading the image to be edited and the editing instructions.

[0045] It should be understood that Figure 1 The user terminals, training devices, image editing devices, image editing models, and data storage shown are merely illustrative. Depending on implementation needs, any number of user terminals, training devices, image editing devices, image editing models, and data storage can be included.

[0046] Figure 2 This is a flowchart illustrating the training method for the image editing model provided in this application embodiment. This method can be performed by... Figure 1 The training device for the image editing model in the system shown is executed. For example... Figure 2 As shown, the training method may include the following steps: Step 210: Obtain training data including multiple training samples. The training samples are triplets consisting of source image samples, editing instruction samples, and target image samples.

[0047] Combination Figure 3 This document provides a schematic diagram of the training framework for the image editing model provided in this application embodiment. First, training data comprising multiple training samples is acquired. Each training sample is a triple consisting of a source image sample, an editing instruction sample, and a target image sample. The source image sample can be a real-world scene text image; the editing instruction sample can be an editing instruction in natural language (such as "replace one text in the image with another text"); and the target image sample is the target image obtained after editing the source image sample according to the editing instruction. Organizing the training samples in triples allows the model to simultaneously perceive the image content before editing, the editing intent, and the expected result after editing during training, providing complete supervision information for subsequent region decoupling and conditional denoising.

[0048] Step 220: Encode the training samples to obtain source image features, target image features, and editing instruction features.

[0049] After acquiring training samples, they are encoded to obtain source image features, target image features, and editing instruction features. This encoding process transforms the training samples from pixel space to latent feature space. For example, a variational autoencoder (VAE) can be used to encode the source and target image samples separately to obtain source image features and target image features; editing instruction samples can be encoded using a text encoder to obtain editing instruction features. Subsequent region decoupling operations, noise addition, and denoising are performed in the latent feature space. Compared to direct operations in pixel space, this significantly reduces computational dimensionality and overhead while preserving the semantic and structural information of the image.

[0050] Step 230: Based on the differences between the source image features and the target image features, generate a region indication map, which includes information indicating the editing area and information indicating the retention area.

[0051] Step 240: Add noise to the target image features to obtain the noise-added features of the target image.

[0052] Noise is added to the target image features to obtain the noisy features of the target image. This noise addition process can be, for example, following the forward noise addition method in the diffusion process, adding Gaussian noise to the target image features at specific time steps. The noise addition process constructs a learning path from noise to target for the Diffusion Transformer (DiT) network, enabling the network to progressively learn the conditional distribution of generating target image features during the denoising process.

[0053] Step 250: Using the source image features and editing instruction features as the first condition, the diffusion transform network is used to denoise the noise features of the target image based on the first condition, and the predicted generation field information is collected during the denoising process.

[0054] By using both source image features and editing instruction features as conditional inputs, the diffusion transform network can simultaneously perceive the original content of the source image and the user's desired editing direction during the denoising process, thereby guiding the network to generate an output that conforms to the editing instructions and maintains consistency with the source image.

[0055] The generated field information mentioned here refers to the intermediate information predicted by the diffusion transform network during the denoising process. Depending on the model architecture used by the diffusion transform network, the generated field information can have different physical meanings (such as velocity field information or noise field information, which will be further explained later). By collecting the generated field information predicted during the denoising process instead of directly collecting the denoising results, fine-grained supervisory constraints can be applied to the intermediate steps of the denoising process during the training phase, thereby more effectively guiding the network to learn the correct generated trajectory.

[0056] Step 260: Using the loss function corresponding to the first training objective, iteratively optimize the parameters of the diffusion transform network. The first training objective includes: minimizing the difference between the predicted generated field information and the generated field information corresponding to the target image features in the editing region, and minimizing the difference between the predicted generated field information and the generated field information corresponding to the source image features in the preserved region; wherein, the diffusion transform network is used to construct the first image editing model after training.

[0057] The parameters of the diffusion transform network are iteratively optimized using the loss function corresponding to the first training objective. The first training objective comprises two independent optimization directions: in the edit region, minimizing the difference between the predicted generated field information and the generated field information corresponding to the target image features; in the preserved region, minimizing the difference between the predicted generated field information and the generated field information corresponding to the source image features. This training objective assigns independent gradient optimization directions to the edit and preserved regions: the generated field information in the edit region is guided to align with the generation trajectory of the target image features, thus ensuring accurate text modification; the generated field information in the preserved region is guided to anchor to the generation trajectory of the source image features, thereby blocking the backpropagation of noise from non-edited regions in the false target image at the gradient level and preventing background degradation.

[0058] The diffusion transform network is a neural network backbone used to denoise the features of a noisy target image and predict the generated field information. Its core function is to gradually recover image content consistent with the editing instructions from the noisy feature state during the denoising process, using the source image features and editing instruction features as the first conditional inputs. The diffusion transform network can employ either a flow matching model or a denoising diffusion probability model.

[0059] A diffusion transform network is used to build the first image editing model after training. The parameters of the diffusion transform network can be iteratively optimized, for example, using low-rank adaptation (LoRA), which updates only the LoRA parameters in the network. After training, the parameter-optimized diffusion transform network serves as the core inference component of the first image editing model. Furthermore, the first image editing model may also include components such as an encoder and a decoder.

[0060] As can be seen from the above process, this application achieves the decoupling and separation of the editing region and the retention region by adaptively generating a region indicator map based on the potential spatial difference between the source image and the target image. During the training stage of the diffusion transform network, independent gradient optimization objectives are assigned to the two types of regions: the editing region is required to align with the features of the target image, while the retention region retains the features of the source image. This ensures that the background region is highly consistent with the source image while achieving text editing, avoiding background degradation and artifacts. It can achieve high-fidelity and high-harmony scene text editing in scenarios without manual masking.

[0061] The above steps will be described in detail below with reference to the embodiments. First, step 210, namely "acquiring training data including multiple training samples", will be described in detail with reference to the embodiments.

[0062] In some embodiments of this application, training samples are obtained in the following manner: source image samples and editing instruction samples are obtained; the source image samples are edited using a second image editing model to obtain candidate target images; the candidate target images are quality evaluated, and candidate target images whose quality evaluation results meet preset quality requirements are selected as target image samples; a triple consisting of source image samples, editing instruction samples, and target image samples is used as a training sample.

[0063] When constructing training samples, source image samples and editing instruction samples are first obtained. The source image samples are real-world scene text images to be edited, and the editing instruction samples are natural language text describing the editing requirements. A pre-trained second image editing model is then invoked, inputting the source image samples and editing instruction samples. The model performs end-to-end editing processing and outputs candidate target images. The second image editing model can be a basic image editing model based on a diffusion probability model or a flow matching model; for example, currently open-source image editing models such as FireRed-Image-Edit and Qwen-Image-Edit can be used.

[0064] The generated candidate target images undergo quality evaluation in at least one dimension. Evaluation dimensions may include, for example, text content accuracy, background consistency, and overall visual naturalness. Unqualified candidates whose evaluation results do not meet a preset quality threshold are removed, and candidate target images that meet the quality requirements are identified as target image samples. In some embodiments, candidate target images corresponding to all source images can be generated in batches before performing batch quality screening, improving the processing efficiency of dataset construction.

[0065] Finally, the source image samples, editing instruction samples, and the filtered target image samples are combined into a triplet structure. Each triplet is a usable training sample. By repeating the above process, a complete training dataset containing multiple training samples can be obtained.

[0066] As an alternative, the quality assessment process can also adopt a fully automated multi-indicator fusion assessment scheme, which automatically filters data by weighting scores from multiple dimensions of indicators without human intervention and is suitable for the automated construction of large-scale training datasets.

[0067] This application's embodiments automatically generate candidate target images through a pre-trained basic editing model, and then construct training triplets by screening out unqualified candidates through quality assessment. This solves the problems of insufficient real pairwise labeled data and high cost of manual labeling in the field of maskless editing, and can quickly build a large-scale high-quality training dataset at a lower cost. At the same time, the quality screening step can remove low-quality samples with serious background misalignment and text errors in advance, reducing the probability of noise introduction from the training data source level.

[0068] In some embodiments of this application, when obtaining editing instruction samples, a visual language model can be used to generate editing instruction samples for the source image samples.

[0069] The source image sample to be used to generate editing instructions is input into a pre-loaded VLM (Vision-Language Model). This VLM supports generating corresponding text descriptions and editing requirements based on the input image. Dedicated generation prompts are set for the visual language model, requiring it to generate reasonable natural language editing instructions based on the existing text content in the input source image. For example, the model is required to recognize the text content in the source image, specify the direction of modification, and generate editing instruction samples that conform to the editing logic. Then, the model's generation capabilities are invoked, outputting candidate editing instructions corresponding to the source image sample.

[0070] Furthermore, rule-based validation can be performed on the candidate editing instructions generated by the visual language model to filter out unqualified candidates with non-standard text format, unreasonable editing logic, or text recognition errors, and the validated candidate editing instructions can be determined as the final editing instruction sample.

[0071] As an example, taking a public street scene text image dataset, a street scene image containing the sign text "Spring Tea House" is selected as the source image sample; the source image is input into VLM, and the generation prompt is set to "identify the text on the sign in the image and generate a reasonable text editing instruction that requires modification of a keyword in the text"; VLM outputs the candidate editing instruction "replace 'tea' with 'coffee' on the sign in the image"; the instruction is verified to confirm that the text content is correctly identified and the editing requirements are clear and reasonable, and finally the instruction is determined as the editing instruction sample corresponding to the source image.

[0072] As an alternative, manually generated editing instruction samples can be used. This involves annotators directly viewing the text content in the source image samples and manually writing natural language editing instructions that meet the requirements according to preset editing rules. These instructions are then manually verified a second time before being stored in the dataset. This method is suitable for constructing small-scale training datasets, ensuring the logical rationality and accuracy of the editing instructions, and reducing the probability of model-generated errors.

[0073] This application utilizes the cross-modal understanding and generation capabilities of visual language models to automatically generate batches of natural language editing instruction samples that conform to editing logic for input source images. It can quickly obtain a large number of well-annotated and logically sound editing instruction samples with low time and manpower costs, adapting to the automated construction needs of large-scale maskless editing training datasets. At the same time, the generated instructions conform to natural language expression habits, which can match the training requirements of instruction-guided image editing models and improve the ability of the trained model to understand and adapt to real user editing instructions.

[0074] In some embodiments of this application, the second image editing model is trained as follows: the source image sample is edited based on the editing instruction sample using the basic image editing model to obtain the initial target image; the reverse editing instruction of the editing instruction sample is obtained; the initial target image and the reverse editing instruction are used as the input of the basic image editing model, and the source image sample is used as the target output of the basic image editing model to iteratively optimize the parameters of the basic image editing model; the basic image editing model after iterative optimization is used as the second image editing model.

[0075] When constructing the second image editing model, the initial basic image editing model is invoked. This model can be a pre-trained image editing model based on a diffusion probability model or a flow matching model. The original real-world scene text image, i.e., the source image sample, is taken as input. The generated forward editing instructions are combined to perform editing processing on the source image sample to obtain the initial target image. The initial target image generated by the initial basic image editing model may have defects such as color shift, texture artifacts, and background misalignment. Then, the editing direction is reversed, and the source editing object and the target object in the forward editing instructions are swapped to obtain the corresponding reverse editing instructions. A reverse training pair is constructed: the initial target image is used as the model input, and the source image sample is used as the supervised target. The parameters of the basic image editing model are iteratively optimized to enable the model to learn the mapping prior from the initial target image to the real natural image, thus completing the reverse refinement training.

[0076] Finally, the base image editing model, after being trained through reverse refinement, is used in conjunction with the initially generated forward editing instructions to perform forward editing on the source image, generating candidate target images.

[0077] As an example, a restaurant poster image containing the text "Grand Opening Special Offer" is selected as the source image. A forward editing instruction, "Change the text 'Special Offer' to 'Offer'", is generated using VLM. The poster image and the forward instruction are input into the initial basic editing model to obtain the initial target image. Then, a reverse editing instruction, "Change the text 'Offer' back to 'Special Offer'", is constructed. Using the initial target image and the reverse editing instruction as input, and the original poster image as the supervised target, iterative parameter optimization is performed on the initial basic editing model, enabling the model to learn the ability to correct artifacts and offsets. By performing the above processing on multiple images, iterative optimization of the initial basic editing model parameters is achieved, ultimately completing the reverse refinement training of the basic editing model.

[0078] In an alternative embodiment, the reverse training process can adopt a batch iterative optimization approach. After generating a certain batch of reverse training pairs, a parameter update is performed once. After repeating multiple rounds of reverse refinement, the forward pseudo-target generation is performed to further improve the quality of pseudo-targets generated by the second image editing model.

[0079] This application adopts a self-supervised inverse-forward synthesis method, which only requires a large number of unlabeled real images to train a second image editing model that can generate high-quality target images. Through inverse training, the model learns in advance the ability to correct artifacts and offsets, which greatly improves the quality of the generated target images and reduces the interference of noise on the subsequent training of the editing model. At the same time, it can automatically generate large-scale high-quality training data without manual annotation, which effectively reduces the cost and cycle of dataset construction.

[0080] In some embodiments of this application, quality evaluation of candidate target images includes at least one of the following: The first method involves identifying target objects in candidate target images, performing a consistency comparison between the editing instruction samples and the target object identification results, and determining the quality assessment results based on the consistency comparison results.

[0081] For example, target object recognition can be optical character recognition (OCR) to identify the text content in candidate target images, and then compare the recognition result with the target text specified in the editing instruction sample. By comparing the target object recognition with the editing instruction, it is verified whether the edited content in the candidate target image meets the expectations of the editing instruction. If they do not match, the quality of the candidate target image is considered unsatisfactory, and the candidate target image is filtered out to ensure the accuracy of the training samples at the level of edited content.

[0082] The second approach involves determining at least one of the PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) of the preserved region in the candidate target image, and then determining the quality assessment result based on at least one of PSNR and SSIM.

[0083] PSNR is an objective metric for measuring image reconstruction quality, assessing the degree of distortion by calculating the mean square value (MSE) of the pixel-level error between two images. SSIM is a metric for measuring image perceptual quality from the perspective of the human visual system, comprehensively comparing the similarity of two images in three dimensions: brightness, contrast, and structure. For example, the PSNR and SSIM of the preserved region (between the source image sample and the candidate target image) can be calculated in the VAE latent space. That is, after encoding the candidate target image and the corresponding source image using VAE, the PSNR and SSIM of the preserved region are calculated in the latent feature space. Then, PSNR and SSIM are compared with corresponding thresholds. If at least one of PSNR and SSIM is less than the corresponding threshold requirement, the candidate target image is considered to have unsatisfactory quality. If both PSNR and SSIM meet the corresponding threshold requirements, the quality assessment is considered successful. By calculating the PSNR and SSIM of the preserved region, the background fidelity of the candidate target image relative to the source image in the non-edited region is quantitatively evaluated, filtering out candidate target images with severely damaged backgrounds and ensuring the background quality of the training samples.

[0084] The third method is to determine the color distribution divergence between the candidate target image and the corresponding source image, and then determine the quality assessment result based on the color distribution divergence.

[0085] Color distribution divergence measures the color consistency between a candidate target image and a source image, detecting color shifts in the candidate target image. For example, color distribution divergence can be calculated based on the Jensen-Shannon Divergence (JS) in the CIELAB color space. This metric is a ground truth-free (GT-free) evaluation method, assessing color consistency without requiring a perfect reference image. By filtering candidate target images with severe color shifts, the color quality of the training samples can be further improved.

[0086] In practical applications, the three quality assessment methods mentioned above can be used individually or in combination to comprehensively filter candidate target images from multiple dimensions such as text correctness, background fidelity, and color consistency, thereby ensuring the quality of the final training samples.

[0087] In an alternative embodiment, the three quality assessment methods described above can be performed sequentially in the following order: First, target object recognition and consistency comparison are performed on the candidate target images. The text content of the editing area in the candidate target image is extracted by using optical character recognition (OCR). The recognized target text is then compared with the target text required by the editing instruction sample. If the comparison results are inconsistent, the candidate target image is directly judged as unqualified and is screened out; only candidates that match the comparison proceed to the next stage of evaluation.

[0088] After completing the consistency check, the regions in the candidate target image that do not need to be edited (i.e., the retained regions) can be determined based on the OCR recognition results. At least one of the following indices—Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM)—is calculated between the retained region and the corresponding retained region in the source image. If the index value is lower than a preset acceptable threshold, it indicates that the retained region of the candidate target image has obvious structural damage or distortion, and the candidate is deemed unqualified and eliminated. Candidates whose indices meet the threshold requirements proceed to the next stage of evaluation.

[0089] Calculate the color distribution divergence (CDD) between the candidate target image and the corresponding source image. This index is used to measure the degree of difference in the overall color distribution of the two images. If the color distribution divergence is higher than the preset offset threshold, it indicates that the candidate target image has a significant overall color shift, and the candidate is judged to be of unqualified quality and is screened out. Candidates whose divergence meets the requirements are judged to be of qualified quality and are determined as target image samples.

[0090] This application embodiment constructs a candidate target image quality evaluation system from dimensions such as content consistency, background structure fidelity, and overall color consistency. It can filter out low-quality samples with text content errors, damaged background structures, and overall color shifts, solve the problem of excessive noise ratio in the process of automatically generating training samples, and greatly improve the sample quality of the final training dataset.

[0091] In some embodiments of this application, when determining the color distribution divergence between a candidate target image and its corresponding source image, editing regions can be cropped from the source image and the candidate target image respectively to obtain a first editing region image and a second editing region image; the first editing region image and the second editing region image are processed using an image segmentation algorithm to obtain a first local binary object mask (i.e., the content to be edited) and a second local binary object mask; the first editing region image and the second editing region image are converted to the CIELAB color space respectively to obtain a first editing region image and a second editing region image in the CIELAB color space; a first object color distribution is generated based on the first editing region and the first local binary object mask in the CIELAB color space, and a second object color distribution is generated based on the second editing region and the second local binary object mask in the CIELAB color space; the color distribution divergence of the images before and after editing is calculated using a distribution divergence algorithm based on the first object color distribution and the second object color distribution.

[0092] Based on the bounding box of the editing region corresponding to the editing instruction, local images containing only the editing region are cropped from the source image and the corresponding candidate target image, respectively denoted as the first editing region image and the second editing region image. Scene text segmentation algorithms such as Hi-SAM are called to segment the two cropped local regions, and the outputs local binary object masks corresponding to the first editing region and the second editing region, respectively. The background pixels in the local region are filtered out by the mask, and only the pixels of the object (i.e. the content to be edited) are retained, eliminating the interference of background color on the object color distribution statistics.

[0093] The cropped first and second edited region images are converted from the native RGB color space to the CIELAB color space, which better matches the human eye's perception of color differences. Based on the extracted binary object mask, the CIELAB color values ​​corresponding only to the objects within each of the two edited regions are extracted. Then, the CIELAB color values ​​are applied to the CIELAB space... L (brightness), a (Red-green hue) b The three channels (yellow and blue hues) are used to independently generate normalized one-dimensional color histograms, which are then used as the source object's color distribution. and the color distribution of the edited object ,in .

[0094] To measure the difference between two color distributions symmetrically and in a bounded manner, this embodiment of the application can use Jensen-Shannon (JS) divergence for calculation: first, calculate the average distribution of the source object color distribution and the edit object color distribution for each channel. The calculation method is as follows: (1) Based on this, the final score of the color distribution divergence (CDD) is defined as the average of the JS divergences of the three CIELAB channels, calculated as follows: (2) in, () is the calculation function for DL ​​divergence. The lower the CDD score, the better the color consistency between the edited object and the object in the source image. In this application, CDD is a pure statistical measure without learnable parameters, and the calculation process is independent of the editing model itself, so it can be used as an objective and reproducible evaluation standard.

[0095] In the sample filtering scenario of this application, the CDD calculation result is compared with the preset offset threshold. If the CDD score is higher than the threshold, it is determined to be a color offset unqualified sample and is screened out.

[0096] This application addresses the shortcomings of existing image quality assessment metrics that rely solely on real reference images and cannot cover the evaluation of color consistency in edited text. It proposes a purely statistical color distribution divergence (CDD) based on the CIELAB color space. This CDD can objectively quantify the color consistency of candidate target images before and after editing, even without real-labeled reference images. It accurately filters out low-quality samples with significant color shifts, filling the gap in the color perception dimension of existing assessment systems. Furthermore, CDD is an unbiased metric without learnable parameters, and its calculation process is independent of the training model, resulting in objective and reproducible results. It can be used for filtering low-quality samples during the dataset construction stage and for automatically evaluating editing results during the inference stage, balancing the needs of dataset construction and model performance verification. This effectively improves the sample quality of the final training dataset, thereby enhancing the visual naturalness and color consistency of the output results of the trained text editing model.

[0097] The following describes step 230, namely "generating a region indication map based on the difference between the source image features and the target image features", in detail with reference to the embodiments.

[0098] After obtaining the source and target image features, a region indicator map is generated based on the differences between them. The region indicator map includes information indicating the editable region and information indicating the retained region. The editable region corresponds to spatial locations where the differences between the source and target image features are significant, while the retained region corresponds to spatial locations where the differences are smaller, i.e., the background. The region indicator map does not require external mask annotations from the user; instead, it automatically infers the spatial distribution of the editable and retained regions from the feature differences within the training samples themselves. This reduces interaction costs and avoids the rigid boundary problems caused by hard masks, providing adaptive spatial guidance for subsequent region segmentation optimization.

[0099] In some embodiments of this application, when generating a region indicator map based on the difference between source image features and target image features, the pixel-level distance between the source image features and target image features in the channel dimension can be determined to obtain a difference map; the difference map is mapped to edit region weights and retain region weights using an exponential weighting algorithm, and the edit region weights and retain region weights constitute the region indicator map.

[0100] The pixel-level distance described above is used to measure the degree of difference between source image features and target image features at each spatial location. Pixel-level distance can be, for example, L2 distance, which calculates the Euclidean distance between source image features and target image features along the channel dimension, resulting in a difference map. By calculating the distance pixel-by-pixel along the channel dimension, the difference map accurately reflects the magnitude of the difference between source image features and target image features at each spatial location. Locations with larger differences correspond to edit regions, while locations with smaller differences correspond to retention regions, providing accurate input for subsequent weight mapping.

[0101] Before mapping the difference map to edit region weights and retain region weights, it can be normalized. For example, min-max normalization can be performed on the difference map to map its value range to [0, 1]. Normalization eliminates the influence of absolute numerical magnitudes in the difference map, allowing subsequent exponential weighting algorithms to map on a uniform scale, thus improving the stability and consistency of weight generation.

[0102] After obtaining the normalized difference map, an exponential weighting algorithm is used to map the difference map into edit region weights and retained region weights. Edit region weights and reserved region weights The following formulas can be used to calculate them respectively: (4) (5) in, This is the normalized difference plot. and These are preset hyperparameters. Compared to linear weighting or hard thresholding binarization, the exponential weighting algorithm can achieve a smooth non-linear transition between the editing and retention areas. That is, it assigns higher editing weights to areas with significant differences and higher retention weights to areas with minor differences, while achieving a continuous gradient in the transition area. This avoids the visually unnatural problems caused by hard boundaries and also avoids the problem of excessive diffusion of editing focus that may be caused by linear weighting.

[0103] In some embodiments, hyperparameters For example, we can choose 1.6. For example, 2.0 can be used. Control the overall scaling of the weights. Control the steepness of the exponential function. Take 1.6, Setting the hyperparameters to 2.0 allows the exponential weighting function to create sufficient weight contrast between regions of significant and minor differences, achieving precise decoupling between the edited and preserved regions while maintaining a smooth transition. Of course, the specific settings for the hyperparameters will vary. The value can be flexibly adjusted by those skilled in the art according to actual needs, and no specific limitation is made here.

[0104] In other embodiments, in addition to the exponential weighting algorithm, the mapping from the difference map to the weight map can also be done using a linear weighting method or a hard threshold binarization method. Different mapping methods can be selected according to the needs of the application scenario.

[0105] This application's embodiments generate region indicator maps by combining pixel-level feature difference calculation with exponential weighted mapping. This can automatically distinguish between edited and retained regions and generate corresponding spatial attention weights. It can provide accurate spatial constraint guidance for model training without manual annotation of edited regions. This reduces the cost of manual annotation and can accurately guide the model to focus on optimizing the generation quality of edited regions. At the same time, it strengthens the constraints on non-edited retained regions, avoiding unnecessary modifications to non-edited regions by the model. This effectively balances the accuracy of edited region content and the fidelity of non-edited region content, improving the overall editing effect of the trained model.

[0106] The following describes step 260, namely "using the loss function corresponding to the first training objective to iteratively optimize the parameters of the diffusion transformation network", in detail with reference to the embodiments.

[0107] In some embodiments of this application, the loss function corresponding to the first training target may include: a first loss term and a second loss term; the first loss term is obtained by weighting the difference between the predicted generated field information and the generated field information corresponding to the target image features using the edit region weight; the second loss term is obtained by weighting the difference between the predicted generated field information and the generated field information corresponding to the source image features using the preserved region weight.

[0108] Here, we take a diffusion transformation network trained based on a flow matching model as an example, where the corresponding generated field information is velocity field information.

[0109] The first loss term is obtained by weighting the difference between the predicted velocity field information and the velocity field information corresponding to the target image features using the edit region weights. It can be represented as: (6) Where E[] refers to the expected value, For the predicted velocity field information, The velocity field information corresponding to the target image features. To edit the region weights, For element-wise multiplication, This represents the square of the 2-norm. The first loss term forces the model to follow the generated trajectory corresponding to the target image features at spatial locations with higher weights in the editing region, thereby ensuring that the content of the object within the editing region is accurately modified.

[0110] The second loss term is obtained by weighting the difference between the predicted velocity field information and the velocity field information corresponding to the features in the source image using the weights of the preserved regions. For example, the second loss term... It can be represented as: (7) in, The velocity field information corresponding to the features of the source image. To preserve region weights, the second loss term forces the model to strictly anchor the predicted velocity field information to the generated trajectory corresponding to the source image features at spatial locations with higher region weights. This effectively blocks the backpropagation of noise from non-edited regions in the target image at the gradient level, preventing background degradation.

[0111] Similarly, if the diffusion transformation network is trained based on the denoised diffusion probability model, then the generated field information is noise field information. When calculating the loss, the velocity field information in the above formulas can be replaced with the corresponding noise field information.

[0112] This application embodiment, by decoupling the training objective loss function which includes the weighted loss of the edited region and the weighted loss of the preserved region, can specifically achieve a balance between the accuracy constraint of the edited region content and the fidelity constraint of the non-edited region content, and blocks the interference of background noise in the target image sample on the model training from the gradient backpropagation level.

[0113] In some embodiments, in addition to spatial domain loss, frequency domain loss constraints can be further introduced to address the loss of high-frequency details and over-smoothing caused by spatial domain mean square error (MSE) loss. Specifically, wavelet transforms at at least one scale level are performed on the predicted generated field information, the generated field information corresponding to the target image features, and the generated field information corresponding to the source image features in at least one direction, respectively, to obtain high-frequency subbands of the predicted generated field information, the generated field information corresponding to the target image features, and the generated field information corresponding to the source image features in at least one direction and at at least one level. The at least one direction may include at least one of the horizontal direction, the vertical direction, and the diagonal direction, and the at least one scale level may include at least one of the first level and the second level. For example, Discrete Wavelet Transform (DWT) can be used to perform cascaded multi-level decomposition of the generated field information, outputting high-frequency subbands in the horizontal (LH), vertical (HL), and diagonal (HH) directions at each level. By using multi-directional and multi-level wavelet transforms, high-frequency details in the generated field information (such as the direction of text edges, the thickness variation of strokes, and the micro-particles of texture) can be captured from different spatial scales and directions, providing multi-granular feature representations for subsequent application of refined supervisory constraints in the frequency domain.

[0114] The wavelet basis used for wavelet transform can be, for example, the Daubechies-4 (DB4) wavelet. The DB4 wavelet has a good balance between spatial and frequency localization, and can more accurately capture text edge features. In other embodiments, other wavelet bases such as the Haar wavelet can also be used. The Haar wavelet has the simplest structure and the highest computational efficiency, making it suitable for scenarios with high real-time requirements. Different wavelet bases can be selected according to application needs. In addition to wavelet transform, high-pass filters based on Fast Fourier Transform (FFT) can also be used to separate high-frequency subbands. FFT high-pass filters can separate high-frequency components in the global frequency domain, making them suitable for scenarios where global frequency features are constrained, providing an alternative for high-frequency feature extraction.

[0115] In embodiments that introduce frequency domain loss, the loss function corresponding to the first training objective may further include a third loss term and a fourth loss term. The third loss term is obtained by using edit region weights to weight the differences between the predicted generated field information and the generated field information corresponding to the target image features in the same direction and at the same level between the high-frequency subbands, and then summing the results of the weighting in at least one direction and at at least one level. Taking the training framework of the flow matching model as an example, the third loss term... For example, it can be represented as: (8) in, Indicates the decomposition hierarchy. Indicates direction ( ), For the predicted velocity field information in the first Level 1 High-frequency subband in the direction, The velocity field information corresponding to the target image features in the first... Level 1 High-frequency subband in the direction, For the first Editing region weights at different levels. The third loss term forces the model to generate high-frequency details consistent with the target image in the editing region in the high-frequency subband of the frequency domain. This forces the model to render sharp object edges and rich textures in the editing region. For example, if the object before and after editing is text, it can render sharp text edges and rich textures in the editing region, effectively combating the over-smoothing caused by the spatial domain MSE loss, i.e., eliminating the "sticker-like" artifacts of the generated text.

[0116] The fourth loss term is obtained by using the preserved region weights to weight the differences between the predicted velocity field information and the velocity field information corresponding to the source image features in the same direction and at the same level among the high-frequency subbands, and then summing the weighted results for at least one direction and at at least one level. For example, it can be represented as: (9) in, The velocity field information corresponding to the source image features in the th... Level 1 High-frequency subband in the direction, For the first The weights of the preserved regions at each level. The fourth loss term forces the model to maintain the same micro-texture as the source image (such as the roughness of stone, the graininess of paper, etc.) in the high-frequency subbands of the frequency domain, preventing the high-frequency details of non-edited regions from being discarded or altered by the model during training.

[0117] This application embodiment extracts high-frequency detail subbands of the generated field through multi-scale, multi-directional wavelet transform, and constructs a frequency domain decoupling loss by combining the weights of the edited region and the retained region. On the basis of spatial domain structural constraints, it supplements the consistency constraints of high-frequency texture details, which can specifically optimize the text edge sharpness of the edited region and the fidelity of the original texture of the non-edited region, further improving the visual naturalness and detail consistency of the generated image.

[0118] The generated field information mentioned in the above embodiments has different physical meanings depending on the model architecture adopted by the diffusion transform network. In some embodiments, if the diffusion transform network is a flow matching model, the generated field information is velocity field information. The flow matching model realizes the generation process by learning the velocity field in the transmission mapping from the noise distribution to the data distribution. The physical meaning of the velocity field information predicted during the denoising process is the instantaneous velocity driving the sample to move from the noise distribution to the data distribution along the optimal transmission path in the probabilistic flow ordinary differential equation. The training objective is to make the predicted velocity field as close as possible to the real velocity field. In other embodiments, if the diffusion transform network is a denoising diffusion probabilistic model (DDPM), the generated field information is noise field information. DDPM realizes denoising generation by learning to predict the noise added during the denoising process. The physical meaning of the noise field information predicted during the denoising process is the Gaussian noise added to the target image features during the forward denoising process. The training objective is to make the predicted noise as close as possible to the actual added noise. The adaptation of the two architectures makes the region decoupling optimization mechanism and frequency domain loss constraint of this application universal.

[0119] This application clarifies the correspondence and adaptation logic of generated field information in different types of continuous-time generative networks. Relying on the inherent unity of the two generation paradigms, the proposed region decoupling optimization and high-frequency texture guidance mechanism can be adapted to both the denoising diffusion probability model and the flow matching model, two mainstream generation paradigms. There is no need to redesign the core constraint mechanism for different paradigms, which significantly improves the versatility and portability of the method. It can flexibly select the basic network architecture according to different task requirements and deployment scenarios, while not changing the constraint effect of the core mechanism. It ensures the balance between the accuracy of the edited region and the fidelity of the non-edited region under different architectures, and reduces the development and modification costs of adapting the method to different generation paradigms.

[0120] The foregoing embodiments have provided a detailed description of the training method for the image editing model. The following describes the method for image editing using the trained first image editing model. Figure 4 This is a flowchart of an image editing process provided in an embodiment of this application. The method can be performed by... Figure 1 The image editing device in the system shown is executed. For example... Figure 4 As shown, the image editing method may include the following steps: Step 410: Obtain the image to be edited and the editing instructions; Step 420: Using the first image editing model, the image to be edited is edited based on the image to be edited and the editing instructions to obtain the edited image; The first image editing model is trained based on any of the aforementioned training methods.

[0121] In one application scenario of the first image editing model, a user wants to edit an image using natural language commands. The edited image needs to accurately reflect the editing intent in the edited area while maintaining the same background fidelity, color consistency, and texture details as the original image in the preserved area. Since the first image editing model obtained in the aforementioned embodiment has internalized the prior capabilities of region decoupling optimization and high-frequency texture preservation during the training phase, it can spontaneously modify the content of the edited area while losslessly protecting the background of the non-edited area and rendering realistic high-frequency textures during the inference phase, thereby achieving a high-fidelity and high-harmony image editing effect.

[0122] The image to be edited and the editing instructions are the inputs provided by the user in actual application. The user only needs to input an image to be edited and a natural language editing instruction, without providing any mask or other auxiliary spatial prior information. This maskless input method reduces the user's usage threshold and interaction cost, and users do not need to have professional image annotation skills to complete image editing operations.

[0123] refer to Figure 5 , Figure 5 This diagram illustrates the inference framework of the image editing model provided in this application embodiment. The first image editing model in this application embodiment mainly includes an encoder (including a VAE encoder and a text encoder), a diffusion transform network, and a decoder. During inference, the acquired source image to be edited is input into the VAE encoder, which compresses it into the latent space to obtain the latent code representation of the source image; simultaneously, the text-based editing instructions are input into the text encoder to obtain the text embedding features corresponding to the editing instructions. The latent code representation of the source image and the text embedding features are input into the diffusion transform network as a second condition, while Gaussian noise is introduced to generate noise features. The diffusion transform network performs denoising processing on the noise features based on the second condition, and outputs the predicted edited image representation. Finally, the edited image representation output by the diffusion transform network is input into the VAE decoder, which maps it from the latent space back to the pixel space, ultimately obtaining the complete edited image.

[0124] This application embodiment uses the first image editing model trained by the aforementioned embodiment to perform inference, which can ensure that the editing area is accurately modified according to the user's instructions, and can preserve the original structure, texture and style of the non-editing area to the greatest extent. It avoids the problems of non-editing area degradation, blurred edges of edited content and inconsistent textures that are common in traditional maskless image editing, and achieves high-fidelity and high-harmony scene text editing.

[0125] In some embodiments of this application, the edited image includes multiple candidate edited images, and the image editing method further includes: evaluating the harmony score of each candidate edited image using a harmony evaluation model based on the multiple candidate edited images; and selecting the final edited image based on the harmony score of each candidate edited image.

[0126] The edited image can include multiple candidate edited images. For example, the first image editing model can generate multiple candidate results in a single editing process using different sampling paths or random Gaussian noise, providing a selection space for subsequent quality screening, so that the system can select the optimal edited result from them, rather than relying solely on the output generated in a single instance.

[0127] For scenarios with multiple candidate edited images, a harmony evaluation model can be used to further evaluate the harmony score of each candidate edited image, and the final edited image can be selected based on the harmony scores of each candidate edited image. The harmony evaluation model can be, for example, a multimodal large language model trained based on human preference alignment. It takes the image to be edited, the candidate edited images, and the editing instructions as input, and outputs a continuous harmony score (such as a Gaussian harmony score). The harmony evaluation model comprehensively assesses candidate results from the perspective of visual coherence between text and background (including dimensions such as perspective accuracy, lighting coherence, and texture fit), providing a multi-dimensional evaluation basis that goes beyond traditional single indicators (such as OCR text correctness alone) for selection. Finally, the edited images of multiple candidates can be sorted according to the harmony score, and the candidate with the highest harmony score is selected as the final edited image output to the user.

[0128] By sorting and filtering by harmony score, the system can automatically recall the editing results with the most accurate perspective, the most consistent lighting, and the best texture fit, reducing the burden on users to manually select from multiple candidate results and improving the user experience.

[0129] In some embodiments of this application, the harmony evaluation model is trained through the following process: (1) Training dataset construction: Source images are collected, and multiple image editing models are used to perform editing on each source image to generate multiple edited samples. Multiple image editing models may include, for example, Qwen-Image-Edit, FireRed-Image-Edit, and the second image editing model refined in the aforementioned embodiments of this application. The annotator uses the source image as a reference and manually scores each edited sample from dimensions such as surface texture fit, lighting coherence, and overall naturalness. Invalid samples that fail to be generated are removed during the annotation process. Then, the editing region is located, and the source image block and the corresponding editing image block are cropped. Unqualified samples with inaccurate positioning are filtered out. Finally, the editing regions generated by different models under the same source image are paired to obtain region-level comparison sample pairs for pairwise preference learning.

[0130] (2) Harmony Assessment Model (HAM) Structure and Training: To adapt to the subjective noise of human scoring, an uncertainty-perceived preference learning training model is adopted. The model takes the source region block, the edit region block, and the edit command as inputs. For example, Qwen3-VL is used as the multimodal backbone to extract cross-modal visual-semantic features. The head output is predicted by MLP (Multilayer Perceptron) to obtain a harmony score that follows a Gaussian distribution. ,in, μ The desired harmony score, σ 2 Characterize the uncertainty of the prediction results. Pairs of samples are constructed during training ( , ),in, For samples with higher quality manual annotation. For samples of lower quality, the preference probability is calculated based on the predicted score distribution of the two samples, and the negative log-likelihood is used as the optimization objective: The training samples are all from the outputs of the above-mentioned multiple image editing models, and do not include the output of the first image editing model trained in this application, so as to avoid the harmony evaluation model from overfitting to a specific output distribution and to ensure the universality of the evaluation.

[0131] This application solves the problem that existing automatic evaluation metrics cannot accurately measure the coherence between edited text and background and the fidelity of local texture by constructing a training dataset that fits human visual preferences and a harmony evaluation model trained with uncertainty perception. It can automatically select editing results that meet human visual preferences, blend naturally with the background, and have high texture fidelity, without the need for manual screening of candidate results one by one, thus improving the processing efficiency of batch image editing scenarios.

[0132] The editing method provided in this application can be applied to image editing in various scenarios. One typical scenario is STE (Scene Text Editing), where the object of editing is the text in an image, i.e., modifying the text content within a natural image. For example, if the image to be edited is a photo of a sign that reads "Today's Special Coffee," the user can input a natural language editing command to change "Today's Special Coffee" to "Today's Special Latte." The image editing model trained using this application can output a target image where the text on the sign has been modified to "Today's Special Latte." The new characters are clear and sharp, blending naturally with the sign's texture, eliminating any blurry "sticker-like" appearance. Simultaneously, the sign's rustic wood grain and the background's depth are perfectly preserved. The entire editing process requires no user masking, making it much simpler.

[0133] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0134] This application embodiment also provides a training device 600 for an image editing model, such as... Figure 6 The diagram shown is a structural schematic of a training device for an image editing model provided in this application embodiment. The training device mainly includes: a first acquisition unit 610, an encoding unit 620, a generation unit 630, a noise-adding unit 640, a noise-reducing unit 650, and an optimization unit 660; the main functions of each component are as follows: The first acquisition unit 610 is configured to acquire training data including multiple training samples, wherein the training samples are triplets consisting of source image samples, editing instruction samples and target image samples.

[0135] The encoding unit 620 is configured to encode the training samples to obtain source image features, target image features, and editing instruction features.

[0136] The generation unit 630 is configured to generate a region indication map based on the difference between the source image features and the target image features, the region indication map including information indicating the editing area and information indicating the retention area.

[0137] The noise-adding unit 640 is configured to add noise to the target image features to obtain the noise-adding features of the target image.

[0138] The denoising unit 650 is configured to use the source image features and the editing instruction features as first conditions, and to perform denoising processing on the noise features of the target image based on the first conditions using a diffusion transform network, and to collect the predicted generation field information during the denoising process.

[0139] The optimization unit 660 is configured to iteratively optimize the parameters of the diffusion transform network using a loss function corresponding to a first training objective. The first training objective includes minimizing the difference between the predicted generated field information and the generated field information corresponding to the target image features in the edit region, and minimizing the difference between the predicted generated field information and the generated field information corresponding to the source image features in the retained region.

[0140] The diffusion transform network is used to construct the first image editing model after training.

[0141] In some embodiments of this application, the first acquisition unit 610 may be specifically configured to: acquire source image samples and editing instruction samples; use a second image editing model to edit the source image samples to obtain candidate target images; perform quality evaluation on the candidate target images, and filter out candidate target images whose quality evaluation results meet preset quality requirements as target image samples; and use the triple formed by the source image samples, the editing instruction samples, and the target image samples as a training sample.

[0142] In some embodiments of this application, the first acquisition unit 610 is further configured to: generate editing instruction samples for the source image samples using a visual language model.

[0143] In some embodiments of this application, the training unit (not shown in the figure) is configured to train the second image editing model in the following manner: using the basic image editing model to edit the source image sample based on the editing instruction sample to obtain an initial target image; obtaining the reverse editing instruction of the editing instruction sample; using the initial target image and the reverse editing instruction as input to the basic image editing model, and using the source image sample as the target output of the basic image editing model, iteratively optimizing the parameters of the basic image editing model; and using the basic image editing model after iterative optimization as the second image editing model.

[0144] In some embodiments of this application, when the first acquisition unit 610 performs quality assessment on the candidate target image, it may be configured to perform at least one of the following: identify the target object in the candidate target image, perform a consistency comparison based on the editing instruction sample and the result of the target object identification, and determine the quality assessment result based on the consistency comparison result; determine at least one of the peak signal-to-noise ratio and structural similarity of the reserved region in the candidate target image, and determine the quality assessment result based on the peak signal-to-noise ratio and structural similarity; determine the color distribution divergence between the candidate target image and the corresponding source image, and determine the quality assessment result based on the color distribution divergence.

[0145] In some embodiments of this application, the generation unit 630 is further configured to: determine the pixel-level distance between the source image features and the target image features in the channel dimension to obtain a difference map; and use an exponential weighting algorithm to map the difference map into edit region weights and retain region weights, wherein the edit region weights and the retain region weights constitute the region indicator map.

[0146] In some embodiments of this application, the loss function corresponding to the first training target includes: a first loss term and a second loss term; the first loss term is obtained by weighting the difference between the predicted generated field information and the generated field information corresponding to the target image features using the edit region weights; the second loss term is obtained by weighting the difference between the predicted generated field information and the generated field information corresponding to the source image features using the preserved region weights.

[0147] In some embodiments of this application, the training device 600 further includes a frequency decomposition unit (not shown in the figure), configured to perform wavelet transforms at at least one scale level on the predicted generated field information, the generated field information corresponding to the target image features, and the generated field information corresponding to the source image features in at least one direction, respectively, to obtain high-frequency subbands of the predicted generated field information, the generated field information corresponding to the target image features, and the generated field information corresponding to the source image features in at least one direction and at at least one level. Accordingly, the loss function corresponding to the first training objective includes a third loss term and a fourth loss term; the third loss term is obtained by using the edit region weights to weight the difference between the predicted generated field information and the generated field information corresponding to the target image features in the same direction and at the same level, and then summing the results of the weighted processing in the at least one direction and at the at least one level; the fourth loss term is obtained by using the preserved region weights to weight the difference between the predicted generated field information and the generated field information corresponding to the source image features in the same direction and at the same level, and then summing the results of the weighted processing in the at least one direction and at the at least one level.

[0148] In some embodiments of this application, if the diffusion transformation network is a flow matching model, then the generated field information is velocity field information; if the diffusion transformation network is a denoising diffusion probability model, then the generated field information is noise field information.

[0149] This application embodiment also provides an image editing device 700, such as... Figure 7 The diagram shown is a structural schematic of an image editing device provided in an embodiment of this application. The image editing device 700 mainly includes a second acquisition unit 710 and an editing unit 720; the main functions of each component are as follows: The second acquisition unit 710 is configured to acquire the image to be edited and the editing instructions.

[0150] The editing unit 720 is configured to use a first image editing model to edit the image to be edited based on the image to be edited and the editing instructions to obtain an edited image.

[0151] The first image editing model is trained based on any of the aforementioned training methods.

[0152] In some embodiments of this application, the edited image includes a plurality of candidate edited images, and the image editing apparatus 700 further includes: an evaluation unit (not shown in the figure), configured to evaluate the harmony score of each candidate edited image using a harmony evaluation model based on the plurality of candidate edited images; and a filtering unit (not shown in the figure), configured to filter out the final edited image based on the harmony score of each candidate edited image.

[0153] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the description of the method embodiments. The system and device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0155] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0156] And an electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0157] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0158] in, Figure 8 An exemplary architecture of an electronic device is shown, which may include a processor 810, a video display adapter 811, a disk drive 812, an input / output interface 813, a network interface 814, and a memory 820. The processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, and memory 820 can communicate with each other via a communication bus 830.

[0159] The processor 810 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0160] The memory 820 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 820 can store the operating system 821 for controlling the operation of the electronic device 800, and the basic input / output system (BIOS) 822 for controlling the low-level operations of the electronic device 800. Additionally, it can store a web browser 823, a data storage management system 824, and an image editing model training device 600 / image editing device 700, etc. The image editing model training device 600 can be an application program that specifically implements the various steps of the aforementioned training method in this embodiment, and the image editing device 700 can be an application program that specifically implements the various steps of the aforementioned image editing method in this embodiment. In summary, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 820 and executed by the processor 810.

[0161] The input / output interface 813 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0162] Network interface 814 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0163] Bus 830 includes a pathway for transmitting information between various components of the device, such as processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, and memory 820.

[0164] It should be noted that although the above-described device only shows the processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, memory 820, bus 830, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0165] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0166] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A training method for an image editing model, characterized in that, The training method includes: Acquire training data comprising multiple training samples, wherein the training samples are triplets consisting of source image samples, editing instruction samples, and target image samples; The training samples are encoded to obtain source image features, target image features, and editing instruction features; Based on the differences between the source image features and the target image features, a region indication map is generated, which includes information indicating the edited region and information indicating the retained region. The target image features are subjected to noise processing to obtain the noisy features of the target image; Using the source image features and the editing instruction features as the first condition, a diffusion transform network is used to denoise the noise features of the target image based on the first condition, and the predicted generation field information is collected during the denoising process. Using the loss function corresponding to the first training objective, the parameters of the diffusion transform network are iteratively optimized. The first training objective includes: minimizing the difference between the predicted generated field information and the generated field information corresponding to the target image features in the edit region, and minimizing the difference between the predicted generated field information and the generated field information corresponding to the source image features in the preserved region. The diffusion transform network is used to construct the first image editing model after training.

2. The training method according to claim 1, characterized in that, The training samples were obtained in the following manner: Obtain source image samples and editing instruction samples; The source image samples are edited using a second image editing model to obtain candidate target images; The candidate target images are subjected to quality assessment, and candidate target images whose quality assessment results meet the preset quality requirements are selected as target image samples; The triple consisting of the source image sample, the editing instruction sample, and the target image sample is used as a training sample.

3. The training method according to claim 2, characterized in that, Obtaining the edit instruction sample includes: Editing instruction samples are generated for the source image samples using a visual language model.

4. The training method according to claim 2, characterized in that, The second image editing model was trained in the following manner: The source image sample is edited using the basic image editing model based on the editing instruction sample to obtain the initial target image; Obtain the reverse editing instructions from the sample editing instructions; The initial target image and the reverse editing instructions are used as inputs to the basic image editing model, and the source image samples are used as the target outputs of the basic image editing model. The parameters of the basic image editing model are iteratively optimized. The base image editing model after iterative optimization is used as the second image editing model.

5. The training method according to claim 2, characterized in that, The quality assessment of the candidate target image includes at least one of the following: The candidate target image is subjected to target object recognition, and a consistency comparison is performed between the editing instruction sample and the target object recognition result. The quality assessment result is determined based on the consistency comparison result. Determine at least one of the peak signal-to-noise ratio (PSNR) and structural similarity between the candidate target image and the source image sample in the reserved region, and determine the quality assessment result based on at least one of the PSNR and structural similarity; The color distribution divergence between the candidate target image and the source image sample is determined, and the quality assessment result is determined based on the color distribution divergence.

6. The training method according to claim 1, characterized in that, The step of generating a region indication map based on the difference between the source image features and the target image features includes: Determine the pixel-level distance between the source image features and the target image features in the channel dimension to obtain a difference map; The difference map is mapped to edit region weights and retain region weights using an exponential weighting algorithm, and the edit region weights and retain region weights constitute the region indicator map.

7. The training method according to claim 6, characterized in that, The loss function corresponding to the first training objective includes: a first loss term and a second loss term; The first loss term is obtained by weighting the difference between the predicted generated field information and the generated field information corresponding to the target image features using the edit region weights; The second loss term is obtained by weighting the difference between the predicted generated field information and the generated field information corresponding to the source image features using the preserved region weights.

8. The training method according to claim 6, characterized in that, The method further includes: performing wavelet transforms at at least one scale level on the predicted generation field information, the generation field information corresponding to the target image features, and the generation field information corresponding to the source image features in at least one direction, respectively, to obtain high-frequency subbands of the predicted generation field information, the generation field information corresponding to the target image features, and the generation field information corresponding to the source image features in at least one direction and at at least one level. The loss function corresponding to the first training objective includes: a third loss term and a fourth loss term; The third loss term is obtained by using the edit region weights to weight the differences between the predicted generated field information and the generated field information corresponding to the target image features in the same direction and at the same level, and then summing the results of the weighting in the at least one direction and at the at least one level. The fourth loss term is obtained by using the preserved region weights to weight the differences between the predicted generated field information and the generated field information corresponding to the source image features in the same direction and at the same level in the high-frequency sub-bands, and then summing the results of the weighting in the at least one direction and at the at least one level.

9. The training method according to any one of claims 1-8, characterized in that, If the diffusion transformation network is a flow matching model, then the generated field information is velocity field information; If the diffusion transformation network is a denoising diffusion probability model, then the generated field information is noise field information.

10. An image editing method, characterized in that, The image editing method includes: Obtain the image to be edited and the editing instructions; The first image editing model is used to edit the image to be edited based on the image to be edited and the editing instructions to obtain the edited image; The first image editing model is trained based on the training method described in any one of claims 1-9.

11. The image editing method according to claim 10, characterized in that, The edited image includes multiple candidate edited images, and the image editing method further includes: Based on the multiple candidate edited images, the harmony score of each candidate edited image is evaluated using a harmony evaluation model. The final edited image is selected based on the harmony score of each candidate edited image.

12. A training device for an image editing model, characterized in that, The training device for the image editing model includes: The first acquisition unit is used to acquire training data including multiple training samples, wherein the training samples are triplets consisting of source image samples, editing instruction samples and target image samples; The encoding unit is used to encode the training samples to obtain source image features, target image features, and editing instruction features; A generation unit is configured to generate a region indication map based on the difference between the source image features and the target image features, the region indication map including information indicating the edited region and information indicating the retained region; A noise-adding unit is used to add noise to the target image features to obtain the noise-adding features of the target image; The denoising unit is used to take the source image features and the editing instruction features as first conditions, and use a diffusion transform network to denoise the noise features of the target image based on the first conditions, and collect the predicted generation field information during the denoising process; An optimization unit is used to iteratively optimize the parameters of the diffusion transform network using the loss function corresponding to the first training objective. The first training objective includes minimizing the difference between the predicted generated field information and the generated field information corresponding to the target image features in the edit region, and minimizing the difference between the predicted generated field information and the generated field information corresponding to the source image features in the retained region. The diffusion transform network is used to construct the first image editing model after training.

13. An image editing device, characterized in that, The image editing device includes: The second acquisition unit is used to acquire the image to be edited and the editing instructions; An editing unit is used to edit the image to be edited based on the image to be edited and the editing instructions using a first image editing model, to obtain an edited image; The first image editing model is trained based on the training method described in any one of claims 1-9.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the training method for the image editing model according to any one of claims 1-9, or the steps of the image editing method according to any one of claims 10-11.

15. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the training method of the image editing model according to any one of claims 1-9, or perform the steps of the image editing method according to any one of claims 10-11.

16. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the training method for the image editing model according to any one of claims 1-9, or the steps of the image editing method according to any one of claims 10-11.

Images