An image editing quality evaluation method fusing perception prior and visual language model
Patent Information
- Application Number
- CN202610897806.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-22
- Publication Date
- 2026-10-30
AI Technical Summary
[0007]本发明的目的是解决现有图像编辑质量评估方法难以同时兼顾细粒度视觉变化感知和高层语义理解的技术问题，提出一种融合感知先验与视觉语言模型的图像编辑质量评估方法
[0049]1、本发明通过将图像编辑感知编码器提取的视觉变化特征作为感知先验注入视觉语言模型，解决了传统视觉语言模型仅依赖原始图像令牌时对细粒度视觉变化不敏感的问题，能够提升对颜色、纹理、小目标添加等细微编辑结果的识别能力。
Smart Images
Figure CN123015731A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an image editing quality assessment method that integrates perceptual priors and visual language models. It belongs to the fields of image editing quality assessment, multimodal learning, computer vision, and visual language model technology, and is applied to automated quality assessment scenarios for instruction-guided image editing results. Background Technology
[0002] With the development of diffusion models, image-to-image generation models, and multimodal large-scale models, users can perform various editing operations on images using natural language commands, such as adding, removing, replacing, modifying backgrounds, and adjusting colors or attributes. Correspondingly, automatically determining whether the edited image correctly executes the text commands, retains unedited areas, and maintains visual plausibility has become a key issue in image editing model training, model selection, and result filtering.
[0003] Existing image editing quality assessment methods mainly include those based on pixel or depth feature distance, those based on visual-language alignment models, and those based on visual-language models. Pixel-level metrics or general perceptual similarity metrics are typically geared towards image reconstruction tasks and struggle to accurately reflect the differences between "parts to be changed" and "parts to be kept" in editing instructions. Metrics based on visual-language alignment models usually map images, text, or image changes to a unified embedding space and obtain quality scores through cosine similarity or dot product; these methods can capture some semantic relevance but often lack contextual understanding of the editing task itself, making it difficult to distinguish between correct large structural changes and erroneous image destruction.
[0004] On the other hand, while visual language models possess strong semantic understanding and reasoning capabilities, enabling them to make comprehensive judgments based on the source image, the edited image, and editing instructions, their visual input is typically segmented, downsampled, or discretized, resulting in insufficient sensitivity to fine-grained visual changes. For example, in scenarios involving slight color changes, local texture changes, or the addition of small objects, visual language models may fail to consistently perceive whether the actual change has occurred, leading to overly high or low evaluation results.
[0005] Therefore, existing methods have the following shortcomings: First, embedding similarity-based metrics lack the ability to make semantic judgments on different editing types, and are prone to systematic biases in tasks with large changes such as object removal and background replacement; Second, visual language models have perceptual bottlenecks in recognizing fine-grained changes, and are prone to missing local or subtle edits; Third, existing methods usually compress visual change signals into single scores or simple splicing features, failing to continuously utilize structured perceptual information in the multi-layered reasoning process of visual language models; Fourth, there is a lack of mechanisms for adaptively calibrating the weights of perceptual information and semantic reasoning information for different editing types.
[0006] Therefore, how to effectively integrate image editing perception features with fine-grained change perception capabilities with the semantic reasoning capabilities of visual language models to improve the accuracy and robustness of instruction-guided image editing quality assessment has become an urgent problem to be solved. Summary of the Invention
[0007] The purpose of this invention is to address the technical problem that existing image editing quality assessment methods cannot simultaneously take into account fine-grained visual change perception and high-level semantic understanding, and to propose an image editing quality assessment method that integrates perceptual priors and visual language models.
[0008] The objective of this invention is achieved through the following technical solution:
[0009] In a first aspect, the present invention discloses an image editing quality assessment method that integrates perceptual priors and visual language models, comprising the following steps:
[0010] Step 1: Obtain the image editing sample to be evaluated, and perform preprocessing operations on the source image Is, the edited image Ie, and the text editing instructions S in the sample respectively; the image editing sample includes the source image, the edited image, and the text editing instructions corresponding to the editing operation;
[0011] The preprocessing operation specifically includes:
[0012] The source and edited images are normalized in size, pixel value, and channel format; these will be used as input to the visual encoder and visual language model; text editing instructions are segmented, special tokens are added, and length is truncated or padded to form a text token sequence; based on the editing task type, image editing samples are divided into one or more categories such as object addition, object removal, object replacement, background modification, and attribute editing for type-aware model training or evaluation calibration;
[0013] Step 2: Combine the preprocessed source image and the edited image into an image pair, concatenate them along the channel dimension, and input them into a pre-trained EditCLIP image editing perceptual encoder with the overall parameters frozen for joint encoding. Extract feature vectors through the encoder's final hidden layer or a specified intermediate layer as perceptual prior features f; the perceptual prior features are used to characterize the local or global visual changes caused by the image editing operation.
[0014] The prior perceptual feature f is represented as:
[0015] f = F(concat(Is, Ie));
[0016] Where concat (·) represents concatenation along the channel dimension; F(·) represents the image editing perceptual encoder;
[0017] Step 3: Using a trainable perceptual prior projection module, the perceptual prior features are mapped to the hidden dimension space of the visual language model and reshaped into continuous soft prefix tokens P consistent with the word embedding dimension of the visual language model; the soft prefix tokens are trained to carry the structured change information between the source image and the edited image, which is used for the attention layer reading of the visual language model.
[0018] The perception prior projection module is composed of a first linear layer, a GELU activation layer, and a second linear layer connected in series.
[0019] The soft prefix token P is represented as: P = reshape(G(f)), P ∈ R^(K×D);
[0020] Where K represents the number of soft prefix tokens, D represents the hidden dimension of the visual language model, G represents the prior projection module, and R represents the real number space.
[0021] Step 4: Input the preprocessed source image and the edited image into the parameter-preserving frozen visual encoder of the visual language model to obtain the image token sequence; input the preprocessed text editing instructions into the parameter-preserving frozen text embedding module of the visual language model, and obtain the text token sequence through the text segmenter and text embedding layer; fuse and splice them in the order of soft prefix token, image token sequence, and text token sequence to form the enhanced input sequence E;
[0022] The enhanced input sequence is represented as: E = [P; V; T];
[0023] Among them, the soft prefix token is P, the image token sequence is V, and the text token sequence is T;
[0024] Step 5: Input the enhanced input sequence into the Transformer attention layer of the visual language model with the subject parameters frozen, and then input the query projection matrix W into the attention layer. up The projection matrix W down A low-rank adaptation module is introduced for parameter fine-tuning. In multi-layer self-attention computation, soft prefix tokens are used as keys and values to participate in reasoning, enabling the visual language model to adaptively fuse perceptual prior information.
[0025] The parameter fine-tuning specifically includes:
[0026] Through a low-rank adaptation module comprising a trainable query projection matrix and a value projection matrix, the original projection matrix is incrementally updated via low-rank decomposition; the query projection matrix is dynamically adjusted for soft prefix token attention strength within the low-rank adaptation module; and perceptual prior information is fused into the multimodal hidden representation for the value projection matrix; specifically, the following conditions are met:
[0027] W' = W + (α / r)·W up ·W down ;
[0028] Where r is the rank of the low-rank matrix, α is the scaling factor; W is the original pre-trained query projection matrix or value projection matrix in the attention layer of the visual language model, and W' is the equivalent projection matrix after introducing the low-rank adaptation module.
[0029] The visual language model adopts the Qwen3-VL multimodal large model;
[0030] Step 6: Extract the hidden state of the last token in the output sequence of the visual language model, the hidden state of the special classification token, or the global hidden representation obtained by pooling as the quality prediction representation h, and output the image editing quality score q with the value within the preset range through the trainable regression prediction module MLP.
[0031] The image editing quality score ranges from [0,1] and is used to characterize the degree of completion and visual quality of the edited image relative to the source image and text editing instructions.
[0032] When the quality score is close to 1, the consistency of the instructions is higher and the visual effect is more natural; when the quality score is close to 0, the editing fails or there are obvious artifacts and deviations from the instructions.
[0033] The regression prediction module is a multilayer perceptron structure, consisting of a first linear layer, a GELU activation layer, a second linear layer, and a Sigmoid normalization layer connected in series.
[0034] The quality score q is expressed as: q = σ( MLP ( h ) );
[0035] Where σ is the Sigmoid function or normalization function;
[0036] Step 7: Construct training samples containing source images, edited images, text editing instructions, and corresponding quality labels; group the training samples based on editing type and introduce type embedding prompts; use mean squared error loss, binary cross-entropy loss, ranking loss, or a weighted combination thereof as loss functions to train the quality assessment model end-to-end, enabling the quality assessment model to learn the weight allocation of perceived prior information under editing type; use the trained quality assessment model for inference and output interpretive evaluation results including dimensions of modification correctness, instruction consistency, and visual rationality;
[0037] The quality label includes at least one of binary quality label, continuous quality score or multidimensional evaluation label; the multidimensional evaluation label includes modification correctness, instruction consistency and visual rationality, wherein modification correctness is used to evaluate whether the target visual change is completed, instruction consistency is used to evaluate whether the editing result conforms to the text editing instruction, and visual rationality is used to evaluate whether the edited image is natural and coherent and without obvious artifacts.
[0038] Secondly, the present invention discloses an image editing quality assessment system that integrates perceptual priors and visual language models to implement the above method; it includes a sample acquisition module, a perceptual prior extraction module, a prior projection module, a multimodal fusion module, a visual language inference module, a scoring output module, and a training and calibration module.
[0039] The sample acquisition module is used to acquire the source image, the edited image, and the text editing instructions in the image editing sample to be evaluated, and to preprocess the source image, the edited image, and the text editing instructions respectively to obtain the preprocessed source image, the preprocessed edited image, and the text token sequence; the preprocessed source image and the preprocessed edited image are used as inputs to the perceptual prior extraction module and the visual encoding part of the visual language model, and the text token sequence is used as inputs to the multimodal fusion module.
[0040] The perceptual prior extraction module is used to receive the preprocessed source image and the preprocessed edited image output by the sample acquisition module, and to use the image editing perceptual encoder to jointly encode the source image and the edited image to extract perceptual prior features that characterize the visual changes before and after editing; the perceptual prior features are used as the input of the prior projection module.
[0041] The prior projection module is used to receive the perceptual prior features output by the perceptual prior extraction module and map the perceptual prior features to one or more consecutive soft prefix tokens under the hidden dimension of the visual language model; the soft prefix tokens serve as the input of the multimodal fusion module.
[0042] The multimodal fusion module is used to receive the soft prefix tokens output by the prior projection module, the image token sequence output by the visual encoding part of the visual language model, and the text token sequence output by the sample acquisition module, and concatenate the soft prefix tokens, image token sequence, and text token sequence in a preset order to form an enhanced input sequence; the enhanced input sequence serves as the input to the visual language inference module.
[0043] The visual language reasoning module is used to receive the enhanced input sequence output by the multimodal fusion module and perform multimodal reasoning based on the multi-layer self-attention structure of the visual language model. During the reasoning process, the perceptual prior information carried by the soft prefix token is adaptively read and fused through the parameter efficient adaptation module to obtain the multimodal hidden representation. The multimodal hidden representation is used as the input of the scoring output module.
[0044] The scoring output module is used to receive the multimodal hidden representation output by the visual language reasoning module and generate an image editing quality score through the regression prediction module. The image editing quality score is used to characterize the degree of completion of the editing of the edited image relative to the source image and the text editing instructions, the consistency of the instructions and the visual rationality.
[0045] The training and calibration module is used to receive labeled image editing training samples and train the prior projection module, parameter efficient adaptation module, and scoring output module based on the labeled image editing training samples. The training and calibration module is also used to perform quality assessment and calibration of the model according to different editing types, so that the model can adaptively utilize perceptual prior information under different editing types such as object addition, object removal, object replacement, background modification, and attribute editing.
[0046] Thirdly, the present invention discloses an electronic device including a processor, a memory, and a computer program stored in the memory, wherein the computer program, when executed by the processor, implements the method described in any of the above-mentioned embodiments.
[0047] Fourthly, the present invention discloses a computer storage medium storing a computer program, which, when run in a processor, implements the method described in any of the above-mentioned embodiments.
[0048] Compared with existing technologies, it has the following advantages:
[0049] 1. This invention injects the visual change features extracted by the image editing perceptual encoder as a perceptual prior into the visual language model, which solves the problem that traditional visual language models are not sensitive to fine-grained visual changes when they only rely on the original image tokens. This invention can improve the recognition ability of subtle editing results such as color, texture, and small object addition.
[0050] 2. This invention converts perceptual priors into soft prefix tokens and inserts them into multimodal input sequences, enabling the visual language model to continuously utilize structured visual change information during multi-layer attention reasoning, thus avoiding information loss caused by compressing perceptual information into a single score.
[0051] 3. This invention solves the problem that evaluation metrics based on embedding similarity are difficult to distinguish between correct large-scale structural changes and erroneous image destruction by combining the semantic reasoning ability of visual language models with the visual change modeling ability of perceptual priors, thereby improving the evaluation accuracy of complex editing types such as object removal, object replacement and background modification.
[0052] 4. This invention introduces a low-rank adaptation module into the visual language model to achieve adaptive utilization of perceptual priors under different editing types. It can dynamically adjust the evaluation criteria according to editing categories such as object addition, object removal, background modification, and attribute editing, thereby improving the robustness of cross-type evaluation.
[0053] 5. This invention adopts a parameter-efficient training method, which only requires training the perceptual prior projection module, the low-rank adaptation module, and the scoring output module, reducing the model training cost and deployment cost. It is suitable for large-scale image editing result screening, model evaluation, and automated data annotation scenarios. Attached Figure Description
[0054] Figure 1 is a schematic diagram of the process of the present invention; Detailed Implementation
[0055] To better illustrate the purpose and advantages of this invention, the invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the implementation of this invention is not limited to the following embodiments, and any modifications or alterations made to this invention will fall within the scope of protection of this invention.
[0056] Example
[0057] This embodiment presents an image editing quality assessment method that integrates perceptual priors and a visual language model. The specific implementation steps are as follows:
[0058] Step 1: Obtain the image editing samples to be evaluated.
[0059] In this embodiment, the sample to be evaluated includes a source image Is, an edited image Ie, and a text editing instruction S. The source image Is represents the original image before editing, and the edited image Ie represents the resulting image generated by the image editing model according to the text editing instruction S. The text editing instruction S is used to describe the desired editing operation, such as "change the motorcycle to pink", "remove the traffic sign", "add a dog to the side of the road", or "change the background to a beach".
[0060] In this embodiment, the source image Is and the edited image Ie are scaled to a preset resolution and normalized. The text editing instructions S are input into the word segmenter of the visual language model to obtain a text token sequence. During the training phase, the samples can be further classified into types such as object addition, object removal, object replacement, background modification, and attribute editing based on text instructions or manual annotation.
[0061] Step 2: Extract perceptual prior features.
[0062] In this embodiment, a pre-trained image editing perceptual encoder F is used to jointly encode the source image Is and the edited image Ie. Specifically, Is and Ie are concatenated along the channel dimension to form a pair of image inputs concat(Is,Ie), which is then input into the image editing perceptual encoder F to obtain the perceptual prior features f:
[0063] f = F(concat(Is,Ie));
[0064] In this context, the perceptual prior feature f represents the visual changes in the region, magnitude, and direction of the changes before and after image editing. Unlike directly outputting a quality score, the perceptual prior feature f does not independently determine whether the editing is correct; instead, it serves as conditional information for subsequent semantic judgments by the visual language model.
[0065] In other embodiments, the image editing perceptual encoder may also use a dual-branch shared weight structure, a cross-attention structure, or a difference perception structure to jointly model Is and Ie. As long as features that characterize the visual changes before and after editing can be obtained, they can be used as perceptual prior features in this invention.
[0066] Step 3: Map the perceptual prior features to soft prefix tokens.
[0067] In this embodiment, a trainable prior projection module G is constructed, which includes a first linear layer, a GELU activation function, and a second linear layer. The perceptual prior features f are input into the prior projection module G to obtain K soft prefix tokens P:
[0068] P = reshape(G(f)), P ∈ R^(K×D).
[0069] Where K represents the number of soft prefix tokens, and D represents the hidden dimension of the visual language model. The soft prefix token P is a continuous vector that does not correspond to discrete lexical units in the natural language vocabulary. It is used to introduce structured visual variation information into the input space of the visual language model.
[0070] Step 4: Construct the enhanced input sequence.
[0071] In this embodiment, the source image Is and the edited image Ie are input into the visual encoder of the visual language model to obtain the image token sequence V; the text editing instructions S are input into the text embedding layer of the visual language model to obtain the text token sequence T. Then, the soft prefix token P, the image token sequence V, and the text token sequence T are concatenated to obtain the enhanced input sequence E.
[0072] E = [P; V; T].
[0073] The enhanced input sequence E is fed into the Transformer layer of the visual language model. In the self-attention computation, the key and value corresponding to the soft prefix token P participate in the attention computation of subsequent tokens, enabling the visual language model to explicitly access visual change information before and after editing when processing image and text semantics.
[0074] Step 5: Achieve adaptive reading of perceptual priors through efficient parameter adaptation.
[0075] In this embodiment, a low-rank adaptation module is introduced into the attention layer of the visual language model. For the query projection matrix W... up Sum projection matrix W down Update it to:
[0076] W' = W + (α / r)·W up ·W down .
[0077] Where r is the rank of the low-rank matrix, α is the scaling factor, and W down and W up It is a trainable low-rank matrix. By adapting the query projection, the model can learn whether more attention needs to be paid to the soft prefix token under different editing types; by adapting the value projection, the model can learn how to fuse visual change information from perceptual priors into the multimodal hidden representation.
[0078] In this embodiment, the main parameters of the visual language model can be frozen, and only the low-rank adaptation module, prior projection module, and scoring output module are trained. This setup can reduce memory overhead and training time while preserving the semantic understanding capability of the pre-trained visual language model.
[0079] Step 6: Output image editing quality score.
[0080] In this embodiment, the visual language model performs multi-level inference on the enhanced input sequence E and outputs a hidden state sequence. The hidden state of the last token or the pooled hidden state is selected as the quality prediction representation h, and it is input into the regression prediction module MLP to obtain the quality score q.
[0081] q = σ(MLP(h)).
[0082] Where σ is the Sigmoid function or other normalization function, and q is between 0 and 1. The closer q is to 1, the more the edited image conforms to the text editing instructions and the more reasonable the visual effect; the closer q is to 0, the more the editing fails, the instructions are inconsistent, or the visual quality is poor.
[0083] Step 7: Train the image editing quality assessment model.
[0084] In this embodiment, a quality-labeled image editing evaluation dataset is constructed. Each training sample includes a source image, a edited image, text editing instructions, and a quality label. The quality label can be a binary label or a continuous human score; it can also further include evaluation results across three dimensions: correctness of modification, consistency of instructions, and visual reasonableness.
[0085] In this embodiment, the mean squared error loss model is used for training:
[0086] L = (1 / N)·Σ|qi - q̂i|^2.
[0087] Where N represents the number of training samples, qi represents the quality label of the i-th sample, and q̂i represents the quality score predicted by the model. For binary quality judgment tasks, binary cross-entropy loss can also be used; for ranking evaluation tasks, pairwise ranking loss can also be used.
[0088] In this embodiment, to improve generalization ability under different editing types, editing type prompts, type embeddings, or type grouping sampling can be introduced during training. This allows the model to learn how to utilize perceptual priors in tasks such as object removal, object replacement, object addition, background modification, and attribute editing. For example, in color or attribute editing, small but locally significant perceptual changes may correspond to successful editing; in object removal or background replacement, large structural changes may not necessarily indicate failure, but rather require judgment of semantic rationality in conjunction with textual instructions.
[0089] Step 8: Deployment and Reasoning.
[0090] In this embodiment, after model training is complete, for any new image editing sample, the system sequentially performs image preprocessing, perceptual prior extraction, soft prefix token generation, multimodal fusion, visual language model inference, and quality score output, ultimately obtaining an image editing quality score. Depending on the specific application requirements, the system can also map the score to quality levels such as "success," "partial success," and "failure," or output explanatory evaluation text related to the correctness of the modification, consistency of instructions, and visual reasonableness.
[0091] like Figure 1As shown, the perceptual prior injection structure in this embodiment includes an image editing perceptual encoder, a prior projection module, a visual language model, and a scoring output module. The image editing perceptual encoder is responsible for extracting change representations from the source image and the edited image; the prior projection module is responsible for converting the change representations into soft prefix tokens; the visual language model is responsible for fusing soft prefix tokens, image tokens, and text tokens and performing semantic reasoning; and the scoring output module is responsible for generating the final quality score based on the hidden state.
[0092] Secondly, this embodiment provides an image editing quality assessment system that integrates perceptual priors and a visual language model to implement the above method. The system includes a sample acquisition module, a perceptual prior extraction module, a prior projection module, a multimodal fusion module, a visual language inference module, a scoring output module, and a training and calibration module.
[0093] The sample acquisition module is used to acquire the source image, the edited image, and the text editing instructions in the image editing sample to be evaluated, and to preprocess the source image, the edited image, and the text editing instructions respectively to obtain the preprocessed source image, the preprocessed edited image, and the text token sequence; the preprocessed source image and the preprocessed edited image are used as inputs to the perceptual prior extraction module and the visual encoding part of the visual language model, and the text token sequence is used as inputs to the multimodal fusion module.
[0094] The perceptual prior extraction module is used to receive the preprocessed source image and the preprocessed edited image output by the sample acquisition module, and to use the image editing perceptual encoder to jointly encode the source image and the edited image to extract perceptual prior features that characterize the visual changes before and after editing; the perceptual prior features are used as the input of the prior projection module.
[0095] The prior projection module is used to receive the perceptual prior features output by the perceptual prior extraction module and map the perceptual prior features to one or more consecutive soft prefix tokens under the hidden dimension of the visual language model; the soft prefix tokens serve as the input of the multimodal fusion module.
[0096] The multimodal fusion module is used to receive the soft prefix tokens output by the prior projection module, the image token sequence output by the visual encoding part of the visual language model, and the text token sequence output by the sample acquisition module, and concatenate the soft prefix tokens, image token sequence, and text token sequence in a preset order to form an enhanced input sequence; the enhanced input sequence serves as the input to the visual language inference module.
[0097] The visual language reasoning module is used to receive the enhanced input sequence output by the multimodal fusion module and perform multimodal reasoning based on the multi-layer self-attention structure of the visual language model. During the reasoning process, the perceptual prior information carried by the soft prefix token is adaptively read and fused through the parameter efficient adaptation module to obtain the multimodal hidden representation. The multimodal hidden representation is used as the input of the scoring output module.
[0098] The scoring output module is used to receive the multimodal hidden representation output by the visual language reasoning module and generate an image editing quality score through the regression prediction module. The image editing quality score is used to characterize the degree of completion of the editing of the edited image relative to the source image and the text editing instructions, the consistency of the instructions and the visual rationality.
[0099] The training and calibration module is used to receive labeled image editing training samples and train the prior projection module, parameter efficient adaptation module, and scoring output module based on the labeled image editing training samples. The training and calibration module is also used to perform quality assessment and calibration of the model according to different editing types, so that the model can adaptively utilize perceptual prior information under different editing types such as object addition, object removal, object replacement, background modification, and attribute editing.
[0100] Thirdly, an electronic device according to this embodiment includes a processor and a memory, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the method described in any of the above-mentioned embodiments.
[0101] Fourthly, in this embodiment, a computer storage medium stores a computer program that, when run in a processor, implements the method described in any of the above-mentioned embodiments.
[0102] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is merely a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, or combinations made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for evaluating image editing quality that integrates perceptual priors and visual language models, characterized in that: Includes the following steps, Step 1: Obtain the image editing sample to be evaluated, and perform preprocessing operations on the source image Is, the edited image Ie, and the text editing instructions S in the sample respectively; the image editing sample includes the source image, the edited image, and the text editing instructions corresponding to the editing operation; Step 2: Combine the preprocessed source image and the edited image into an image pair, concatenate them along the channel dimension, and input them into a pre-trained EditCLIP image editing perceptual encoder with the overall parameters frozen for joint encoding. Extract feature vectors through the encoder's final hidden layer or a specified intermediate layer as perceptual prior features f; the perceptual prior features are used to characterize the local or global visual changes caused by the image editing operation. Step 3: Using a trainable perceptual prior projection module, the perceptual prior features are mapped to the hidden dimension space of the visual language model and reshaped into continuous soft prefix tokens P consistent with the word embedding dimension of the visual language model. The soft prefix token is trained to carry structural change information between the source image and the edited image, and is used for the attention layer reading of the visual language model; The perception prior projection module is composed of a first linear layer, a GELU activation layer, and a second linear layer connected in series. Step 4: Input the preprocessed source image and the edited image into the parameter-preserving frozen visual encoder of the visual language model to obtain the image token sequence; input the preprocessed text editing instructions into the parameter-preserving frozen text embedding module of the visual language model, and obtain the text token sequence through the text segmenter and text embedding layer. The enhanced input sequence E is formed by fusing and concatenating the soft prefix token, image token sequence, and text token sequence in that order. Step 5: Input the enhanced input sequence into the Transformer attention layer of the visual language model with the subject parameters frozen, and then input the query projection matrix W into the attention layer. up The projection matrix W down A low-rank adaptation module is introduced for parameter fine-tuning. In multi-layer self-attention computation, soft prefix tokens are used as keys and values in inference, enabling the visual language model to adaptively fuse perceptual prior information. The parameter fine-tuning specifically includes incrementally updating the original projection matrix through low-rank decomposition using a low-rank adaptation module that includes a trainable query projection matrix and a value projection matrix. The low-rank adaptation module also dynamically adjusts the attention intensity of soft prefix tokens on the query projection matrix. The logarithmic projection matrix fuses perceptual prior information into the multimodal hidden representation; The visual language model adopts the Qwen3-VL multimodal large model; Step 6: Extract the hidden state of the last token in the output sequence of the visual language model, the hidden state of the special classification token, or the global hidden representation obtained by pooling as the quality prediction representation h. Output the image editing quality score q, which is located within a preset value range, through the trainable regression prediction module MLP. The image editing quality score ranges from [0,1] and is used to characterize the degree of editing completion and visual quality of the edited image relative to the source image and text editing instructions. When the quality score approaches 1, the instruction consistency is higher and the visual effect is more natural. When the quality score approaches 0, the editing fails or there are obvious artifacts and instruction deviations. The regression prediction module is a multilayer perceptron structure, which is composed of a first linear layer, a GELU activation layer, a second linear layer, and a Sigmoid normalization layer connected in series. Step 7: Construct training samples containing source images, edited images, text editing instructions, and corresponding quality labels; group the training samples based on editing type and introduce type embedding prompts; use mean squared error loss, binary cross-entropy loss, ranking loss, or a weighted combination thereof as loss functions to train the quality assessment model end-to-end, enabling the quality assessment model to learn the weight allocation of perceived prior information under the editing type; use the trained quality assessment model for inference and output interpretive evaluation results including dimensions of modification correctness, instruction consistency, and visual rationality; the quality labels include at least one of binary quality labels, continuous quality scores, or multi-dimensional evaluation labels; the multi-dimensional evaluation labels include modification correctness, instruction consistency, and visual rationality, where modification correctness is used to evaluate whether the target visual change is completed, instruction consistency is used to evaluate whether the editing result conforms to the text editing instructions, and visual rationality is used to evaluate whether the edited image is natural and coherent without obvious artifacts.
2. The image editing quality assessment method that integrates perceptual priors and visual language models as described in claim 1, characterized in that: In step 1, the preprocessing operation specifically includes, The source and edited images are normalized in size, pixel value, and channel format. It will serve as input to the visual encoder and visual language model; it will perform word segmentation, special tag addition, and length truncation or padding on the text editing instructions to form a text token sequence; according to the type of editing task, the image editing samples will be divided into one or more categories among object addition, object removal, object replacement, background modification, and attribute editing for type-aware model training or evaluation calibration.
3. The image editing quality assessment method that integrates perceptual priors and visual language models as described in claim 1, characterized in that: In step 2, the perceptual prior feature f is represented as follows: f = F(concat(Is, Ie)); Where concat (·) represents concatenation along the channel dimension; F(·) represents the image editing perceptual encoder.
4. The image editing quality assessment method that integrates perceptual priors and visual language models as described in claim 1, characterized in that: In step 3, the soft prefix token P is represented as follows: P = reshape(G( f)), P ∈ R^(K×D); Where K represents the number of soft prefix tokens, D represents the hidden dimension of the visual language model, G represents the prior projection module, and R represents the real number space.
5. The image editing quality assessment method that integrates perceptual priors and visual language models as described in claim 1, characterized in that: In step 4, the enhanced input sequence is represented as follows: E = [P; V; T]; Among them, the soft prefix token is P, the image token sequence is V, and the text token sequence is T.
6. The image editing quality assessment method that integrates perceptual priors and visual language models as described in claim 1, characterized in that: In step 5, the parameter fine-tuning specifically satisfies the following conditions: W' = W + ( α / r )·W up ·W down ; Where r is the rank of the low-rank matrix, α is the scaling factor, W is the original pre-trained query projection matrix or value projection matrix in the attention layer of the visual language model, and W' is the equivalent projection matrix after introducing the low-rank adaptation module.
7. The image editing quality assessment method that integrates perceptual priors and visual language models as described in claim 1, characterized in that: In step 6, the quality score q is represented as, q = σ( MLP ( h ) ); Where σ is the Sigmoid function or normalization function.
8. An image editing quality assessment system that integrates perceptual priors and visual language models to implement the method described in any one of claims 1-7, characterized in that: It includes a sample acquisition module, a perceptual prior extraction module, a prior projection module, a multimodal fusion module, a visual language reasoning module, a scoring output module, and a training and calibration module; The sample acquisition module is used to acquire the source image, the edited image, and the text editing instructions in the image editing sample to be evaluated, and to preprocess the source image, the edited image, and the text editing instructions respectively to obtain the preprocessed source image, the preprocessed edited image, and the text token sequence. The preprocessed source image and the preprocessed edited image are used as inputs to the perceptual prior extraction module and the visual encoding part of the visual language model, and the text token sequence is used as input to the multimodal fusion module. The perceptual prior extraction module is used to receive the preprocessed source image and the preprocessed edited image output by the sample acquisition module, and to use the image editing perceptual encoder to jointly encode the source image and the edited image to extract perceptual prior features that characterize the visual changes before and after editing; the perceptual prior features are used as the input of the prior projection module. The prior projection module is used to receive the perceptual prior features output by the perceptual prior extraction module and map the perceptual prior features to one or more consecutive soft prefix tokens under the hidden dimension of the visual language model. The soft prefix token serves as the input to the multimodal fusion module; The multimodal fusion module is used to receive the soft prefix tokens output by the prior projection module, the image token sequence output by the visual encoding part of the visual language model, and the text token sequence output by the sample acquisition module, and to concatenate the soft prefix tokens, the image token sequence, and the text token sequence in a preset order to form an enhanced input sequence. The enhanced input sequence serves as the input to the visual language reasoning module; The visual language reasoning module is used to receive the enhanced input sequence output by the multimodal fusion module and perform multimodal reasoning based on the multi-layer self-attention structure of the visual language model. During the inference process, the perceptual prior information carried by the soft prefix token is adaptively read and fused through the parameter efficient adaptation module to obtain the multimodal hidden representation; the multimodal hidden representation is used as the input of the scoring output module. The scoring output module is used to receive the multimodal hidden representation output by the visual language reasoning module, and generate an image editing quality score through the regression prediction module. The image editing quality score is used to characterize the degree of editing completion, instruction consistency and visual rationality of the edited image relative to the source image and text editing instructions. The training and calibration module is used to receive labeled image editing training samples and to train the prior projection module, the parameter efficient adaptation module, and the scoring output module based on the labeled image editing training samples. The training and calibration module is also used to perform quality assessment and calibration of the model according to different editing types, so that the model can adaptively utilize perceptual prior information under different editing types such as object addition, object removal, object replacement, background modification and attribute editing.
9. An electronic device comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program, when executed by the processor, implements the method as claimed in any one of claims 1-7.
10. A computer storage medium storing a computer program that, when executed in a processor, implements the method as described in any one of claims 1-7.