Method for evaluating performance of instruction-based image editing model

Abstract

The invention discloses a method for evaluating the performance of an instruction-based image editing model. The instruction-based image editing model can be comprehensively evaluated. The invention provides an I2EBench evaluation standard, which comprises the following steps of: 1, randomly collecting at least 2000 original images from an existing public data set, marking editing instructions on the original images, and carrying out diversified processing on the editing instructions through ChatGPT so as to construct an evaluation data set; step 2, inputting the evaluation data set into each instruction-based image editing model to be evaluated, evaluating the image editing models based on 16 editing dimensions including 8 high-level editing dimensions and 8 low-level editing dimensions, and calculating a benchmark evaluation score; 3, performing user investigation through the questionnaire, obtaining the ranking of the user for the image editing effect of the model, and calculating a human evaluation score; and comparing the benchmark evaluation score with the human evaluation score, and calculating the relevancy between the benchmark evaluation score and the human evaluation score.

Description

Technical Field

[0001] The present invention relates to the technical field of model performance evaluation, and in particular to a method for evaluating the performance of an instruction-based image editing model. Background Art

[0002] With the development of generative adversarial networks (GANs) and diffusion models, text-to-image models have made significant progress in recent years. This technological advancement caters to the growing demand for image editing and has spawned many image editing models based on text descriptions. Among them, prompt-based image editing (PIE) is a task that requires users to provide both a target description and an original image. The PIE model analyzes the target description and modifies the original image accordingly to generate a target image that matches the target description. Although this method lowers the technical threshold for image editing, users still need to describe the target image more comprehensively when interacting, which is a challenge in itself and places certain requirements on the user's proficiency in using the model.

[0003] To overcome the above problems, instruction-based image editing (IIE) has emerged, which enables image editing by simplifying user input - only providing the original image and modification instructions (such as "remove the dog in the picture"). In particular, InstructPix2Pix introduced a large-scale dataset and proposed an instruction-based diffusion model for image editing based on this dataset. However, the automatic generation and screening process of this dataset has raised concerns about its quality and potential noise. To improve the quality of the dataset, MagicBrush proposed a manually annotated instruction-guided image editing dataset. The InstructAny2Pix model not only accepts text instructions, but also handles instructions in other modalities such as audio and images. MGIE aims to improve the level of detail of instructions and the accuracy of editing results by introducing a multimodal large language model (MLLM). Despite significant research progress in related technologies, evaluating the editing performance of IIE models remains a key issue.

[0004] In the existing technology, the evaluation of text-based image editing models usually relies on indicators such as CLIP score, PSNR, SSIM and LPIPS. Although some studies have proposed benchmarks such as TedBench, EditBench and EditVal to evaluate the performance of image editing models, these benchmarks have limitations: (1) TedBench contains only 100 images and limited editing types; (2) EditBench only focuses on evaluating mask-guided editing methods that require additional masks; (3) The evaluation of some dimensions of EditVal relies on manual operations, which limits its reproducibility. In addition, most dimensions focus on the modification of object size or position, and fail to fully cover all possible editing tasks. (4) Although MagicBrush and Emu Edit proposed test sets to evaluate editing performance, they still rely on traditional evaluation metrics that may not accurately capture the nuances of all editing types; (5) Although SmartEdit has developed a benchmark for complex editing scenarios, it is not applicable to all editing scenarios.

[0005] In summary, the existing evaluation benchmarks for instruction-based image editing have the disadvantages of small dataset size and limited evaluation dimensions, which leads to insufficient alignment between the evaluation results of these benchmarks and human perception, and incomplete model performance evaluation results caused by imperfect evaluation dimensions. Therefore, there is an urgent need for a comprehensive benchmark to effectively evaluate the performance of these models. Summary of the invention

[0006] The purpose of the present invention is to provide a method for evaluating the performance of an instruction-based image editing model, to solve the problems existing in the prior art, to comprehensively evaluate the instruction-based image editing model, and to improve the degree of alignment with human perception.

[0007] In order to achieve the above object, the solution of the present invention is: A method for evaluating the performance of instruction-based image editing models is proposed. 2 The EBench evaluation criteria include the following steps: Step 1: Randomly collect at least 2,000 original images from existing public datasets, annotate the original images with editing instructions, the number of editing instructions is at least 4,000, and diversify the editing instructions through ChatGPT to construct an evaluation dataset; Step 2: Input the evaluation dataset into each instruction-based image editing model to be evaluated, evaluate these image editing models based on 8 high-level editing dimensions and 8 low-level editing dimensions, a total of 16 editing dimensions, and calculate the benchmark evaluation score; among them, the 8 high-level editing dimensions include counting, direction perception, object removal, object replacement, background replacement, color modification, style transformation and regional editing accuracy, and the 8 low-level editing dimensions include deblurring, defogging, low-light enhancement, noise removal, rain removal, shadow removal, snow removal and watermark removal; Step 3: Conduct a user survey through a questionnaire to obtain users' rankings of the model's image editing effects and calculate the human evaluation score; compare the baseline evaluation score with the human evaluation score and calculate the correlation between the two.

[0008] In step 2, the advanced editing dimensions - counting, direction perception, object removal, object replacement, background replacement and color modification - are all evaluated using GPT-4V, and the calculation formula is: ; ; ; ; ; in, represents the edited image; represents the model used to edit the original image; represents the original image; represents the answer of GPT-4V, yes abbreviation of; Represents the GPT-4V model; Expressing questions about GPT-4V, yes abbreviation of; Indicates that ChatGPT determines whether GPT-4V's answer is consistent with the correct answer. yes abbreviation of; Represents the ChatGPT model; Indicates the correct answer, yes abbreviation of; Indicates whether each image is edited successfully. A value of 1 indicates success, and a value of 0 indicates failure. Represents the dimension score of the model's specified advanced editing dimension. to denoting the dimension scores for counting, direction perception, target removal, target replacement, background replacement, and color, respectively; Indicates the sequence number of the image; represents the total number of images for which the model is evaluated for a given high-level edit dimension, to Represents the total number of images evaluated for count, orientation awareness, object removal, object replacement, background replacement, and color, respectively.

[0009] In step 2, for the advanced editing dimension - style transformation, the calculation formula is: ; ; ; in, represents the edited image; represents the model used to edit the original image; represents the original image; Indicates the CLIP model's score for the consistency between image and text description. yes abbreviation of; represents the CLIP model; A text description of the image content; represents the dimension score of style transformation, yes abbreviation of; Indicates the sequence number of the image; Denotes the total number of images for which the style transfer is evaluated.

[0010] In step 2, for the advanced editing dimension - regional editing accuracy, the calculation formula is: ; ; ; ; ; in, represents the edited image; represents the model used to edit the original image; represents the original image; Represents the specified area of ​​the original image; Represents the Mask model; represents the area in the edited image that corresponds to the original image; represents the regional score, yes abbreviation of; represents the SSIM model; The dimension score representing the region editing accuracy, yes abbreviation of; Indicates the sequence number of the image; Represents the total number of images for which the editing accuracy of the region was evaluated.

[0011] In step 2, for the low-level editing dimensions - deblurring, defogging, low-light enhancement, noise removal, rain removal, shadow removal, snow removal and watermark removal, the calculation formula is: ; ; ; in, represents the edited image; represents the model used to edit the original image; represents the original image; It represents the score obtained by using the SSIM model to evaluate the similarity between the correct image editing result and the image after model editing. yes abbreviation of; represents the SSIM model; Indicates the correct editing results of the image specified by the model for the low-level editing dimensions; represents the dimension score of the model-specified low-level editing dimension, to Represents the dimension scores of deblurring, dehazing, low-light enhancement, noise removal, rain removal, shadow removal, snow removal, and watermark removal respectively; Indicates the sequence number of the image; represents the total number of images for which the model is evaluated for a given low-level edit dimension, to denotes the total number of images evaluated for deblurring, dehazing, low-light enhancement, noise removal, rain removal, shadow removal, snow removal, and watermark removal, respectively.

[0012] In step 3, when conducting a questionnaire survey, an editing instruction, an original image, and a group of edited images generated by several different IIE models are displayed to the user, and then the user ranks the image editing effects of the models based on his or her own subjective judgment, and each IIE model obtains a corresponding score based on the ranking.

[0013] Preferably, in step 3, for the number of models M, when evaluating the image editing effect of a single editing dimension and a single image, the score of the model ranked first is M, and the score of the model ranked last is 1. Finally, the score of the image editing effect of each model for all images in a single editing dimension is averaged to obtain the corresponding human evaluation score.

[0014] After adopting the above technical solution, the present invention has the following technical effects: The present invention proposes 2 The EBench evaluation standard can more comprehensively evaluate the performance of the model in various editing tasks, and its rich evaluation dimensions can show in detail the performance advantages and disadvantages of different models in different editing tasks, so that users have a more comprehensive reference when selecting models for image editing; the present invention can achieve strong alignment with human perception, and its evaluation results are more in line with the general public's performance requirements for image editing models. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of a specific embodiment of the present invention; Figure 2 A score table showing the evaluation results of the present invention on the low-level dimensional editing effects of each model; Figure 3 This is a score table of the results of evaluating the high-level dimensional editing effects of each model in the present invention. DETAILED DESCRIPTION

[0016] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0017] refer to Figure 1 As shown, the present invention discloses a method for evaluating the performance of an instruction-based image editing model, and proposes I 2 The EBench evaluation criteria include the following steps: Step 1: Randomly collect at least 2,000 original images from existing public datasets, annotate the original images with editing instructions, the number of editing instructions is at least 4,000, and diversify the editing instructions through ChatGPT to construct a large-scale evaluation dataset; Step 2: Input the evaluation dataset into each instruction-based image editing model to be evaluated, evaluate these image editing models based on 8 high-level editing dimensions and 8 low-level editing dimensions, a total of 16 editing dimensions, and calculate the benchmark evaluation score; among them, the 8 high-level editing dimensions include counting, direction perception, object removal, object replacement, background replacement, color modification, style transformation and regional editing accuracy, and the 8 low-level editing dimensions include deblurring, defogging, low-light enhancement, noise removal, rain removal, shadow removal, snow removal and watermark removal, so that the performance of various editing tasks of the model can be evaluated more comprehensively; Step 3: Conduct user surveys through questionnaires to obtain users’ rankings of the model’s image editing effects and calculate human evaluation scores; compare the baseline evaluation scores with the human evaluation scores and calculate the correlation between the two, thereby verifying I 2 EBench evaluates the reliability of the criteria and how well they align with human perception.

[0018] Through the above scheme, the present invention proposes I 2 The EBench evaluation standard can more comprehensively evaluate the performance of the model in various editing tasks, and its rich evaluation dimensions can show in detail the performance advantages and disadvantages of different models in different editing tasks, so that users have a more comprehensive reference when selecting models for image editing; the present invention can achieve strong alignment with human perception, and its evaluation results are more in line with the general public's performance requirements for image editing models.

[0019] Specific embodiments of the present invention are shown below.

[0020] In step 2 above, the advanced editing dimensions - counting, direction perception, object removal, object replacement, background replacement, and color modification - are all evaluated using GPT-4V, and the calculation formula is: ; ; ; ; ; in, represents the edited image; represents the model used to edit the original image; represents the original image; represents the answer of GPT-4V, yes abbreviation of; Represents the GPT-4V model; represents a question to GPT-4V (for example, for the technical editing dimension, the question could be “how many dogs are there in the picture”), yes abbreviation of; Indicates that ChatGPT determines whether GPT-4V's answer is consistent with the correct answer. yes abbreviation of; Represents the ChatGPT model; Indicates the correct answer, yes abbreviation of; Indicates whether each image is edited successfully. A value of 1 indicates success, and a value of 0 indicates failure. Represents the dimension score of the model's specified advanced editing dimension. to denoting the dimension scores for counting, direction perception, target removal, target replacement, background replacement, and color, respectively; Indicates the sequence number of the image; represents the total number of images for which the model is evaluated for a given high-level edit dimension, to Represents the total number of images evaluated for count, orientation awareness, object removal, object replacement, background replacement, and color, respectively.

[0021] In step 2 above, for the advanced editing dimension - style transformation, the calculation formula is: ; ; ; in, represents the edited image; represents the model used to edit the original image; represents the original image; Indicates the CLIP model's score for the consistency between image and text description. yes abbreviation of; It represents the CLIP (Contrastive Language-Image Pre-Training) model; A text description of the image content; represents the dimension score of style transformation, yes abbreviation of; Indicates the sequence number of the image; Denotes the total number of images for which the style transfer is evaluated.

[0022] In step 2 above, the calculation formula for the advanced editing dimension - regional editing accuracy is: ; ; ; ; ; in, represents the edited image; represents the model used to edit the original image; represents the original image; Represents the specified area of ​​the original image; Represents the Mask model; represents the area in the edited image that corresponds to the original image; represents the regional score, yes abbreviation of; represents the SSIM model; The dimension score representing the region editing accuracy, yes abbreviation of; Indicates the sequence number of the image; Represents the total number of images for which the editing accuracy of the region was evaluated.

[0023] In step 2 above, for the low-level editing dimensions - deblurring, defogging, low-light enhancement, noise removal, rain removal, shadow removal, snow removal, and watermark removal, the calculation formula is: ; ; ; in, represents the edited image; represents the model used to edit the original image; represents the original image; It represents the score obtained by using the SSIM model to evaluate the similarity between the correct image editing result and the image after model editing. yes abbreviation of; represents the SSIM model; Indicates the correct editing results of the image specified by the model for the low-level editing dimensions; represents the dimension score of the model-specified low-level editing dimension, to Represents the dimension scores of deblurring, dehazing, low-light enhancement, noise removal, rain removal, shadow removal, snow removal, and watermark removal respectively; Indicates the sequence number of the image; represents the total number of images for which the model is evaluated for a given low-level edit dimension, to denotes the total number of images evaluated for deblurring, dehazing, low-light enhancement, noise removal, rain removal, shadow removal, snow removal, and watermark removal, respectively.

[0024] In the above step 3, when conducting the questionnaire survey, an editing instruction, an original image, and a set of edited images generated by several different IIE models are displayed to the user, and then the user ranks the image editing effects of the model based on his or her own subjective judgment, and each IIE model obtains a corresponding score based on the ranking.

[0025] Furthermore, for the number of models M, when evaluating the image editing effect of a single editing dimension and a single image, the first-ranked model scores M, and the last-ranked model scores 1. Finally, the image editing effect scores of each model for all images in a single editing dimension are averaged to obtain the corresponding human evaluation scores.

[0026] In step 1 above, the specific steps for diversifying the editing instructions through ChatGPT are: Input the editing instruction into ChatGPT, and provide ChatGPT with a text prompt so that ChatGPT can modify the input editing instruction into an expression form corresponding to the text prompt according to the text prompt. For example, the following prompt structure can be constructed - "Please give different expressions of the following sentence: <Editing instruction>", so that ChatGPT can generate a new sentence with the same meaning but different expression according to the original instruction.

[0027] In this embodiment, the instruction-based image editing models are mainly HIVE, InstrctDiffusin, InstructPix2Pix, MagicBrush, MGIE, InsturctEdit, InstructAny2Pix and HQ-Edit. Figure 2 , Figure 3 It can be seen that no single model can perform well in all evaluation dimensions, and each has its own advantages and disadvantages.

[0028] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A method for evaluating the performance of an instruction-based image editing model, characterized in that Propose I 2 The EBench evaluation criteria include the following steps: Step 1: Randomly collect at least 2,000 original images from existing public datasets, annotate the original images with editing instructions, the number of editing instructions is at least 4,000, and diversify the editing instructions through ChatGPT to construct an evaluation dataset; Step 2: Input the evaluation dataset into each instruction-based image editing model to be evaluated, evaluate these image editing models based on 8 high-level editing dimensions and 8 low-level editing dimensions, a total of 16 editing dimensions, and calculate the benchmark evaluation score; among them, the 8 high-level editing dimensions include counting, direction perception, object removal, object replacement, background replacement, color modification, style transformation and regional editing accuracy, and the 8 low-level editing dimensions include deblurring, defogging, low-light enhancement, noise removal, rain removal, shadow removal, snow removal and watermark removal; Step 3: Conduct a user survey through a questionnaire to obtain users' rankings of the model's image editing effects and calculate the human evaluation score; compare the baseline evaluation score with the human evaluation score and calculate the correlation between the two.

2. The method for evaluating the performance of an instruction-based image editing model according to claim 1, characterized in that: In step 2, the advanced editing dimensions - counting, direction perception, object removal, object replacement, background replacement and color modification - are all evaluated using GPT-4V, and the calculation formula is: ； ； ； ； ； in, represents the edited image; represents the model used to edit the original image; represents the original image; represents the answer of GPT-4V, yes abbreviation of; Represents the GPT-4V model; Expressing questions about GPT-4V, yes abbreviation of; Indicates that ChatGPT determines whether GPT-4V's answer is consistent with the correct answer. yes abbreviation of; Represents the ChatGPT model; Indicates the correct answer, yes abbreviation of; Indicates whether each image is edited successfully. A value of 1 indicates success, and a value of 0 indicates failure. Represents the dimension score of the model's specified advanced editing dimension. to denoting the dimension scores for counting, direction perception, target removal, target replacement, background replacement, and color, respectively; Indicates the sequence number of the image; represents the total number of images for which the model is evaluated for a given high-level edit dimension, to Represents the total number of images evaluated for count, orientation awareness, object removal, object replacement, background replacement, and color, respectively.

3. The method for evaluating the performance of an instruction-based image editing model according to claim 1, wherein: In step 2, for the advanced editing dimension - style transformation, the calculation formula is: ； ； ； in, represents the edited image; represents the model used to edit the original image; represents the original image; Indicates the CLIP model's score for the consistency between image and text description. yes abbreviation of; represents the CLIP model; A text description of the image content; represents the dimension score of style transformation, yes abbreviation of; Indicates the sequence number of the image; Denotes the total number of images for which the style transfer is evaluated.

4. The method for evaluating the performance of an instruction-based image editing model according to claim 1, wherein: In step 2, for the advanced editing dimension - regional editing accuracy, the calculation formula is: ； ； ； ； ； in, represents the edited image; represents the model used to edit the original image; represents the original image; Represents the specified area of ​​the original image; Represents the Mask model; represents the area in the edited image that corresponds to the original image; represents the regional score, yes abbreviation of; represents the SSIM model; The dimension score representing the region editing accuracy, yes abbreviation of; Indicates the sequence number of the image; Represents the total number of images for which the editing accuracy of the region was evaluated.

5. The method for evaluating the performance of an instruction-based image editing model according to claim 1, wherein: In step 2, for the low-level editing dimensions - deblurring, defogging, low-light enhancement, noise removal, rain removal, shadow removal, snow removal and watermark removal, the calculation formula is: ； ； ； in, represents the edited image; represents the model used to edit the original image; represents the original image; It represents the score obtained by using the SSIM model to evaluate the similarity between the correct image editing result and the image after model editing. yes abbreviation of; represents the SSIM model; Indicates the correct editing results of the image specified by the model for the low-level editing dimensions; represents the dimension score of the model-specified low-level editing dimension, to Represents the dimension scores of deblurring, dehazing, low-light enhancement, noise removal, rain removal, shadow removal, snow removal, and watermark removal respectively; Indicates the sequence number of the image; represents the total number of images for which the model is evaluated for a given low-level edit dimension, to denotes the total number of images evaluated for deblurring, dehazing, low-light enhancement, noise removal, rain removal, shadow removal, snow removal, and watermark removal, respectively.

6. The method for evaluating the performance of an instruction-based image editing model according to claim 1, wherein: In step 3, when conducting a questionnaire survey, an editing instruction, an original image, and a group of edited images generated by several different IIE models are displayed to the user, and then the user ranks the image editing effects of the models based on his or her own subjective judgment, and each IIE model obtains a corresponding score based on the ranking.

7. The method for evaluating the performance of an instruction-based image editing model according to claim 6, wherein: In step 3, for the number of models M, when evaluating the image editing effect of a single editing dimension and a single image, the score of the model ranked first is M, and the score of the model ranked last is 1. Finally, the score of the image editing effect of each model for all images in a single editing dimension is averaged to obtain the corresponding human evaluation score.

Images