Method and system for testing and output repair of instruction guided image editing models based on transmutation testing techniques

Abstract

This invention discloses a testing and output repair method for an instruction-guided image editing model based on metamorphic testing technology, comprising: collecting the original input image and editing instructions as original test cases; collecting the output images of the original test cases; comparing the input image and the output image to obtain the editing region of the original test cases; transforming the input image according to the metamorphic relationship to generate a candidate image set; selecting equivalent derived test images and combining them with the original editing instructions to form new derived test cases; collecting the output images of all derived test cases; comparing the input image and the output image to obtain the editing region of the derived test cases; determining whether the editing regions of all derived test cases are consistent with those of the original test cases; cross-validating the outputs of all derived test cases with those of the original test cases; selecting the output of the best test case and mapping it onto the original test case as the original test output for repair; summarizing the test results and outputting the repair results.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a testing and output repair method and system for an instruction-guided image editing model based on metamorphic testing technology. Background Technology

[0002] With the rapid development of computer vision, natural language processing, and generative artificial intelligence, image editing models based on natural language commands (hereinafter referred to as image editing systems) are becoming an important research direction in the field of computer vision. These models, through deep integration of semantic understanding and image generation technologies, enable ordinary users to precisely control the image modification process using everyday language, such as commands like "Replace the cat in the image with the dog." Compared to traditional methods requiring specialized software, command-guided image editing systems break down the barriers to tool usage, freeing creative expression from the constraints of technical complexity and significantly improving the intelligence level of image processing. In practical applications, this technology has demonstrated broad potential. For example, Adobe's Firefly system supports intelligent image retouching via text commands; Runway's Gen-2 model enables text-driven video editing; and in the field of medical imaging, researchers are exploring the use of natural language commands to assist in lesion annotation and visualization analysis. With the evolution of open-source models such as Stable Diffusion, individual users can now achieve professional-grade image synthesis and style transfer using simple prompts, which is reshaping the way digital content is created.

[0003] Currently, methods for verifying the correctness of output images from image editing systems typically require manual intervention in constructing test instructions. Some verification methods directly use user surveys to verify the correctness of the model's output; others require a reference edited image and then verify the correctness of the model's output image by calculating the similarity between the model's output image and the reference image; still others require manual division of the editing area and background area of ​​a test case, and then calculating the similarity between the background area of ​​the output image and the input image, as well as the image-text similarity between the edited area of ​​the output image and the editing instructions. Clearly, this manual testing process is time-consuming and labor-intensive, making it unsuitable for large-scale application.

[0004] On the other hand, repairing the output of image editing systems is often done manually, through retraining, etc., requiring manual annotation costs and hardware costs for retraining. Another part of the work involves identifying and repairing suspicious neurons in deep neural networks to fix the model. These tasks either cannot be fully automated for locating and repairing errors in deep learning models, or they rely on neural networks and code for white-box repair. Without access to the internal details of the model, these repair methods cannot conveniently correct errors in the model's output.

[0005] To address the above situation, a feasible solution is to apply metamorphic testing techniques from the field of software testing. This allows for preliminary correctness checks and output repair of the image editing system's output when the reference edited image is unknown. Metamorphic testing determines whether the software output is likely to be erroneous by examining the software's performance on multiple interrelated inputs. For example, when checking the correctness of a program calculating the specific value of the sine function sin with an input of 120°, metamorphic testing does not directly check the correctness of the program output itself. Instead, based on the designed metamorphic relationship, it compares the program output with the output at 120° input and with the outputs at (180°-120°), -120°, etc., to see if they meet the expected relationship, thus making a preliminary judgment on the reliability of the program output. Therefore, metamorphic testing can perform preliminary reliability checks on the edited images output by the image editing system even without a standard answer, and can also be used to perform preliminary repairs on the model's output.

[0006] Applying metamorphic testing to verify and repair the reliability of image editing system output presents the following specific technical challenges:

[0007] There is currently no metamorphic relation for image editing systems: no metamorphic relation has been found that is suitable for checking the reliability of the output of image editing systems. Therefore, it is necessary to design a dedicated metamorphic relation that is specific to the input and output characteristics of image editing systems.

[0008] Currently, there is no method to verify the consistency of different outputs from an image editing system. The key to metamorphic testing is to determine whether the outputs of multiple corresponding test cases generated by a metamorphic relation are consistent. If they are inconsistent, the model output is considered to be erroneous. Since no suitable method has been found for verifying the consistency of image editing system outputs, it is necessary to design a method to verify consistency based on the output characteristics of the model.

[0009] Currently, there is no method to automatically repair the output of an image editing system: existing repair methods often require knowledge of the model's weights and deployment details, and a method needs to be designed to perform black-box, automated output repair of the image editing system. Summary of the Invention

[0010] To overcome the shortcomings of the prior art, this invention provides a method for output reliability checking and repair of image editing systems. It utilizes a large language model to generate appropriate editing instructions for a user-given image, designs suitable metamorphic relationships to generate new test cases to verify the reliability of the image editing system output, and uses a multi-test case cross-validation method to repair the original output.

[0011] According to one aspect of the present invention, a method for testing and output restoration of an image editing system based on metamorphic testing technology is provided, comprising:

[0012] Step 1: Collect the given original input image and editing instructions as the original test case, and collect the output image of the original test case, which is obtained by inputting the original test case into the image editing system under test; compare the input image and the output image of the original test case to obtain the difference grayscale image, and after noise reduction processing, obtain the smooth difference image, and further calculate the editing area of ​​the original test case;

[0013] Step 2: Based on the selected metamorphic relationship, transform the input image of the original test case to generate a series of candidate image sets. Select equivalent derived test images from the candidate image sets and combine them with the original editing instructions to form a series of new derived test cases.

[0014] Step 3: Collect the output images of all derived test cases, obtain the editing area of ​​the derived test cases by comparing the input and output images of the derived test cases, and determine whether the editing area of ​​all derived test cases is consistent with the editing area of ​​the original test cases by the output relationship;

[0015] Step 4: If the editing area of ​​any derived test case is inconsistent with the editing area of ​​the original test case, cross-validate the outputs of all derived test cases with the original test cases, select the output of the best test case, and then map it onto the original test case as the original test output for repair.

[0016] Step 5: Display the test and repair results.

[0017] As a further technical solution, step 1 also includes:

[0018] Calculate the difference between the input and output images of the original test cases, and perform grayscale binarization to obtain the grayscale image of the difference between the input and output images of the original test cases.

[0019] Gaussian blur is applied to the obtained difference grayscale image to obtain a blurred difference grayscale image;

[0020] The blurred difference grayscale image is binarized again to obtain a smooth difference grayscale image between the input and output images of the original test case;

[0021] A morphological closing operation is performed on the obtained smoothed difference grayscale image to obtain the final difference grayscale image;

[0022] Edge detection methods are used to calculate the edges of the final difference grayscale image, thus obtaining the editing area of ​​the original test case.

[0023] As a further technical solution, step 2, based on the selected metamorphic relationship, transforms the input image of the original test case to generate a series of candidate image sets, and also includes:

[0024] remember The metamorphosis relationship currently selected by the user;

[0025] when At that time, the original input image is subjected to multiple different cropping transformations to form a candidate image set;

[0026] when At that time, the original input image is subjected to multiple different stretching transformations along the vertical direction, and the resulting stretched images are used as a candidate image set.

[0027] when At that time, rotate the original input image clockwise. The generated rotated images are used as a candidate image set.

[0028] As a further technical solution, step 2, which involves selecting an equivalent derived test image from the candidate image set, also includes:

[0029] For all candidate images transformed from the original input image, the pixel changes during the image transformation process are recorded as a transformation matrix of the same size as the original input image and assigned values;

[0030] For all candidate images, determine whether the edit region of the original test case disappears due to image transformation, and select the candidate images that do not cause the edit region to disappear as the set of derived input images obtained by transforming the original input image.

[0031] As a further technical solution, step 3, which determines whether the editing areas of all derived test cases are consistent with the editing areas of the original test cases based on the output relationship, also includes:

[0032] Calculate the intersection-union ratio (IUU) of the edit regions of the original test cases and the edit regions of the derived test cases. ;

[0033] Compare the crossover and union ratios Does it exceed a predefined threshold? If the output exceeds the limit, the original test case and the derived test case are considered to be consistent.

[0034]

[0035] Calculate whether the outputs of the test cases in the original test case set and the derived test case set are consistent to obtain the consistency set. .

[0036] As a further technical solution, step 4, determining whether the editing area of ​​any derived test case is consistent with the editing area of ​​the original test case, also includes:

[0037]

[0038] when If the test results are not found, proceed to step 5.

[0039] when At that time, output repair will be performed.

[0040] As a further technical solution, step 4, which involves cross-validating the outputs of the original test cases and derived test cases, also includes:

[0041] For a derived test case and another derivative test case in the derivative test case set The consistency of the outputs is determined by calculating the intersection-union ratio of the edited regions of the two entities.

[0042]

[0043] Calculate the set of derived test cases and the derived test cases. The output should consistently reflect the number of derived test cases.

[0044] As a further technical solution, step 4 also includes:

[0045] when At that time, the output image of the best test case is directly overlaid on the corresponding position of the original test case output image;

[0046] when When the image is stretched, it is restored, and the restored output image is overlaid on the corresponding position of the original test case output image;

[0047] when When the rotation is complete, the rotated image is restored, and the restored output image is overlaid on the corresponding position of the original test case output image;

[0048] Among them, when , or At time, position The value corresponding to the above is determined by the transformation matrix of the optimal test case, which determines whether the original test case output image or the optimal test case output image is used.

[0049] As a further technical solution, step 5 also includes:

[0050] The interactive interface displays the output images and corresponding editing areas of the original test cases from the image editing system, as well as the output images and corresponding editing areas of several generated derivative test cases from the image editing system.

[0051] Display the results of whether the output images of several derived test cases are consistent with the output image of the original test case, and show the test results based on whether the editing area of ​​any derived test case is consistent with the original test case:

[0052] like If the system fails to pass the test for the original test cases, the user is prompted that the output of the image editing system is unreliable; otherwise, the system passes the test for the original test cases and the output is basically reliable.

[0053] According to one aspect of the present invention, a testing and output repair system for a command-guided image editing model based on metamorphic testing technology is provided, comprising:

[0054] The first main module is used to collect the given original input image and editing instructions as the original test cases, and at the same time collect the output image of the original test cases, which is obtained by inputting the original test cases into the image editing system under test; compare the input image and the output image of the original test cases to obtain the difference grayscale image, and after noise reduction processing, obtain the smooth difference image, and further calculate the editing area of ​​the original test cases;

[0055] The second main module is used to transform the input image of the original test case according to the selected metamorphic relationship to generate a series of candidate image sets, select equivalent derived test images from the candidate image sets, and form a series of new derived test cases together with the original editing instructions;

[0056] The third main module is used to collect the output images of all derived test cases, obtain the editing area of ​​the derived test cases by comparing the input and output images of the derived test cases, and determine whether the editing area of ​​all derived test cases is consistent with the editing area of ​​the original test cases by the output relationship.

[0057] The fourth main module is used to cross-validate the outputs of all derived test cases with the original test cases if the editing area of ​​any derived test case is inconsistent with the editing area of ​​the original test case, select the output of the best test case, and then map it onto the original test case as the original test output for repair.

[0058] The fifth main module is used to display the test and repair results.

[0059] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0060] 1. This invention aims to perform reliability testing and repair on the output images of an image editing system without requiring manual intervention to obtain test instructions or the expected output of test cases. First, it collects the user-provided original test cases and the output images of the image editing system. The editing region of the original test cases is obtained by calculating the difference in grayscale images between the input and output images. Then, based on the metamorphic relationship selected by the user, the input images of the original test cases are transformed to generate a set of candidate test cases, which are further selected to obtain a set of derived test cases. Next, all user-provided derived test cases and the output images provided by the image editing system are collected, and the difference in grayscale images between the input and output images of all derived test cases is calculated to obtain the editing regions of all derived test cases. Subsequently, the editing regions of the original test cases and all derived test cases are compared to determine if the outputs of the original and derived test cases are consistent. If inconsistencies are found, a best test case is selected through cross-validation, and its output is mapped to the original test case. Finally, the output testing and repair status is summarized and reported to the user. The core idea of ​​this method is that, under the condition that the editing instructions remain unchanged and the input image editing region is unaffected, the original test cases and derived test cases should exhibit consistent behavior, i.e., the editing regions should remain consistent. If the editing area of ​​the image editing system for the derived test case is inconsistent with that of the original test case, it indicates that at least one of the system's outputs for the original test case and the derived test case is incorrect.

[0061] 2. The method provided by this invention is based on a novel responsive metamorphic relationship, that is, the generation of derived test cases not only requires the original test cases, but also the output of the original test cases to assist in the generation. The reliability of the image editing system is checked by examining the consistency of the output of the image editing system to the original test cases and each derived test case. This process does not require manual intervention and can be fully automated by scripts. This solves the problems of existing image editing system testing methods based on traditional test precepts, which require manual intervention, are time-consuming and labor-intensive, and cannot provide real-time checks during the use of the image editing system.

[0062] 3. The method provided by this invention solves the problem of lacking a suitable metamorphic relationship when applying metamorphic testing technology to image editing systems, thus making it possible to test image editing systems automatically; it designs a method for calculating the editing area of ​​image editing test cases, solving the problem of lacking a suitable method to verify the consistency of image editing system output; and it designs a method for repairing image editing system output based on multi-test case cross-validation, thus solving the problem of lacking a method for black-box, automated repair of image editing system output.

[0063] 4. The method formed by the present invention can ultimately enable users to preliminarily verify the reliability of the output images of the image editing system without human intervention during the use of the image editing system, and repair unreliable responses, thereby achieving the goal of testing and repairing the output returned by the system without prior access to the reference output image or prerequisites of the image editing system. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a schematic diagram of the method flow provided in an embodiment of the present invention.

[0066] Figure 2 Example illustration of calculating the editing region for the input and output images of the comparative test cases provided in this embodiment of the invention;

[0067] Figure 3 This is an example illustration of the metamorphic relationship of transforming the input image of the original test case according to an embodiment of the present invention;

[0068] Figure 4 This is an example illustration of generating derived test cases based on original test cases, provided in an embodiment of the present invention.

[0069] Figure 5 An example illustration of the method for generating general interrogative sentences provided in the embodiments of the present invention;

[0070] Figure 6 This is an example illustration of repairing the output of the original test cases provided in an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0072] The purpose of this invention is to perform reliability testing and repair on the output images of an image editing system without requiring manual intervention to obtain test instructions or the expected output of test cases. This alleviates the problems of current image editing system testing processes requiring manual intervention, which are time-consuming and labor-intensive, making them unsuitable for large-scale application, and current output repair methods relying on model details and unable to conveniently repair the output. This invention designs a novel responsive metamorphic relationship based on metamorphic testing technology, and designs a method for calculating the editing region of image editing test cases to perform consistency checks on the outputs of original and derived test cases. It also designs a method for repairing the output of an image editing system based on multi-test case cross-validation, performing black-box, automated repair of the image editing system's output. This allows users to instantly verify the reliability of the image editing system's output images without manual intervention.

[0073] The following is combined with Figures 1 to 6 The specific implementation of this invention is as follows: This invention provides a method for output testing and repair of an image editing system based on responsive metamorphic relationships, see [link to relevant documentation]. Figure 1 ,include:

[0074] Step 1: Collect the user-provided original input image and editing instructions as the original test cases. Simultaneously, collect the output image of the user-provided original test cases, which is obtained by the user inputting the original test cases into the image editing system under test. Then, by comparing the input and output images of the original test cases, a difference grayscale image is obtained. After noise reduction processing, a smooth difference image is obtained, and then the editing area of ​​the original test cases is further calculated.

[0075] Preferably, the original test cases given by the user and the output image returned by the image editing system to the user as input in step 1 are as follows:

[0076] The original test cases input by the user are The original input image is The editing instructions are: ;

[0077] The image editing system under test is: ;

[0078] The image editing system returns the following edited image as the input original test case: .

[0079] The grayscale image representing the difference between the original input image and the original output image is:

[0080] Through calculation and The difference between the input and output images of the original test cases is calculated and then binarized to obtain a grayscale image representing the difference between the input and output images. .

[0081] In practical implementation, one specific method for implementing the above process using an instruction-guided image editing model is as follows:

[0082] The `absdiff` function provided by the python-opencv library is used to calculate the difference between the input and output images. Then, the Korean-style `cvtColor` function is used to convert this difference image into a grayscale image, resulting in the grayscale difference image between the input and output images. As attached Figure 2 (c) shows the input image. Figure 2 (a) and output image Figure 2 (b) The difference grayscale image.

[0083] The smoothed difference grayscale image of the original test case input image and output image mentioned in step 1 is as follows:

[0084] First, the difference grayscale image of the original test cases. Gaussian blurring involves using a Gaussian smoothing operator kernel to reduce the saliency of isolated pixels, resulting in a blurred difference grayscale image. :

[0085]

[0086] Then, the grayscale image with blur difference is processed. The image is binarized again to eliminate sporadic noise, resulting in a smooth difference grayscale image of the input and output images of the original test cases. :

[0087]

[0088] in, The grayscale image of the blur difference for the original test cases. Middle position The pixel values ​​on, where )) , )) . It is based on the blur difference grayscale image of the original test cases. This is derived from the Otsu algorithm, an adaptive image binarization thresholding calculation method:

[0089] For any candidate threshold The blur difference image of the original test cases All pixels are divided into two categories and :

[0090]

[0091] Then calculate at the threshold Below, the inter-class tolerance of two types of pixel sets in an image. :

[0092]

[0093] in, For blur difference grayscale images China is classified as The proportion of pixels that make up all pixels. For blur difference grayscale images China is classified as The proportion of pixels that make up all pixels. for The average value of the middle pixels. for The mean of the middle pixels, for all possible candidate thresholds. Choose the threshold that maximizes the inter-class tolerance:

[0094]

[0095] In practical implementation, one specific method for implementing the above process using an instruction-guided image editing model is as follows:

[0096] Use the GaussianBlur function provided by the python-opencv library to process differential grayscale images. Perform a Gaussian blur, where ksize is set to (5, 5), to obtain a grayscale image with the blurred difference. ;

[0097] The threshold function is used to calculate the blur difference in grayscale images. Adaptive thresholding for binarization The `type` parameter is set to `cv2.THRESH_BINARY+cv2.THRESH_OTSU`, indicating that an adaptive threshold is calculated using the Otsu algorithm, and then the blur difference grayscale image is processed based on this threshold. Binarization is performed to obtain a smoothed difference grayscale image of the input and output images of the original test cases. .

[0098] The editing area for the original test cases described in step 1 is:

[0099] Smoothed difference grayscale image of the input and output images of the original test cases Morphological closing is performed, which uses structure operators to perform dilation followed by erosion on the image to fill it. The tiny holes in the image yield the final difference grayscale image. :

[0100]

[0101] in This represents the structure operator used to perform morphological closing operations on smoothed difference grayscale images, namely 5. A 5-fold convolution kernel. This indicates an expansion operation, specifically... In the image Swipe up to position At that time, if A part and the image When positions overlap, position Added to the expanded collection middle:

[0102]

[0103] This indicates an etching operation; specifically, it only occurs when... Completely covered When it is a certain part, the corresponding position Only then will it be added to the eroded set. middle:

[0104]

[0105] Finally, edge detection methods are used to calculate the final difference grayscale image. The edge of the code provides the editing area for the original test cases. .

[0106] In practical implementation, one specific method for implementing the above process using an instruction-guided image editing model is as follows:

[0107] First, define a `ones` function provided by NumPy. 5 A matrix of 5, where all values ​​are 1, is used to smooth the difference grayscale image using the morphologyEx function provided by python-opencv. Perform a closing operation, where the parameter op is defined as cv2.MORPH_CLOSE, indicating that a closing operation needs to be performed to obtain the final difference grayscale image. ;

[0108] Then, the `findContours` function provided by python-opencv is used to draw the final grayscale image of the difference. The outline of the edit region is defined, and the edit region mask image is obtained using the drawContours function. .

[0109] Appendix Figure 2 This diagram illustrates the entire process of obtaining the test case editing area based on the input and output images of a test case. Figure 2 (d) shows the editing area of ​​the test case, which is the part enclosed by the green box.

[0110] Preferably, the original input image in step 2 A set of candidate images generated by transformation for:

[0111] remember The metamorphosis relationship currently selected by the user;

[0112] when At that time, the original input image After multiple different cropping transformations, the candidate image set is... ,in This represents the predefined number of candidate images to be generated;

[0113] when At that time, the original input image The image undergoes multiple stretching transformations along the vertical direction. To ensure the transformed image remains undistorted, the proportion of the original input image retained in the stretching transformation must be no less than 0.6. The resulting n stretched images are then used as a candidate image set. ;

[0114] when At that time, the original input image Rotate clockwise Similarly, to ensure the realism of the transformed image, this rotation transformation requires the image to be rotated by a certain angle. The generated n rotated images are used as a candidate image set, i.e. .

[0115] In practical implementation, one specific method for implementing the above process using an instruction-guided image editing model is as follows:

[0116] First, the user's chosen transformation relationship is obtained through the interactive interface. ;

[0117] when At that time, a series of top-left and bottom-right corner points are randomly selected, and then the input image of the original test case is directly based on each pair of top-left and bottom-right corner points. Perform cropping and then use the resize function provided by python-opencv to modify the size of the cropped image. By maintaining consistency, a candidate image set for cropping transformation is obtained. ;

[0118] when At that time, a series of upper and lower bound points are randomly selected along the vertical axis of the image, and then the input image of the original test case is cropped. The pixel values ​​above the upper bound and below the lower bound are selected, and then the resize function is used to modify the resulting image size. To maintain consistency and create a stretching effect, a set of candidate images for the stretching transformation is obtained. ;

[0119] when At that time, firstly A series of rotation angles are randomly selected within the range, and then the getRotationMatrix2D and warpAffine functions provided by python-opencv are used to process the input image of the original test cases. Perform a rotation operation to obtain a set of candidate images for the rotation transformation. .

[0120] Appendix Figure 3 Examples of transformations are shown for the input image of an original test case. Figure (a) is the given input image, Figure (b) is an example of cropping the input image, Figure (c) is an example of stretching the input image, and Figure (d) is an example of rotating the input image.

[0121] Step 2 describes the process of selecting from the candidate image set. Select equivalent derived test image for:

[0122] For all images derived from the original input image Candidate images derived from transformation The pixel changes during the image transformation process are recorded as a series of data. Transformation matrices of the same size ,if In position The corresponding pixels are exist, In position The corresponding value is 1, otherwise In position The corresponding value above is 0:

[0123]

[0124] For candidate image set All candidate images It determines whether the edit region of the original test case disappears due to image transformation, and then selects the candidate image that does not cause the edit region to disappear as the original input image. The derived input image set obtained by transformation :

[0125]

[0126] in This is a predefined, adjustable threshold.

[0127] The generated derivative test cases described in step 2 It consists of the generated derived input image and the editing instructions for the original test cases:

[0128]

[0129] In practical implementation, one specific method for implementing the above process using an instruction-guided image editing model is as follows:

[0130] For any transformed image in the candidate image set, record the changes of each pixel during the transformation process, and define a NumPy matrix to record the retained pixel values, thus obtaining the transformation matrix of this transformed image. Then, the edit region of the original test case is calculated using the where function provided by NumPy. and The sum of the matrix and the sum of the matrix are then used to calculate the sum of the matrix and the sum of the matrix. The total pixel value is calculated, and finally, it is determined whether the proportion of pixels left by this transformed image to the total pixel value of the edited area is higher than a threshold. If so, it is added to the derived input image set corresponding to this original test case. middle;

[0131] Then, the derived input image and the editing instructions for the original test case are combined to form a derived test case, attached. Figure 4 This example demonstrates an original test case and how it generates a derived test case.

[0132] Preferably, the editing area for the derived test cases obtained in step 3 is:

[0133] The output image of the user-returned derivative test cases is ,in It is the set of derived input images generated in step 2. The Middle A derived input image Input image editing system Obtained;

[0134] Similar to calculating the edit region of the original test case in step 2, calculate the edit region of all derived test cases in the derived test case set. Specifically, for the first... One derivative test case Its input image is The output image is Then its editing area for:

[0135] Derivative test cases Input image and output image The difference grayscale image is ;

[0136] Difference grayscale image Gaussian blurring was used to obtain derived test cases. Blur difference grayscale image ;

[0137] Blur difference grayscale image The derived test cases are obtained after binarization and smoothing. Smoothed difference grayscale image ;

[0138] Smoothed difference grayscale image Derived test cases are obtained after morphological closing operations and adaptive threshold binarization. The final difference grayscale image ;

[0139] Derivative test cases are calculated using edge detection methods. The final difference grayscale image At the edge, derived test cases are obtained. Editing area .

[0140] For the derived test case set Calculate the edit area of ​​all derived test cases to obtain .

[0141] In practical implementation, one specific method for implementing the above process using an instruction-guided image editing model is as follows:

[0142] For each derived test case Similar to calculating the edit region of the original test cases in step 1, the corresponding function provided by python-opencv is used to obtain the blur difference grayscale image. Smoothing difference grayscale image and the final difference grayscale image ;

[0143] Then, the findContours function is used to draw the final grayscale image of the difference. The edit region outline is defined, and the edit region mask image for each derived test case is obtained using the drawContours function. After obtaining the edit areas for all derived test cases, a set is obtained. .

[0144] The consistency judgment method between the original test cases and the derived test cases in step 3 is as follows:

[0145] Calculate the editing area of ​​the original test cases and derived test cases Editing area The degree of overlap, i.e., the crossover ratio of the two, is denoted as . :

[0146]

[0147] By comparing the original test cases and the derived test cases The editing area intersection ratio Does it exceed a predefined threshold? To determine whether the outputs of the original test cases and the derived test cases are consistent:

[0148]

[0149] Calculate whether the outputs of the test cases in the original test case set and the derived test case set are consistent to obtain the consistency set. ,in , indicating the original test case and the first Whether the outputs of the derived test cases are consistent.

[0150] In practical implementation, one specific method for implementing the above process using an instruction-guided image editing model is as follows:

[0151] For a pair of original test cases and derived test cases, the editing area for the original test cases is first obtained. and derived test cases Editing area Then, first calculate using the where function provided by NumPy. and AND matrix and OR matrix;

[0152] Then, the sum of pixel values ​​of the matrix and the matrix sum or matrix is ​​calculated using the sum function, and the result is divided to obtain the intersection-union ratio. Then With predefined threshold If a comparison is made, If the threshold is not exceeded, the outputs of the original test case and the derived test case are considered inconsistent; otherwise, the outputs are considered consistent.

[0153] Preferably, the method for determining whether the edit area of ​​any derived test case is consistent with that of the original test case in step 4 is as follows:

[0154]

[0155] when If the output of the original test case is consistent with the output of all derived test cases, then the output of the original test case is considered to have passed this verification. There is no need to perform the output repair step, and we can directly jump to step 5 to show the test results to the user.

[0156] when If the output of the original test case fails the verification, the output repair steps need to be performed.

[0157] The method for cross-validating the outputs of the original test cases and derived test cases described in step 4 is as follows:

[0158] For a derived test case and another derivative test case in the derivative test case set Similarly, the consistency of the outputs is determined by calculating the intersection-union ratio of the two editing regions:

[0159]

[0160] Calculate the set of derived test cases and the derived test cases. The number of derived test cases that are expected to remain consistent with the output is denoted as . :

[0161]

[0162] in For indicator functions, when the given expression is When the condition is met, the indicator function returns 1; otherwise, it returns 0.

[0163]

[0164] Step 4, which describes finding the optimal test case output, first requires finding one that makes... The largest derived test case index is denoted as :

[0165]

[0166] The first in the derivative test case set The best test case is selected from the derived test cases, denoted as _____. Its output image is .

[0167] In practical implementation, one specific method for implementing the above process using an instruction-guided image editing model is as follows:

[0168] For each derived test case Iterate through all test cases in the derived test set:

[0169] For another derivative test case in the derivative test case set Obtain derived test cases Editing area and derived test cases Editing area Then calculate using the where function provided by NumPy. and The sum of pixel values ​​in the AND and OR matrices is calculated using the SUM function, and the result is divided to obtain the intersection-union ratio (IUGR). ;

[0170] A dictionary records the derivative test cases in the derivative test case set. Consistent number of derived test cases After obtaining the number of all derived test cases, obtain the set that makes Largest derived test case index And obtain the first test case in the derived test case set. A number of derivative test cases.

[0171] Step 4 involves mapping the output of the optimal test case back to the original test case to obtain the repaired original test case output. for:

[0172] when At that time, the output image of the best test case is directly overlaid onto the corresponding position of the original test case output image. The corresponding value is either the original test case output image or the optimal test case output image, determined by the transformation matrix of the optimal test case. Decide:

[0173]

[0174] Location This is the location of the original test case output image. The corresponding position of the pixel in the output image of the best test case;

[0175] when Then, the first step is to process the stretched image. To restore, we perform a scaling operation in the same direction to obtain the restored image, denoted as... Then, the restored output image is overlaid onto the corresponding position of the original test case output image. Similarly, the position... The corresponding value is either the original test case output image or the optimal test case output image, determined by the transformation matrix of the optimal test case. Decide:

[0176]

[0177] Location This is the location of the original test case output image. The corresponding position of the pixel in the restored output image;

[0178] when At that time, in order to rotate the image To restore it, rotate your clock hand. The restored image is obtained. Then, the restored output image is overlaid onto the corresponding position of the original test case output image. Similarly, the position... The corresponding value is either the original test case output image or the optimal test case output image, determined by the transformation matrix of the optimal test case. Decide:

[0179]

[0180] Location This is the location of the original test case output image. The corresponding position of each pixel in the reconstructed output image.

[0181] In practical implementation, one specific method for implementing the above process using an instruction-guided image editing model is as follows:

[0182] when First, the resize function provided by NumPy is used to adjust the size of the optimal test case output image to the size of the shape enclosed by the top left and bottom right corners. These top left and bottom right corners are determined by the optimal test case transformation matrix. Then, NumPy features are used to cover the area enclosed by the top left and bottom right corners of the original test case output image with the optimal test case output image.

[0183] when When the best test case output image height is adjusted to the area from the upper bound to the lower bound using the resize function provided by NumPy, the width remains unchanged. This upper and lower bound is determined by the best test case transformation matrix. Then, using the features of NumPy, the output image of the best test case is overlaid on the area between the upper and lower bounds of the original test case output image height.

[0184] when At that time, the getRotationMatrix2D and warpAffine functions provided by python-opencv were used to process the input images of the original test cases. The image is rotated to obtain a rotated reconstruction image, with the angle being the opposite of the angle of the input image of the best test case. Then, using NumPy's where function, based on the transformation matrix of the best test case, if the transformation matrix value is 1 at a certain position, the pixel value of the reconstruction image is obtained from the rotated reconstruction image; otherwise, it is obtained from the output image of the original test case.

[0185] Preferably, step 5, which involves summarizing and presenting the inspection results and reporting the test results to the user, is as follows:

[0186] The image editing system is presented to the user through an interactive interface. Output image of the original test cases With the corresponding editing area and image editing systems For the generated Output image of each derivative test case With the corresponding editing area , ;

[0187] show Output image of each derivative test case The result of judging whether the output image is consistent with the original test case output image. And based on the result of judging whether the edit area of ​​any derived test case is consistent with that of the original test case. Test results are displayed:

[0188] like The system prompts the user to use the image editing system. If the output of the original test cases fails the check, the response is unreliable; otherwise, report to the image editing system. The output of the original test cases passed the inspection, and the answers are basically reliable.

[0189] Step 5, reporting the repair results to the user, is as follows:

[0190] The user interface is used to display the repaired output image of the original test case to the user, i.e., the repaired output image. .

[0191] The implementation of the various embodiments of the present invention is based on programmed processing through a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide a testing and output repair system for an instruction-guided image editing model based on metamorphic testing technology. This system is used to execute the testing and output repair method for the instruction-guided image editing model based on metamorphic testing technology in the above method embodiments.

[0192] The system includes: a first main module, used to collect a given original input image and editing instructions as original test cases, and simultaneously collect the output image of the original test cases, which is obtained by inputting the original test cases into the image editing system under test; comparing the input and output images of the original test cases to obtain a difference grayscale image, which is then processed by noise reduction to obtain a smooth difference image, and further calculating the editing area of ​​the original test cases; and a second main module, used to transform the input image of the original test cases according to the selected transformation relationship to generate a series of candidate image sets, select equivalent derived test images from the candidate image set, and combine them with the original editing instructions to form a series of new derived test images. The test cases are divided into five main modules: the third module, the fourth module, and the fifth module. The fifth module is used to collect the output images of all derived test cases, obtain the editing area of ​​the derived test cases by comparing the input and output images of the derived test cases, and determine whether the editing area of ​​all derived test cases is consistent with the editing area of ​​the original test cases by the output relationship. If the editing area of ​​any derived test case is inconsistent with the editing area of ​​the original test case, the module performs cross-validation of the output of all derived test cases with the output of the original test cases, selects the output of the best test case, and then maps it to the original test case as the original test output for repair. The sixth module is used to display the test and repair results.

[0193] The present invention provides a test and output repair system for an instruction-guided image editing model based on metamorphic testing technology. This system addresses the problem of applying metamorphic testing to verify and repair the reliability of image editing system output. It employs several modules, utilizes a large language model to generate appropriate editing instructions for the user-given image, designs appropriate metamorphic relationships to generate new test cases to verify the reliability of the image editing system output, and uses a multi-test case cross-validation method to repair the original output.

[0194] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is that corresponding functional modules are set. The principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art can improve the modules in the above system embodiments by referring to the specific technical solutions in other method embodiments and combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, on the basis of the above system embodiments, and on the premise of ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments.

[0195] In summary, this invention proposes a testing and output repair method for an instruction-guided image editing model based on metamorphic testing technology. First, the invention collects the user-provided original input image and editing instructions as original test cases, and simultaneously collects the output images of the user-provided original test cases. The editing region of the original test cases is obtained by comparing the input and output images. Second, based on the metamorphic relationship selected by the user, the input image of the original test cases is transformed to generate a series of candidate image sets. Without affecting the editing region of the original test cases, equivalent derived test images are selected and combined with the original editing instructions to form a series of new derived test cases, which are returned to the user. Third, the output images of all derived test cases returned by the user are collected. The editing region of the derived test cases is obtained by comparing the input and output images of the derived test cases, and the consistency between the editing regions of all derived test cases and the editing regions of the original test cases is determined by the output relationship. Fourth, all derived test cases are cross-validated with the outputs of the original test cases. The output of the best test case is selected and mapped onto the original test case as the original test output for repair. Finally, the test results and output repair results are summarized and returned to the user. This invention can test and repair the output of the model under test without the expected output of test cases, thereby enhancing the reliability of the model under test.

[0196] Those skilled in the art, upon learning the basic inventive concept, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this invention.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for testing and outputting repair of an instruction-guided image editing model based on metamorphic testing technology, characterized in that, include: Step 1: Collect the given original input image and editing instructions as the original test case, and collect the output image of the original test case, which is obtained by inputting the original test case into the image editing system under test; compare the input image and the output image of the original test case to obtain the difference grayscale image, and after noise reduction processing, obtain the smooth difference image, and further calculate the editing area of ​​the original test case; Step 2: Based on the selected metamorphic relationship, transform the input image of the original test case to generate a series of candidate image sets. Select equivalent derived test images from the candidate image sets and combine them with the original editing instructions to form a series of new derived test cases. Step 3: Collect the output images of all derived test cases, obtain the editing area of ​​the derived test cases by comparing the input and output images of the derived test cases, and determine whether the editing area of ​​all derived test cases is consistent with the editing area of ​​the original test cases by the output relationship; Step 4: If the editing area of ​​any derived test case is inconsistent with the editing area of ​​the original test case, cross-validate the outputs of all derived test cases with the original test cases, select the output of the best test case, and then map it onto the original test case as the original test output for repair. Step 5: Display the test and repair results.

2. The testing and output repair method for the instruction-guided image editing model based on metamorphosis testing technology according to claim 1, characterized in that, Step 1 also includes: Calculate the difference between the input and output images of the original test cases, and perform grayscale binarization to obtain the grayscale image of the difference between the input and output images of the original test cases. Gaussian blur is applied to the obtained difference grayscale image to obtain a blurred difference grayscale image; The blurred difference grayscale image is binarized again to obtain a smooth difference grayscale image between the input and output images of the original test case; A morphological closing operation is performed on the obtained smoothed difference grayscale image to obtain the final difference grayscale image; Edge detection methods are used to calculate the edges of the final difference grayscale image, thus obtaining the editing area of ​​the original test case.

3. The testing and output repair method for the instruction-guided image editing model based on metamorphosis testing technology according to claim 1, characterized in that, Step 2, based on the selected metamorphic relationship, transforms the input image of the original test case to generate a series of candidate image sets, and also includes: remember The metamorphosis relationship currently selected by the user; when At that time, the original input image is subjected to multiple different cropping transformations to form a candidate image set; when At that time, the original input image is subjected to multiple different stretching transformations along the vertical direction, and the resulting stretched images are used as a candidate image set. when At that time, rotate the original input image clockwise. The generated rotated images are used as a candidate image set.

4. The testing and output repair method for the instruction-guided image editing model based on metamorphosis testing technology according to claim 1, characterized in that, Step 2, which involves selecting an equivalent derived test image from the candidate image set, also includes: For all candidate images transformed from the original input image, the pixel changes during the image transformation process are recorded as a transformation matrix of the same size as the original input image and assigned values; For all candidate images, determine whether the edit region of the original test case disappears due to image transformation, and select the candidate images that do not cause the edit region to disappear as the set of derived input images obtained by transforming the original input image.

5. The testing and output repair method for the instruction-guided image editing model based on metamorphosis testing technology according to claim 1, characterized in that, Step 3, which determines whether the editing areas of all derived test cases are consistent with the editing areas of the original test cases based on the output relationship, also includes: Calculate the intersection-union ratio (IUU) of the edit regions of the original test cases and the edit regions of the derived test cases. ; Compare the crossover ratios Does it exceed a predefined threshold? If the output exceeds the limit, the original test case and the derived test case are considered to be consistent. ， Calculate whether the outputs of the test cases in the original test case set and the derived test case set are consistent to obtain the consistency set. .

6. The testing and output repair method for the instruction-guided image editing model based on metamorphosis testing technology according to claim 5, characterized in that, Step 4, determining whether the edit area of ​​any derived test case is consistent with the edit area of ​​the original test case, also includes: ， when If the test results are not found, proceed to step 5. when At that time, output repair will be performed.

7. The testing and output repair method for the instruction-guided image editing model based on metamorphosis testing technology according to claim 6, characterized in that, Step 4 involves cross-validating the outputs of the original test cases and derived test cases, which also includes: For a derived test case and another derivative test case in the derivative test case set The consistency of the outputs is determined by calculating the intersection-union ratio of the edited regions of the two entities. ， Calculate the set of derived test cases and the derived test cases. The output should consistently reflect the number of derived test cases.

8. The testing and output repair method for the instruction-guided image editing model based on metamorphosis testing technology according to claim 1, characterized in that, Step 4 also includes: when At that time, the output image of the best test case is directly overlaid on the corresponding position of the original test case output image; when When the image is stretched, it is restored, and the restored output image is overlaid on the corresponding position of the original test case output image; when When the rotation is complete, the rotated image is restored, and the restored output image is overlaid on the corresponding position of the original test case output image; Among them, when , or At time, position The value corresponding to the above is determined by the transformation matrix of the optimal test case, which determines whether the original test case output image or the optimal test case output image is used.

9. The testing and output repair method for the instruction-guided image editing model based on metamorphosis testing technology according to claim 1, characterized in that, Step 5 also includes: The interactive interface displays the output images and corresponding editing areas of the original test cases from the image editing system, as well as the output images and corresponding editing areas of several generated derivative test cases from the image editing system. Display the results of whether the output images of several derived test cases are consistent with the output image of the original test case, and show the test results based on whether the editing area of ​​any derived test case is consistent with the original test case: like If the system fails to pass the test for the original test cases, the user is prompted that the output of the image editing system is unreliable; otherwise, the system passes the test for the original test cases and the output is basically reliable.

10. A testing and output restoration system for a command-guided image editing model based on metamorphic testing technology, characterized in that, include: The first main module is used to collect the given original input image and editing instructions as the original test cases, and at the same time collect the output image of the original test cases, which is obtained by inputting the original test cases into the image editing system under test; compare the input image and the output image of the original test cases to obtain the difference grayscale image, and after noise reduction processing, obtain the smooth difference image, and further calculate the editing area of ​​the original test cases; The second main module is used to transform the input image of the original test case according to the selected metamorphic relationship to generate a series of candidate image sets, select equivalent derived test images from the candidate image sets, and form a series of new derived test cases together with the original editing instructions; The third main module is used to collect the output images of all derived test cases, obtain the editing area of ​​the derived test cases by comparing the input and output images of the derived test cases, and determine whether the editing area of ​​all derived test cases is consistent with the editing area of ​​the original test cases by the output relationship. The fourth main module is used to cross-validate the outputs of all derived test cases with the original test cases if the editing area of ​​any derived test case is inconsistent with the editing area of ​​the original test case, select the output of the best test case, and then map it onto the original test case as the original test output for repair. The fifth main module is used to display the test and repair results.

Images