Test and output restoration method and system of instruction-guided image editing model based on metamorphic test technology

Abstract

The invention discloses a test and output restoration method for an instruction-guided image editing model based on a metamorphic test technology, and the method comprises the steps: collecting an original input image and an editing instruction as an original test case, collecting an output image of the original test case, and comparing the input image with the output image to obtain an editing region of the original test case; converting the input image according to the metamorphic relation to generate a candidate image set, selecting an equivalent derivative test image, and forming a new derivative test case together with an original editing instruction; collecting output images of all the derivative test cases, comparing the input images with the output images to obtain editing areas of the derivative test cases, and judging whether the editing areas of all the derivative test cases are consistent with the editing areas of the original test cases or not; carrying out cross validation on all derivative test cases and the output of the original test case, selecting the output of the optimal test case, and mapping the output of the optimal test case to the original test case to serve as the restored original test output; and summarizing a test result and outputting a repair result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and particularly designs a test and output repair method and system for an instruction-guided image editing model based on a decay test technology. BACKGROUND

[0002] With the rapid development of computer vision technology, natural language processing technology and generative artificial intelligence, image editing models based on natural language instructions (hereinafter referred to as image editing systems) are becoming an important research direction in the field of computer vision. Such models deeply integrate semantic understanding and image generation technology, enabling ordinary users to use daily language to accurately control the image modification process, such as the instruction "Replace the cat in the image with dog." Compared with traditional methods that require professional software operation, instruction-guided image editing systems break through the tool use threshold, making creative expression no longer limited by technical complexity, and greatly improving the intelligent level of image processing. In practical application scenarios, this technology has shown extensive potential, such as Adobe's Firefly system supporting intelligent retouching through text instructions; Runway's Gen-2 model enabling text-driven video editing; in the medical imaging field, researchers are exploring the use of natural language instructions to assist in lesion labeling and visual analysis. With the evolution of open-source models such as Stable Diffusion, individual users can achieve professional-level image synthesis and style transfer through simple prompt words, which is reshaping the production method of digital content creation.

[0003] Currently, methods for verifying the correctness of the output images of image editing systems usually require human involvement in the construction of test instructions. Some verification methods directly use user surveys to verify the correctness of the model output, some verification methods require a reference editing image to be given, and then the similarity between the model output image and the reference image is calculated to verify the correctness of the model output image; while another part of the verification method also needs human to divide the editing area and the background area of a test case, and then calculate the similarity between the output image background area and the input image and the similarity between the output image editing area and the text of the editing instruction. Obviously, this testing process that requires human involvement is time-consuming and labor-intensive, and is not easy to apply on a large scale.

[0004] On the other hand, the output of the image editing system is often repaired by manual repair, retraining, etc., which requires manual annotation costs and hardware costs for retraining, and another part of the work also needs to be repaired by identifying and repairing suspicious neurons on the deep neural network. These works or cannot completely automate error positioning and repair of deep learning models, or need to rely on neural networks and code for white-box repair. Without access to the internal details of the model, these repair methods cannot conveniently repair errors in the output of the model.

[0005] In view of the above, a feasible solution is to apply metamorphic testing technology in the field of software testing to preliminarily check the correctness and repair the output of the image editing system without knowing the reference editing image. Metamorphic testing technology checks the performance of software on multiple interrelated inputs to determine whether the software output is likely to be incorrect. For example, when checking whether the output of a program for calculating the sine function sin is correct when the input is 120°, metamorphic testing does not directly check whether the program output is correct, but compares the program output with the output under the input of (180°-120°), -120°, etc. to determine whether the expected relationship is met, thereby preliminarily determining the reliability of the program output. Therefore, metamorphic testing can preliminarily check the reliability of the editing image output by the image editing system without knowing the standard answer, and can also preliminarily repair the output of the model based on this.

[0006] There are the following specific technical problems in applying metamorphic testing to the reliability verification and repair of the output of the image editing system:

[0007] There is no metamorphic relationship for the image editing system at present: no metamorphic relationship suitable for checking the reliability of the output of the image editing system has been found, so it is necessary to design a special metamorphic relationship for the characteristics of the input and output of the image editing system.

[0008] There is no method to verify the consistency of different outputs of the image editing system at present: the key of metamorphic testing is to determine whether the outputs of multiple corresponding test cases generated by a metamorphic relationship are consistent. If not, it is considered that the model output has an error. No method suitable for verifying the consistency of the output of the image editing system has been found, so it is necessary to design a method to verify the consistency according to the output characteristics of the model.

[0009] There is no method to automatically repair the output of the image editing system at present: the existing repair methods often need to understand the weights and deployment details of the model, and need to design a method to automatically repair the output of the image editing system in a black box. SUMMARY

[0010] In order to overcome the shortcomings of the prior art, the present application provides an output reliability check and repair method for an image editing system, which uses a large language model to generate appropriate editing instructions for a given image, designs appropriate metamorphic relationships to generate new test cases to verify the reliability of the output of the image editing system, and uses a multi-case cross-validation method to repair the original output.

[0011] According to an aspect of the present application, a test and output repair method for an image editing system based on metamorphic testing technology is provided, comprising:

[0012] Step 1: Collect the given original input image and editing instructions as the original test case, and collect the output image of the original test case, which is obtained by inputting the original test case into the image editing system under test; compare the input image and the output image of the original test case to obtain a difference gray image, and after noise reduction processing, obtain a smooth difference image, and further calculate the editing region of the original test case;

[0013] Step 2: According to the selected metamorphic relationship, transform the input image of the original test case to generate a series of candidate image sets, select equivalent derivative test images from the candidate image sets, and together with the original editing instructions, form a series of new derivative test cases;

[0014] Step 3: Collect the output images of all derivative test cases, compare the input images and the output images of the derivative test cases to obtain the editing regions of the derivative test cases, and judge whether the editing regions of all derivative test cases are consistent with the editing region of the original test case through the output relationship;

[0015] Step 4: If the editing region of any derivative test case is inconsistent with the editing region of the original test case, cross-verify the outputs of all derivative test cases with the output of the original test case, select the output of a best test case, and then map it to the original test case as the repaired original test output;

[0016] Step 5: Show the test and repair results.

[0017] As a further technical solution, step 1 further comprises:

[0018] Calculate the difference between the input image and the output image of the original test case, and perform gray scale binarization processing to obtain the difference gray image of the input image and the output image of the original test case;

[0019] Gaussian blur is performed on the obtained difference gray image to obtain a blurred difference gray image;

[0020] The blurred difference gray image is binarized again to obtain a smooth difference gray image of the input image and the output image of the original test case;

[0021] The smooth difference gray image is subjected to a morphological closing operation to obtain a final difference gray image;

[0022] An edge of the final difference gray image is calculated using an edge detection method to obtain an edit region of the original test case.

[0023] As a further technical solution, in step 2, a series of candidate image sets are generated by transforming the input image of the original test case according to the selected metamorphosis relationship, and further comprising:

[0024] Let be the metamorphosis relationship currently selected by the user;

[0025] When , the original input image is subjected to multiple different cropping transformations to serve as the candidate image set;

[0026] When , the original input image is subjected to multiple different stretching transformations along the vertical direction, and the generated stretching images serve as the candidate image set;

[0027] When , the original input image is rotated clockwise by degrees, and the generated rotation images serve as the candidate image set.

[0028] As a further technical solution, in step 2, the equivalent derivative test image is selected from the candidate image set, further comprising:

[0029] For all candidate images transformed from the original input image, the pixel change in the image transformation process is recorded as a transformation matrix with the same size as the original input image and is assigned a value;

[0030] For all candidate images, it is determined whether the edit region of the original test case disappears due to image transformation, and the candidate image that does not cause the edit region to disappear is selected as the derivative input image set obtained by transforming the original input image.

[0031] As a further technical solution, in step 3, it is determined whether the edit region of all derivative test cases is consistent with the edit region of the original test case by the output relationship, further comprising:

[0032] The intersection-over-union ratio of the edit region of the original test case and the edit region of the derivative test case is calculated ;

[0033] The intersection-over-union ratio is compared whether a predefined threshold is exceeded if exceeded, determine that the original test case and the derived test case output are consistent:

[0034]

[0035] calculate whether the test case outputs in the original test case and the derived test case set are consistent, and obtain a consistency set .

[0036] As a further technical solution, step 4 further includes:

[0037]

[0038] when , jump to step 5 to display the test results;

[0039] when , execute the output repair.

[0040] As a further technical solution, step 4 further includes:

[0041] For a derived test case and another derived test case in the derived test case set , whether the outputs are consistent is determined by calculating the intersection-over-union of the two editing regions:

[0042]

[0043] Calculate the number of derived test cases in the derived test case set that are expected to remain consistent with the derived test case .

[0044] As a further technical solution, step 4 further includes:

[0045] when , directly overlay the output image of the best test case onto the corresponding position of the original test case output image;

[0046] when , restore the stretched image, and overlay the restored output image onto the corresponding position of the original test case output image;

[0047] when , restore the rotated image, and overlay the restored output image onto the corresponding position of the original test case output image;

[0048] wherein when , or the position The corresponding value is the original test case output image or the best test case output image, which is determined by the transformation matrix of the best test case.

[0049] As a further technical solution, step 5 further comprises:

[0050] The output image of the original test case and the corresponding editing area of the image editing system are displayed through the interactive interface, and the output image of the generated several derivative test cases and the corresponding editing area of the image editing system are displayed;

[0051] The judgment result of whether the output image of the several derivative test cases is consistent with the output image of the original test case is displayed, and the test result is displayed according to the judgment result of whether the editing area of any derivative test case is consistent with the editing area of the original test case:

[0052] If , prompt the user that the output of the image editing system for the original test case does not pass the check, and the answer is unreliable; otherwise, report that the output of the image editing system for the original test case passes the check, and the answer is basically reliable.

[0053] According to an aspect of the present application, a test and output repair system for guiding image editing model based on transmutation test technology is provided, comprising:

[0054] The first main module is used for collecting a given original input image and editing instruction as an original test case, and collecting an output image of the original test case, which is obtained by inputting the original test case into the image editing system under test; comparing the input image and the output image of the original test case to obtain a difference gray image, and obtaining a smooth difference image after noise reduction processing, and further calculating to obtain an editing area of the original test case;

[0055] The second main module is used for transforming the input image of the original test case to generate a series of candidate image sets according to the selected transmutation relationship, selecting equivalent derivative test images from the candidate image sets, and composing a series of new derivative test cases together with the original editing instruction;

[0056] The third main module is used for collecting the output images of all derivative test cases, obtaining the editing area of the derivative test case by comparing the input image and the output image of the derivative test case, and judging whether the editing area of all derivative test cases is consistent with the editing area of the original test case through the output relationship;

[0057] a fourth main module for cross verifying all the derived test cases with the output of the original test case if the edit region of any derived test case is not consistent with the edit region of the original test case, selecting an output of a best test case, and then mapping the output to the original test case as the repaired original test output;

[0058] a fifth main module for displaying the test and repair results.

[0059] Compared with the prior art, the present application has the beneficial effects that:

[0060] 1. The present application can reliably test and repair the output image of an image editing system without the need for manual intervention to obtain the expected output of the test oracle or test case. First, the original test case given by the user and the output image of the image editing system are collected, and the edit region of the original test case is obtained by calculating the difference gray image of the input image and the output image of the original test case. Then, the input image of the original test case is transformed according to the metamorphic relationship selected by the user to generate a candidate test case set, and further selection is performed to obtain a derived test case set. Then, all the derived test cases given by the user and the output image given by the image editing system are collected again, and the edit region of all the derived test cases is obtained by calculating the difference gray image of the input image and the output image of all the derived test cases. Subsequently, the edit regions of the original test case and all the derived test cases are compared to determine whether the outputs of the original test case and the derived test cases are consistent. If there is an inconsistency, a best test case is selected through cross verification, and the output of the best test case is mapped to the original test case. Finally, the output test and repair situation is summarized and reported to the user. The idea of the method is that under the condition that the edit instruction remains unchanged and the input image edit region is not affected, the original test case and the derived test case should have consistent behavior, i.e., the edit region should remain consistent. If the edit region of the image editing system for the derived test case is inconsistent with that of the original test case, it means that at least one of the outputs of the image editing system for the original test case and the derived test case is incorrect.

[0061] 2. The method provided by the present application is based on a novel responsive metamorphic relationship, i.e., the generation of a derived test case not only needs an original test case, but also needs the output of the original test case to assist. The reliability of the image editing system is checked by checking the output consistency of the image editing system for the original test case and each derived test case. The process does not need manual intervention throughout, and can be automatically run and returned through a script. This solves the problem that the existing image editing system test method based on traditional test oracle needs manual intervention, is time-consuming and laborious, and cannot provide immediate checking during the use of the image editing system.

[0062] 3. The method provided by the present application solves the problem of no suitable transmutation relationship when applying the transmutation test technology to the image editing system, thereby making it possible to automatically test the image editing system; a method for calculating the image editing test case editing area is designed, thereby solving the problem of no suitable method for verifying the output consistency of the image editing system; and a method for repairing the output of the image editing system based on multi-case cross verification is designed, thereby solving the problem of no method for automatically repairing the output of the image editing system in a black box manner.

[0063] 4. The method formed by the present application can ultimately preliminarily check the reliability of the output image of the image editing system in the scenario where the user does not need to participate in the process of using the image editing system, and repair the unreliable answer, thereby achieving the purpose of testing and repairing the output returned by the system without obtaining the reference output image of the image editing system in advance or the precondition. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0065] Figure 1 The method flowchart provided by the embodiment of the present application is shown.

[0066] Figure 2 The example display diagram of the input image and the output image of the comparative test case calculated by the embodiment of the present application is shown.

[0067] Figure 3 The transmutation relationship example display diagram of the input image of the original test case provided by the embodiment of the present application is shown.

[0068] Figure 4 The example display diagram of the original test case provided by the embodiment of the present application is shown.

[0069] Figure 5 The example display diagram of the general question generation method provided by the embodiment of the present application is shown.

[0070] Figure 6 The example display diagram of the output repair of the original test case provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0071] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present application. In addition, the technical features in each of the embodiments or in a single embodiment provided by the present application can be combined with each other at will to form new technical solutions, and the combination is not restricted by the order of steps and / or structure mode, but should be based on the realization by those of ordinary skill in the art. When the combination of technical solutions appears contradictory or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the protection scope of the present application.

[0072] The present application aims to reliably test and repair the output images of the image editing system without manual intervention to obtain the expected output of the test oracle or test case, so as to alleviate the problems that the current image editing system test needs manual intervention, the test process is time-consuming and laborious, is not easy to be applied on a large scale, and the current output repair method depends on model details and cannot conveniently repair the output. The present application designs a novel responsive metamorphic relationship based on metamorphic testing technology, and designs a method for calculating the editing area of the image editing test case to check the consistency of the output of the original test case and the derived test case, and also designs an image editing system output repair method based on multi-case cross-validation to automatically repair the output of the image editing system in a black box, so that the user can check the reliability of the output image of the image editing system in real time without manual intervention.

[0073] The specific embodiments of the present application are described below in combination with Figures 1 to 6 The specific embodiments of the present application are described below in combination with Figure 1 , including:

[0074] Step 1: Collect the original input image and editing instruction given by the user as the original test case, and collect the output image of the original test case given by the user, which is obtained by inputting the original test case into the image editing system to be tested by the user. Then, by comparing the input image and the output image of the original test case, a difference gray image is obtained, and after noise reduction processing, a smooth difference image is obtained, and then the editing area of the original test case is further calculated;

[0075] As a preferred, the original test case given by the user and the output image returned by the image editing system to be tested for the user given input in step 1 are:

[0076] The original test case input by the user is , wherein the original input image is , and the editing instruction is ;

[0077] The image editing system to be tested is: ;

[0078] The output edited image returned by the image editing system for the input original test case is: .

[0079] The difference gray image of the original input image and the original output image is:

[0080] The difference gray image of the original input image and the original output image is obtained by calculating the difference between and and performing gray scale binarization processing. .

[0081] In the specific implementation process, a specific method for implementing the above process by the instruction-guided image editing model is:

[0082] The function absdiff provided in the python-opencv library is used to calculate the difference image of the input image and the output image, and then the Korean cvtColor is used to convert the difference image into a gray image, thereby obtaining the difference gray image of the input image and the output image , as shown in FIG. 8 (c) is the difference gray image of the input image Figure 2 (a) and the output image Figure 2 (b). Figure 2

[0083] The smoothed difference gray image of the original test case input image and the output image in step 1 is:

[0084] First, the difference gray image of the original test case is subjected to Gaussian blur, that is, a Gaussian smoothing operator kernel is used to reduce the prominence of isolated pixel points, thereby obtaining a blurred difference gray image :

[0085]

[0086] Then, the blurred difference gray image is subjected to binarization processing again to eliminate the influence of sporadic noise, thereby obtaining a smoothed difference gray image of the input image and the output image of the original test case :

[0087] ​

[0088] wherein, is the fuzzy difference gray image of the original test case the pixel value in the middle position wherein, )), ))). is the fuzzy difference gray image of the original test case is obtained according to the adaptive image binarization threshold calculation method Otsu algorithm:

[0089] For any candidate threshold , all pixels in the fuzzy difference image of the original test case are divided into two categories and :

[0090]

[0091] Then, the inter-class tolerance of the two pixel sets in the image under the threshold is calculated:

[0092]

[0093] wherein, is the proportion of pixels classified as in the fuzzy difference gray image , is the proportion of pixels classified as in the fuzzy difference gray image , is the mean value of pixels in , is the mean value of pixels in , and for all possible candidate thresholds , the threshold that maximizes the inter-class tolerance is selected:

[0094]

[0095] In the specific implementation process, a specific method for implementing the above process by the instruction-guided image editing model is:

[0096] The difference gray image is Gaussian blurred using the function GaussianBlur provided in the python-opencv library, wherein ksize is set to (5, 5), to obtain the fuzzy difference gray image ;

[0097] The function threshold is used to calculate the fuzzy difference gray image​ Adaptive thresholding for binarization where type is set to cv2.THRESH_BINARY+cv2.THRESH_OTSU, means that the adaptive threshold is calculated by otsu algorithm, and then the fuzzy difference gray image is binarized according to this threshold Binarization is performed to obtain the smoothed difference gray image of the input image and the output image of the original test case .

[0098] The editing area of the original test case in step 1 is:

[0099] The smoothed difference gray image of the input image and the output image of the original test case is Morphological closing operation is performed, which uses a structure operator to perform the operation of first expansion and then corrosion on the image to fill small holes in it to obtain the final difference gray image :

[0100]

[0101] where represents the structure operator used for morphological closing operation on the smoothed difference gray image, that is, a convolution kernel of 5 . represents the expansion operation, specifically, when is slid to position on the image , if some part overlaps the image position, the position is added to the expanded set :

[0102]

[0103] represents the corrosion operation, specifically, only when is completely covered in some part of , the corresponding position is added to the eroded set :

[0104]

[0105] Finally, the edge detection method is used to calculate the edge of the final difference gray image , and the editing area of the original test case is obtained .

[0106] In the specific implementation process, a specific method for implementing the above process by the instruction guided image editing model is as follows:

[0107] First, define a matrix of 5x5 using the ones function provided by numpy, where all values are 1, and then use the morphEx function provided by python-opencv to perform a closing operation on the smoothed difference gray image , where the parameter op is defined as cv2.MORPH_CLOSE, indicating that the closing operation needs to be performed, to obtain the final difference gray image . Then use the findContours function provided by python-opencv to draw the editing area contour in the final difference gray image , and use the drawContours function to obtain the editing area mask image

[0108] . The full flowchart of obtaining the editing area of a test case from the input image and output image of the test case is shown in the accompanying , where

[0109] (d) shows the editing area of the test case, which is the part enclosed by the green box. Figure 2 Figure 2 As a preferred embodiment, the original input image in step 2 is transformed to generate a set of candidate images

[0110] , which are:

[0111] Let be the current user-selected transmutation relationship;

[0112] When , the original input image is subjected to multiple different cropping transformations to generate a set of candidate images, i.e. , where represents the pre-defined number of generated candidate images;

[0113] When , the original input image is subjected to multiple different stretching transformations along the vertical direction. To ensure that the transformed images are not distorted, the stretching transformation requires that the proportion of the image to be preserved in the original input image cannot be less than 0.6. The generated n stretching images are used as a set of candidate images, i.e. .

[0114] When , the original input image​​ Clockwise rotation Similarly, in order to ensure the authenticity of the transformed image, this rotation transformation requires the image to be rotated by an angle of The generated n rotated images are used as the candidate image set, that is, .

[0115] In a specific implementation process, a specific method for implementing the above process for the instruction-guided image editing model is:

[0116] First, the transformation relationship selected by the user is obtained through the interactive interface ;

[0117] when When , a series of upper left corner points and lower right corner points are randomly selected, and then the input image of the original test case is directly matched according to each pair of upper left corner and lower right corner points. Crop and capture, and then use the resize function provided by python-opencv to modify the cropped image size and Keep consistent and get the candidate image set of cropping transformation ;

[0118] when When , a series of upper and lower boundary points are randomly selected along the vertical axis of the image, and then the input image of the original test case is intercepted The pixel values ​​above the upper limit and below the lower limit are then modified using the resize function to obtain the image size. Keep it consistent to form a stretching effect and get a set of candidate images for stretching transformation ;

[0119] when When, first Randomly select a series of rotation angles within the range, and then use the getRotationMatrix2D function and warpAffine function provided by python-opencv to transform the input image of the original test case Perform the rotation operation to obtain a set of candidate images for rotation transformation .

[0120] Attachment Figure 3 The following figure shows an example of transforming the input image of an original test case. Figure (a) is a given input image, Figure (b) is an example of cropping the input image, Figure (c) is an example of stretching the input image, and Figure (d) is an example of rotating the input image.

[0121] Step 2: From the candidate image set Select the equivalent derivative test image for:

[0122] For all the original input images Transformed candidate images , records the pixel changes during the image transformation process as a Transformation matrices of the same size ,if In position The corresponding pixel is exist, In position The corresponding value is 1, otherwise In position The corresponding value is 0:

[0123]

[0124] For the candidate image set All candidate images in , judge whether the editing area of ​​the original test case disappears due to image transformation, and then select the candidate image that does not make the editing area disappear as the original input image The set of derivative input images obtained by transformation :

[0125]

[0126] in is a predefined adjustable threshold.

[0127] Derivative test cases generated in step 2 It consists of the generated derivative input image and the editing instructions of the original test case:

[0128]

[0129] In a specific implementation process, a specific method for implementing the above process for the instruction-guided image editing model is:

[0130] For any transformed image in the candidate image set, record the changes of each pixel during the transformation process, and define a numpy structure matrix to record the retained pixel values ​​to obtain the change matrix of this transformed image , and then calculate the edit area of ​​the original test case through the where function provided by numpy and The sum matrix is ​​then calculated using the sum function. the total pixel value of the transformed image, and finally, whether the proportion of the pixels left by the transformed image in the total pixel value of the editing region is higher than a threshold value, if yes, add to the derived input image set corresponding to the original test case ;

[0131] Then, the derived input image and the editing instruction of the original test case are combined to form a derived test case, and the derived test case is attached to the original test case Figure 4 An original test case and an example of generating a derived test case therefrom are shown.

[0132] As preferred, the editing region of the derived test case obtained in step 3 is:

[0133] The output image of the derived test case returned by the user is , wherein is the derived input image set generated in step 2 , the th derived input image is input into the image editing system under test to obtain the output image ;

[0134] , the th derived test case in the derived test case set is , the input image is , and the output image is , and the editing region of the derived test case is:

[0135] The difference gray image of the input image and the output image of the derived test case is ;

[0136] The difference gray image is subjected to Gaussian blur to obtain the blurred difference gray image of the derived test case ;

[0137] The blurred difference gray image is subjected to binaryzation and smoothing operation to obtain the smoothed difference gray image of the derived test case ;

[0138] The smoothed difference gray image is subjected to morphological closing operation and adaptive threshold binaryzation to obtain the final difference gray image of the derived test case ;

[0139] Use edge detection methods to calculate derived test cases The final difference grayscale image The edge of the derivative test case Editing area .

[0140] For derived test case sets , calculate the editing area of ​​all derived test cases to get .

[0141] In a specific implementation process, a specific method for implementing the above process for the instruction-guided image editing model is:

[0142] For each derived test case , similar to calculating the editing area of ​​the original test case in step 1, use the corresponding function provided by python-opencv to obtain the fuzzy difference grayscale image , smoothed difference grayscale image and the final difference grayscale image ;

[0143] Then use the findContours function to draw the final difference grayscale image The edit area outline in , and use the drawContours function to get the edit area mask image of each derived test case , get the editing area of ​​all derived test cases and get a collection .

[0144] The consistency determination method for the original test case and the derived test case in step 3 is as follows:

[0145] Compute the edit area of ​​the original test case and derived test cases Editing area The overlap degree of the two is calculated, that is, the intersection ratio of the two is recorded as :

[0146]

[0147] By comparing the original test case and the derived test case Intersection-over-Union of Editing Regions Whether it exceeds the predefined threshold , to determine whether the output of the original test case and the derived test case are consistent:

[0148]

[0149] Consistency set is obtained by calculating whether the outputs of the test cases in the original test case and the derived test case set are consistent , wherein represents whether the outputs of the original test case and the first derived test case are consistent.

[0150] In the implementation process, a specific method for implementing the above process by the instruction-guided image editing model is as follows:

[0151] For a pair of original test case and derived test case, the editing region of the original test case and the editing region of the derived test case are obtained first. Then, the and matrix sum or matrix of and are calculated first by using the where function provided by numpy.

[0152] Then, the pixel value sum of the and matrix sum or matrix is calculated by using the sum function, and the intersection-over-union ratio is obtained by division. Then, the is compared with a predefined threshold . If the does not exceed the threshold, it is considered that the outputs of the original test case and the derived test case are inconsistent, otherwise, it is considered that the outputs are consistent.

[0153] As a preferred, the method for judging whether the editing region of any derived test case is consistent with the original test case in step 4 is as follows:

[0154]

[0155] When , it is considered that the output of the original test case is consistent with the outputs of all derived test cases, and it is considered that the output of the original test case passes the verification, and the output repair step is not needed, and the test result is directly shown to the user in step 5.

[0156] When , it is considered that the output of the original test case does not pass the verification, and the output repair step needs to be performed.

[0157] The method for cross-verifying the outputs of the original test case and the derived test case in step 4 is as follows:

[0158] For a derived test case and another derived test case in the derived test case set , whether the outputs are consistent is also judged by calculating the intersection-over-union ratio of the editing regions of the two test cases.

[0159]

[0160] Calculate the derived test case set and the derived test case The number of derived test cases whose outputs are expected to be consistent is denoted as :

[0161]

[0162] in is the indicator function. When the expression given is , the result returned by the indicator function is 1, otherwise it is 0:

[0163]

[0164] To find the best test case output as described in step 4, we first need to find The largest derivative test case index is denoted by :

[0165]

[0166] The first The derived test cases are taken as the best test cases, denoted as , the output image is .

[0167] In a specific implementation process, a specific method for implementing the above process for the instruction-guided image editing model is:

[0168] For each derived test case , iterate over all test cases in the derived test collection:

[0169] For another derived test case in a derived test case set , get the derived test case Editing area and derived test cases Editing area , and then calculate it using the where function provided by numpy and The sum function is used to calculate the sum of the pixel values ​​of the matrix and the matrix, and the intersection and union ratio is obtained by dividing them. ;

[0170] Record the derived test case collection and the derived test case through a dictionary Consistent number of derived test cases After obtaining the number of all derived test cases, obtain the set that makes Maximum derived test case index , and the th derived test case in the derived test case set is obtained.

[0171] The output of the best test case is mapped to the original test case in step 4, and the repaired original test case output is obtained :

[0172] When , the output image of the best test case is directly overlaid on the corresponding position of the original test case output image, and the corresponding value at the position is determined by the transformation matrix of the best test case: :

[0173]

[0174] Wherein the position is the corresponding position of the pixel at the position of the original test case output image in the output image of the best test case.

[0175] When , the stretched image needs to be restored first, that is, a scaling operation in the same direction is performed to obtain a restored image, denoted as , and then the restored output image is overlaid on the corresponding position of the original test case output image. Similarly, the corresponding value at the position is determined by the transformation matrix of the best test case: :

[0176]

[0177] Wherein the position is the corresponding position of the pixel at the position of the original test case output image in the restored output image.

[0178] When , the rotated image needs to be restored by rotating it by degrees to obtain a restored image , and then the restored output image is overlaid on the corresponding position of the original test case output image. Similarly, the corresponding value at the position is determined by the transformation matrix of the best test case: :

[0179]

[0180] Where Is the original test case output image location The corresponding position of the pixel in the restored output image.

[0181] In a specific implementation process, a specific method for implementing the above process for the instruction-guided image editing model is:

[0182] when When resizing the best test case output image, first use the resize function provided by numpy to resize the best test case output image to the shape enclosed by the upper left corner and lower right corner points. The upper left corner and lower right corner points are determined by the best test case transformation matrix. Then, directly use the characteristics of numpy to overwrite the best test case output image to the area enclosed by the upper left corner points and lower right corner points of the original test case output image.

[0183] when When the image height of the best test case output is adjusted to the area from the upper bound to the lower bound, the width remains unchanged. The upper and lower bounds are determined by the best test case transformation matrix. Then, the features of numpy are used directly to overwrite the output image of the best test case to the area between the upper and lower bounds of the original test case output image height.

[0184] when When using the getRotationMatrix2D function and warpAffine function provided by python-opencv to the input image of the original test case The restored image is rotated to obtain the reverse angle of the best test case input image. Then, the where function of numpy is used as the basis for the transformation matrix of the best test case. If the transformation matrix value at a certain position is 1, the pixel value of the repaired image is obtained from the rotated restored image, otherwise it is obtained from the output image of the original test case.

[0185] Preferably, the summary and display of the inspection results and reporting of the test results to the user in step 5 is as follows:

[0186] Present the image editing system to users through an interactive interface Output image for the original test case Corresponding editing area , and image editing systems For generated Output image of the derived test case Corresponding editing area , ;

[0187] displaying output images of the individual derived test cases a result of a determination of whether the output is consistent with the output of the original test case , and a result of a determination of whether the edit region of any derived test case is consistent with the original test case showing the test results:

[0188] If , prompting the user to the image editing system the output of the original test case does not pass the check, the answer is unreliable; otherwise, reporting the image editing system the output of the original test case passes the check, the answer is basically reliable.

[0189] Step 5 reports the repair result to the user as:

[0190] showing the repair result of the output image of the original test case to the user through the interactive interface, i.e. the repair output image .

[0191] The implementation basis of each embodiment of the present application is realized by programmed processing of a device with processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present application are packaged into various modules. Based on this actual situation, on the basis of each of the above embodiments, the embodiments of the present application provide a test and output repair system of an instruction guided image editing model based on transmutation testing technology, which is used to execute the test and output repair method of the instruction guided image editing model based on transmutation testing technology in the above method embodiment.

[0192] The system comprises: a first main module for collecting a given original input image and editing instructions as an original test case, and collecting an output image of the original test case, which is obtained by inputting the original test case into the image editing system under test; comparing the input image and the output image of the original test case to obtain a difference gray image, and obtaining a smooth difference image after noise reduction processing, and further calculating to obtain an editing region of the original test case; a second main module for transforming the input image of the original test case according to a selected metamorphosis relationship to generate a series of candidate image sets, selecting equivalent derivative test images from the candidate image sets, and combining the equivalent derivative test images with the original editing instructions to form a series of new derivative test cases; a third main module for collecting output images of all the derivative test cases, comparing the input images and the output images of the derivative test cases to obtain editing regions of the derivative test cases, and judging whether the editing regions of all the derivative test cases are consistent with the editing region of the original test case through an output relationship; a fourth main module for, if the editing region of any derivative test case is inconsistent with the editing region of the original test case, cross- verifying the outputs of all the derivative test cases and the original test case, selecting an output of a best test case, and then mapping the output of the best test case to the original test case as a repaired original test output; and a fifth main module for displaying test and repair results.

[0193] The test and output repair system for the instruction guided image editing model based on the metamorphic test technology provided by the embodiment of the application faces the problem that metamorphic test is applied to image editing system output reliability verification and repair, adopts the foregoing modules, generates appropriate editing instructions for the image given by a large language model, designs appropriate metamorphic relationships to generate new test cases for image editing system output reliability verification, and uses a multi-case cross-verification method to repair the original output.

[0194] It should be noted that the system embodiments provided by the application are used to implement the methods in the method embodiments, and are also used to implement the methods in other method embodiments provided by the application. The difference is only that the corresponding functional modules are set, and the principle is basically the same as that of the foregoing system embodiments provided by the application. As long as the person skilled in the art improves the modules in the foregoing system embodiments by combining technical features to obtain corresponding technical means and technical solutions composed of these technical means on the premise of ensuring the practicability of the technical solutions, the corresponding system embodiments are obtained, which are used to implement the methods in other method embodiments.

[0195] Based on the above embodiments, the application proposes a test and output repair method of an instruction guide image editing model based on metamorphic testing technology. First, the application collects the original input image and editing instruction given by the user as the original test case, collects the output image of the original test case given by the user, and obtains the editing area of the original test case by comparing the input image and the output image of the original test case. Second, according to the metamorphic relationship selected by the user, a series of candidate image sets are generated by transforming the input image of the original test case, and under the premise of not affecting the editing area of the original test case, an equivalent derivative test image is selected, which is combined with the original editing instruction to form a series of new derivative test cases and returned to the user. Third, the output images of all the derivative test cases returned by the user are collected, the editing area of the derivative test case is obtained by comparing the input image and the output image of the derivative test case, and whether the editing area of all the derivative test cases is consistent with the editing area of the original test case is judged through the output relationship. Fourth, the outputs of all the derivative test cases and the original test case are cross-verified, the output of a best test case is selected, and then it is mapped to the original test case as the repaired original test output. Finally, the test results and the output repair results are returned to the user. The application can test and repair the output of the tested model without the expected output of the test case, so as to enhance the reliability of the tested model.

[0196] Those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all the changes and modifications falling within the scope of the application.

[0197] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the application.

Claims

1. A method for testing and outputting repairs of an instruction-guided image editing model based on a metamorphic testing technique, characterized in that: include: Step 1: Collect the given original input image and editing instructions as the original test case, and also collect the output image of the original test case, which is obtained by inputting the original test case into the tested image editing system; compare the input image and output image of the original test case to obtain a difference grayscale image, and then perform noise reduction to obtain a smooth difference image, and further calculate the editing area of ​​the original test case; Step 2: Based on the selected transformation relationship, the input image of the original test case is transformed to generate a series of candidate image sets. Equivalent derivative test images are selected from the candidate image sets and combined with the original editing instructions to form a series of new derivative test cases. Step 3: Collect the output images of all derived test cases, obtain the edited regions of the derived test cases by comparing their input and output images, and determine whether the edited regions of all derived test cases are consistent with the edited regions of the original test cases through the output relationship. Step 4: If the edited region of any derived test case is inconsistent with the edited region of the original test case, cross-validate the outputs of all derived test cases with the original test case, select the output of the best test case, and then map it to the original test case as the repaired original test output; Step 5: Display the test and repair results.

2. The method for testing and outputting a repair command-guided image editing model based on the degradation testing technology according to claim 1, characterized in that: Step 1 also includes: Calculate the difference between the input image and the output image of the original test case, and perform grayscale binarization processing to obtain the difference grayscale image of the input image and the output image of the original test case; Performing Gaussian blur on the obtained difference grayscale image to obtain a blurred difference grayscale image; The fuzzy difference grayscale image is binarized again to obtain a smooth difference grayscale image of the input image and output image of the original test case; Perform morphological closing operation on the obtained smooth difference grayscale image to obtain the final difference grayscale image; The edge detection method is used to calculate the edges of the final difference grayscale image and obtain the edited area of ​​the original test case.

3. The method for testing and outputting repair of an instruction-guided image editing model based on the degradation testing technology according to claim 1, characterized in that: In step 2, the input image of the original test case is transformed according to the selected transformation relationship to generate a series of candidate image sets, which also includes: remember The transformation relationship currently selected by the user; when When , the original input image is subjected to multiple different cropping transformations and used as a candidate image set; when When , the original input image is subjected to multiple different stretching transformations along the vertical direction, and the generated stretched images are used as the candidate image set; when When , the original input image is rotated clockwise The generated several rotated images are used as candidate image sets.

4. The method for testing and outputting repair of an instruction-guided image editing model based on the degradation testing technology according to claim 1, characterized in that: Selecting an equivalent derivative test image from the candidate image set in step 2 also includes: For all candidate images transformed from the original input image, the pixel changes during the image transformation process are recorded as a transformation matrix with the same size as the original input image and assigned; For all candidate images, determine whether the edited area of ​​the original test case disappears due to image transformation, and select candidate images that do not make the edited area disappear as the derived input image set obtained by transforming the original input image.

5. The method for testing and outputting repair of an instruction-guided image editing model based on the degradation testing technology according to claim 1, characterized in that: In step 3, the output relationship is used to determine whether the editing areas of all derived test cases are consistent with the editing areas of the original test cases, which also includes: Calculate the intersection of the original test case edit region and the derived test case edit region. ; Compare the intersection-union ratio Whether it exceeds the predefined threshold If it exceeds, the original test case and the derived test case output are consistent: ， Calculate whether the test case outputs in the original test case and the derived test case set are consistent, and obtain the consistency set .

6. The method for testing and outputting repair of an instruction-guided image editing model based on the degradation testing technology according to claim 5, characterized in that: In step 4, determining whether the editing area of ​​any derived test case is consistent with the editing area of ​​the original test case also includes: ， when When , skip to step 5 to display the test results; when , performs output repair.

7. The method for testing and outputting repair of an instruction-guided image editing model based on the degradation testing technology according to claim 6, characterized in that: The cross-validation of the outputs of the original test case and the derived test case in step 4 also includes: For a derived test case and another derived test case in the derived test case collection , we can determine whether the outputs are consistent by calculating the intersection-over-union ratio of the two edited regions: ， Calculate the derived test case set and the derived test case Output the number of derived test cases that are expected to remain consistent.

8. The method for testing and outputting repair of an instruction-guided image editing model based on the degradation testing technology according to claim 1, characterized in that: Step 4 also includes: when When , the output image of the best test case is directly overlaid on the corresponding position of the output image of the original test case; when , the stretched image is restored and the restored output image is overlaid on the corresponding position of the original test case output image; when , the rotated image is restored and the restored output image is overlaid on the corresponding position of the original test case output image; Among them, when 、 or Time, location Whether the corresponding value above is the original test case output image or the best test case output image is determined by the transformation matrix of the best test case.

9. The method for testing and outputting repair of an instruction-guided image editing model based on the degradation testing technology according to claim 1, characterized in that: Step 5 also includes: Displaying the output image and corresponding editing area of ​​the image editing system for the original test case, and the output images and corresponding editing areas of the image editing system for several generated derivative test cases through an interactive interface; Displays the results of the judgment on whether the output images of several derived test cases are consistent with the output images of the original test cases, and displays the test results based on whether the editing area of ​​any derived test case is consistent with that of the original test case: like , prompting the user that the output of the image editing system for the original test case has not passed the inspection and the answer is unreliable; otherwise, it reports that the output of the image editing system for the original test case has passed the inspection and the answer is basically reliable.

10. A test and output repair system for an instruction-guided image editing model based on metamorphic testing technology, characterized in that: include: The first main module is configured to collect a given original input image and editing instructions as an original test case, and simultaneously collect an output image of the original test case, the output image being obtained by inputting the original test case into the tested image editing system; compare the input image and the output image of the original test case to obtain a difference grayscale image, perform noise reduction processing to obtain a smoothed difference image, and further calculate the editing area of ​​the original test case; The second main module is used to transform the input image of the original test case according to the selected transformation relationship to generate a series of candidate image sets, select equivalent derived test images from the candidate image sets, and form a series of new derived test cases together with the original editing instructions; The third main module is used to collect the output images of all derived test cases, obtain the editing areas of the derived test cases by comparing the input images and output images of the derived test cases, and determine whether the editing areas of all derived test cases are consistent with the editing areas of the original test cases through the output relationship; The fourth main module is used to cross-validate the outputs of all derived test cases with the original test cases if the edited region of any derived test case is inconsistent with the edited region of the original test case, select the output of the best test case, and then map it to the original test case as the repaired original test output; The fifth main module is used to display test and repair results.

Images