An artificial intelligence component-oriented test case generation method

Abstract

The application discloses a kind of test case generation methods for artificial intelligence component, belong to image processing field;Specifically: first, for original picture, based on conditional generative adversarial network cGAN, using one-hot encoding splicing to original picture generates test sample;Then, through the loss function of cGAN, adopt fuzzy test method to add disturbance item to the generated test sample and expand;Through latent space sampling constraint and training constraint, and design benefit function to estimate the quality of the test sample set after expansion, guarantee sample set to meet the test requirements. Then, build metamorphic relationship to reduce the sample set that meets the requirements, generate the final test case for artificial intelligence component;Finally, the reliability, sufficiency and coverage of three indexes are used to verify the final generated test case;The application can fully cover the functional constraints of the component to be tested, can test the functional constraints of the component to be tested, and can also test the functional constraints of the component to be tested.

Description

Technical Field

[0001] This invention belongs to the field of test case generation for image processing, specifically a method for generating test cases for artificial intelligence components. Background Technology

[0002] Artificial intelligence (AI) technology has been a focal point of computer science research in recent years. With the improvement of computing power, AI is now widely used in image recognition, natural language processing, and other fields. In specific scenarios such as autonomous driving, facial recognition, speech recognition, and medical assistance, AI has unparalleled advantages. In the era of big data, AI can extract valuable information from large amounts of diverse, low-value-density data quickly, efficiently, and cost-effectively, and its related achievements have been widely applied in all aspects of life.

[0003] In traditional software system development, testing is a crucial step, directly impacting the software's ability to function correctly after deployment. With the rapid development of the internet, software has grown to an unimaginable scale, and the consequences of software problems are unpredictable. In safety-critical fields such as aviation, aerospace, healthcare, and autonomous driving, testing involves not only significant economic interests but also the safety of users; therefore, thorough testing is of paramount importance.

[0004] Traditional software is algorithm-driven; the internal logic of the software system is determined by the algorithm. Traditional testing techniques, based on test coverage criteria, can identify algorithmic defects by rigorously covering program branches. Artificial intelligence components, represented by neural network systems, are data-driven. Their internal logic is determined by neurons trained with specific weights and the connection methods of these neurons. Therefore, traditional software testing methods and metrics cannot fully test and evaluate the internal logic of a specific neural network system. For neural networks, the same input and the same model may produce different outputs under different training conditions; therefore, testing the algorithm alone cannot yield stable, reliable, and effective conclusions.

[0005] Currently, testing technologies for artificial intelligence components are mainly divided into two categories. The first category is to use the black-box method of testing traditional software to test artificial intelligence components. This method does not care about the internal structure of the artificial intelligence component, treats the component under test as a black box, and only cares about its input and output.

[0006] Black-box methods mainly include the following: 1. Transfer model-based attack methods, which mainly use the gradient of the local model to generate adversarial examples to attack unknown target models with similar functions. This method requires the training set of the target model; 2. Score-based attack methods, which use the final classification probability of the model to attack; 3. Decision-based attack methods, which only require the final output result of the model to launch an attack on the target model.

[0007] Current black-box methods can effectively construct attack samples, allowing trained models to expose problems in some generated examples. However, these methods focus more on the attack itself, and the number and coverage of test cases are insufficient for testing AI components. The reliability of test results obtained using this method is low, and it is difficult to automate the testing of AI components that require large-scale testing.

[0008] The second type of method employs a white-box approach to test artificial intelligence components, such as neural networks. This method generates test cases by establishing an optimization function based on the internal neuron states of the deep neural network, aiming to maximize neuron coverage. The objective function of the white-box method is the difference between the predicted value and the original label. Gradient descent optimization is applied to this function to obtain test cases with different neuron coverage and different predicted values. The advantage of the white-box method is that it establishes a more interpretable coverage metric to guide test case generation. However, it is ineffective in scenarios where the network structure is unknown.

[0009] There are two main problems to be faced during the specific testing process:

[0010] First, there's test case evaluation. Test cases are endless, requiring a metric to assess whether they can uncover potential defects and faults in the system under test, and whether they adequately cover all functionalities to be tested. In traditional white-box testing of software, coverage criteria such as statement coverage, branch coverage, or condition coverage can be used to determine the sufficiency of test cases. In black-box testing, since only inputs and outputs are available, only some functional tests can be performed according to testing requirements, but the sufficiency of these tests is difficult to determine.

[0011] In evaluating the quality of test suites for artificial intelligence, the DeepMutation method borrows the idea of ​​mutation testing. This involves mutating deep neural networks to generate new networks with structures different from the original ones, and then using these new networks to test the test suites. Based on this approach, corresponding test case evaluation metrics are proposed. Experimental results show that DeepMutation can effectively and quantitatively evaluate the quality of test case sets and, according to the evaluation metrics, help developers improve the value of test data.

[0012] Second, test case generation. Current technologies employ a coverage-based test case generation method, migrated from traditional software testing techniques. Coverage metrics typically start with the structure of deep neural networks, defining the number of neurons activated in the test case set and the structure of those neurons from different perspectives to determine coverage. Test cases are then generated with the goal of maximizing coverage. This method requires in-depth observation of the output value of each neuron within the neural network structure. Another approach is adversarial testing, which adds subtle perturbations to the original data, creating changes that are difficult for humans to discern, yet elicit drastically different outputs from the AI ​​components. This type of method is called adversarial testing. Adversarial methods can utilize white-box methods to construct optimization functions using internal neural network information, such as L-BFGS and FGSM algorithms, which can quickly generate adversarial examples. C&W can bypass defense distillation techniques to attack deep neural networks. Adversarial methods can also be implemented using black-box methods. The decision-based black-box adversarial algorithm proposed by Brendel et al. can find adversarial examples near the decision boundary through boundary attack methods. These adversarial methods have achieved good results.

[0013] However, existing technologies lack functionality specifically for test cases. Functional testing requires verifying whether the software's behavior meets expectations to determine if it fulfills the requirements of the functional specification. Therefore, the test case set for software testing should be as comprehensive as possible to discover as many component defects as possible. Currently, whether it's coverage testing achieved through migration from traditional testing techniques or adversarial test generation techniques, the sample sets generated by these techniques cannot fully cover the functional constraints in the specification, and contain a large number of invalid test cases that would not appear in real-world application scenarios.

[0014] Furthermore, when the application scenarios of the samples are limited to adversarial training and adversarial attack scenarios, the test samples may not meet the sufficiency and reliability requirements necessary for functional testing. Summary of the Invention

[0015] To address the issue that current test cases for AI components may not adequately cover functional constraints, this invention proposes a test case generation method for AI components. This method generates test cases that fully cover functional constraints and are uniformly accurate, even without knowing the internal structure of the AI ​​component under test. Guided by a fuzzy framework, it can not only detect whether the component under test has implemented its intended functions but also whether it has implemented functions that it "shouldn't have." Furthermore, this invention proposes a method for reducing the test case set, which reduces the size of the test case set while maintaining its functionality, thereby lowering the time cost of testing.

[0016] The specific steps of the test case generation method for artificial intelligence components are as follows:

[0017] Step 1: For the original image, use a conditional generative adversarial network (cGAN) and one-hot encoding to stitch the image together to generate test samples.

[0018] The objective function of cGAN is shown below:

[0019]

[0020] G(z|y) is the generator function, D(x|y) is the discriminator function; E x～pdata(x) Indicates the expected value;

[0021] Step 2: Using the loss function of cGAN, a perturbation term is added to expand the generated test cases through fuzz testing.

[0022] The loss function of cGAN is: L cGAN =L G +L D ;

[0023] L G Let be the generator loss function.

[0024] Where ε is the disturbance term, c i For the target category generated by the generator in the i-th iteration, x i is the original sample input in the i-th iteration; m is the total number of generator iterations;

[0025] L D The discriminator loss function;

[0026]

[0027] Samples generated by the generator;

[0028] The discriminator will give higher scores to correct conditions and real use cases, and lower scores to correct condition-generated use cases and incorrect condition-real use cases, so that the samples generated by the generator gradually become closer to real samples.

[0029] The loss function for the disturbance term is: L ε =max(ε max ,‖ε‖2);

[0030] ε max The maximum value of the disturbance term; ||ε||2 indicates taking the L2 norm;

[0031] Step 3: Estimate the quality of the expanded test sample set using the payoff function to ensure that the sample set meets the testing requirements.

[0032] Profit function F i(TestCase j The formula is:

[0033] Where i = 1, 2, 3, ..., m is the test case set corresponding to the constraint function; TestCase j This is the set of use cases for the current j-th iteration.

[0034] To evaluate the quality of the expanded test case set, the softmax output of the alternative model is used.

[0035] Step 4: Construct metamorphic relationships. Reduce the sample set that meets the requirements to generate the final test cases for artificial intelligence components.

[0036] Metamorphotic relationships include:

[0037] MR1: When a use case is classified into a certain category with extremely high confidence, it means that the use case is in the center of the decision space. At this time, applying a small perturbation to it should not change its classification result.

[0038] MR2: When a use case is classified into a certain category with sufficient confidence and is different from other examples classified into that category, it means that the use case is not in the center of the decision space but is also on the edge of the decision space. Applying a small perturbation to it should not change its classification result.

[0039] MR3: When a use case is classified into two categories with similar probabilities, it indicates that it is at the decision boundary. Slightly perturbing it will cause the result to be incorrect.

[0040] MR4: When the probabilities of a use case for each class are close, it means that it cannot be classified and is outside the decision space. If perturbation is applied to the sample, the classification result will still be incorrect. These use cases are invalid use cases.

[0041] The test cases selected through MR1 can test whether the model under test has achieved the expected function and can accurately classify relatively clear test cases; the test cases selected through MR2 can test whether the model under test has the ability to resist noise; MR3 tests whether the model under test can resist obfuscation; and MR4 can select the most representative and high-value test cases from the test case set.

[0042] Step 5: Verify the final generated test cases using three metrics: reliability, adequacy, and coverage.

[0043] Reliability describes whether the set of test cases is accurate and effective, and whether the correctness of the test case execution results is determinable;

[0044] Sufficiency describes whether a set of use cases can uncover as many problems as possible;

[0045] Coverage measures whether test cases cover all functional constraints.

[0046] Reliability index: g(X) R,model )=a*g1(X R,model )+(1-a)*g2(X R,model )

[0047] Among them, g1(X) R,model ) is the correctness evaluation function. This metric is the ratio of the number of correct test cases to the size of the effective test case set, used to evaluate the proportion of test cases with the correct expected results; g2(X) R,model ) is the effectiveness evaluation function. This indicator uses the ratio of the number of test cases that satisfy MR4 to the overall size of the test case set to evaluate the proportion of effective test cases in the test case set; a is the weight of the correctness indicator in the calculation of the overall reliability indicator. When a is larger, it means that the correctness indicator occupies a higher weight in the overall evaluation indicator.

[0048] Sufficiency index: Mutation tests are conducted using mutants at the data, model, and program levels. The results of these tests are given by the ratio of the total number of eliminated mutants to the size of the mutant set.

[0049]

[0050] mutant killed This indicates the number of mutants eliminated, where "mutants" represents the total number of mutants.

[0051] Coverage metric: Derived from the ratio of the number of covered functional constraints to the total set of functional constraints.

[0052]

[0053] R i "is covered" indicates that the functional constraints in the functional constraint set are covered, and "Requirement" is the functional constraint set.

[0054] The advantages of this invention are:

[0055] 1) A test case generation method for artificial intelligence components, which can fully cover the functional constraints of the components under test, and can test the functional constraints that the components under test should have, as well as the functional constraints that the components under test should not have.

[0056] 2) A test case generation method for artificial intelligence components, which can automatically generate test cases for neural networks in the field of image processing, reducing the time cost required for manual generation.

[0057] 3) A test case generation method for artificial intelligence components, which can fully reduce the test case set and reduce the time required for testing while ensuring the functionality of the test case set. Attached Figure Description

[0058] Figure 1 This is a flowchart of a test case generation method for artificial intelligence components according to the present invention;

[0059] Figure 2 This is a schematic diagram of the basic structure of GANs in the existing technology;

[0060] Figure 3 This is a labeled MNIST handwritten digit example generated by the cGAN of this invention;

[0061] Figure 4 This is a diagram of the cGAN structure with added perturbation used in this invention. Detailed Implementation

[0062] The present invention will now be described in further detail with reference to the accompanying drawings and examples.

[0063] Current AI testing methods involve splitting a portion of the training set into a test set and testing it after training. The resulting test set has a data distribution that is basically the same as the training set. However, it cannot represent a wider range of data in real-world environments. Even if some new data with some obfuscation is generated through augmentation techniques, from a testing perspective, the data generated in this way still cannot fully cover the functional constraints in the specification. The AI ​​after testing may not meet the requirements for sufficiency and reliability.

[0064] Based on this, this invention proposes a generation method using cGAN as the primary example, employing a fuzzy framework to guide the overall generation process, expanding the generated test case set, providing latent space sampling constraints and training constraints, and designing a reward function to evaluate the richness of the test case set, ensuring the sufficiency and reliability of the test case set. Simultaneously, to address the test prediction problem under large test sets, this invention proposes four general metamorphic relations based on the metamorphic testing principle to ensure the accuracy of expected test results. Furthermore, to measure the sufficiency and reliability of the newly generated test cases, a set of evaluation methods and metrics for the generated cases is proposed to guarantee the quality of the final generated test case set.

[0065] In summary, this invention provides a test case generation method for artificial intelligence components, comprising three parts: test case generation, test case set reduction, and test case set evaluation. It achieves automatic generation of test cases for neural networks in the field of image processing, such as... Figure 1 As shown, the specific steps are as follows:

[0066] Step 1: For the original image, use a conditional generative adversarial network (cGAN) to stitch the image together with one-hot encoding to generate test cases based on functional constraints.

[0067] like Figure 2 As shown, Generative Adversarial Networks (GANs) learn the feature distribution of the training set to generate new data with a distribution similar to the original training set. However, GANs generate data by simply sampling random noise without any constraints, and therefore do not carry label information.

[0068] like Figure 3 and Figure 4 As shown, this invention uses cGAN to accept specified constraints and generate new examples based on existing constraints. For the label information of the original examples, it is generally stitched to the original image using one-hot encoding to generate new examples.

[0069] The objective function of cGAN is shown below:

[0070]

[0071] G(z|y) is the generator function, D(x|y) is the discriminator function; E x～pdata(x) This is the expected value;

[0072] Step 2: Using the loss function of cGAN, a perturbation term is added to expand the generated test cases through fuzz testing.

[0073] cGAN's loss function L cGAN It consists of two parts: generator loss L G And discriminator loss L D ;

[0074] The discriminator assigns higher scores to correct conditions and real use cases, and lower scores to correct condition-generated use cases and incorrect condition-real use cases, so that the generated examples gradually resemble real samples.

[0075] L cGAN =L G +L D ;

[0076] The generator loss function is:

[0077] Where ε is the disturbance term, c i For the target category generated by the generator in the i-th iteration, x i is the original sample input in the i-th iteration; m is the total number of generator iterations;

[0078] The discriminator loss function is:

[0079]

[0080] Samples generated by the generator;

[0081] In cGAN, a perturbation term is added to mutate the generated examples, but this mutation should be kept within a certain range and should not produce excessive errors, i.e., it should not generate unnatural use cases; the loss function of the perturbation term is: L ε =max(ε max ,‖ε‖2);

[0082] ε max ε represents the maximum value of the disturbance term; ||ε|2 indicates taking the L2 norm.

[0083] The final loss function is expressed as: L = L cGAN +L ε .

[0084] The examples generated by cGAN pass the test model with very high accuracy and are basically in the center of the original model's decision range. The test example set is expanded with the examples generated by cGAN as the core.

[0085] Step 3: Estimate the quality of the expanded test sample set using the payoff function to ensure that the sample set meets the testing requirements.

[0086] This application employs a fuzzy testing method to expand the sample set. The original fuzzy testing adds a random perturbation to the input sample and then adds the perturbed sample to the sample set as an expansion. However, the random perturbation cannot meet the test work's requirement for sufficient coverage in the decision space. Therefore, this application proposes a reward function to estimate the quality of the test sample set, thereby ensuring that the final sample set can meet the requirements of the test work.

[0087] Profit function F i (TestCase j The formula is:

[0088] Where i = 1, 2, 3, ..., m is the test case set corresponding to the constraint function; TestCase j This is the set of use cases for the current j-th iteration.

[0089] The payoff function needs to ensure that the sample decision space for each constraint is explored as much as possible; therefore, the variance of the test case set for each class should be as large as possible. When generating a batch of test case sets, the difference between the softmax output values ​​of the new test case set and the original test case set on the alternative model is evaluated.

[0090] Alternative models are an important concept in black-box testing. Simply put, models with the same functionality have similar decision spaces. Therefore, this invention uses alternative models to calculate the output of test cases, thereby replacing the position of the test cases in the decision space of the model under test. Alternative models must use the same training set as the model under test and achieve the same expected metrics before they can be used in testing.

[0091] Step 4: Construct metamorphic relationships. Reduce the sample set that meets the requirements to generate the final test cases for artificial intelligence components.

[0092] In the training process of artificial intelligence, a large number of samples can enable the model to be fully trained and achieve good results in actual work. However, in the testing work, the test set needs to reflect the problems of the artificial intelligence components. A large number of repetitive samples will make the testing work lengthy and inefficient. In addition, the test prediction of artificial intelligence—that is, whether it can give the expected test results for a given test set—is also a problem that troubles test engineers.

[0093] This invention reduces the sample set by constructing metamorphic relations, and since metamorphic relations are essentially equivalent transformations, it also solves the problem of test predictions; the metamorphic relations are shown in Table 1, and there are mainly four types:

[0094] Table 1

[0095] MR1 Slightly perturb the use cases at the center of the decision space without affecting the prediction results. MR2 Slightly perturb use cases that are neither central nor boundary-oriented within the decision space without affecting prediction. MR3 Slightly perturbing the use cases at the decision space boundary leads to incorrect prediction results. MR4 Even with slight perturbations to use cases outside the decision space, the prediction results are still incorrect.

[0096] MR1: When a use case is classified into a certain category with extremely high confidence, it means that the use case is in the center of the decision space. At this time, applying a small perturbation to it should not change its classification result.

[0097] MR2: When a use case is classified into a certain category with sufficient confidence and is different from other examples classified into that category, it means that the use case is not in the center of the decision space but is also on the edge of the decision space. Applying a small perturbation to it should not change its classification result.

[0098] MR3: When a use case is classified into two categories with similar probabilities, it indicates that it is at the decision boundary. Slightly perturbing it will cause the result to be incorrect.

[0099] MR4: When the probabilities of a use case for each class are close, it means that it cannot be classified and is outside the decision space. If perturbation is applied to the sample, the classification result will still be incorrect. These use cases are invalid use cases.

[0100] The test cases selected through MR1 can test whether the model under test has achieved the expected function and can accurately classify relatively clear test cases; the test cases selected through MR2 can test whether the model under test has the ability to resist noise; MR3 tests whether the model under test can resist obfuscation; and MR4 can select the most representative and high-value test cases from the test case set.

[0101] Step 5: Verify the final generated test cases using three metrics: reliability, adequacy, and coverage.

[0102] Reliability describes whether the set of test cases is accurate and effective, and whether the correctness of the test case execution results is determinable;

[0103] Sufficiency is used to describe whether a set of use cases can discover as many problems as possible;

[0104] Coverage is a measure of whether test cases cover all functional constraints.

[0105] Reliability metrics include: Correctness: the ratio of correctly labeled test cases to the total number of valid test cases in the entire test case set; Validity: the ratio of valid test cases to the total number of test case sets. Multiplying these two by their respective weights yields the reliability metric.

[0106] g(X R,model )=a*g1(X R,model )+(1-a)*g2(X R,model )

[0107] Among them, g1(X) R,model ) is the correctness evaluation function:

[0108] x correct X is the number of correct use cases; R,model x represents the overall size of the use case set. MR4 To meet the number of use cases required for MR4;

[0109] This metric is the ratio of the number of correct use cases to the size of the effective use case set, used to evaluate the proportion of use cases that have the correct expected results.

[0110] g2(X R,model ) is the validity evaluation function:

[0111] This metric uses the ratio of the number of test cases that meet MR4 to the overall size of the test case set to evaluate the proportion of valid test cases in the test case set;

[0112] 'a' represents the weight of the correctness indicator in the calculation of the overall reliability indicator. The larger 'a' is, the higher the weight of the correctness indicator in the overall evaluation indicator.

[0113] Sufficiency index: Mutation tests are conducted using mutants at the data, model, and program levels. The results of these tests are given by the ratio of the total number of eliminated mutants to the size of the mutant set.

[0114]

[0115] mutant killed This indicates the number of mutants eliminated, where "mutants" represents the total number of mutants.

[0116] Coverage metric: Derived from the ratio of the number of covered functional constraints to the total set of functional constraints.

[0117]

[0118] R i is the number of functional constraints that have been covered for the i-th time; Requirement is the total number of functional constraints.

[0119] Example:

[0120] First, test cases are generated for the AI ​​component under test. Then, the test case set is reduced to remove invalid and duplicate test cases. Finally, the generated test case set is evaluated.

[0121] Step 1: Alternative model training.

[0122] Since black-box testing cannot perceive the internal workings of the model under test, this embodiment uses a substitute model with the same objective and similar performance to describe the decision space of the model under test.

[0123] This process requires feeding the training set of the tested model into the surrogate model for training until the accuracy of the surrogate model reaches the expected accuracy of the target component. The process is as follows:

[0124] Input: training set X n Functional constraint set R m The target model's expected accuracy is acc.

[0125] The output is: Alternative model f sub Alternative model indicators;

[0126] Based on the functional constraint set R m Select an alternative model and use the training set X n Train the selected alternative model, when the alternative model f sub If the accuracy reaches acc, the iteration process ends, and the alternative model index result is output; otherwise, based on the constraint set R... m Replace the model with an alternative one until the accuracy is achieved;

[0127] Step 2: Test case generation.

[0128] Test case generation follows a fuzzy testing framework, where each iteration perturbs the test cases on the existing test set to generate new cases; specifically:

[0129] Select the original sample set X n Determine the functional constraint set R m Determine the sampling parameter d and the profit function F. i (TestCase j ), i = 1, 2, 3, ..., m (i is the i-th functional constraint), j = 1, 2, 3, ..., k (j is the iteration number).

[0130] In the original sample set X n Regarding functional constraints R i After selecting d samples, and inputting a total of m*d samples into the trained substitution model, the softmax output of the substitution model is obtained, which is then processed by the profit function F. i After calculation (TestCase1), d new sample directions are generated. d Then input it into cGAN to generate test cases;

[0131] Then, the generated test cases are added to the existing test case set TestCase. j A new test set, TestCase, is obtained. j+1 Then input it into the alternative model to obtain the softmax value of the functional constraint for that time, and recalculate the payoff function F at this time. i (TestCase2), continue to the next iteration.

[0132] After k iterations, if the profit function F i (TestCase j No further improvement is needed, resulting in R for the functional constraint. i TestCase k .

[0133] In each acquisition of the profit function F i (TestCase j When this happens, it is necessary to use the existing test case set TestCase. j If d samples are obtained, and the current reward function value is small, it indicates that the similarity between the test cases in the test case set is high, and the sampling direction needs to be changed to improve the sufficiency of the test case set; if the current reward function value is high, it indicates that the test case set is sufficiently divergent in the decision space and has good sufficiency. At this time, random sampling can be adopted to keep the reward function at a high level.

[0134] The sampling process is as follows:

[0135] The input is: the revenue function F of the use case set. i (TestCase j ), the threshold ε of the profit function, and the number of random samples t;

[0136] The output consists of: latent space variable z, and intermediate model p trained by the generator.

[0137] First, calculate the benefit function value f of the current use case, and determine whether the benefit satisfies f>ε. If so, random sampling is performed; otherwise, random sampling is performed t times based on the newly generated sampling results, and the latent space variable z with the largest difference from the original sampling results is selected.

[0138] At the same time, based on the softmax correctness of the newly generated test cases, model parameters are randomly selected from a set that is better than the current intermediate model p;

[0139] Step 3: Reduce the set of use cases.

[0140] To address the issues of excessively large test case sets leading to high testing costs and test oracle problems, this embodiment proposes a test case set reduction technique based on metamorphic relationships. This technique removes duplicate and invalid test cases from large-scale test case sets while ensuring that the quality of the test set is maintained during the reduction process, giving it the same testing capabilities as the test case set before reduction.

[0141] This embodiment will use four metamorphic relationships to filter each use case set under each functional constraint.

[0142] For MR1, the set of use cases is input into the alternative model, and the use cases that are closer to the class center are selected, that is, the use cases whose predicted probability for the current class is higher than the expected accuracy and are far from the boundary.

[0143] For MR2, the set of use cases is input into the alternative model, and use cases that deviate from the class center but are far from the decision boundary are selected. Specifically, use cases whose maximum classification probability output by the softmax function for a set of use cases exceeds the maximum probability that satisfies MR3 (i.e., 50%), but is lower than the expected accuracy are selected.

[0144] For MR3, the set of use cases is input into the alternative model, and all use cases whose outputs are close to the decision boundary are selected. That is, the highest n classification probabilities are very close, and it is not possible to classify the use case into that category with a relatively certainty.

[0145] For MR4, the use cases cannot be effectively distinguished and are therefore invalid, so all use cases are filtered out.

[0146] The above four steps are performed sequentially. The test cases selected in the first step can test whether the model under test has achieved the expected function and can accurately classify relatively clear test cases. The test cases selected in the second step can test whether the model under test has the ability to resist noise. The third step tests whether the model under test can resist obfuscation. Through these four steps, the most representative and high-value test cases can be selected from the test case set.

[0147] Step 4, Use Case Set Evaluation; Use Case Set Evaluation is conducted from three perspectives: sufficiency, reliability, and coverage.

Claims

1. A method for generating test cases for artificial intelligence components, characterized in that, The specific steps are as follows: First, for the original image, one-hot encoding is used to stitch the image together with the original image to generate test cases based on the conditional generative adversarial network cGAN. Then, by using the loss function of cGAN, a perturbation term is added to the generated test examples using a fuzzy testing method; by using latent space sampling constraints and training constraints, and designing a benefit function, the quality of the expanded test example set is estimated to ensure that the example set meets the testing requirements. Next, the metamorphic relationship is constructed to reduce the sample set that meets the requirements, generating the final test cases for the artificial intelligence components; Metamorphotic relationships include: MR1: When a use case is classified into a certain category with extremely high confidence, it means that the use case is in the center of the decision space. At this time, applying a small perturbation to it should not change its classification result. MR2: When a use case is classified into a certain category with sufficient confidence and is different from other examples classified into that category, it means that the use case is neither in the center nor on the edge of the decision space. Applying a small perturbation to it should not change its classification result. MR3: When a use case is classified into two categories with similar probabilities, it indicates that it is at the decision boundary. Slightly perturbing it will cause the result to be incorrect. MR4: When the probabilities of a use case for each class are close, it means that it cannot be classified and is outside the decision space. If perturbation is applied to the use case and the classification result is still incorrect, these use cases are invalid use cases. The test cases selected through MR1 can test whether the model under test has achieved the expected function and can accurately classify relatively clear test cases; the test cases selected through MR2 can test whether the model under test has the ability to resist noise; MR3 tests whether the model under test can resist obfuscation; and MR4 can select the most representative and high-value test cases from the test case set. Finally, the generated test cases are validated using three metrics: reliability, adequacy, and coverage. Reliability describes whether the set of test cases is accurate and effective, and whether the correctness of the test case execution results is determinable; Sufficiency describes whether a set of use cases can uncover as many problems as possible; Coverage measures whether test cases cover all functional constraints; Reliability metrics: in, This is a correctness evaluation function. The metric is the ratio of the number of correct use cases to the size of the effective use case set, used to evaluate the proportion of use cases with the correct expected results. As a validity evaluation function, this metric uses the ratio of the number of test cases that satisfy MR4 to the overall size of the test case set to evaluate the proportion of valid test cases in the test case set; The correctness index is the weight of the overall reliability index in the calculation. The larger the value of 𝑎, the higher the weight of the correctness index in the overall evaluation index. Sufficiency index: Mutation tests are conducted using mutants at the data, model, and program levels. The results of these tests are given by the ratio of the total number of eliminated mutants to the size of the mutant set. This indicates the number of mutants eliminated. Indicates the number of variants; Coverage metric: Derived from the ratio of the number of covered functional constraints to the total set of functional constraints. This indicates the functional constraints that are covered in the set of functional constraints. It is a set of functional constraints.

2. The test case generation method for artificial intelligence components as described in claim 1, characterized in that, The objective function of cGAN is shown below: For generator functions, This is the discriminator function.

3. The test case generation method for artificial intelligence components as described in claim 1, characterized in that, The loss function of the cGAN is: ; Let be the generator loss function. ; in, For the disturbance term, For the first The target category generated by the secondary generator. It is the first The original sample input; This represents the total number of generator iterations. The discriminator loss function; Samples generated by the generator; The discriminator will give higher scores to correct conditions and real use cases, and lower scores to correct condition-generated use cases and incorrect condition-real use cases, so that the samples generated by the generator gradually approach the real samples. The loss function for the disturbance term is: ; This represents the maximum value of the disturbance term; This indicates taking the 2-norm.

4. The test case generation method for artificial intelligence components as described in claim 1, characterized in that, The profit function The formula is: ； in, This is the set of use cases corresponding to the constraint functions; For the current number The set of use cases for the next iteration; The mean; To evaluate the quality of the expanded test case set, the softmax output of the alternative model is used.

Images