Fuzzy test priority ranking method and device based on metamorphic relation
Through adaptive seed selection strategy and differential-driven metamorphosis relationship selection strategy, the seed and metamorphosis relationship are dynamically adjusted, test cases are generated and the results are verified, which solves the problem of insufficient seed mutation in the existing technology, and improves the failure of fuzzy testing Reveal efficiency and coverage.
Patent Information
- Application Number
- CN202410979788.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-12
AI Technical Summary
In the prior art, it is impossible to guide seed mutation to generate test cases through metamorphosis relationships, which affects the fault revelation efficiency of fuzzy testing.
The adaptive seed selection strategy and differential-driven metamorphosis relationship selection strategy are adopted to dynamically adjust the seed selection probability and metamorphosis relationship strategy, generate test cases, and update the strategy through verification results to improve the fault revelation efficiency of fuzzy testing.
Improved the efficiency of fuzz testing to reveal deep learning model failures, achieving higher neuronal coverage and lower time overhead.
Smart Images

Figure CN118939540A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of software testing technology, and in particular to a fuzzy testing priority sorting method and device based on metamorphosis relationship. Background Art
[0002] Deep learning models learn decision logic through data, so data has an important impact on the development and testing of deep learning models. In practice, data engineers usually collect labeled data to build training data and test data sets. However, data collection includes multiple steps such as obtaining and processing data, which may require a lot of resources and manpower, so data collection will incur high costs.
[0003] Fuzz testing is essentially a test case generation technology that generates a large amount of well-formatted, unexpected test data by mutating a small number of test cases (seeds) to cover as many boundary conditions as possible. This technology saves the manpower overhead of manually building test cases and has efficient fault revealing capabilities.
[0004] At present, the research on using fuzz testing to test deep learning models mainly focuses on proposing new mutation strategies or coverage criteria, ignoring the impact of seed selection and mutation strategy selection on the efficiency of fuzz testing fault disclosure. Fuzz testing requires selecting a seed from a seed queue, then selecting a mutation strategy, and mutating the seed according to the guidance of the mutation strategy to generate test cases. Therefore, seed selection and mutation strategy selection are key steps in generating test cases for fuzz testing. In terms of mutation strategies, researchers have proposed a large number of mutation strategies for models that handle different types of problems using methods such as domain knowledge and search algorithms, but have ignored the different fault disclosure efficiencies of seeds under different mutation strategies.
[0005] The test expectation problem refers to the difficulty in distinguishing between correct behavior and potential erroneous behavior of the software under given input during the testing process. Like traditional software testing, deep learning model testing also has test expectation problems, that is, for a given test input, it is difficult to determine whether the output of the deep learning model meets expectations. Since the decision logic of deep learning models is difficult to understand and deep learning models often help humans solve uncertain problems, the test result judgment is the difficulty of deep learning model fuzz testing.
[0006] In view of this, how to provide a fuzz test priority sorting method to use the metamorphosis relationship to determine the fuzz test results, thereby improving the fault revealing efficiency of fuzz testing, has become a technical problem that needs to be solved urgently. Summary of the invention
[0007] The embodiments of the present application provide a fuzzy test priority sorting method based on metamorphosis relationship, a multifunctional transmitting component device, an electronic device and a computer storage medium, so as to solve the problem in the prior art that it is impossible to generate test cases by guiding seed mutation through metamorphosis relationship, thereby affecting the fault revealing efficiency of fuzzy testing.
[0008] In a first aspect of an embodiment of the present application, a fuzzy test priority sorting method based on a metamorphosis relationship is provided, comprising:
[0009] Selecting a target seed from a seed queue based on an adaptive seed selection strategy, wherein the adaptive seed selection strategy is a strategy for dynamically adjusting a seed selection probability based on relevant information generated during the execution of the seed, and selecting a target seed based on the seed selection probability;
[0010] Based on the metamorphosis relationship selection strategy, a target metamorphosis relationship is selected from the metamorphosis relationship set, and based on the target metamorphosis relationship, the target seed is mutated to generate a test case set, wherein the metamorphosis relationship selection strategy is to quantify the distance between the seed and the test case generated under the guidance of different metamorphosis relationships according to the canonical compression distance between the test case and the seed corresponding to each metamorphosis relationship in the metamorphosis relationship set, and determine the strategy of the target metamorphosis relationship based on the distance obtained after quantification;
[0011] Input the target seed and the test case set without labels into the model to be tested for execution, and output the execution result, wherein the execution result carries the target seed execution result and the test case execution result;
[0012] Based on the target metamorphosis relationship, verify the target seed execution result and the test case execution result to obtain a verification result, and based on the verification result and the relevant information generated by the seed during the execution process, update the adaptive seed selection strategy and the metamorphosis relationship selection strategy;
[0013] Based on the updated adaptive seed selection strategy and the updated metamorphosis relationship selection strategy, the target seed and the target strategy are repeatedly selected until the number of iterations reaches a preset iteration threshold, the fuzzy test is stopped, and the test results are output.
[0014] In a second aspect of an embodiment of the present application, a fuzzy test priority sorting device based on a metamorphosis relationship is provided, comprising:
[0015] A seed selection module is configured to select a target seed from a seed queue based on an adaptive seed selection strategy, wherein the adaptive seed selection strategy is a strategy for dynamically adjusting a seed selection probability based on relevant information generated during the execution of the seed, and selecting a target seed based on the seed selection probability;
[0016] The relationship selection module is configured to select a target metamorphic relationship from a metamorphic relationship set based on a metamorphic relationship selection strategy, and mutate the target seed based on the target metamorphic relationship to generate a test case set, wherein the metamorphic relationship selection strategy is to quantify the distance between the seed and the test case generated under the guidance of different metamorphic relationships according to the canonical compression distance between the test case and the seed corresponding to each metamorphic relationship in the metamorphic relationship set, and determine the strategy of the target metamorphic relationship based on the distance obtained after quantification;
[0017] An execution module is configured to input the target seed and the test case set without a label into the model to be tested for execution, and output an execution result, wherein the execution result carries the target seed execution result and the test case execution result;
[0018] A verification module is configured to verify the target seed execution result and the test case execution result based on the target metamorphosis relationship to obtain a verification result, and update the adaptive seed selection strategy and the metamorphosis relationship selection strategy based on the verification result and the relevant information generated by the seed during the execution process;
[0019] The iteration module is configured to repeatedly select target seeds and target strategies based on the updated adaptive seed selection strategy and the updated metamorphosis relationship selection strategy until the number of iterations reaches a preset iteration threshold, stop the fuzzy test, and output the test results.
[0020] The fuzzy test priority sorting method based on metamorphosis relationship provided in the present application is applied. First, the target seed and target metamorphosis relationship are selected by using the adaptive seed selection strategy and the difference-driven metamorphosis relationship selection strategy; then, the target seed is mutated according to the target metamorphosis relationship to obtain the test case, and the seed and the test case are executed; secondly, the result is verified by comparing whether the output results of the seed and the test case conform to the selected metamorphosis relationship; finally, the seed selection strategy and the metamorphosis relationship selection strategy are further updated according to the test results and the test process information, so as to improve the probability of fuzzy testing revealing deep learning model failures.
[0021] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0023] Figure 1 A flowchart of a fuzzy test priority sorting method based on a metamorphosis relationship provided in an embodiment of the present application;
[0024] Figure 2 A structural diagram of a fuzzy test priority sorting method based on a metamorphosis relationship provided in an embodiment of the present application;
[0025] Figure 3 An example diagram of a fuzzy test priority sorting method based on a metamorphosis relationship provided in an embodiment of the present application;
[0026] Figure 4 A schematic diagram of the structure of a fuzzy test priority sorting device based on a metamorphosis relationship provided in an embodiment of the present application;
[0027] Figure 5 A structural block diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0028] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0029] Metamorphosis testing is a widely used testing technology to alleviate the expected problems of deep learning model testing. This technology first identifies the metamorphosis relationship through the metamorphosis properties of the program to be tested, and then generates derivative test cases under the guidance of the metamorphosis relationship from the original test case. After executing the original test case and the derivative test case, it is verified whether the output result violates the metamorphosis relationship. If the metamorphosis relationship is violated, it indicates that there is a fault in the program to be tested. The metamorphosis relationship can not only be used as a test expectation, but also generate new test data, and has been successfully applied in many fields. Some researchers combine fuzz testing technology with metamorphosis testing technology and use metamorphosis relationships to determine test results. However, existing studies have ignored that metamorphosis relationships can guide seed mutations and generate test cases, affecting the fault disclosure efficiency of fuzz testing.
[0030] This application aims to solve the technical problem of how to prioritize metamorphosis relations to select efficient metamorphosis relations, and use metamorphosis relations to determine fuzz testing results to improve the fault revealing efficiency of fuzz testing. Considering the process of deep learning model fuzz testing and the principle of metamorphosis testing, a fuzz testing priority sorting method based on metamorphosis relations is proposed, and a difference-driven metamorphosis relation selection strategy is designed. Compared with the baseline method, the proposed priority sorting method can generate more adversarial inputs, achieve higher neuron coverage, and have a smaller time overhead.
[0031] See also Figure 1 , Figure 1 A flowchart of a fuzzy test priority sorting method based on a metamorphosis relationship provided in an embodiment of the present application. Figure 1 As shown, the specific steps include the following steps.
[0032] Step S102: Select a target seed from the seed queue based on an adaptive seed selection strategy, wherein the adaptive seed selection strategy is a strategy for dynamically adjusting the seed selection probability based on relevant information generated during the execution of the seed, and selecting the target seed based on the seed selection probability.
[0033] The embodiment of the present application uses an adaptive seed selection strategy to select seeds for fuzz testing from a seed queue. The adaptive seed selection strategy aims to dynamically adjust the seed selection probability by using valuable information generated by the seed during execution (such as coverage information, number of seed mutations, etc.), so that seeds with a higher probability of revealing faults or triggering new coverage have a higher probability of being selected.
[0034] Specifically, the method selects a target seed from a seed queue based on an adaptive seed selection strategy, including: obtaining seed-related information generated by various seeds in the seed queue during execution, and dynamically adjusting the seed selection probability based on the seed-related information to generate a probability selection list, wherein the seed-related information carries seed coverage information and seed variation information; based on the probability selection list, determining a seed whose probability value meets a preset condition as a target seed, wherein the preset condition is a condition for sorting the various seed selection probabilities in the probability selection list to determine the seed with the largest probability value as the target seed.
[0035] More specifically, the method dynamically adjusts the seed selection probability based on the seed-related information and generates a probability selection list, including: generating a seed variation list based on the test cases generated by various sub-variations in the seed queue and the time corresponding to the generation of the test cases, and generating an initial probability selection list based on the probability of various sub-variations being selected in the seed variation list; dividing the seed variation list into a first seed coverage list and a second seed coverage list based on the seed coverage information generated by various sub-variations during execution, wherein the first seed coverage list is a list corresponding to the number of times a seed is selected and a new coverage is triggered, and the second seed coverage list is a list corresponding to the number of times a seed is selected and a new coverage is not triggered; based on the initial probability selection list, the first coverage list and the second coverage list, dynamically adjust the seed selection probability and generate a probability selection list.
[0036] More specifically, the method of dynamically adjusting the seed selection probability based on the initial probability selection list, the first coverage list and the second coverage list to generate a probability selection list includes: selecting a first seed in the initial probability selection list for seed mutation, and generating a test case set based on the first seed and the seed mutation strategy; in response to the presence of at least one test case in the test case set triggering new coverage, dynamically adjusting the seed selection probability based on a first formula to generate a first probability selection list; in response to the absence of a test case in the test case set triggering new coverage, dynamically adjusting the seed selection probability based on a second formula to generate a second probability selection list; generating a probability selection list based on the first probability selection list and the second probability selection list.
[0037] In practical applications, assume that there is a seed queue S, whose size is n, L i (1≤i≤n) storage based on seed s i The test cases generated by the mutation and the corresponding generation time. For example, if the test cases s'1 and s'2 generated based on seeds s1 and s2 at time t1 and t2 trigger new coverage of the model under test, then s'1 and t1, s'2 and t2 are constructed into two tuples and added to L1 and L2. During the test process, this strategy maintains an L i The probability of selecting a list P, for an element p in P i (1≤i≤n), indicating the list L i In addition, in order to utilize the information of seed-triggered coverage, the strategy maintains two lists E and E' that record the new coverage information triggered by the seed. The element e in E i (1≤i≤n) represents L i The number of times a seed in is selected and triggers a new overlay; the element e' in E' i Indicates L iThe number of times a seed in was selected but did not trigger a new overwrite.
[0038] It should be noted that if based on seed s i The test cases generated by the mutation can trigger new coverage of the model under test, and the corresponding selection probability p i will increase; otherwise, p i In addition, the fewer times p is selected, the i The greater the increase; conversely, the more times it is selected, the greater the increase in p i The smaller the increase.
[0039] At the beginning of the test, the probability selection list P and two lists E and E' that record the new coverage information triggered by the seed are initialized. During the test, it is assumed that the strategy selects L according to P. i , and selected L i The latest seeds generated in i Let T be a i And the mutation strategy generates a test case set containing k test cases. After executing all the test cases in T, if there is a test case in T that triggers new coverage, then set e i =e i +1, and update p using the first formula j and p i (In order to facilitate the calculation and distinction of the values before and after the selection probability changes, use p' j and p' i Indicates p j and p i updated value).
[0040] The first formula includes formula (1) and formula (2):
[0041]
[0042] Among them, ε is the probability adjustment factor.
[0043]
[0044] If no test cases in T can trigger the new coverage, update e' i =e' i +1, and use the second formula to update the probability of seed selection.
[0045] The second formula includes formula (3) and formula (4):
[0046]
[0047] During the test, the strategy continuously updates the seed selection probability according to the test process information, so that seeds with a higher probability of triggering new coverage are more likely to be selected; conversely, seeds that are difficult to trigger new coverage have a lower selection probability.
[0048] Step S104: Based on the metamorphosis relationship selection strategy, a target metamorphosis relationship is selected from the metamorphosis relationship set, and based on the target metamorphosis relationship, the target seed is mutated to generate a test case set, wherein the metamorphosis relationship selection strategy is to quantify the distance between the seed and the test case generated under the guidance of different metamorphosis relationships according to the canonical compression distance between the test case and the seed corresponding to each metamorphosis relationship in the metamorphosis relationship set, and determine the strategy of the target metamorphosis relationship based on the distance obtained after quantification.
[0049] The embodiment of the present application uses a difference-driven metamorphosis relationship selection strategy to select a metamorphosis relationship from a metamorphosis relationship set, and the metamorphosis relationship can be used as a mutation rule for a seed. The metamorphosis relationship can be identified based on the mathematical properties or domain knowledge of the deep learning model, and current research work has proposed a large number of metamorphosis relationships to solve different problems existing in the deep learning model. Among them, the difference-driven metamorphosis relationship selection strategy quantifies the distance between the seed and the test case generated under the guidance of different metamorphosis relationships according to the canonical compression distance between the test case generated by each metamorphosis relationship and the seed, and preferentially selects the metamorphosis relationship that can make the generated test case differ the most from the seed as the next metamorphosis relationship to be used.
[0050] Specifically, the transformation relationship selection strategy is based on selecting a target transformation relationship from a transformation relationship set, and based on the target transformation relationship, mutating the target seed to generate a test case set, including: mutating the target seed according to each transformation relationship in the transformation relationship set to generate an initial test case set; quantifying the compression distance between each test case in the initial test case set and the target seed based on a third formula; based on the compression distance obtained after quantification, selecting a target transformation relationship from the transformation relationship set, and mutating the target seed based on the target transformation relationship to generate a test case set.
[0051] In practical applications, assume that there is a seed queue S of size n and a metamorphosis relation set M of size h, s i (1≤i≤n) and m j (1≤j≤h) represents the i-th seed and j-th metamorphosis relation in S and M respectively. Any seed s in S i Corresponding to a set M of applicable transformation relations of size g i .
[0052] Assume that during the test, a seed s is selected from S using the adaptive seed selection strategy introduced in step 1. i , then according to the guidance of the transformation relationship, a test case set T of size g can be i (Assuming that the selected seed mutates only once according to the metamorphosis relationship), where T i Elements in Yes i In a Transforming Relationship Then, s i and The difference can be quantified by the third formula.
[0053] The third formula is expressed by formula (5):
[0054]
[0055] Where C is the compression program suitable for the seed type; For seeds i and seeds connection.
[0056] When using the third formula to quantify the difference between the seed and the test case, it is necessary to select the best compression program according to the data format of the seed. When established, Selected as the next transformation relation to be used.
[0057] When selecting a metamorphosis relationship using the difference-driven metamorphosis relationship selection strategy, the difference between the test case generated by each metamorphosis relationship and the seed is considered. The metamorphosis relationship with a greater degree of difference is more likely to discover defects or abnormal behaviors in the model to be tested, and therefore has a higher priority. Through the above priority sorting strategy, the best metamorphosis relationship can be effectively selected to improve the fault revealing efficiency of fuzz testing.
[0058] Step S106: input the target seed and the test case set without a label into the model to be tested for execution, and output an execution result, wherein the execution result carries the target seed execution result and the test case execution result.
[0059] The embodiment of the present application sends the selected seeds and the generated test cases to the model to be tested for execution. If the seeds in the seed queue contain tags, the seeds may not be executed; otherwise, the seeds need to be executed to verify whether the execution results of the seeds and the execution results of the test cases meet the corresponding transformation relationship.
[0060] Step S108: Based on the target metamorphosis relationship, the target seed execution result and the test case execution result are verified to obtain a verification result, and based on the verification result and the relevant information generated by the seed during the execution process, the adaptive seed selection strategy and the metamorphosis relationship selection strategy are updated.
[0061] The embodiment of the present application sends the selected seeds and the generated test cases to the model to be tested for execution. If the seeds in the seed queue contain tags, the seeds may not be executed; otherwise, the seeds need to be executed to verify whether the execution results of the seeds and the execution results of the test cases meet the corresponding transformation relationship.
[0062] In the embodiment of the present application, the obtained seeds and the execution results of the test cases are verified to verify whether the execution results of the seeds and the execution results of the test cases are consistent with the selected metamorphosis relationship.
[0063] In addition, the embodiment of the present application also decides to retain or discard the generated test case based on the verification result. If the execution result of the seed and the test case meets the selected transformation relationship, the test passes and the test case is discarded; otherwise, the test fails and the test case is added to the adversarial input set.
[0064] Specifically, based on the target metamorphosis relationship, the target seed execution result and the test case execution result are verified to obtain a verification result, including:
[0065] In response to the target seed execution result and the test case execution result being consistent with the target metamorphosis relationship, discarding the test case;
[0066] In response to the target seed execution result and the test case execution result being inconsistent with the target metamorphosis relationship, the test case is added to the adversarial input set.
[0067] Step S110: Based on the updated adaptive seed selection strategy and the updated metamorphosis relationship selection strategy, repeatedly select target seeds and target strategies until the number of iterations reaches a preset iteration threshold, stop fuzzy testing, and output test results.
[0068] The embodiment of the present application records the test process information, uses the test process information as feedback information to guide the test process, and updates the seed selection strategy and the metamorphosis relationship selection strategy based on the test information to adapt to the uncertainty of the failure probability revealed by the seed and metamorphosis relationship.
[0069] The output of the deep learning model is probabilistic and the input that triggers the fault will not cause the deep learning model to produce obvious abnormal behavior (such as model crash), which increases the difficulty of judging the test results.
[0070] The fuzzy test priority sorting method based on metamorphosis relationship proposed in the present application includes a feedback loop consisting of test case generation, test execution and test result judgment components, wherein the controller consists of a seed selection strategy and a metamorphosis relationship selection strategy. This method can not only guide the controller to select seeds and metamorphosis relationships based on test history information, but also continuously improve the controller to adapt to the ability of revealing faults by the ever-changing seeds and metamorphosis relationships. The adaptability proposed in the embodiment of the present application enables fuzzy testing to gradually improve and improve the fault revealing efficiency of the test in continuous iterations.
[0071] See also Figure 2 , Figure 2 A structural diagram of a fuzzy test priority sorting method based on metamorphosis relationship provided in an embodiment of the present application.
[0072] like Figure 2 As shown, first, based on the data set, the seed queue is determined, and the controller selects seeds from the seed queue based on the adaptive seed selection strategy; then, based on the model to be tested, the metamorphosis relationship set is determined, and the controller selects the metamorphosis relationship from the metamorphosis relationship set based on the metamorphosis relationship selection strategy; secondly, the selected seeds are mutated based on the selected metamorphosis relationship to generate test cases; thirdly, the seeds and test cases are input into the model to be tested for execution, the execution results are output, and the execution results are verified; finally, the test cases are retained or discarded according to the verification results, the seed selection strategy and the metamorphosis relationship selection strategy are updated, and finally a test report is generated.
[0073] The metamorphosis relationship provides guidance for seed mutation, indicates the relationship between the seed and the test case output, and is the core component of the fuzzy test priority method based on the metamorphosis relationship in the embodiment of the present application. While using the metamorphosis relationship to mutate the seed and determine the test results, the method also provides a method for selecting the "best" metamorphosis relationship. The commonly used metamorphosis relationships in the image field are as follows.
[0074] MR1: Change the contrast of the original seed, and the output results of the mutated seed are consistent with the original seed.
[0075] MR2: Change the brightness of the original seed, and the output results of the mutated seed are consistent with the original seed.
[0076] MR3: The original seed is appropriately blurred, and the output results of the mutated seed are consistent with the original seed.
[0077] MR4: Add noise to the original seed, and the output of the mutated seed is consistent with the original seed.
[0078] MR5: The original seed is translated, and the output result of the mutated seed is consistent with the original seed.
[0079] MR6: Scale the original seed so that the output of the mutated seed is consistent with the original seed.
[0080] MR7: Prune the original seed so that the output of the mutated seed is consistent with the original seed.
[0081] MR8: Rotate the original seed, and the output of the mutated seed is consistent with the original seed.
[0082] Based on the above transformation relationship, a transformation relationship set is constructed, and a deep learning model is selected for testing. Assume that a certain number of test cases have been selected from the test case data set in a certain way (such as random selection) to construct a seed queue.
[0083] After that, an adaptive seed selection strategy is used to select a seed from the seed queue, and a difference-driven metamorphosis relationship selection strategy is used to select a metamorphosis relationship from the metamorphosis relationship set. A certain number of test cases are generated by mutating the seed k times according to the selected metamorphosis relationship, and a test case set is formed. The unlabeled seed and the generated test cases are input into the deep learning model for testing, and the output results of the model are obtained, and test information such as neuron coverage is collected.
[0084] After the above steps, the output results of the seed and the test case are obtained, and the execution results of the seed and the test case are compared to see if they violate the currently selected transformation relationship. If they do, the test case is added to the adversarial input set; otherwise, the generated test case is discarded.
[0085] Finally, the seed selection strategy and metamorphosis relationship selection strategy are self-updated according to the test results and the information collected during the test process (including coverage information, number of seed mutations, etc.), and the above steps are repeated until the stop condition of the fuzz test is reached, such as "execute 1,000 test cases".
[0086] by Figure 3 For example, Figure 3 An example diagram of a fuzzy test priority sorting method based on metamorphosis relationship provided in an embodiment of the present application.
[0087] In this exemplary embodiment, the open source deep learning model CNN-20 is tested, and the test case set used is the commonly used data set CIFAR10. The data set consists of 60,000 32x32 color images, divided into 10 categories. 100 pictures are randomly selected from each category to build a seed queue containing 1,000 pictures.
[0088] Specifically, first, select an image as a seed from the seed queue according to the adaptive seed selection strategy. Suppose the selected seed is Figure 3(a); Then, the metamorphosis relation MR2 is selected from the metamorphosis relation set according to the difference-driven metamorphosis relation selection strategy; under the guidance of MR2, the seed is adjusted after brightness, and the following is obtained: Figure 3 (b)(c)(d) Test cases shown. Since the seed has the label "bird", only the three test cases generated by mutation need to be sent to the model CNN-20 for testing, and check whether the output result of the model is "bird". After that, verify whether the seed label and the output results of the three test cases satisfy the metamorphosis relationship MR2. If the seed label and the output result of a test case are both "bird", that is, the output relationship between the seed and the test case satisfies the metamorphosis relationship MR2, it means that the test case is not an adversarial input, so the generated test case is discarded; otherwise, it means that the test case is an adversarial input and it is added to the adversarial input set.
[0089] Afterwards, the seed selection strategy and metamorphosis relationship selection strategy are updated according to the neuron coverage, test results and other information during the test, and the seed selection and metamorphosis relationship selection steps are repeated. When the number of executed test cases reaches 1000, the fuzzy test is stopped and the test report is output.
[0090] This application aims at the test expectation problems in the fuzzy testing of deep learning models, and proposes a fuzzy test priority sorting method based on metamorphosis relationship. The deep learning model has the characteristics of complex problems to be solved, probabilistic output of the model, and fault input will not cause obvious abnormal behavior, which leads to the prominent existence of test expectation problems in the deep learning testing process. The embodiment of the present application combines the general process of fuzzy testing and the principle of metamorphosis testing, and respectively selects seeds and metamorphosis relationships using adaptive seed selection strategy and difference-driven metamorphosis relationship selection strategy; obtains test cases based on metamorphosis relationship mutation seeds, and executes seeds and test cases; verifies the results by comparing whether the output results of seeds and test cases conform to the selected metamorphosis relationship; and further updates the seed selection strategy and metamorphosis relationship selection strategy according to the test results and test process information. This application not only provides a way for fuzzy testing to generate test cases and determine test results using metamorphosis relationships, but also uses test history information to sort the priorities of seeds and metamorphosis relationships, thereby improving the efficiency of fuzzy testing in revealing deep learning model faults.
[0091] Corresponding to the above method embodiment, this specification also provides a fuzzy test priority sorting device embodiment based on the metamorphosis relationship, Figure 4 A schematic diagram of the structure of a fuzzy test priority sorting device based on a metamorphosis relationship provided in an embodiment of the present application. Figure 4 As shown, the device comprises:
[0092] The seed selection module 402 is configured to select a target seed from the seed queue based on an adaptive seed selection strategy, wherein the adaptive seed selection strategy is a strategy for dynamically adjusting the seed selection probability based on the relevant information generated during the execution of the seed, and selecting the target seed based on the seed selection probability;
[0093] The relationship selection module 404 is configured to select a target metamorphic relationship from the metamorphic relationship set based on a metamorphic relationship selection strategy, and mutate the target seed based on the target metamorphic relationship to generate a test case set, wherein the metamorphic relationship selection strategy is to quantify the distance between the test case and the seed corresponding to each metamorphic relationship in the metamorphic relationship set and the test case generated under the guidance of different metamorphic relationships, and determine the strategy of the target metamorphic relationship based on the distance obtained after quantification;
[0094] The execution module 406 is configured to input the target seed and the test case set without a label into the model to be tested for execution, and output an execution result, wherein the execution result carries the target seed execution result and the test case execution result;
[0095] The verification module 408 is configured to verify the target seed execution result and the test case execution result based on the target metamorphosis relationship, obtain a verification result, and update the adaptive seed selection strategy and the metamorphosis relationship selection strategy based on the verification result and the relevant information generated by the seed during the execution process;
[0096] The iteration module 410 is configured to repeatedly select target seeds and target strategies based on the updated adaptive seed selection strategy and the updated metamorphosis relationship selection strategy until the number of iterations reaches a preset iteration threshold, stop the fuzzy test, and output the test results.
[0097] In an optional embodiment, the seed selection module 402 is further configured to:
[0098] Obtain seed-related information generated by various seeds in the seed queue during execution, and dynamically adjust the seed selection probability based on the seed-related information to generate a probability selection list, wherein the seed-related information carries seed coverage information and seed variation information;
[0099] Based on the probability selection list, a seed whose probability value meets a preset condition is determined as a target seed, wherein the preset condition is a condition for sorting various sub-selection probabilities in the probability selection list and determining a seed with the largest probability value as the target seed.
[0100] In an optional embodiment, the seed selection module 402 is further configured to:
[0101] Generate a seed variation list based on the test cases generated by various sub-variations in the seed queue and the time corresponding to the generation of the test cases, and generate an initial probability selection list based on the probability of various sub-variations being selected in the seed variation list;
[0102] Based on the seed coverage information generated by various seeds during execution, the seed variation list is divided into a first seed coverage list and a second seed coverage list, wherein the first seed coverage list is a list corresponding to the number of times the seed is selected and triggers a new coverage, and the second seed coverage list is a list corresponding to the number of times the seed is selected and does not trigger a new coverage;
[0103] Based on the initial probability selection list, the first coverage list and the second coverage list, the seed selection probability is dynamically adjusted to generate a probability selection list.
[0104] In an optional embodiment, the seed selection module 402 is further configured to:
[0105] Selecting a first seed in the initial probability selection list to perform seed mutation, and generating a test case set based on the first seed and the seed mutation strategy;
[0106] In response to the presence of at least one test case in the test case set triggering new coverage, dynamically adjusting the seed selection probability based on the first formula to generate a first probability selection list;
[0107] In response to the absence of a test case in the test case set triggering new coverage, dynamically adjusting the seed selection probability based on a second formula to generate a second probability selection list;
[0108] A probabilistic selection list is generated based on the first probabilistic selection list and the second probabilistic selection list.
[0109] In an optional embodiment, the relationship selection module 404 is further configured to:
[0110] According to each metamorphosis relation in the metamorphosis relation set, the target seed is mutated to generate an initial test case set;
[0111] Based on the third formula, quantify the compression distance between each test case in the initial test case set and the target seed;
[0112] Based on the compression distance obtained after quantization, a target metamorphosis relationship is selected from the metamorphosis relationship set, and based on the target metamorphosis relationship, the target seed is mutated to generate a test case set.
[0113] In an optional embodiment, the verification module 408 is further configured to:
[0114] In response to the target seed execution result and the test case execution result being consistent with the target metamorphosis relationship, discarding the test case;
[0115] In response to the target seed execution result and the test case execution result being inconsistent with the target metamorphosis relationship, the test case is added to the adversarial input set.
[0116] The device proposed in this application is applied, combined with the general process of fuzz testing and the principle of metamorphosis testing, and the seeds and metamorphosis relations are selected using the adaptive seed selection strategy and the difference-driven metamorphosis relationship selection strategy respectively; test cases are obtained by mutating seeds according to the metamorphosis relationship, and the seeds and test cases are executed; the results are verified by comparing whether the output results of the seeds and test cases conform to the selected metamorphosis relationship; and the seed selection strategy and the metamorphosis relationship selection strategy are further updated according to the test results and test process information. This application not only provides a method for fuzzy testing to generate test cases and determine test results using metamorphosis relationships, but also uses test history information to sort the priorities of seeds and metamorphosis relationships, thereby improving the efficiency of fuzzy testing in revealing deep learning model failures.
[0117] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the fuzzy test priority sorting device based on the metamorphosis relationship, since it is basically similar to the fuzzy test priority sorting method embodiment based on the metamorphosis relationship, the description is relatively simple, and the relevant parts can be referred to the partial description of the fuzzy test priority sorting method embodiment based on the metamorphosis relationship.
[0118] Figure 5 This is a block diagram of a computing device provided in an embodiment of the present application. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.
[0119] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).
[0120] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0121] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 500 may also be a mobile or stationary server.
[0122] The processor 520 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned fuzzy test priority sorting method based on the metamorphosis relationship.
[0123] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the fuzzy test priority sorting method embodiment based on the metamorphosis relationship, the description is relatively simple, and the relevant parts can be referred to the partial description of the fuzzy test priority sorting method embodiment based on the metamorphosis relationship.
[0124] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned fuzzy test priority sorting method based on the metamorphosis relationship.
[0125] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the fuzzy test priority sorting method embodiment based on the metamorphosis relationship, the description is relatively simple, and the relevant parts can be referred to the partial description of the fuzzy test priority sorting method embodiment based on the metamorphosis relationship.
[0126] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned fuzzy test priority sorting method based on the metamorphosis relationship.
[0127] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer program embodiment, since it is basically similar to the fuzzy test priority sorting method embodiment based on the metamorphic relationship, the description is relatively simple, and the relevant parts can be referred to the partial description of the fuzzy test priority sorting method embodiment based on the metamorphic relationship.
[0128] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0129] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0130] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present specification.
[0131] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0132] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.
Claims
1. A fuzzy test priority sorting method based on metamorphosis relationship, characterized in that: include: Selecting a target seed from a seed queue based on an adaptive seed selection strategy, wherein the adaptive seed selection strategy is a strategy for dynamically adjusting a seed selection probability based on relevant information generated during the execution of the seed, and selecting a target seed based on the seed selection probability; Based on the metamorphosis relationship selection strategy, a target metamorphosis relationship is selected from the metamorphosis relationship set, and based on the target metamorphosis relationship, the target seed is mutated to generate a test case set, wherein the metamorphosis relationship selection strategy is to quantify the distance between the seed and the test case generated under the guidance of different metamorphosis relationships according to the canonical compression distance between the test case and the seed corresponding to each metamorphosis relationship in the metamorphosis relationship set, and determine the strategy of the target metamorphosis relationship based on the distance obtained after quantification; Input the target seed and the test case set without labels into the model to be tested for execution, and output the execution result, wherein the execution result carries the target seed execution result and the test case execution result; Based on the target metamorphosis relationship, verify the target seed execution result and the test case execution result to obtain a verification result, and based on the verification result and the relevant information generated by the seed during the execution process, update the adaptive seed selection strategy and the metamorphosis relationship selection strategy; Based on the updated adaptive seed selection strategy and the updated metamorphosis relationship selection strategy, the target seed and the target strategy are repeatedly selected until the number of iterations reaches a preset iteration threshold, the fuzzy test is stopped, and the test results are output.
2. The method according to claim 1, characterized in that The step of selecting a target seed from a seed queue based on an adaptive seed selection strategy includes: Obtain seed-related information generated by various seeds in the seed queue during execution, and dynamically adjust the seed selection probability based on the seed-related information to generate a probability selection list, wherein the seed-related information carries seed coverage information and seed variation information; Based on the probability selection list, a seed whose probability value meets a preset condition is determined as a target seed, wherein the preset condition is a condition for sorting various sub-selection probabilities in the probability selection list and determining a seed with the largest probability value as the target seed.
3. The method according to claim 2, characterized in that The dynamically adjusting the seed selection probability based on the seed related information and generating a probability selection list comprises: Generate a seed variation list based on the test cases generated by various sub-variations in the seed queue and the time corresponding to the generation of the test cases, and generate an initial probability selection list based on the probability of various sub-variations being selected in the seed variation list; Based on the seed coverage information generated by various seeds during execution, the seed variation list is divided into a first seed coverage list and a second seed coverage list, wherein the first seed coverage list is a list corresponding to the number of times the seed is selected and triggers a new coverage, and the second seed coverage list is a list corresponding to the number of times the seed is selected and does not trigger a new coverage; Based on the initial probability selection list, the first coverage list and the second coverage list, the seed selection probability is dynamically adjusted to generate a probability selection list.
4. The method according to claim 3, characterized in that The step of dynamically adjusting the seed selection probability based on the initial probability selection list, the first coverage list, and the second coverage list to generate a probability selection list includes: Selecting a first seed in the initial probability selection list to perform seed mutation, and generating a test case set based on the first seed and the seed mutation strategy; In response to the presence of at least one test case in the test case set triggering new coverage, dynamically adjusting the seed selection probability based on the first formula to generate a first probability selection list; In response to the absence of a test case in the test case set triggering new coverage, dynamically adjusting the seed selection probability based on a second formula to generate a second probability selection list; A probabilistic selection list is generated based on the first probabilistic selection list and the second probabilistic selection list.
5. The method according to claim 1, characterized in that The transformation relationship selection strategy is based on the transformation relationship, and the target transformation relationship is selected from the transformation relationship set. Based on the target transformation relationship, the target seed is mutated to generate a test case set, including: According to each metamorphosis relation in the metamorphosis relation set, the target seed is mutated to generate an initial test case set; Based on the third formula, quantify the compression distance between each test case in the initial test case set and the target seed; Based on the compression distance obtained after quantization, a target metamorphosis relationship is selected from the metamorphosis relationship set, and based on the target metamorphosis relationship, the target seed is mutated to generate a test case set.
6. The method according to claim 1, characterized in that The target seed execution result and the test case execution result are verified based on the target metamorphosis relationship to obtain a verification result, including: In response to the target seed execution result and the test case execution result being consistent with the target metamorphosis relationship, discarding the test case; In response to the target seed execution result and the test case execution result being inconsistent with the target metamorphosis relationship, the test case is added to the adversarial input set.
7. A fuzzy test priority sorting device based on metamorphosis relationship, characterized in that: include: A seed selection module is configured to select a target seed from a seed queue based on an adaptive seed selection strategy, wherein the adaptive seed selection strategy is a strategy for dynamically adjusting a seed selection probability based on relevant information generated during the execution of the seed, and selecting a target seed based on the seed selection probability; The relationship selection module is configured to select a target metamorphic relationship from a metamorphic relationship set based on a metamorphic relationship selection strategy, and mutate the target seed based on the target metamorphic relationship to generate a test case set, wherein the metamorphic relationship selection strategy is to quantify the distance between the seed and the test case generated under the guidance of different metamorphic relationships according to the canonical compression distance between the test case and the seed corresponding to each metamorphic relationship in the metamorphic relationship set, and determine the strategy of the target metamorphic relationship based on the distance obtained after quantification; An execution module is configured to input the target seed and the test case set without a label into the model to be tested for execution, and output an execution result, wherein the execution result carries the target seed execution result and the test case execution result; A verification module is configured to verify the target seed execution result and the test case execution result based on the target metamorphosis relationship to obtain a verification result, and update the adaptive seed selection strategy and the metamorphosis relationship selection strategy based on the verification result and the relevant information generated by the seed during the execution process; The iteration module is configured to repeatedly select target seeds and target strategies based on the updated adaptive seed selection strategy and the updated metamorphosis relationship selection strategy until the number of iterations reaches a preset iteration threshold, stop the fuzzy test, and output the test results.
8. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the fuzzy testing priority sorting method based on metamorphosis relationship described in any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the fuzzy testing priority sorting method based on metamorphosis relationship as described in any one of claims 1 to 6.
Citation Information
Cited By
Dynamic simulation execution extension and verification method for instruction
CN121478646A
