A Simulink Test Case Reduction Method Based on Result Backtracking

Abstract

The present invention discloses a Simulink test case reduction method based on result backtracking, including: generating test cases that can reflect errors or defects in the software under test, establishing a test case set to be reduced, and collecting a third-party real defect feedback case set of the software under test for expanding the test case set to be reduced; running the test cases in the test case set to be reduced, and classifying the test cases according to the running results; adopting a result-oriented module deletion equivalent test case reduction scheme for the test cases that fail to compile in the running results, and adopting a result-oriented data flow equivalent test case reduction scheme for the test cases that are successfully compiled in the running results; summarizing the reduced test case set that is consistent with the running results of the original test case set, recording the induced cause report, and uploading it to the official technical support of the software under test; collecting the opinions and suggestions of the official technical support of the software under test, and optimizing and iterating this method.

Description

Technical Field

[0001] The present invention relates to the field of software testing, and particularly to a Simulink test case reduction method based on result backtracking. Background Art

[0002] Simulink has the advantages of wide adaptability, clear structure and process, fine simulation, close to reality, high efficiency, flexibility, etc. Based on these advantages, it is widely applied to core fields such as aerospace, automobile manufacturing, and life and health. Therefore, if there are bugs in the Simulink toolchain, it will cause compilation errors and may potentially bury unexpected latent problems in the system. In addition, compiler errors make debugging more difficult because it is very hard for developers to determine whether the software failure is caused by the software they are developing or the compiler they are using. Therefore, it is crucial to ensure the quality of the Simulink toolchain compiler. In the worst case, a subtle error in the Simulink toolchain may lead to unexpected behaviors in core safety applications such as cars or airplanes, causing huge casualties and losses.

[0003] Currently, in the field of software testing, there is no test case reduction method for the Simulink compiler of the software under test. Engineers may find errors (or defects) in the Simulink compiler of the software under test in actual engineering projects. However, due to the overly large models used in engineering operations and other objective factors such as project duration, it is impossible to reduce the model with a large amount of functions to a scale and size that meets the official technical support (in fact, the official technical support cannot analyze the complete model either, which is too complex, so engineers often need to reduce it to a small scale by themselves and then submit it). Most of the existing solutions are manual reduction by engineers, with extremely high reduction difficulty and greatly increased time consumption for engineers. Summary of the Invention

[0004] According to the problems existing in the prior art, the present invention discloses a Simulink test case reduction method based on result backtracking, which specifically includes the following steps:

[0005] Generate test cases that can reflect the errors or defects in the software under test, establish a test case set to be reduced, and collect a set of real defect feedback cases of third parties of the software under test for expanding the test case set to be reduced;

[0006] Run the test cases in the test case set to be reduced, and classify the test cases according to the running results;

[0007] Delete the modules in the test cases with compilation failures in the running results using a result-oriented module deletion equivalent test case reduction scheme; delete the modules in the test cases with successful compilation in the running results using a result-oriented data flow equivalent test case reduction scheme;

[0008] Run the reduced test case set, compare the differences in the running results with the original test case set, and record the reasons for errors or defects as well as the test case modules that induce problems;

[0009] Summarize the reduced test case set with the same running results as the original test case set, record the inducement reports, and upload them to the official technical support end of the software under test;

[0010] Collect the opinions and suggestions from the official technical support end of the software under test, and optimize and iterate the Simulink test case reduction method.

[0011] Furthermore, the test cases with errors or defects are embodied as models in the software under test;

[0012] Generate a large number of software models under test through existing methods of randomly generating software models under test and existing model mutation methods;

[0013] Select the software models under test with errors or defects from the above-mentioned large number of software models under test through differential testing means to form a test case set to be reduced;

[0014] Collect the models with specific error reasons and error-triggering methods publicly disclosed by the official technical end of the software under test and the model cases of real defect feedback from third-party open sources to expand the test case set to be reduced for the software under test.

[0015] Furthermore, according to the running results of the test cases, the test cases are divided into test cases with compilation failures and test cases with successful compilation; compilation failure means that the test case crashes or encounters an error during the running on the software under test and cannot continue to be compiled, and the software under test forcibly terminates the running of the test case by popping up a prompt message or a crash message; compilation success means that the test case runs normally on the software under test and outputs a result.

[0016] Furthermore, when running the reduced test cases, the results should be the same as those of the original test cases. Specifically, if the original test case fails to compile, the reduced test case set should also fail to compile and output the same prompt message; if the original test case compiles successfully, the reduced test case should also compile successfully and output the same result after running.

[0017] Due to the adoption of the above technical solution, a Simulink test case reduction method based on result backtracking provided by the present invention classifies and summarizes test cases containing errors (defects), and selects targeted reduction schemes for different error reasons (running results). To a certain extent, it is beneficial to improve the success rate of test case reduction. Using this method to reduce test cases also greatly reduces the labor expenditure, saves the time spent by engineers, contributes to the development of automated testing, more accurately locates the error reasons and modules (groups) hidden in the software under test, and improves the test effect. Brief Description of the Drawings

[0018] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 is a flowchart of the method of the present invention;

[0020] Figure 2 is a schematic diagram of test cases in the present invention;

[0021] Figure 3 is a schematic diagram of the reduced test cases in the present invention; Detailed Embodiments

[0022] To make the technical solutions and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention:

[0023] As Figure 1 shown, a Simulink test case reduction method based on result backtracking specifically includes the following steps:

[0024] In step S101: Collect and expand the test case set to be reduced;

[0025] Specifically, in this step, a test case set to be reduced is established, which mainly comes from the test cases that can expose or cause errors in the tested software and have been verified by the official technical support of the tested software during the previous testing process of the tested software. Errors usually refer to the situation where the compilation fails and the equivalent film input is inconsistent. Among them, compilation failure means that during the running process of the test case on the tested software, a vulnerability in the compiler of the tested software is triggered, resulting in compilation failure and the test case being forcibly terminated on the tested software with a prompt box popped up. More seriously, it may directly cause the tested software to crash and a crash message to be popped up. The situation of inconsistent equivalent film input usually occurs between two test cases with the same performance, and there is a difference in the same external environment and configuration of the tested software. This difference is reflected in that one test case runs normally while the other fails, or both test cases can run but the output results are different. The output results include but are not limited to the signal difference and time vector difference of the modules in the test case. However, relying solely on these cannot provide data support for this method. Therefore, it is also necessary to obtain and screen out the models that can reproduce the errors from the publicly available bug reports of the tested software as test cases, as well as the third-party real defect feedback case sets of the tested software publicly available on open source websites such as Github.

[0026] In step S102: Run the test cases and classify them;

[0027] In this step, all the test cases in the test case set to be reduced are run, and they are classified according to their result performance. Specifically, they can be divided into two major categories: compilation failure and inconsistent running results. The specific classification method is designed according to the reduction processing scheme in the subsequent steps. For compilation failure, the tested software usually directly helps to locate near the module of the faulty test case, which helps to quickly screen out irrelevant modules in large quantities and greatly improves the reduction efficiency. However, this does not mean that it can be directly reported without reduction (in fact, the tested software may have false error reports or compilation failures caused by multiple complex modules). Therefore, for the test cases with compilation failure, the error can be quickly located in the subsequent steps through prompts or the key of whether the compilation is successful. Therefore, we classify them into one category. Compilation failure in this step specifically refers to the test case being forcibly terminated on the tested software and an error well cover message or a crash message being popped up. The other category is inconsistent running results, which is usually more hidden. In fact, the more complex the test case, the more likely it is that a single move affects the whole situation. The running result cannot directly point to a single module in a certain test case. Instead, the module that causes the error will lead to a large number of errors in signal data, time vectors, etc. This is also the main reason why test case reduction is difficult and time-consuming.

[0028] In step S103: Reduce the test cases;

[0029] After classifying the test case set to be reduced in the early stage, for the classified test cases, we can take targeted measures to break them one by one. For the test cases with compilation failures in the running results, we adopt a result-oriented module deletion equivalent test case reduction scheme. For the test cases with successful compilation in the running results, we adopt a result-oriented data flow equivalent test case reduction scheme.

[0030] Specifically, the result-oriented module deletion equivalent test case reduction scheme quickly reduces the scale and locates by proportionally deleting modules in the test cases with the goal of maintaining compilation failures. Since compilation failure errors usually have prompt messages and help locate a specific module, the test cases are automatically arranged (usually the software under test can be automatically implemented through the API). First, batch delete the irrelevant modules that do not have data and control dependencies with the module. Then, batch specify the module group after deleting the module. Finally, delete the module group before deleting the module. After each deletion, the model needs to be run to observe whether the compilation failure problem can be reproduced. If it cannot be reproduced, backtrack to the previous stage and gradually reduce the amount of deletion (the minimum is in units of standard modules) until the compilation failure problem can be exactly reproduced.

[0031] Figure 2 As a schematic diagram of a test case to be reduced, we assume that an error occurs in module C23, causing the test case to fail compilation on the software under test. At this time, since module F26 and module G27 are the subsequent signal outputs of module C23 and will not affect the result of module C23, we delete them. Further, the subsequent modules D24, E25, and H28 after module C23 will affect module C because of the loop. It is necessary to carefully judge whether the test case cannot normally reproduce the compilation failure after deleting these modules. Therefore, after each deletion, it is necessary to consider whether to delete the module according to whether the compilation failure can be normally reproduced in the running results. Assume that module D24, module E25, and module H28 will not affect module C23, then delete them, and consider module B22 before module C23. After deleting it and running, the compilation failure cannot be reproduced, so it cannot be deleted and is retained. The same is true for module A21 after deletion. Finally, after reduction, as Figure 3 shown.

[0032] The result-oriented data flow equivalent test case reduction scheme usually occurs in a more complex test environment. At this time, due to signal value or time vector errors caused by a certain module, a batch of error chain reactions occur in the same test case model under different modes, and it is difficult to judge the specific error cause and error location. The following steps need to be taken:

[0033] 1. Automatically arrange the test case set;

[0034] 2. Delete the modules and paths that are completely irrelevant to the error module group in the signal data flow;

[0035] 3. Delete all subsequent modules of the error module whose signal is close to the total output of the model in the known chain reaction;

[0036] 4. Delete the error modules close to the total output of the model in the test case one by one and run to see if the results can be reproduced. If not, go back to the state before deletion.

[0037] 5. For the model that meets 4, delete the erroneous modules close to the total input of the model one by one and observe and run to see if the problem can be reproduced. If not, go back to the state before deletion.

[0038] 6. Finally, we get the simplified test case set.

[0039] Similarly, Figure 2 For example, assume that module A 21, module B 22, module C 23, and module D 24 have inconsistent signals in their running results under different modes of the tested software. At this time, the test case set is first automatically arranged (arrangement has been completed in this test case diagram), and completely irrelevant modules and paths are deleted. At this time, module F 26 and module G 27 do not affect the above-mentioned inconsistent result modules, and are directly deleted according to step 2. Next, according to step 3, delete module E 25 and module H 28 after the error module D 24 closest to the total output of the model, because they do not produce inconsistent situations. Next, according to step 4, delete module D 24 and run the model. At this time, the inconsistent results can still be reproduced, which proves that module D 24 is just affected and is not the module that causes the error, so it is deleted. Then delete module C 23 and run the test case. At this time, it is observed that the inconsistent situation disappears, so module C 23 cannot be deleted. Enter step 5 to delete module A21 and module B 22. After running, no inconsistent situation is observed, so they cannot be deleted. According to step 6, the simplified test case model is as follows: Figure 3 shown.

[0040] In step S104: comparing and marking the error-inducing causes;

[0041] Run the reduced test case set, compare the differences in the running results with the original test case set, and record the reasons for errors (or defects) and the test case modules (groups) that induce problems. It should be particularly noted that. In this step, it is necessary to verify whether the problems reported by the reduced test cases have been made public by the official, and whether there is a situation where the test cases have not been reduced to the simplest form, that is, whether it is possible to continue to simplify, or whether it can be regarded as two different errors for reporting. It is necessary to manually confirm again and delete the duplicate test cases from the test case set, rename and label the modules for the inducing reasons, and save the error information and documents for the subsequent writing and submission of the bug analysis report.

[0042] In step S105: Summarize and report test cases;

[0043] In this step, for the test cases that have been successfully reduced and have not been publicly announced by the official of the software under test, write a bug analysis report according to the requirements of the official of the software under test and summarize and submit it. In addition, classify and summarize the types of reasons for errors, count information such as the reduction time, and conduct a chart analysis to examine whether the current method can meet the test case reduction work of the software under test. Timely summarize and iteratively supplement this method for the newly discovered and undiscovered test case error types.

[0044] The above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered by the protection scope of the present invention.

Claims

1. A Simulink test case reduction method based on result backtracking, characterized in that Including: Generate test cases that can reflect errors or defects in the software under test, establish a test case set to be reduced, and collect a set of third-party real defect feedback cases of the software under test for expanding the test case set to be reduced; Run the test cases in the test case set to be reduced, and classify the test cases according to the running results; Adopt a result-oriented module deletion equivalent test case reduction scheme to delete modules in the test cases with compilation failures in the running results; adopt a result-oriented data flow equivalent test case reduction scheme to delete modules in the test cases with successful compilation in the running results; Run the reduced test case set, compare the difference in running results with the original test case set, and record the reasons for errors or defects and the test case modules that induce problems; Summarize the reduced test case set with the same running results as the original test case set, record the inducement report, and upload it to the official technical support end of the software under test; Collect opinions and suggestions from the official technical support end of the software under test, and optimize and iterate this Simulink test case reduction method; The test cases of the errors or defects are embodied as models in the software under test; Generate a large number of models of the software under test through the existing methods of randomly generating models of the software under test and the existing method of model mutation; Screen out the models of the software under test with errors or defects from the above-mentioned large number of models of the software under test through differential testing means to form a test case set to be reduced; Collect models with specific error reasons and error-triggering methods publicly disclosed by the official technical end of the software under test and third-party open-source real defect feedback case models to expand the test case set to be reduced of the software under test.

2. The method according to claim 1, wherein: According to the results of running test cases, the test cases are divided into test cases with compilation failures and test cases with successful compilation; Compilation failure means that the test case crashes or encounters an error during running on the software under test and cannot continue compilation, and the software under test forcibly terminates the running of the test case by popping up a prompt message or a crash message, resulting in failure; successful compilation means that the test case runs normally on the software under test and outputs a result.

3. The method according to claim 1, wherein: Running the reduced test cases, the results should be the same as the original test cases. Specifically, if the original test case fails to compile, the reduced test case set should also fail to compile and output the same prompt message; if the original test case compiles successfully, the reduced test cases should also compile successfully and output the same result after running.

Images