Modeling specification and coverage test automation method based on SimulLink simulation model

Abstract

The invention belongs to the technical field of simulation test analysis, and relates to a modeling specification and coverage test automation method based on a SimulLink simulation model. A set of model building specifications is designed in a customized mode, specification detection can be automatically conducted on the model, meanwhile, automatic testing of the SimulLink model is completed on the basis of coverage testing requirements, and correctness and logic completeness of SimulLink model building are guaranteed. The invention innovatively provides a thought for customized modeling normative development so as to adapt to SimulLink module modeling design of different professions. Meanwhile, the internal structure of the model is analyzed in advance when the test cases are generated, effective cases are screened out to form a formal test case set by taking beneficial structure coverage as a screening rule, and for subsequent tests such as model upgrading and changing, the coverage test efficiency is greatly improved, a coverage approximate result can be predicted, and the test efficiency is improved. And meanwhile, defects of model structure logic can be accurately positioned.

Description

Technical Field

[0001] The invention belongs to the technical field of simulation test analysis, and relates to a modeling specification and coverage test automation method based on a SimuLink simulation model. Background Art

[0002] SimuLink is an important extension component of Matlab, focusing on providing a graphical modeling and simulation environment, especially suitable for the design, simulation and verification of dynamic systems and embedded systems. Its modeling function is powerful and can realize system modeling of linear, nonlinear, continuous, discrete, multivariable and hybrid structures. It is easy to operate and widely used. SimuLink itself has a rich model library, which users can call directly without delving into the underlying logic of each module in the model library. However, the convenient graphical operation makes the construction of the entire model architecture random and scattered, especially those multi-layer nested models with huge structures. There are many design risks in the grammatical correctness, logical completeness and rationality of the execution order. Therefore, to ensure that the design model is accurate and available, a large number of simulation tests must be done. At present, there is a lack of an automated testing method on the market to conduct a large number of comprehensive tests on SimuLink simulation models to ensure the correctness and logical integrity of the model construction.

[0003] One of the important indicators for measuring software testing quality in engineering is coverage testing. Good coverage can ensure the completeness and effectiveness of the test. Conventional coverage testing mainly analyzes three types of structural coverage: decision branch coverage DC (Decision Coverage), condition branch coverage CC (Condition Coverage), and modified condition / decision branch coverage MCDC (Modified Condition / Decision Coverage). Simulink has provided relevant toolboxes to help with effective testing. Simulink Design Verifier (SLDV) provides related functions such as detecting design errors and generating test cases. When generating test cases, the CC, DC, and MC / DC principles can be set through the configuration interface to select coverage. However, considering that the actual model is large and complex, it is difficult to generate complete coverage test cases. In addition, the underlying SLDV mainly relies on formal analysis, in which nonlinear modules will cause uncertainty in formal analysis, which will eventually lead to the failure of test case generation and the inability to complete coverage testing. Based on the above reasons, it is necessary to design a method that can be applied to all models and quickly achieve maximum coverage testing. Summary of the invention

[0004] Purpose of the Invention

[0005] In terms of modeling specifications, although some common specifications provided by SimuLink can be directly used for modeling specification checking, modeling specifications need to be customized according to the characteristics of each profession. This method follows the idea of ​​SimuLink modeling specifications, customizes the design checking specifications and can be automatically executed, which is more in line with design needs.

[0006] In terms of automatic test models, the traditional idea is generally to define interface variables, randomly deviate each variable, and randomly combine them into a large number of test cases for automated testing. However, the purpose of this method is to statistically analyze the impact of input parameter uncertainty on simulation results through a large number of tests, and the case generation is mainly to randomly deviate the input parameters in large quantities. The automated test in this article and the traditional method first aim to achieve fast and effective coverage testing. In addition, in terms of specific operation methods, this method first generates random test cases, then iteratively optimizes and selects test cases that are beneficial to coverage and non-repetitive, and then performs automatic testing. Finally, traditional coverage testing tools often cannot achieve coverage testing of nonlinear modules, but this method can solve this problem. Therefore, this method has a wider range of applications, can ensure maximum coverage testing, and can point out where logical branches are unreasonable, while the execution process is efficient and non-redundant.

[0007] Technical Solution

[0008] A modeling specification and coverage test automation method based on SimuLink simulation model, which customizes a set of model building specifications and can automatically perform specification detection on the model. At the same time, it completes the automated testing of the SimuLink model based on the coverage test requirements to ensure the correctness and logical completeness of the SimuLink model construction.

[0009] Furthermore, the method includes model specification customized design and inspection, and coverage test scheme. Through the SIMULINK component GUI tool design platform interactive interface, it is divided into two major items according to function: specification customized design and inspection, and coverage test scheme. Secondly, the two functional schemes are specifically implemented respectively.

[0010] Furthermore, the model specification customized design and inspection includes the following steps:

[0011] Step 1: Customize the design of modeling specification test inspection items according to the requirements, form a test specification set, and add it to the inspection specification library provided by SIMULINK. That is, according to the modeling specification test requirements, design specific test inspection specifications corresponding to the test requirements, form specific inspection execution items through the test software interactive interface, and formulate inspection result evaluation criteria and disposal methods for unqualified results. Generally, the inspection execution items should include: inspection specification name, inspection description, inspection action execution, inspection failure disposal method, etc. For details, please refer toFigure 2 According to different requirements of modeling specification testing, multiple test check items are formed and merged into a modeling specification set. The specification set is added to the test specification library provided by the SIMULINK platform, and the specification check can be performed through the SIMULINK built-in specification check tool.

[0012] Further, step 2: perform specification checking. Open the test model through the software test interactive interface, and open the SIMULINK built-in specification checking tool. From the inspection specification library, select the corresponding inspection specification items according to the test requirements, automatically batch execute the modeling specification checking and generate the result report.

[0013] Further, step three: check result evaluation. If the standard items designed in step one are selected for inspection, the test result report will display the inspection results according to the evaluation criteria designed in step one, which generally include correct, error, and warning.

[0014] Furthermore, the coverage test scheme includes the following steps:

[0015] Step 1): Load the model to be tested through the test software interface, automatically parse the internal structure of the model, and search for logical branches related to coverage. That is, parse the entire structure of the model, including the relevant modules in the sub-level that meet the coverage criteria, extract the predecessor and successor connection relationship of the module and the parameter information within the module.

[0016] Step 2): Automatically parse the external input interface of the model to be tested, set the input variable value range and random distribution through the test software interactive interface, and automatically generate a set of preliminary test cases. That is, obtain the model interface variables, and set the input range and waveform (sine, pulse, step, square wave, triangle wave, etc.) of the variables. A test case case needs to generate random numbers for all input interface variables according to their set waveforms. If the number of deviations N and the simulation time t are designed according to the size of the model, N preliminary test cases with a duration of t will be formed. These N cases are the preliminary test case set.

[0017] Step 3): Design screening criteria to form a formal test case set. Substitute the prepared test cases in step 2) into the model for simulation analysis, and set the number of simulation iterations n through the interactive interface, which means that n cases will be selected from N test cases for simulation screening. After each simulation is completed, the contribution value of the case to the coverage is counted. If the coverage is increased, the case is added to the formal test case set. Otherwise, the case is discarded and cases are continued to be selected for simulation screening. If the number of simulations reaches n times, or the coverage has reached 100%, the simulation screening is stopped. According to the above steps, the number of cases in the formal test case set is <= n.

[0018] Step 4): After the formal use case set is formed, the model test is automatically executed. After the model to be tested is opened, one or more cases in the formal use case set are selected through the software test interactive interface, and the test case input and model simulation are automatically executed in batches, and the simulation results are automatically saved and a test report is generated.

[0019] Step 5) Test result analysis: After the test is completed, a test coverage result report for the model is automatically generated. In addition to statistical coverage, the report can also locate the specific locations of logical branches in the model where coverage does not reach 100%.

[0020] Furthermore, the model specification customized design and inspection also includes step 4, which is the disposal of the inspection results, that is, whether to automatically modify or not, which depends on whether there is a design disposal item in step 1;

[0021] Furthermore, before step 1) of the coverage test plan, the model should contain a logical structure that can perform logic coverage testing and have an external input interface for designing input cases.

[0022] Furthermore, in the coverage test scheme, there are two special cases that cause the coverage to never reach 100%. Case 1 is the test case design problem, that is, the number of simulation iterations is insufficient or the numerical range or random distribution of the designed input variables is not appropriate, resulting in the generated cases causing some logical branches to never be executed, resulting in the coverage not reaching 100%; Case 2 is that the logical branch design of the model itself is not closed, resulting in the coverage not reaching 100%.

[0023] Furthermore, when the test result coverage does not reach 100%, first check the test report, locate the specific logic branch that has not been executed, and analyze and confirm which of the above two situations it is. If it is situation 1, optimize and adjust the numerical range of the input variables and the random deviation method according to step 2). If necessary, you can also increase the number of simulation iterations n; if it is situation 2, it means that there is a logical problem with the model itself, and the model can be modified and optimized. After the change is completed, you can use the formal use case set for automatic testing again, and check the coverage statistics to determine whether the change is correct.

[0024] The beneficial effects of this application are:

[0025] SimuLink model modeling has the characteristics of prominent individuality and scatteredness, which is not conducive to the standardization and inheritance of a professional design, and the modeling specifications provided by SimuLink do not meet the needs of professional modeling specification checking in various industries. The customization method in the present invention can help all professions realize customized modeling specification checking.

[0026] In terms of automated testing, conventional automated testing methods focus on randomly pulling the model input signal to form a large number of test cases, and then simulating through distributed batch processing to improve test efficiency. The purpose of the present invention is to achieve structural coverage testing, and after the test case is generated, the test case that is beneficial to the coverage increase is first screened out through an iterative algorithm. Therefore, the application of this method to form test cases can make subsequent testing work more effective and faster to test the maximum coverage of the object module, and can locate and find out the logical incompleteness of the model. Practical engineering applications show that the present invention is easy to use and the conclusions are reliable.

[0027] The key point and protection point of the present invention is that it innovatively provides a way of thinking for customized modeling normative development to adapt to the SimuLink module modeling design of different professions. At the same time, when generating test cases, the internal structure of the model is parsed in advance, and the effective cases are screened out with the structure coverage as the screening rule to form a formal test case set. For subsequent model upgrade and modification tests, the coverage test efficiency is greatly improved, and the approximate coverage results can be predicted, and the defects of the model structure logic can be accurately located. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Flowchart diagram;

[0029] Figure 2 Schematic diagram of customized modeling specifications (taking the example of checking whether the module name is under the module);

[0030] Figure 3 Schematic diagram of coverage test results (taking the MRef_Taem model as an example);

[0031] Figure 4 Schematic diagram of problem location in coverage test results. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the implementation of the present invention clearer, the technical scheme in the embodiment of the present invention will be described in more detail in combination with the embodiment of the present invention. In the example, the same or similar reference numerals throughout represent the same or similar originals or elements with the same or similar functions. The described embodiments are part of the embodiments of the present invention, not all of them. The embodiments described below by reference are exemplary and intended to be used to explain the present invention, and should not be construed as limitations on the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present invention. The following is a detailed description in combination with the embodiments of the present invention.

[0033] After the technical solution is implemented, any Simulink model can be automatically modeled and evaluated. The evaluation steps are as follows (the flowchart can be seen Figure 1 ):

[0034] Step A: Modeling specification check - define modeling specifications. On the customized modeling specification interface, define the inspection specifications according to the model inspection requirements, including the inspection specification name, inspection description, inspection action execution, inspection failure treatment, and write the corresponding code. You can design multiple specification requirements to form a modeling specification set.

[0035] Step B: Modeling specification check - automatically perform modeling specification check; in the check specification configuration, select the specification items that need to be checked for this test, execute with one click, and view the result report.

[0036] Step C: Modeling specification check - Handle items that fail the inspection: After the inspection is completed, the user can confirm whether the items that fail the inspection should be automatically modified according to the disposal plan specified in step A.

[0037] Step D, coverage test - check whether it is necessary to generate a formal test case set: If the structure related to coverage in the model structure has not changed and a formal test case set has been generated before, you can proceed directly to step F. If not, start iterating to generate a test case set according to step E.

[0038] Step E, coverage test - generation of preliminary test cases; set the input range and waveform (sine, pulse, step, square wave, triangle wave, etc.) of the model input variables, randomly deviate each variable, design the number of deviations N, simulation time t, and simulation iterations according to the size of the model, and form N preliminary test case cases with a time of t.

[0039] Step F, coverage test - formal test case generation: parse the structural branches in the model again, extract the logic related to the structural coverage, and screen the input cases that contribute to the total coverage of the model. If the case is beneficial to increase the coverage and is not repeated, add it to the formal test case set, otherwise discard the test case. If the number of iterations is reached or the coverage rate has reached 100%, the iteration stops. At the same time, the coverage statistics are completed, which should be consistent with the simulation test results to be performed later.

[0040] Step G, coverage test - automatically execute test cases; select any number of test cases from the formal test case set and automatically perform batch simulation tests.

[0041] Step H, coverage test - test result analysis and tracing of reasons for failure; after the automatic test is completed, a coverage result report is generated. The test result coverage is consistent with the coverage calculation when the test case is generated. For coverage that does not reach 100%, the reasons can be found in the result analysis report, and the corresponding modules can be traced back in the model, so that users can make modifications according to the actual situation.

[0042] It is understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of ordinary technicians in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and unless defined as here, will not be interpreted with idealized or overly formal meanings. The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Within the spirit and principle of the present invention, any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment of equivalent changes for application in other fields, but any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the technical solution content of the present invention, any modification, equivalent replacement, improvement, etc., made according to the technical essence of the present invention, should be included in the protection scope of the present invention.

Claims

1. A modeling specification and coverage test automation method based on SimuLink simulation model, characterized in that: A set of customized model building specifications can be designed and automatically tested for model specifications. At the same time, the automated testing of the SimuLink model can be completed based on the coverage test requirements to ensure the correctness and logical completeness of the SimuLink model construction.

2. The method according to claim 1, characterized in that The method includes model specification customized design and inspection, and coverage test plan; through the SIMULINK component GUI tool design platform interactive interface, it is divided into two major items according to function: specification customized design and inspection, and coverage test plan; secondly, the two functional plans are specifically implemented respectively.

3. The method according to claim 2, characterized in that The model specification customized design and inspection includes the following steps: Step 1: Customize the design of modeling specification test inspection items according to the requirements, form a test specification set, and add it to the inspection specification library that comes with SIMULINK; that is, design specific test inspection specifications corresponding to the modeling specification test requirements, form specific inspection execution items through the test software interactive interface, and formulate inspection result evaluation standards and disposal methods for unqualified results; the inspection execution items should include: inspection specification name, inspection description, inspection action execution, and disposal methods for inspection failed items; according to different requirements of modeling specification testing, form multiple test inspection items, merge them into a modeling specification set, and add the specification set to the test specification library that comes with the SIMULINK platform, so that the specification inspection can be performed through the SIMULINK built-in specification inspection tool.

4. The method according to claim 3, characterized in that Step 2: Perform specification check; open the test model through the software test interactive interface, and open the SIMULINK built-in specification check tool. From the check specification library, select the corresponding check specification item according to the test requirements, automatically batch execute the modeling specification check and generate a result report.

5. The method according to claim 4, characterized in that Step 3: Check result evaluation: If the standard items designed in step 1 are selected for inspection, the test result report will display the inspection results according to the evaluation criteria designed in step 1, including correct, error, and warning.

6. The method according to claim 2, characterized in that in, The coverage test scheme comprises the following steps: Step 1): Load the model to be tested through the test software interface, automatically parse the internal structure of the model, and search for logical branches related to coverage; that is, parse the entire structure of the model, including the relevant modules in the sub-level that meet the coverage criteria, extract the predecessor and successor connection relationship of the module and the parameter information within the module; Step 2): Automatically parse the external input interface of the model to be tested, set the input variable value range and random distribution through the test software interactive interface, and automatically generate a preliminary test case set; that is, obtain the model interface variables, and set the variable input range and waveform. A test case case needs to generate a random number for all input interface variables according to their set waveforms; if the number of deviations N and the simulation time t are designed according to the size of the model, N preliminary test case cases with a time of t will be generated, and these N cases are the preliminary test case set; Step 3): Design screening criteria to form a formal test case set; substitute the prepared test cases in step 2) into the model for simulation analysis, and set the number of simulation iterations n through the interactive interface, which means that n cases will be selected from N test cases for simulation screening; After each simulation is completed, the contribution value of the case to the coverage is counted. If the coverage is increased, the case is added to the formal use case set. Otherwise, the case is discarded and the case is selected for simulation screening. If the number of simulations reaches n times or the coverage has reached 100%, the simulation screening is stopped. According to the above steps, the number of cases in the formal use case set is <= n; Step 4): After the formal use case set is formed, the model test is automatically executed; after the model to be tested is opened, one or more cases in the formal use case set are selected through the software test interactive interface, and the test case input and model simulation are automatically executed in batches, and the simulation results are automatically saved and a test report is generated; Step 5) test result analysis; After the test is completed, the model test coverage result report is automatically generated; in addition to statistical coverage, the report can also locate the specific locations of logical branches in the model where coverage does not reach 100%.

7. The method according to claim 6, characterized in that The model specification customized design and inspection also includes step four, which is the disposal of the inspection results, that is, whether to automatically modify or not, which depends on whether there is a design disposal item in step one.

8. The method according to claim 7, characterized in that Before step 1 of the coverage test plan, the model should contain a logical structure that can perform logic coverage testing and have an external input interface for designing input cases.

9. The method according to claim 8, characterized in that In the coverage test plan, there are two special situations that cause the coverage to never reach 100%; Case 1, the test case design problem, that is, the number of simulation iterations is insufficient or the designed input variable numerical range or random distribution is inappropriate, resulting in the generated case causing some logical branches to not be executed, resulting in the coverage not reaching 100%; Case 2, the logical branch design of the model itself is not closed, resulting in the coverage not reaching 100%.

10. The method according to claim 9, characterized in that When the test result coverage does not reach 100%, first check the test report, locate the specific logic branch that has not been executed, and analyze and confirm which of the above two situations it is; if it is situation 1, optimize and adjust the numerical range of the input variables and the random deviation method according to step 2), and if necessary, increase the number of simulation iterations n; if it is situation 2, it means that the model itself has a logical problem, and the model can be modified and optimized; after the change is completed, the formal use case set can be used for automatic testing again, and the coverage statistics can be checked to determine whether the change is correct.

Images