Test system and test method

Abstract

To structure an environment where even an object to be tested for which a test is not provided by AUTOSAR can be tested by selecting and individually handling the object to be tested.SOLUTION: A test system is configured to perform: a test object selection step of sorting software to be tested into a first test object based upon software architecture and a second test object not based upon the software architecture, and generating incidental information representing the second test object; a standard conversion step of converting configuration parameters not based upon the standard specifications included in the second test object, into a format based upon the standard specifications; a first test step of creating a first test scenario from a first base test scenario and testing the first test object; and a second test step of creating a second test scenario from a second base test scenario and testing the second test object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a test tool. [Background technology]

[0002] In order to improve the efficiency of software development, standardization is being promoted in various fields and standards are being established. As a result, the impact of standardization is not limited to the software itself, but also applies to software testing, and tests based on the items in the standards are required.

[0003] For testing targeting AUTOSAR (AUTomotive Open System ARchitecture), AUTOSAR has defined and provided specifications for Acceptance Tests (AT) for each AUTOSAR function. However, since the tests targeted in acceptance tests are limited to the basic functions of AUTOSAR, it is necessary to create test scenarios for the configuration implemented in the product.

[0004] In addition, in AUTOSAR, when configuration parameters are changed, all source code is regenerated, and although the amount of source code generated is enormous, it is necessary to verify all patterns when conducting testing to ensure quality.

[0005] The following prior art exists as background art in this technical field: Patent Document 1 (JP 2000-000000) describes a test tool having configuration parameters that set an environment for an abstraction layer that abstracts hardware, API parameters, a combination extraction means that comprehensively extracts combinations of configuration parameters and combinations of API parameters, a test case generation means that generates test cases based on processes included in a processing sequence, an interface model that models the interface of the abstraction layer, and a test execution means that applies combinations of configuration parameters, combinations of API parameters, or test cases to the abstraction layer to test the abstraction layer. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-221803 Summary of the Invention [Problem to be solved by the invention]

[0007] The test tool described in Patent Document 1 uses the all-pairs method for testing AUTOSAR, using configuration parameters as factors in the method, selecting two factors from four target factors, and constructing a system that ensures 100% combinations of tests for the two selected factors. However, as mentioned above, this method cannot verify all patterns, and there are concerns about reduced test coverage and omission of test scenarios due to bias in the selected configuration parameter / API parameter combinations.

[0008] Therefore, there is a need to build an automated testing environment that has the function of converting and standardizing non-standard configuration parameters based on additional information, and the function of automatically generating test scenarios based on the additional information. [Means for solving the problem]

[0009] A representative example of the invention disclosed in the present application is as follows: That is, a test system for testing software in compliance with a software architecture based on a predetermined standard, the test system comprising a computer having an arithmetic unit that executes predetermined processing and a storage device accessible by the arithmetic unit, the arithmetic unit executing a test target selection step of sorting software to be tested into a first test target that complies with the software architecture and a second test target that does not comply with the software architecture and generating accompanying information indicating the second test target, a standard conversion step of converting configuration parameters included in the second test target that do not comply with the standard into a format that complies with the standard, a first test step of creating a first test scenario from a pre-registered first base test scenario and testing the first test target, and a second test step of creating a second test scenario from a pre-registered second base test scenario and testing the second test target. [Effects of the Invention]

[0010] According to one aspect of the present invention, it is possible to create an environment in which tests can be executed even for test targets for which tests are not provided in standards such as AUTOSAR, by selecting test targets and taking individual measures. Problems, configurations, and effects other than those described above will become clear from the description of the following embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating the overall configuration of a test tool according to a first embodiment. [Figure 2] FIG. 10 is a diagram showing the structure of a list according to the first embodiment. [Figure 3A] FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. [Figure 3B] FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. [Figure 3C] FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. [Figure 3D] FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. [Figure 3E] FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. [Figure 3F] FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. [Figure 3G] FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. [Figure 4] FIG. 1 is a diagram showing a test subject selection step in Example 1. [Figure 5] FIG. 1 is a diagram illustrating an overall configuration of test scenario creation steps according to a first embodiment. [Figure 6] FIG. 10 is a diagram showing the contents of a data table for creating a test scenario according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing the contents of a test definition file according to the first embodiment. [Figure 8] FIG. 10 is a diagram showing a test method and information on whether or not the test can be performed in the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating definition information of a CAN according to the first embodiment. [Figure 10] FIG. 1 is a diagram showing test results of Example 1. DETAILED DESCRIPTION OF THE INVENTION

[0012] Example 1 Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0013] FIG. 1 is a diagram showing the overall configuration of a test tool according to a first embodiment of the present invention.

[0014] The test tool of the first embodiment will be explained below by dividing it into the test target selection step 104, the standard conversion step 107, the test scenario creation step 113, and the test step 117.

[0015] In the test target selection step 104, the test tool of this embodiment targets the AUTOSAR DCM (Diagnostic Communication Manager), which requires testing for each configuration parameter and requires all pattern verification. Therefore, as many tests as there are configuration parameters are required, resulting in a huge number of test scenarios. Therefore, the test targets are selected as either configuration parameters that determine the behavior of the entire function or requirement-specific configuration parameters, making the test targets easier to handle.

[0016] Next, with reference to FIG. 2, a list 101 in which configuration parameters used by the test tool of this embodiment are described will be described.

[0017] 2 is a diagram showing the configuration of list 101. List 101 includes Parameters 202, session-service 203, session-localId 204, DID_List(OBD) 205, DID_List(OBD) 206, RID_List 207, and TID_List 208, and each list records configuration parameters used by the test tool of this embodiment, with the information shown in FIGS. 3A to 3G entered in each list.

[0018] FIG. 4 is a diagram showing the test subject selection step 104.

[0019] In the test target selection step 104, the list 101 is sorted into a first test target 102 and a second test target 103. For each selected test target, the first test target 102 is designated as a common part, and the second test target 103 is designated as a specific part. The first test target 102, which is the common part, collects basic configuration parameters required for operation as an AUTOSAR DCM, such as the basic communication parts and constant sizes used in AUTOSAR. On the other hand, the second test target 103, which is the specific part, collects detailed configuration parameters associated with the AUTOSAR basic configuration parameters and configuration parameters that change depending on product-specific specifications.

[0020] Next, we will explain the type determination step 201, which selects common parts and specific parts in the test object selection step 104. The "Specification" included in Parameters 202 shown in FIG. 3A is used as additional information. If any of session-service 203, session-localId 204, DID_List(UDS) 205, DID_List(OBD) 206, RID_List 207, and TID_List 208 is set in this item, it is determined to be a specific part. Then, additional information indicating a specific function is entered in Parameters 202 of FIG. 2, which is included in the second test object 103. As a result of the selection, as shown in FIG. 4, the list 101 is divided into the first test object 102, which is a common part, and the second test object 103, which is a specific part. The specific part is separated as a specific part because the number of tests and the tests to be performed change dynamically depending on the product-specific specifications, making it easier to manage.

[0021] In this embodiment, as a measure for automatically executing the test tool, an AUTOSAR-compliant list is created in the standard conversion step 107. The standard conversion step 107 identifies the configuration parameters specific to each test platform included in the above-mentioned list 101, and establishes a mechanism for converting them into configuration parameters that comply with AUTOSAR.

[0022] The overall configuration of the standard conversion step 107 will be described with reference to Fig. 1. For the first test target 102 and the second test target 103 selected in the test target selection step 104, a standard conversion tool 105 is used to combine multiple lists to create a single configuration list 106 that complies with AUTOSAR. For example, the standard conversion tool 105 has a correspondence table between unique configuration parameters and common configuration parameters, and converts the unique configuration parameters into common configuration parameters according to the designer's operations.

[0023] In the standard conversion step 107, the target of conversion is the configuration parameters for utilizing the unique specifications of each test platform. If these configuration parameters are reflected as they are in the list, it will no longer be a standard list for AUTOSAR.

[0024] Regarding the configuration parameters described above, the designer looks at the accompanying information described in Parameters 202 in FIG. 2, and considers the setting values ​​of the configuration parameters shown in FIG. 2 for the configuration parameters of the specific functions, and converts them into the values ​​considered when creating the configuration list 106.

[0025] Also, as mentioned above, if the unique specification is enabled, it will no longer be a standardized list, but since it may be used, it is possible to enable the setting itself. On the other hand, for configuration parameters that do not need to be converted, both the common part and the unique part are transcribed into the configuration list 106.

[0026] Next, in test scenario creation step 113, test scenarios are created for the first test target 102 and the second test target 103 using the additional information in the configuration list 106. For example, it is advisable to create a base test scenario in advance and create test scenarios based on the contents registered in the configuration list 106.

[0027] In test step 117, expected values ​​are incorporated into each base test scenario in advance to automatically judge the results obtained from the tool. The expected values ​​of the base test scenarios should be created by the designer based on the ISO (International Organization for Standardization) or AUTOSAR standard 301. The expected values ​​to be used are automatically set according to the configuration parameters registered in the configuration list 106.

[0028] 5 is a diagram showing the overall configuration of the test scenario creation step 113. Based on the configuration list 106, a first base test scenario 109 created from the AUTOSAR standard 301 prepared in advance is used to create a first test scenario 111 for the first test target 102 (common part), and a second base test scenario 110 created from the test scenario creation data table 302 is used to create a second test scenario 112 for the second test target 103 (specific part). Each test scenario creation method is described below.

[0029] The first test scenario 111 is a test scenario that targets configuration parameters required for AUTOSAR behavior in the test target selection step, and is created based on the first base test scenario 109 created from the AUTOSAR standard 301. Furthermore, the test scenario creation tool 108 selects necessary test scenarios and expected values ​​from the first base test scenario 109 in accordance with the configuration parameters registered in the configuration list 106, and creates the first test scenario 111.

[0030] The second test scenario 112 is created using the second base test scenario 110 based on the information in the configuration list 106. The second base test scenario 110 prepares a test scenario creation data table 302 for each of the lists included in the configuration list 106: session-service 203, DID_List(UDS) 205, DID_List(OBD) 206, RID_List 207, and TID_List 208.

[0031] Here, the test scenario creation data table 302 will be explained. Fig. 6 shows the contents of the test scenario creation data table 302 for the session-service 203, DID_List(UDS) 205, DID_List(OBD) 206, RID_List 207, and TID_List 208. As shown in Fig. 6, for each test scenario creation data table 302, a structure that can be defined as a structure is prepared in advance based on the number of valid IDs, data size, and whether or not they exist in the configuration list 106. Therefore, a configuration parameter with an invalid value entered in the configuration list 106 is determined to be an unnecessary test scenario. Furthermore, since session-service 203 defines the test area to be implemented, IDs that are not implemented are excluded from the test scenario.

[0032] Furthermore, in test step 117, it is necessary to define the ID and addressing method of the CAN (Controller Area Network) communication used by the test tool, so a test definition file 303 shown in FIG. 7 is created. The test definition file 303 shown in FIG. 7 includes test information 401, an ID list 402, and a CAN definition list 403. The test information 401 is created based on the test method and test implementation information shown in FIG. 8, and the test scenario name is registered in the test information 401. The test scenario registered in the test information 401 is executed in the subsequent test step 117. The CAN ID list 402 and the CAN definition list 403 are definition information of the CAN used for communication with the tool in test step 117. This definition information defines the CAN ID and addressing method for each test scenario based on the CAN definition information shown in FIG. 9. This information is also incorporated into the test scenario creation data table 302, and the test definition file 303 is created.

[0033] Next, the related information extraction step will be described. The related information extraction step is performed by referring to the configuration list 106. The configuration list 106 is written so that the characteristics of each configuration parameter can be understood, so when a designer changes or checks a test scenario, he or she can easily select the configuration parameters of the corresponding test scenario. In addition, since the configuration list 106 is a list organized by AUTOSAR configuration parameters, the designer can easily check the test scenario related information.

[0034] In this embodiment, in test step 117, the test scenario created in test scenario creation step 113 is executed. The test is executed in accordance with the contents of the test definition file 303. The test results 116 obtained by executing the test are compared with the expected values ​​defined in the base test scenario, and the test judgment results are notified for each test scenario.

[0035] The overall configuration of the test step 117 will be described with reference to Fig. 5. In the test step 117, the first test scenario 111 created in the test scenario creation step 113 is tested in a first test step 114 based on the test definition file 303, and the expected values ​​described in the first base test scenario 109 are compared with the test result 116 to make a test judgment. Also, the second test scenario 112 created in the test scenario creation step 113 is tested in a second test step 115 based on the test definition file 303, and the expected values ​​described in the second base test scenario 110 are compared with the test result 116 to make a test judgment.

[0036] Figure 10 shows the test results 116, which is a file that records the results of the test. The values ​​below the expected values ​​in the figure are the expected values ​​obtained from the base test scenario, and the actual measured values ​​are the test results. The results are judged by comparing the CAN data and using a tool to determine whether the sent and received data match, and the result is recorded as normal or incorrect in the judgment section in the table.

[0037] In addition, since all test scenarios are associated with the configuration list 106, the correspondence is clear regarding the test of the configuration parameter whose test judgment result is invalid, and it is possible to use the information in the configuration list 106 to analyze the invalid setting value and the cause.

[0038] Furthermore, in the test step 117, it is preferable to extract test objects whose test results indicate an abnormality based on the test results in the first test step and the test results in the second test step. In this way, test objects whose test results indicate an abnormality can be extracted, and the efficiency of the designer's work to address the problem can be improved.

[0039] Next, a computer on which the test tool of this embodiment operates will be described. The test tool of this embodiment is a program executed on a computer having a processor (CPU), memory, auxiliary storage device, and a communication interface. The computer on which the test tool runs constitutes a test system.

[0040] The processor is a computing device that executes programs stored in memory. The processor executes various programs to realize various functions of the test tool. Note that some of the processing performed by the processor by executing the programs may be executed by another computing device (for example, hardware such as an ASIC or FPGA).

[0041] The memory includes ROM, which is a non-volatile storage element, and RAM, which is a volatile storage element. ROM stores unchanging programs (e.g., BIOS), etc. RAM is a high-speed, volatile storage element such as DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 1 and data used when the programs are executed.

[0042] The auxiliary storage device is a large-capacity, non-volatile storage device such as a magnetic storage device (HDD) or flash memory (SSD). The auxiliary storage device also stores data used by the processor when it executes a program, and the program to be executed by the processor. That is, the program is read from the auxiliary storage device, loaded into memory, and executed by the processor to realize each function of the test tool.

[0043] The communication interface is a network interface device that controls communication with other devices in accordance with a predetermined protocol. The computer may have an input interface and an output interface.

[0044] The program executed by the processor is provided to the computer from removable media (CD-ROM, flash memory, etc.) or via a network and stored in a non-volatile auxiliary storage device, which is a non-transitory storage medium. For this reason, the computer should have an interface for reading data from removable media.

[0045] The test tool executed by the computer of this embodiment is a computer system configured on one physical computer, or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. For example, each functional unit may operate on a separate physical or logical computer, or multiple functional units may be combined to operate on a single physical or logical computer.

[0046] As described above, the test tool of this embodiment proposes a method and environment for efficiently creating test scenarios in order to verify all patterns of a huge number of test scenarios targeted for AUTOSAR, and the test tool includes a test target selection step 104 in which a computing device selects software to be tested into a first test target 102 that conforms to a software architecture (for example, the AUTOSAR standard) and is a test target targeted for basic configuration parameters 203 to 208 of the software architecture, and a second test target 103 that does not conform to the software architecture and is targeted for detailed configuration parameters associated with the basic configuration parameters 203 to 208, and generates additional information indicating that the software is the second test target; The method includes a standard conversion step 107 in which a configuration parameter that does not comply with the standard is converted into a format that complies with the standard, a first test step 114 in which the computing device creates a first test scenario 111 from a first base test scenario 109 that has been registered in advance and tests a first test target 102, and a second test step 115 in which the computing device creates a second test scenario 112 from a second base test scenario 110 that has been registered in advance and tests a second test target 103. Therefore, the test targets in the list 101 are selected based on whether they comply with the software architecture, and further, test targets that do not comply with the software architecture are converted into a format that complies with the software architecture, thereby expanding the scope of software test automation, reducing the man-hours required to create test scenarios, reducing the burden on designers, and shortening the test period.

[0047] In addition, since test scenarios are created from base test scenarios according to the setting values ​​of the configuration list, all patterns can be verified, and the omission of test scenarios can be suppressed, thereby suppressing a decrease in test coverage. In other words, if all patterns are verified for each configuration parameter, the number of test scenarios will become enormous, but by adding ingenuity in the creation of configuration parameters and test scenarios, it is possible to achieve efficient testing.

[0048] Furthermore, the test object selection step 104 includes a type determination step 201 in which the calculation device determines the type of the test object based on the additional information of the first test object 102 and the additional information of the second test object 103, so that selection can be performed easily and with high accuracy based on the additional information.

[0049] In addition, in the standard conversion step 107, the computing device converts non-standard configuration parameters into standard-compliant configuration parameters for the test target whose type has been determined, and creates a configuration list 106 that includes the characteristics of the converted configuration parameters. For example, in the case of a DCM that uses CAN communication, the CAN ID, etc. is converted into a format that complies with the software architecture. Therefore, conversion according to the type of test target can improve conversion accuracy and conversion efficiency.

[0050] Furthermore, in the test scenario creation step 113, the computing device selects the tests necessary for the first test subject 102 based on the additional information in the common part of the configuration list 106, creates a first test scenario 111 using a first base test scenario 109 that has been registered in advance, and selects the tests necessary for the second test subject 103 based on the additional information in the specific part of the configuration list 106, and creates the second test scenario using a second base test scenario 110 that has been registered in advance, so that the necessary tests can be automatically selected based on whether they are listed in the configuration list 106 and test scenarios can be created.

[0051] In addition, in the related information extraction step, the computing device extracts information related to the base test scenario used to create the second test scenario 112. Therefore, when the designer creates the second test scenario by hand-coding, the automatically extracted related information can be used, thereby improving work efficiency.

[0052] Furthermore, in the test target extraction step, test targets whose test results indicate an abnormality are extracted based on the test results in the first test step 114 and the test results in the second test step 115, so that test targets whose test results indicate an abnormality can be extracted, improving the efficiency of the designer's work in dealing with defects.

[0053] The present invention is not limited to the above-described embodiments, but includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0054] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0055] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0056] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0057] 101 Configuration List 102 First Test Subject 103 Second Test Subject 104 Test Subject Selection Steps 105 Configuration Parameter Standard Conversion Tool 106 Configuration List 107 Standard Conversion Steps 108 Test scenario creation tool 109 First base test scenario file 110 Second base test scenario file 111 First test scenario file 112 Second test scenario file 113 Test Scenario Creation Steps 114 First Test Step 115 Second Test Step 116 Test Results 117 test steps 201 Classification determination step 202, 202, 203, 204, 205, 206, 207, 208 Configuration parameter files 301 AUTOSAR Standard 302 Data table for creating test scenarios 303 Test Definition File 401 Test Information 402 CAN ID List 403 CAN definition list

Claims

1. A test system for testing software in compliance with a software architecture based on a predetermined standard, The test system includes: a computer having an arithmetic unit that executes predetermined processing and a storage device that can be accessed by the arithmetic unit; The computing device a test target selection step of sorting the software to be tested into a first test target that conforms to the software architecture and a second test target that does not conform to the software architecture, and generating supplemental information indicating that the software is the second test target; A standard conversion step of converting configuration parameters that do not comply with the standard included in the second test target into a format that complies with the standard; a first test step of creating a first test scenario from a pre-registered first base test scenario and testing the first test target; a second test step for creating a second test scenario from a second base test scenario registered in advance and testing the second test target.

2. 10. The test system of claim 1, A test system characterized in that in the test target selection step, the calculation device determines the type of test target based on additional information of the first test target and additional information of the second test target.

3. 3. The test system of claim 2, In the standard conversion step, the arithmetic device converts configuration parameters that do not comply with the standard included in the second test target into configuration parameters that comply with the standard, and creates a configuration list including characteristics of the converted configuration parameters. A test system characterized by the above.

4. 4. The test system of claim 3, the computing device executes a test scenario creation step of creating a test scenario; In the test scenario creation step, the computing device selects tests necessary for the first test target based on additional information in the common part of the configuration list, and creates the first test scenario using a first base test scenario registered in advance; A test system characterized in that the calculation device selects tests necessary for the second test target based on additional information in the unique part of the configuration list, and creates the second test scenario using a pre-registered second base test scenario.

5. 5. The test system of claim 4, A test system, comprising: a related information extraction step in which the computing device extracts information related to a base test scenario used to create the second test scenario.

6. 10. The test system of claim 1, A test system comprising a test object extraction step of extracting test objects whose test results show an abnormality based on the test results in the first test step and the test results in the second test step.

7. A testing method in which a computer tests software in compliance with a software architecture based on a predetermined standard, comprising: the computer has an arithmetic unit that executes predetermined processing and a storage device that can be accessed by the arithmetic unit; The test method comprises: a test target selection step in which the computing device sorts the software to be tested into a first test target that conforms to the software architecture and a second test target that does not conform to the software architecture, and writes additional information indicating that the software is the second test target; A standard conversion step in which the arithmetic device converts configuration parameters that do not comply with the standard and are included in the second test target into a format that complies with the standard; a first test step in which the computing device creates a first test scenario from a pre-registered first base test scenario and tests the first test target; a second test step in which the computing device creates a second test scenario from a pre-registered second base test scenario and tests the second test target.

Images