Test equipment, test methods, and test programs

Abstract

To improve the accuracy of determining pass / fail of a test.SOLUTION: A test apparatus is configured to: execute a test for a test target according to a test scenario in which a first expectation value required for operation and instructions to cause the test target to execute the operation and the order of the instructions are defined; acquire, from the test target, first time-series data which are time-series first measurement values regarding the operation associated with the order; identify a correspondence relation between a first measurement time of the first measurement values and a second measurement time of second measurement values, on the basis of the first time-series data and second time-series data in which the time-series second measurement values are defined which are associated with the order of the operation according to the test scenario; adjust the first measurement time on the basis of a time difference between the first measurement time and the second measurement time; and determine whether the test target has passed or failed the test, on the basis of the first measurement values corresponding to the adjusted first measurement time and the first expectation value corresponding to the order in the adjusted first measurement time.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a test apparatus, a test method, and a test program for testing a test object. [Background technology]

[0002] Generally, when performing automated testing, it is necessary to create a test scenario in advance. This test scenario must describe the operating procedure, the expected value used for judgment, and the judgment point. The above test scenario is input to the test subject, time-series output waveform data is acquired, the output waveform data is compared with the expected value at the corresponding judgment point (time), and the pass / fail result is output.

[0003] Patent Document 1 discloses a misfire detection method for detecting the occurrence of a misfire in an internal combustion engine. This misfire detection method includes a measurement step of measuring the exhaust temperature of the internal combustion engine, a synchronization step of synchronizing a target waveform of the exhaust temperature measured in the measurement step with a predetermined reference waveform so that the cross-correlation is high, and a determination step of aligning the synchronized target waveform with the synchronized reference waveform based on a dynamic time stretching method, and determining whether or not a misfire has occurred based on the difference between the aligned target waveform and the reference waveform.

[0004] Patent Document 2 discloses a verification system for verifying the operation of a program executed in a device. In this verification system, a tester can visually recognize test specifications representing the verification method, for example, in the form of a sequence diagram, via a UML modeling tool 50 (hereinafter referred to as the tool). The information representing the sequence diagram created by the tester via the tool is converted into a test script that can be recognized and executed by a CAN bus monitor by the script generation function of the tool. The CAN bus monitor monitors the operation of the ECU to be verified, determines whether the actual operation of the ECU matches the test specifications represented by the test script, and outputs a test report representing the determination result to an external source. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2012-97603 [Patent Document 2] Japanese Patent Publication No. 2010-15240 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, in the case of real-time simulation (especially real-time simulation including hardware), even if the exact same test scenario is performed, the acquired output waveform data will not be perfectly identical. This is because latency, a hardware factor, and processing delays due to the processing accuracy (real-time performance) of executing the scenario occur. As a result, deviations from the expected judgment point occur, leading to a failure of the test result.

[0007] Currently, if a test fails, the test scenario is reviewed again (the evaluation points are adjusted), and the automated test is run again. Alternatively, an explanation of the reason for failure is required, and it must be proven that the test is properly designed and without problems.

[0008] Furthermore, the tests at the above-mentioned judgment points cannot include waveform characteristics (shape) as an automated test item. Currently, when conducting tests that include waveform characteristics (shape), the acquired data must be visualized, and the waveform shape must be determined visually.

[0009] The present invention aims to improve the accuracy of pass / fail determination in tests. [Means for solving the problem]

[0010] A test apparatus comprising one aspect of the invention disclosed herein is a test apparatus having a processor that executes a program and a storage device that stores the program, wherein the processor performs a test of the test subject according to a test scenario that defines instructions for causing the test subject to perform an operation, the order of the instructions, and a first expected value required for the operation; an acquisition process that acquires first time-series data from the test subject, which is a time-series first measurement value relating to the operation associated with the order obtained by the test process; the first time-series data acquired by the acquisition process and the test scenario Based on a second time-series data in which a second time-series measurement is defined that corresponds to the sequence of the operations, the correspondence between the first measurement time of the first measurement and the second measurement time of the second measurement is identified, and an adjustment process is performed to adjust the first measurement time based on the time difference between the first measurement time and the second measurement time, a determination process is performed to determine whether the test for the test subject passes or fails based on the first measurement value corresponding to the first measurement time after adjustment by the adjustment process and the first expected value corresponding to the sequence of operations at the first measurement time after adjustment, and an output process is performed to output the determination result from the determination process. [Effects of the Invention]

[0011] According to a typical embodiment of the present invention, the accuracy of pass / fail determination in tests can be improved. Problems, configurations, and effects other than those mentioned above will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 is a block diagram showing an example of the hardware configuration of the test apparatus. [Figure 2] Figure 2 is a block diagram showing an example of the functional configuration of the test apparatus. [Figure 3] Figure 3 is an explanatory diagram showing an example of the first automated test execution decision process. [Figure 4] Figure 4 is an explanatory diagram showing an example of the first test scenario. [Figure 5]FIG. 5 is an explanatory diagram showing an example of time-series data. [Figure 6] FIG. 6 is an explanatory diagram showing an example of the first test result. [Figure 7] FIG. 7 is an explanatory diagram showing an example of the second automatic test execution determination process. [Figure 8] FIG. 8 is an explanatory diagram showing an example of the second test scenario. [Figure 9] FIG. 9 is an explanatory diagram showing an example of reference time-series data. [Figure 10] FIG. 10 is an explanatory diagram showing an example of the second test result. [Figure 11] FIG. 11 is a graph showing an example of data mapping by DTW. [Figure 12] FIG. 12 is a flowchart showing an example of a reference time-series data generation processing procedure by a test apparatus. [Figure 13] FIG. 13 is a flowchart showing an example of a time-series data generation processing procedure by a test apparatus. [Figure 14] FIG. 14 is a flowchart showing an example of an automatic determination point adjustment test execution processing procedure by a test apparatus.

MODE FOR CARRYING OUT THE INVENTION

[0013] <Example of the hardware configuration of the test apparatus> Figure 1 is a block diagram showing an example of the hardware configuration of a test apparatus. The test apparatus 100 includes a processor 101, a storage device 102, an input device 103, an output device 104, and a communication interface (communication IF) 105. The processor 101, storage device 102, input device 103, output device 104, and communication IF 105 are connected by a bus 106. The processor 101 controls the test apparatus 100. The storage device 102 serves as the work area for the processor 101. The storage device 102 is a non-temporary or temporary recording medium that stores various programs and data. Examples of storage devices 102 include ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and flash memory. The input device 103 inputs data. Examples of input devices 103 include a keyboard, mouse, touch panel, numeric keypad, scanner, microphone, and sensor. The output device 104 outputs data. Output devices 104 include, for example, a display, a printer, and a speaker. The communication interface 105 connects to a network and sends and receives data.

[0014] <Example of functional configuration of test apparatus 100> Figure 2 is a block diagram showing an example of the functional configuration of the test apparatus 100. The test apparatus 100 includes an automatic test execution unit 201, a determination unit 202, a first communication unit 203, a test scenario 204, time-series data 205, test results 206, and reference time-series data 207. The test apparatus 100 has an automatic determination point adjustment function that automatically adjusts the determination point taking into account the delay of the simulation operation of the test object 210, and enables test determination at the expected determination point.

[0015] The automated test execution unit 201, the determination unit 202, and the first communication unit 203 are functions that are specifically realized by, for example, causing the processor 101 to execute a program stored in the memory device 102 shown in Figure 1. The test scenario 204, time-series data 205, test results 206, and reference time-series data 207 are data stored in the memory device 102.

[0016] Furthermore, the test subject 210 is a device that is tested by the test apparatus 100. The test subject 210 can be any device that operates over time. The test subject 210 is, for example, a mobile object such as an automobile. In this embodiment, a simulation operation test of an automobile will be used as an example. The test subject 210 includes software 211 that simulates the operation of the test subject 210, and a second communication unit 212.

[0017] <First Automated Test Execution Decision Process> Figure 3 is an explanatory diagram showing an example of the first automated test execution determination process. The first automated test execution determination process 310 is a process executed by the test device 100, but it is the process in which the above-mentioned automated determination point adjustment function is not applied. Specifically, for example, the test device 100 executes the test scenario execution process 311, the first communication process 312, the time-series data acquisition process 313, and the test determination process 314 through the first automated test execution determination process 310.

[0018] The test scenario execution process 311 is a process in which the automated test execution unit 201 reads the first test scenario 304 and executes the test target 210 according to the instructions described in the first test scenario 304.

[0019] The first communication process 312 is a process in which the first communication unit 203, which is the communication IF 106, communicates with the test subject 210.

[0020] The time-series data acquisition process 313 is a process that acquires the time-series data 205 received from the test target 210 to the first communication process 312 by the test scenario execution process 311.

[0021] The test determination process 314 is a process in which the determination unit 202 compares the time series data 205 with the expected value at the corresponding determination point (time) to determine whether the test passed or failed, and outputs the time series data 205 and the first test result 306.

[0022] The test environment 320 is the test environment in the test subject 210, and the software 211 controls the second communication unit 212, which is a communication interface, to execute the second communication process 321 with the first communication unit 203.

[0023] <First Exam Scenario 304> Figure 4 is an explanatory diagram showing an example of the first test scenario 304. The first test scenario 304 is a test scenario that does not support the automatic judgment point adjustment function. The first test scenario 304 is data that defines the commands and the order of commands that cause the test subject 210 to perform an action, and the expected value required for the action.

[0024] The first test scenario 304 has, as a field, sequence No. 401, instruction 402, set value 403, and expected value 404.

[0025] Sequence No. 401 is an ascending number indicating the order in which the actions to be performed by the test subject 210 are to be executed, with instruction 402 being executed in ascending order. The value of Sequence No. 401, # (where # is an integer greater than or equal to 1), is sometimes denoted as "Sequence (#)". Note that "s" is an abbreviation for seconds (sec) (the same applies to other figures).

[0026] Instruction 402 is a command that causes the test subject 210 to perform an action. The set value 403 is the value set for the test subject 210 in instruction 402. The expected value 404 is the value that is expected when instruction 402 is given with the set value 403. In this example, the expected value 404 is the Fail_flag, which indicates whether or not a failure occurred with the instruction 402 of sequence (5), which is a failure-injection command. "0" indicates that no failure occurred, and "1" indicates that a failure occurred.

[0027] The first test scenario 304 will be explained in sequence No. 401 order.

[0028] Sequence (1) The test device 100 turns on the IGN (ignition) of the test subject 210.

[0029] Sequence (2) Wait 5 seconds.

[0030] Sequence (3) Increase the amount of pressure required to depress the accelerator pedal by 40%.

[0031] Sequence (4) Wait 5 seconds.

[0032] Sequence (5) The command that causes a failure (Fail_Injection) is turned ON to induce a failure in the test subject 210.

[0033] Sequence (6)~(8) JudgePoint is a function that determines whether a failure has occurred in the test object 210 due to a failure-inducing command (Fail_Injection) after the cumulative time specified in the setting value 403 has elapsed.

[0034] In sequence (6), the expected value 404 0.1 seconds after the failure-causing instruction (Fail_Injection) is turned ON is set to "0". In sequence (7), the expected value 404 0.2 seconds after the failure-causing instruction (Fail_Injection) is turned ON is set to "1". In sequence (8), the expected value 404 0.3 seconds after the failure-causing instruction (Fail_Injection) is turned ON is set to "0".

[0035] Thus, the first test scenario 304 is a test scenario to confirm that Fail_flag becomes "1" in 0.1 seconds, 0.2 seconds after the instruction 402 of sequence (5), which is the instruction causing the failure (Fail_Injection), is turned ON (set value 403).

[0036] <Time series data 205> Figure 5 is an explanatory diagram showing an example of time-series data 205. Time-series data 205 is a time-series measurement of the operation of the test subject 210 according to the instruction 402 and set value 403 associated with the sequence (#). Time-series data 205 is a log obtained as a result of the first test scenario 304 being input to the test subject 210. Time-series data 205 has the following fields: simulation time 501, sequence No. 401, and judgment value 502.

[0037] The simulation time 501 is the actual elapsed time from the start of the test. The judgment value 502 is a measured value indicating whether or not a failure occurred due to a failure-inducing instruction (Fail_Injection) (Fail_flag).

[0038] The time series data 205 shows that during the simulation time 501, from 10.5(s) to 10.7(s), the instruction 402 of sequence (5), which causes a failure (Fail_Injection), was turned ON (set value 403) (indicated by 510).

[0039] Furthermore, time series data 205 shows the period from when the simulation time 501 is 10.5(s) until the judgment value 502 becomes "1" (symbol 520).

[0040] In other words, the time-series data 205 indicates that a processing delay occurred in sequence (5) in the test subject 210, causing the processing of instruction 402 in sequence (5), which is the instruction that causes the failure (Fail_Injection), to take 0.3(s). Therefore, the time-series data 205 is different from the expected value 404.

[0041] <First Test Result 306> Figure 6 is an explanatory diagram showing an example of the first test result 306. The first test result 306 has, as a field, sequence No. 401, instruction 402, set value 403, and first judgment result 600. The first judgment result 600 shows the result of comparing the expected value 404 and the judgment value 502.

[0042] The "-" indicates that instruction 402 is not "JudgePoint," and therefore no comparison is made between the expected value 404 and the judgment value 502.

[0043] "OK" indicates that the judgment value 502 and the expected value 404 match. "NG x(y)" indicates that the judgment value 502 and the expected value 404 do not match. "x" represents the judgment value 502, and "(y)" represents the expected value 404.

[0044] In other words, sequence (7) indicates a mismatch because the judgment value 502 is "0" and the expected value 404 is "1," and sequence (8) indicates a mismatch because the judgment value 502 is "1" and the expected value 404 is "0." Therefore, the test result is a failure.

[0045] <Second automated test execution decision process> Figure 7 is an explanatory diagram showing an example of the second automated test execution determination process. The second automated test execution determination process 710 is a process executed by the test device 100, and is the process when the automated determination point adjustment function described above is applied. Specifically, for example, the test device 100 executes the test scenario execution process 311, the first communication process 312, the time-series data acquisition process 313, the automated determination point adjustment process 711, and the test determination process 314 through the second automated test execution determination process 710.

[0046] The test scenario execution process 311 is a process in which the automated test execution unit 201 reads the second test scenario 704 and executes the test target 210 according to the instructions described in the second test scenario 704.

[0047] The automatic judgment point adjustment process 711 is a process in which the judgment unit 202 automatically adjusts the judgment points (time) in the time series data 205 using the reference time series data 207.

[0048] The test determination process 314 is a process in which the determination unit 202 determines whether the test passes or fails by comparing the time series data 205 with the expected value at the corresponding determination point (time) based on the adjustment result from the automatic determination point adjustment process 711, and outputs the time series data 205 and the second test result 706.

[0049] <Second Exam Scenario 704> Figure 8 is an explanatory diagram showing an example of the second test scenario 704. The second test scenario 704 is a test scenario corresponding to the automatic judgment point adjustment function. The second test scenario 704 has as fields sequence No. 401, instruction 402, set value 403, expected value 404, and DTW expected value 805. The values ​​of sequence No. 401, instruction 402, set value 403, and expected value 404 in the second test scenario 704 are the same as in the first test scenario 304.

[0050] The DTW expected value 805 is the expected value supplied to the waveform showing the time-dependent change of Fail_flag according to sequence No. 401 when instruction 402 is given with a set value of 403. DTW is an abbreviation for Dynamic Time Warping.

[0051] In this example, since the DTW expectation value 805 is "0", it is required that the waveform showing the change in Fail_flag over time in the reference time series data 207 matches the waveform showing the change in Fail_flag over time in the second test scenario 704. The DTW expectation value 805 is set within the range of 0 to 1.

[0052] <Reference time series data 207> Figure 9 is an explanatory diagram showing an example of reference time series data 207. Reference time series data 207 is, for example, time series data obtained as a result of simulating the test subject 210 in advance using the same test scenario as the second test scenario 704. Alternatively, it may be time series data from when a test result was determined to be a pass in the past.

[0053] The reference time series data 207 has the following fields: simulation time 501, sequence No. 401, and reference decision value 902.

[0054] The reference judgment value 902 is a measured value indicating whether or not a failure occurred (Fail_flag) due to a failure-causing instruction (Fail_Injection) at the time of generation of the reference time series data 207.

[0055] The reference time series data 207 shows that no processing delay occurred in sequence (5). Therefore, at 10.7(s), 0.2 seconds after the simulation time 501 in which sequence (5) was executed (10.5(s)) (indicated by 910), the reference judgment value 902 becomes "1" (indicated by 920).

[0056] <Results of the second examination: 706> Figure 10 is an explanatory diagram showing an example of the second test result 706. The second test result 706 has as fields: sequence No. 401, instruction 402, set value 403, second judgment result 1000, and DTW judgment result 1001. The second judgment result 1000 shows the result of comparing the expected value 404 with the judgment value 502 adjusted by the automatic judgment point adjustment process 711.

[0057] "OK(zs)" indicates that the adjusted judgment value 502 by the automatic judgment point adjustment process 711 matches the expected value 404. "z" is the adjustment time (s) at JudgePoint corrected by DTW based on the time series data 205.

[0058] In other words, in the time series data 205 in Figure 5, the judgment values ​​502(0,0,1) (sign 520) at simulation times 501, 10.6(s), 10.7(s), and 10.8(s) were compared with the expected value 404. However, due to the DTW correction, the judgment values ​​502(0,1,0) (sign 530) at simulation times 501, 10.7(s), 10.8(s), and 10.9(s) are compared with the expected value 404. In this example, in sequences (6) to (8), the judgment timing of JudgePoint increased by 0.1(s) due to the DTW correction, so "z" (=+0.1) is added to the second judgment result 1000 as the adjustment time (s).

[0059] In the first test result 306 in Figure 6, the first judgment result 600 for JudgePoint in sequences (7) and (8) is "NG", but in the second test result 706, it becomes "OK".

[0060] The DTW result 1001 is an index value that indicates whether the DTW value matches the expected DTW value 805 for the second trial scenario 704. For example, since the expected DTW value 805 is "0", if the DTW value is "0", the DTW result 1001 will be "OK", and if the DTW value is greater than "0", the DTW result 1001 will be "NG".

[0061] Note that the DTW expected value 805 is not limited to "0", but may be in a range such as 0.0 ≤ DTW expected value 805 < threshold th. Since the DTW value takes a value of 0.0 or greater, the threshold th can be set to any value of 0.0 or greater. However, even when the DTW value is close to 0.0, it is preferable to set the threshold th to a value of 0.5 or less in order to make the DTW judgment result 1001 "OK".

[0062] Here, we will explain how to calculate the DTW value. Let x and y be two time series data. The value of the time series data x and y at time t (1 ≤ t ≤ T) is x t ,y t The waveform will be represented as follows: Time t = w of time series data x x and the time t=w of the time series data yy corresponds by alignment as w=(w x ,w y ), and represents the entire waveform alignment as a set of K correspondence relations by a warping path W={w1, w2, …, w K}. Here, if w k =(w k x ,w k y )(where k is an integer satisfying 1 ≤ k ≤ K), the DTW value is defined by the following formula (1).

[0063] [Number]

[0064] The DTW value indicates the similarity of time series data x and y. The DTW value, like the DTW expected value 805, takes a range from 0 to 1. By setting two time series data x and y as the Fail_flag (reference determination value 902) in the reference time series data 207 and the Fail_flag (determination value 502) in the second test scenario 704, the DTW value is calculated.

[0065] The smaller the DTW expected value 805, the more similar the waveform showing the temporal change of the Fail_flag (reference determination value 902) in the reference time series data 207 is to the waveform showing the temporal change of the Fail_flag (determination value 502) in the second test scenario 704.

[0066] [Data Mapping by DTW] Figure 11 is a graph showing an example of data mapping using DTW. Graph 1100 has a horizontal axis 1110, a first vertical axis 1121, and a second vertical axis 1122. The horizontal axis 1110 is the simulation time 501. The first vertical axis 1121 is the reference judgment value 902 for the reference time series data 207. The second vertical axis 1122 is the judgment value 502 for the time series data 205. (#)tt.ts near the circle indicates the value tt.t for the simulation time 501 of sequence (#). For example, (5)10.6s indicates the value 10.6s for the simulation time 501 of sequence (5).

[0067] To avoid overlap between the first waveform W1 and the second waveform W2, the second waveform W2 is conveniently shifted in the direction of the first vertical axis 1121 for display purposes.

[0068] The first waveform W1 shows the change over time of the reference judgment value 902 in the reference time series data 207. The first waveform W1 is the waveform before adjustment. The second waveform W2 shows the change over time of the judgment value 502 in the time series data 205. The second waveform W2 is the waveform after adjustment. In the first waveform W1 and the second waveform W2, points enclosed by the same circular shape are mapped to each other.

[0069] By performing DTW, specifically, for example, the correspondence between sequence (6) 10.6s with a value of "0" in the first waveform W1 and sequence (5) 10.7s with a value of "0" in the second waveform W2, and the time difference between them, +1.0s (=10.7s-10.6s), are identified.

[0070] Furthermore, the correspondence between sequence (7) 10.7s, where the value in the first waveform W1 is "1", and sequence (6) 10.8s, where the value in the second waveform W2 is "1", and the time difference between them, +1.0s (=10.8s-10.7s), are identified.

[0071] Furthermore, the correspondence between sequence (8) 10.8s, where the value in the first waveform W1 is "0", and sequence (7) 10.9s, where the value in the second waveform W2 is "0", and the time difference between them, +1.0s (=10.8s-10.7s), are identified.

[0072] Furthermore, using the above equation (1), if we calculate the DTW value for the simulation time 501 within the interval of 9s to 12s indicated by the horizontal axis 1110, the DTW value becomes DTW=0.

[0073] In this way, by applying DTW, the test apparatus 100 can acquire the correspondence between points in the first waveform W1 and the second waveform W2, as well as DTW values ​​related to the first waveform W1 and the second waveform W2.

[0074] <Reference time series data generation process> Figure 12 is a flowchart illustrating an example of the reference time series data generation procedure by the test device 100. In the second automated test execution determination process, Figure 12 shows the process by which the test device 100 generates reference time series data 207 in a simulation through the test scenario execution process 311, the first communication process 312, and the time series data acquisition process 313. The entries for the reference time series data 207 are empty before the simulation is executed.

[0075] The test apparatus 100 loads the second test scenario 704 and performs analysis (conversion to a script file) of sequence No. 401, instruction 402, and setpoint 403 (step S1201).

[0076] The test device 100 starts measuring the measurement signal used for the automatic test determination, in this example, the reference determination value 902 (Fail_flag) (step S1202).

[0077] The test apparatus 100 determines whether the execution of all sequences of sequence No. 401 has been completed (step S1203). If the execution of all sequences of sequence No. 401 has not been completed (step S1203: No), it determines whether the current simulation time 501 (current simulation time) is greater than or equal to the execution time of sequence No. (step S1204).

[0078] The current simulation time is "0.0s" at the start of step S1202, and increases by one step (0.1s) in increments during the simulation in step S1207.

[0079] Sequence No. execution time is the cumulative time up to the execution of sequence No. 401 (hereinafter referred to as the current sequence (#)). Since sequences (#) are executed in ascending order of sequence No. 401, the current sequence (#) is the youngest sequence No. 401 that has not yet been executed. The sequence No. execution time is "0.0s" at the start of step S1202 and is accumulated in step S1206 as sequences (#) are executed.

[0080] If the current simulation time is equal to or greater than the execution time of sequence No. (Step S1204: Yes), proceed to step S1205. On the other hand, if the current simulation time is not equal to or greater than the execution time of sequence No. (Step S1204: No), proceed to step S1207.

[0081] At the start of step S1202, both the current simulation time and the execution time of sequence No. are "0.0s", so the program proceeds to step S1204.

[0082] The test device 100 executes instruction 402 of the current sequence (#) (step S1205). At the start of step S1202, the current sequence (#) is sequence (1). Therefore, the test device 100 executes the process of changing instruction 402 of sequence (1), "IGN," to the set value 403, "ON." As a result, the reference judgment value 902 is output as the value of the measurement signal.

[0083] Sequence (1) does not simulate a failure because instruction 402 is not a fail-injection instruction. Therefore, the value of the reference judgment value 902 (Fail_flag), which indicates whether or not a failure has occurred, is "0".

[0084] The test apparatus 100 updates the current sequence number execution time (step S1206). Specifically, for example, the test apparatus 100 updates the sequence number execution time by adding the time required to execute instruction 402 of the current sequence (#) in step S1204 to the sequence number execution time.

[0085] For example, suppose sequence (1) takes 0.1 seconds. In this case, the execution time of sequence No. will be updated from "0.0s" to "0.1s".

[0086] Next, the test apparatus 100 advances the current simulation time by one step (step S1207). Specifically, for example, the test apparatus 100 adds 0.1s to the current simulation time. As a result, the current simulation time is updated from "0.0s" to "0.1s".

[0087] Next, the test apparatus 100 writes the current simulation time, the current sequence (#), and the value of the measurement signal, which is the result of step S1205, to the reference time series data 207 (step S1208). If the current simulation time is "0.1s", then the current simulation time "0.1s", the sequence No. 401 of the current sequence (1) which is "1", and the value of the measurement signal, which is the result of the execution of the current sequence (1), which is "0", are written to the reference time series data 207 as entries. Then, the process returns to step S1203.

[0088] Since the current sequence (#) has a sequence number of "1" for sequence No. 401, the execution of all sequences has not finished (step S1203: No). Therefore, proceed to step S1204.

[0089] In step S1204, the current simulation time is "0.1s" and the execution time of sequence No. is also "0.1s", so step S1204 is satisfied (Step S1204: Yes). Therefore, we proceed to step S1205.

[0090] In step S1205, the test device 100 executes sequence (2). In this case, the test device 100 performs the process of changing the instruction 402 of sequence (2), "Wait," to the set value 403, "5s." As a result, the reference judgment value 902 is output as the value of the measurement signal.

[0091] In sequence (2), since instruction 402 is not a fail-injection instruction, no failure is simulated. Therefore, the value of the reference judgment value 902 (Fail_flag), which indicates whether or not a failure has occurred, is "0".

[0092] The test device 100 updates the current sequence No. execution time (step S1206). Since instruction 402 of sequence (2), which is "Wait," is a waiting process, the time required to execute "Wait" is the set value 403, which is "5s." Therefore, the test device 100 updates the sequence No. execution time to "5.1s" by adding the time required to execute instruction 402 of the current sequence (2) in step S1204, which is "5s," to the sequence No. execution time, which is "0.1s."

[0093] Next, the test apparatus 100 adds 0.1s to the current simulation time of "0.1s" (step S1207). As a result, the current simulation time is updated from "0.1s" to "0.2s".

[0094] Next, the test device 100 writes the current simulation time "0.2s", the sequence No. 401 of the current sequence (2) "2", and the measurement signal value "0", which is the result of step S1205, as entries to the reference time series data 207 (step S1208). Then, it returns to step S1203.

[0095] Since the current sequence (#) has a sequence number of "2" for sequence No. 401, the execution of all sequences has not finished (step S1203: No). Therefore, proceed to step S1204.

[0096] In step S1204, the current simulation time is "0.2s" and the execution time of sequence No. is "5.1s", so step S1204 is not satisfied (step S1204: No). Therefore, proceed to step S1207.

[0097] In step S1207, the test apparatus 100 adds 0.1s to the current simulation time of "0.2s" (step S1207). As a result, the current simulation time is updated from "0.2s" to "0.3s".

[0098] The test apparatus 100 will continue the loop of steps S1204:No, S1207, S1208, and S1203:No until the current simulation time reaches or exceeds the execution time of sequence No. "5.1s" as determined by step S1207.

[0099] After this, if the current simulation time becomes equal to or greater than the execution time of sequence No. "5.1s" (step S1204: Yes), the test apparatus 100 executes steps S1205 to S1208 and returns to step S1203.

[0100] Finally, when the test device 100 determines that the execution of the entire sequence is complete (step S1203: Yes), it terminates the reference time series data generation process. This generates the reference time series data 207 as shown in Figure 9.

[0101] <Time-series data generation process> Figure 13 is a flowchart illustrating an example of the time-series data generation process by the test device 100. In the second automated test execution determination process, Figure 13 shows the process by which the test device 100 generates time-series data 205 using a real-time simulation with the test target 210 through the test scenario execution process 311, the first communication process 312, and the time-series data acquisition process 313. The entries for the time-series data 205 are empty before the execution of the real-time simulation with the test target 210.

[0102] The test apparatus 100 loads the second test scenario 704 and performs analysis (conversion to a script file) of sequence No. 401, instruction 402, and setpoint value 403 (step S1301).

[0103] The test device 100 starts measuring the measurement signal used for the automated test judgment, in this example, the reference judgment value 902 (Fail_flag) (step S1302). Step S1302 starts the elapsed time of the real-time simulation time 501.

[0104] The test device 100 transmits the measurement start signal from step S1302 to the test subject 210 (C1).

[0105] The test apparatus 100 determines whether the execution of all sequences of sequence No. 401 has been completed (step S1303). If the execution of all sequences of sequence No. 401 has not been completed (step S1303: No), it determines whether the current simulation time 501 (current simulation time) is greater than or equal to the execution time of sequence No. (step S1204).

[0106] Unlike in Figure 12, the current simulation time is the actual elapsed time from the start of measurement in step S1302. It is "0.0s" at the start of step S1302 and increases as real time progresses. The execution time of sequence No. and sequence (#) are the same as in Figure 12. The execution time of sequence No. is "0.0s" at the start of step S1302 and is accumulated in step S1306 by the execution of sequence (#).

[0107] If the current simulation time is equal to or greater than the execution time of sequence No. (Step S1304: Yes), proceed to step S1305. On the other hand, if the current simulation time is not equal to or greater than the execution time of sequence No. (Step S1304: No), proceed to step S1303.

[0108] At the start of step S1302, both the current simulation time and the execution time of sequence No. are "0.0s", so the program proceeds to step S1305.

[0109] The test device 100 executes instruction 402 of the current sequence (#) (step S1305). At the start of step S1302, the current sequence (#) is sequence (1). Therefore, the test device 100 executes the process of setting the instruction 402 of sequence (1), "IGN," to the set value 403, "ON," on the test subject 210. That is, the test device 100 sends a command to the test subject 210 to set "IGN" to "ON" (C2).

[0110] After the test subject 210 receives a measurement start signal (C1) from the test device 100 via the second communication processing 321, the software 211 executes an operation based on the command (C2) each time it receives a command (C2) from the test device 100, and sequentially reads out the value of the measurement signal generated by that operation (step S1310).

[0111] Sequence (1) does not simulate a failure because instruction 402 is not a command that causes a failure (Fail_Injection). Therefore, the value of the judgment value 502 (Fail_flag) indicating whether or not a failure has occurred is "0". The test subject 210 transmits the value of the read measurement signal as the judgment value 502 to the test device 100 via the second communication process 321 (C3).

[0112] Next, the test device 100 writes the measurement results (current simulation time, current sequence (#), and the value of the measurement signal which is the result of executing step S1310) as entries to the time series data 205 (step S1306). Suppose the current simulation time becomes "0.1s" as time passes. In this case, the test device 100 writes the current simulation time "0.1s", the sequence No. 401 of the current sequence (1) which is "1", and the value of the measurement signal which is the result of executing the current sequence (1) which is "0" as entries to the time series data 205.

[0113] The test device 100 updates the current sequence number execution time (step S1306). Specifically, for example, the test device 100 updates the sequence number execution time by adding the time taken from sending a command from the test device 100 (C2) to receiving the value of the measurement signal (C3) as the time required to execute instruction 402 of the current sequence (#) to the sequence number execution time.

[0114] For example, suppose sequence (1) takes 0.1 seconds. In this case, the execution time of sequence No. will be updated from "0.0s" to "0.1s".

[0115] Since the current sequence (#) has a sequence number of "1" for sequence No. 401, the execution of all sequences has not finished (step S1303: No). Therefore, proceed to step S1304.

[0116] In step S1304, the current simulation time is "0.1s" and the execution time of sequence No. is also "0.1s", so step S1304 is satisfied (Step S1304: Yes). Therefore, we proceed to step S1305.

[0117] In step S1305, the test device 100 executes sequence (2). In this case, the test device 100 sends a command to the test subject 210 to change the instruction 402 of sequence (2), which is "Wait", to the set value 403, which is "5s" (C2).

[0118] In sequence (2), since instruction 402 is not a command that causes a failure (Fail_Injection), no failure is simulated. Therefore, the value of the judgment value 502 (Fail_flag) indicating whether or not a failure has occurred is "0". The test subject 210 transmits the value of the read measurement signal to the test device 100 as the judgment value 502 (C3).

[0119] Furthermore, due to the instruction 402 "Wait" in sequence (2), the test subject 210 continues to wait until the set value 403 "5s" has elapsed. During this time, the test subject 210 continues to transmit the value of the read measurement signal "0" to the test device 100 as the judgment value 502 (C3).

[0120] The measurement results (current simulation time, current sequence (#), and the value of the measurement signal, which is the result of executing step S1310) are written to the time series data 205 as entries (step S1306). Suppose the current simulation time becomes "0.2s" as time passes. In this case, the test device 100 writes the current simulation time "0.2s", the sequence No. 401 of the current sequence (1) which is "2", and the value of the measurement signal, which is the result of executing the current sequence (2) which is "0", to the time series data 205 as entries.

[0121] The test device 100 updates the current sequence No. execution time (step S1307). Since instruction 402 of sequence (2), "Wait," is a process that causes the test subject 210 to wait, the time required to execute "Wait" is the set value 403, which is "5s." Therefore, the test device 100 updates the sequence No. execution time to "5.1s" by adding the set value 403, which is "5s," to the sequence No. execution time, which is "0.2s." Then, it returns to step S1303.

[0122] Since the current sequence (#) has a sequence number of "2" for sequence No. 401, the execution of all sequences has not finished (step S1303: No). Therefore, proceed to step S1304.

[0123] In step S1304, the current simulation time is "0.2s" and the execution time of sequence No. is "5.1s", so step S1304 is not satisfied (step S1304: No). Therefore, proceed to step S1303.

[0124] The test apparatus 100 will continue the loop of steps S1304:No and S1303:No until the current simulation time is equal to or greater than the execution time of sequence No., which is "5.1s".

[0125] After this, if the current simulation time exceeds the execution time of sequence No. "5.1s" due to the passage of time (step S1304: Yes), the test apparatus 100 executes steps S1305 to S1307 and returns to step S1303.

[0126] Finally, when the test device 100 determines that the execution of the entire sequence is complete (step S1303: Yes), it terminates the time-series data generation process. This generates time-series data 205 as shown in Figure 5.

[0127] <Automatic Judgment Point Adjustment Test Execution Process> Figure 14 is a flowchart illustrating an example of the procedure for executing an automatic judgment point adjustment test by the test device 100. In the second automatic test execution judgment process, Figure 14 shows the process by which the test device 100 executes an automatic judgment point adjustment test using the reference time series data 207 and the time series data 205 acquired in Figure 13, through the automatic judgment point adjustment process 711 and the test judgment process 314. Although Figure 14 shows an example of executing an automatic judgment point adjustment test for one type of measurement signal, if there are multiple types of measurement signals in a single second test scenario 704, the automatic judgment point adjustment test will be executed for each type of measurement signal.

[0128] The test device 100 reads the reference time series data 207 and the time series data 205 acquired in Figure 13 (step S1401).

[0129] The test device 100 obtains time-series values ​​of the Judge Point to be adjusted from the reference time-series data and the time-series data (step S1402). Specifically, for example, the test device 100 obtains the reference judgment value 902 for the simulation time 501 of the reference time-series data 207 between 0.1s and 12.0s, and the judgment value 502 for the simulation time 501 of the time-series data 205 between 0.1s and 12.0s.

[0130] The test apparatus 100 uses the DTW algorithm to identify the target time and the adjustment time from the source time for each Judge Point and calculates the DTW value (step S1403). The DTW algorithm associates the source time with the target time.

[0131] As mentioned above, the source of adjustment is the reference judgment value 902 of the time-series Judge Point in the reference time-series data 207. The time of the source of adjustment is the simulation time 501 at that reference judgment value 902. In the example in Figure 11, the time of the source of adjustment is the sequence (6) 10.6s, sequence (7) 10.7s, and sequence (8) 10.8s of the first waveform W1.

[0132] As mentioned above, the adjustment target is the judgment value 502 of the time-series Judge Point in the time-series data 205, and the adjustment target time is the simulation time 501 at that judgment value 502. In the example in Figure 11, the adjustment target times are sequence (5) 10.7s, sequence (6) 10.8s, and sequence (7) 10.9s of the second waveform W2.

[0133] The adjustment time from the source is the time difference obtained by subtracting the corresponding source time from the time of the destination adjustment.

[0134] The test apparatus 100 compares the expected value 404 of the second test scenario 704 with the judgment value 502 of the time series data 205 at a time-adjusted Judge Point to generate a second judgment result 1000 (step S1404).

[0135] The time-adjusted Judge Points are sequences (5) to (7) in time series data 205, which correspond to sequences (6) to (8) in the reference time series data 207, which is the source of the adjustment. The judgment value 502 for sequences (5) to (7) is (0,1,0). The expected value 404 for sequences (6) to (8) in the second trial scenario 704 is (0,1,0).

[0136] Therefore, the determination value 502 for sequence (5) in time-adjusted time series data 205, derived from sequence (6) in reference time series data 207, is "0", and the expected value 404 for sequence (6) in the second test scenario 704 is "0", so the two values ​​match.

[0137] Furthermore, the adjustment time from sequence (6) in the source time series data 207 to sequence (5) in the target time series data 205 is +1.0s. Therefore, the value of the second judgment result 1000 for sequence (6) in the second test scenario 704 is "OK + 1.0s".

[0138] Furthermore, the judgment value 502 for sequence (6) in time-adjusted time series data 205, derived from sequence (7) in reference time series data 207, is "1", and the expected value 404 for sequence (7) in the second test scenario 704 is "1", so the two values ​​match.

[0139] Furthermore, the adjustment time from sequence (7) in the source time series data 207 to sequence (6) in the target time series data 205 is +1.0s. Therefore, the value of the second judgment result 1000 for sequence (7) in the second test scenario 704 is "OK + 1.0s".

[0140] Furthermore, the judgment value 502 for sequence (7) in time-adjusted time series data 205, derived from sequence (8) in reference time series data 207, is "0", and the expected value 404 for sequence (8) in the second test scenario 704 is "0", so the two values ​​match.

[0141] Furthermore, the adjustment time from sequence (8) in the source time series data 207 to sequence (7) in the target time series data 205 is +1.0s. Therefore, the value of the second judgment result 1000 for sequence (8) in the second test scenario 704 is "OK + 1.0s".

[0142] The test apparatus 100 compares the DTW expected value 805 with the DTW value to generate a DTW judgment result 1001 (step S1405). As shown in Figure 8, the DTW expected value 805 is "0". Also, since the first waveform W1 and the second waveform W2 match due to DTW, the DTW value becomes "0". Therefore, the DTW expected value 805 and the DTW value match, and the DTW judgment result 1001 becomes "OK".

[0143] The test apparatus 100 outputs the second test result 706 as shown in Figure 10 (step S1406).

[0144] Thus, according to this embodiment, the test device 100 automatically adjusts the judgment point taking processing delay into account and performs the test judgment at the expected judgment point. Specifically, the test device 100 automatically adjusts the judgment point by using a DTW algorithm that measures the similarity between time-series data, and uses the calculated DTW value for the test judgment. Therefore, the accuracy of the test can be improved.

[0145] Specifically, even if latency, a hardware factor of the 210 test subject, or processing delays occur when executing the test scenario, the waveform characteristics of the time-series data are also included in the automated test items. Therefore, misjudgments (failures) of test results due to shifts in judgment points can be suppressed. Furthermore, by suppressing misjudgments (failures) of test results, unnecessary work such as reviewing test scenarios, investigating the cause of failure, and conducting retests that result from misjudgments (failures) is reduced. Consequently, the test period can be shortened.

[0146] In particular, in the V-model process for vehicle development, by utilizing the second test scenario 704 and the reference time-series data 207, which are the output results of the tests conducted in the earlier stage using a highly abstract simulation environment (PC simulator), it becomes possible to achieve a seamless automated test configuration throughout the entire process.

[0147] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail to make the present invention easier to understand, and the present invention is not necessarily limited to having all of the described configurations. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Furthermore, some of the configurations of one embodiment may be added to those of another embodiment. Furthermore, some of the configurations of each embodiment may be added, deleted, or replaced with other configurations.

[0148] Furthermore, each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each function.

[0149] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs (Solid State Drives), or on recording media such as IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs).

[0150] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of Symbols]

[0151] 100 Test equipment 101 Processors 102 Storage Devices 201 Automated Test Execution Unit 202 Judgment section 203 First Communications Department 204 Exam Scenarios 205 Time series data 206 Test Results 207 Reference time series data 210 Test subjects 211 Software 212 Second Communications Department 304 First Exam Scenario 306 First Exam Results 310 First Automated Test Execution Decision Process 311 Test Scenario Execution Process 312 First Communication Processing 313 Time-series data acquisition process 314 Test result processing 320 Test Environments 321 Second Communication Processing 704 Second Exam Scenario 706 Second Exam Results

Claims

1. A test apparatus having a processor for executing a program and a storage device for storing the program, The aforementioned processor, A test process that performs a test on the test subject according to a test scenario in which the instructions for causing the test subject to perform an action, the order of the instructions, and a first expected value required for the action are defined, An acquisition process for acquiring first time-series data, which is a time-series first measurement value related to the operation associated with the sequence obtained by the test process, from the test subject; Based on the first time-series data acquired by the acquisition process and the second time-series data in which a second measurement value is defined that corresponds to the sequence of the operation according to the test scenario, the correspondence between the first measurement time of the first measurement value and the second measurement time of the second measurement value is identified, and the first measurement time is adjusted based on the time difference between the first measurement time and the second measurement time. A determination process for determining whether the test for the test subject is successful or not, based on the first measurement value corresponding to the first measurement time after adjustment by the adjustment process and the first expected value corresponding to the order at the first measurement time after adjustment. An output process that outputs the determination result obtained by the aforementioned determination process, A test apparatus characterized by performing the following actions.

2. A test apparatus according to claim 1, The determination result includes a first comparison result between the first measured value corresponding to the adjusted first measurement time and the first expected value corresponding to the sequence of operations at the adjusted first measurement time, and the time difference. A test apparatus characterized by the following features.

3. A test apparatus according to claim 1, The aforementioned test scenario specifies the second expected value required for the sequence of the first expected values ​​in the aforementioned order, In the adjustment process, the processor calculates the similarity between the first time series data and the second time series data. In the determination process, the processor determines whether the test is pass or fail based on the similarity calculated by the adjustment process and the second expected value. A test apparatus characterized by the following features.

4. The test apparatus according to claim 3, The determination result includes a second comparison result between the similarity and the second expected value. A test apparatus characterized by the following features.

5. A test apparatus according to claim 1, In the adjustment process, the processor identifies the correspondence based on a dynamic time stretching method. A test apparatus characterized by the following features.

6. The test apparatus according to claim 3, In the adjustment process, the processor calculates the similarity based on the dynamic time stretching method. A test apparatus characterized by the following features.

7. A test method using a test apparatus having a processor for executing a program and a storage device for storing the program, The aforementioned processor, A test process that performs a test on the test subject according to a test scenario in which the instructions for causing the test subject to perform an action, the order of the instructions, and a first expected value required for the action are defined, An acquisition process for acquiring first time-series data, which is a time-series first measurement value related to the operation associated with the sequence obtained by the test process, from the test subject; Based on the first time-series data acquired by the acquisition process and the second time-series data in which a second measurement value is defined that corresponds to the sequence of the operation according to the test scenario, the correspondence between the first measurement time of the first measurement value and the second measurement time of the second measurement value is identified, and the first measurement time is adjusted based on the time difference between the first measurement time and the second measurement time. A determination process for determining whether the test for the test subject is successful or not, based on the first measurement value corresponding to the first measurement time after adjustment by the adjustment process and the first expected value corresponding to the order at the first measurement time after adjustment. An output process that outputs the determination result obtained by the aforementioned determination process, A test method characterized by performing the following.

8. In the processor, A test process that performs a test on the test subject according to a test scenario in which the instructions for causing the test subject to perform an action, the order of the instructions, and a first expected value required for the action are defined, An acquisition process for acquiring first time-series data, which is a time-series first measurement value related to the operation associated with the sequence obtained by the test process, from the test subject; Based on the first time-series data acquired by the acquisition process and the second time-series data in which a second measurement value is defined that corresponds to the sequence of the operation according to the test scenario, the correspondence between the first measurement time of the first measurement value and the second measurement time of the second measurement value is identified, and the first measurement time is adjusted based on the time difference between the first measurement time and the second measurement time. A determination process for determining whether the test for the test subject is successful or not, based on the first measurement value corresponding to the first measurement time after adjustment by the adjustment process and the first expected value corresponding to the order at the first measurement time after adjustment. An output process that outputs the determination result obtained by the aforementioned determination process, A test program characterized by causing the execution of a specific action.

Images