Vehicle control software test method

Abstract

In the present invention, design test case information that relates to a combination of input data which is created in advance and which is to be used in a test and an expected value of output data is prepared. In a control device installed in a vehicle, vehicle data that is generated in accordance with actual travel of the vehicle is collected, a new set of input data that is obtained by masking data of a specific type on the basis of a synthesis rule given in advance is acquired from the input data of the design test case information and the collected vehicle data, and new test case information is generated from the new set of input data and the expected value of the output data of the design test case information. Control software to be tested is operated, and a test for the control software to be tested is executed on the basis of the new test case information.

Description

Test Method for Vehicle Control Software

[0001] The present invention relates to a software test method, and particularly to a test method for control software implemented in a vehicle control device mounted on a vehicle such as an automobile.

[0002] Control software installed in a vehicle such as an automobile to control it is subjected to various tests during its development stage to detect defects such as bugs. For example, in a verification test, the control software is operated using assumed input data or combinations thereof, and it is evaluated whether the output value becomes the expected value.

[0003] In recent years, vehicles have been equipped with advanced functions such as driving assistance systems that make the vehicle follow a vehicle traveling ahead on an automatic brake or a highway, or automatically perform some driving operations. As a result, the scale of the control software installed in the vehicle control device has increased, and the content of control has become more complex. Therefore, the test man-hours for control software have increased, and verification using comprehensive input data has become difficult.

[0004] On the other hand, regarding the test of control software, for example, in Patent Document 1, on the vehicle control device of an actually running vehicle, the control software to be tested is operated so as not to affect the operation of the actually operating control software, and its evaluation is performed, so that the evaluation of the control software can be efficiently performed.

[0005] Japanese Patent Application Laid-Open No. 2023-38697

[0006] According to the technique described in Patent Document 1, the control software can be tested using vehicle data obtained from actual driving, and it is possible to perform a test of the control software using comprehensive data. However, since the evaluation of the output value of the control software to be tested is performed by comparing it with the output value of the actually operating control software, the validity of the operation can be evaluated, but it cannot be used for verification of whether the expected output value is obtained.

[0007] In view of the problems in the prior art described above, the object of the present invention is to provide a test method that can efficiently perform verification tests of control software using comprehensive input data.

[0008] The present invention provides a test method for control software that performs the functions of a vehicle control device mounted on a vehicle. In one preferred embodiment, the method involves preparing design test case information relating to a combination of expected values ​​of input and output data that has been created in advance for use in the test, collecting vehicle data generated during actual vehicle operation, obtaining a new set of input data obtained by masking specific types of data based on a pre-given synthesis rule from the input data of the design test case information and the collected vehicle data, generating new test case information from the new set of input data and the expected values ​​of the output data of the design test case information, operating the control software to be tested, and performing a test of the control software based on the new test case information.

[0009] According to the present invention, it is possible to efficiently perform verification tests of control software using comprehensive input data. Other novel features of the present invention and the technical problems solved thereby will become apparent from the description and drawings herein.

[0010] This is a schematic diagram showing an overview of control software testing in one embodiment. This is a schematic block diagram showing the logical configuration of the vehicle control system used for testing the control software. This is a flowchart showing the flow of vehicle data acquisition processing performed by the vehicle data acquisition unit. This is a schematic diagram of vehicle data stored in the vehicle data storage unit. This is a flowchart showing the flow of test case generation processing performed by the test case synthesis unit. This is a schematic diagram showing the data structure of design test case information stored in the design test case information storage unit. This is a schematic diagram showing the concept of the first method of test case generation. This is a schematic diagram showing the concept of the second method of test case generation. This is a flowchart showing the flow of processing performed by the execution control unit. This is a schematic diagram showing the data structure of test result information obtained as a result of evaluation in the evaluation unit. This is a schematic block diagram showing the configuration of the control software test environment in the second embodiment.

[0011] Hereinafter, representative embodiments of the present invention will be described with reference to the drawings. It should be noted that the embodiments and drawings described below are illustrative examples for explaining the present invention, and have been omitted and simplified as appropriate for clarity of explanation.

[0012] Figure 1 is a schematic diagram illustrating the testing of control software in one embodiment of the present invention.

[0013] Cloud 1 possesses computing resources such as computers and memory devices, and a development environment is built on it for use in developing control software. In this embodiment, the control software to be subjected to verification testing (hereinafter simply referred to as "test") is developed using the development environment built on Cloud 1 and is distributed to electronic control units (ECUs) installed in one or more vehicles 2 via wireless communication using a technology called OTA (Over the Air), for example. The control software distributed to the vehicles 2 is executed in the background using vehicle data obtained while the vehicles 2 are running, so as not to affect the execution of the control software that is actually controlling the driving of the vehicles 2. The data output from the control software as a result of the execution is evaluated based on the expected value information distributed from Cloud 1 to the vehicles 2, similar to the control software under test.

[0014] The information obtained as a result of the evaluation is stored in a memory device within or connected to the ECU, and is sent to a development environment built on Cloud 1 as needed, for example, via wireless communication. The control software developer can verify whether the control software is operating as expected by analyzing the evaluation results collected on Cloud 1.

[0015] Vehicle 2 is a vehicle used as a test environment for control software. Vehicle 2 may be an experimental vehicle operated by the control software developer for testing purposes, or it may be a vehicle that is sold on the market and in general use.

[0016] Figure 2 is a schematic block diagram showing the logical configuration of the control system of the vehicle 2 used for testing the control software in this embodiment.

[0017] The vehicle control unit (ECU) 10 is mounted on the vehicle 2 and controls at least some of the functions of the vehicle 2. The ECU 10 is communicated with the sensor 30, actuator 40, and gateway 50 via the communication bus 20.

[0018] The communication bus 20 constitutes a network in which communication is carried out according to communication standards such as CAN (Controller Area Network) and Ethernet. The communication bus 20 is not limited to conforming to these communication standards, but may be configured using any standard or protocol.

[0019] Sensor 30 may be a stereo camera or radar device mounted on the vehicle 2 that acquires information about the surrounding environment of the vehicle 2. Alternatively, sensor 30 may be mounted on the vehicle 2 that acquires information about the operating status of various parts of the vehicle 2, such as the vehicle's speed, acceleration, and engine speed, as well as information about the driver's operation of the vehicle 2, such as an accelerator pedal sensor or a brake pedal sensor. Sensor 30 may be the various sensors themselves, or it may be configured as a sensor system including an ECU that processes the signals detected by the sensors. If sensor 30 is, for example, a stereo camera, the vehicle data transmitted and received via the communication bus 20 may be image information captured by the stereo camera, or information recognized from the image information, such as the direction, distance, and type of an object recognized from the captured image.

[0020] The actuator 40 receives output values ​​from the ECU 10 or other control devices connected to the communication bus 20, and operates mechanisms for maintaining and changing the driving state of the vehicle 2. The actuator 40 also sends control values ​​to the communication bus 20 that represent the operating state of the mechanisms for maintaining and changing the driving state of the vehicle 2. Like the sensor 30, the actuator 40 may be the actuator itself that operates the mechanism, or it may be configured as an actuation system that includes a control device that controls the actuator.

[0021] The gateway 50 controls communication between the vehicle 2 and external sources such as the cloud 1. In this embodiment, design test case information, including information on the control program under test and test conditions, distributed from the cloud 1, is sent to the ECU 10 via the gateway 50. Test result information obtained as a result of tests performed in the ECU 10 is uploaded from the ECU 10 to the cloud 1 via the gateway 50.

[0022] The ECU 10 executes the first control software 11. The first control software 11, for example, acquires and processes vehicle data related to vehicle control handled by the first control software 11 from vehicle data sent from sensors 30 and actuators 40 via the communication bus 20, and sends output data for controlling the actuators 40 to the communication bus 20. In other words, the first control software is control software executed by the ECU 10 to drive the vehicle 2. The first control software 11 may be, for example, sensor fusion that processes information about the vehicle's surroundings acquired by multiple sensors 30, or AD / ADAS (Automated Driving / Advanced Driver-Assistance System) control software that controls the actuators 40 based on the acquired surrounding information.

[0023] The second control software 12 is the control software to be tested and can be downloaded from the cloud 1 via the gateway 50. The second control software 12 may take the form of a single function that is part of the functions related to the control of the vehicle 2, a module that combines multiple single functions, or a system that integrates all the functions of a certain ECU. The second control software 12 may also be provided in the form of a test unit that includes test tools used to test the control software.

[0024] The test execution control software 13 is software that controls the execution of tests for the second control software 12. The test execution control software 13 monitors the utilization rate of the computing resources of the ECU 10, and when the utilization rate of the computing resources of the ECU 10 is low, it executes the second control software 12 in a manner that does not affect the execution of the first control software 11, and acquires and evaluates its output data. The evaluation results are sent to the cloud 1 via the gateway 50. The test execution control software 13 includes a vehicle data acquisition unit 100, a test case synthesis unit 200, an execution control unit 300, an evaluation unit 400, and an upload unit 500 as functions for performing tests on the second control software 12.

[0025] The vehicle data acquisition unit 100 reads and acquires vehicle data from the communication bus 20. The test case synthesis unit 200 generates test case information that has input data to be used to test the second control software 12 and output expected values ​​for output data to be output as an execution result, based on the design test case information prepared for the verification of the second control software 12 and distributed from the cloud 1, and the vehicle data acquired by the vehicle data acquisition unit 100. The execution control unit 300 controls the execution of the second control software 12 on the ECU 10. The evaluation unit 400 evaluates the output data output as a result of the execution of the second control software 12 based on the output expected values. The upload unit 500 uploads the evaluation results from the evaluation unit 400 to the cloud 1 via the gateway 50.

[0026] The ECU 10 also has a memory unit 14. The memory unit 14, used by the test execution control software, includes a vehicle data memory unit 600, a design test case information memory unit 610, a test case information memory unit 620, and a test result memory unit 630.

[0027] The vehicle data storage unit 600 stores vehicle data acquired from the communication bus 20 by the vehicle data acquisition unit 100. The design test case information storage unit 610 stores design test case information sent from the cloud 1. The test case information storage unit 620 stores new test case information generated by the test case synthesis unit 200. In addition, the test result storage unit 630 stores test case information as result information when the output value and the expected output value of the second control software do not match among the evaluation results from the evaluation unit 400.

[0028] In this embodiment, test case information is generated by the test case synthesis unit 200 using vehicle data acquired by the vehicle data acquisition unit 100, and the second control software 12 is tested under the control of the execution control unit 300 using the generated test case information. The output data obtained as a result of the test is evaluated by the evaluation unit 400, and the evaluation results are uploaded to the cloud 1 by the upload unit 500. The processing of each of these units does not necessarily have to start processing in the next processing unit only after processing in the preceding processing unit is completed. Some or all of the processing units may operate in parallel, and processing may be performed in a pipeline manner using the results obtained from the previous processing.

[0029] Physically, the ECU 10 consists of a CPU (Central Processing Unit) and memory. The first control software 11, the second control software 12, and the test execution control software 13 are programs stored in memory and executed by the CPU. The functions of the vehicle data acquisition unit 100, test case synthesis unit 200, execution control unit 300, evaluation unit 400, and upload unit 500 described above are realized by the execution of the test execution control software 13 on the CPU.

[0030] The storage unit 14 can be implemented by a non-volatile storage device such as a hard disk drive or flash memory device, which is built into the ECU 10 or can be connected to the outside of the ECU 10. From the viewpoint of processing speed, a storage device using a high-speed volatile memory element such as DRAM (Dynamic Random Access Memory) may be temporarily used for the storage unit 14. The vehicle data storage unit 600, the design test case information storage unit 610, the test case information storage unit 620, and the test result storage unit 630 may be provided in the same storage device, or different storage devices may be used for some or each of the storage units.

[0031] Figure 3 is a flowchart showing the flow of the vehicle data acquisition process performed by the vehicle data acquisition unit 100.

[0032] Prior to acquiring vehicle data, the vehicle data acquisition unit 100 reads acquisition rules indicating the acquisition conditions for vehicle data from the design test case information storage unit 610. The acquisition rules, like the design test case information, are distributed in advance from the cloud 1 and stored in the design test case information storage unit 610.

[0033] The acquisition rules include trigger information indicating the timing of vehicle data acquisition. The trigger information may include information indicating the period of vehicle data acquisition. If the trigger information specifies a period, the vehicle data acquisition unit 100 will periodically acquire vehicle data from the communication bus 20 at the specified period in subsequent processing. The trigger information may also monitor one or more specific vehicle data and specify the acquisition of vehicle data or the start of vehicle data acquisition when the conditions specified by that or those vehicle data are met. The specified conditions may include when a specific vehicle data takes a specific value, or exceeds a specific value, or when the number of occurrences of a specific value reaches a predetermined number of times. For example, when acquiring vehicle information during emergency braking, the condition may be that vehicle data related to braking force exceeds a predetermined value.

[0034] The acquisition rules may include information such as the vehicle data to be acquired, the size of the vehicle data to be acquired, and the period, in addition to trigger information. For example, if the trigger information specifies a period or the timing to start acquiring vehicle data, it may also include information specifying the conditions for ending the acquisition of vehicle data (step S100).

[0035] The vehicle data acquisition unit 100 monitors the vehicle data transmitted and received on the communication bus 20, determines whether the conditions for acquiring vehicle data are met according to the acquired rules it has read, and waits until the conditions are met (step S105).

[0036] In step S105, if it is determined that the acquisition conditions have been met, the vehicle data acquisition unit 100 acquires vehicle data from the communication bus 20 (S110). Subsequently, the vehicle data acquisition unit 100 adds information such as priority and tags to the vehicle data acquired from the communication bus 20, which can be used to determine whether or not to save the test case information generated in the test case generation process described later as test case information. The addition of priority and tags here can be set in the acquisition rules. For example, if it is determined that the vehicle is traveling in a specific area based on the vehicle's location information, a high priority may be added, and if it is in another area, a low priority may be added. In addition, the vehicle's driving status may be determined based on the vehicle data, and information such as "turn right at intersection" may be added as a tag. The necessity of adding priority and tags may also be specified in the acquisition rules (step S115).

[0037] The vehicle data acquisition unit 100 then determines whether the acquired vehicle data can be stored in the vehicle data storage unit 600. For example, if the memory area allocated to the vehicle data storage unit 600 is full, it is determined that new vehicle data cannot be stored in the vehicle data storage unit 600. If it is determined that the vehicle data cannot be stored, the vehicle data acquisition unit 100 discards the vehicle data and terminates the process (step S120).

[0038] If it is possible to store vehicle data, the vehicle data acquisition unit 100 stores the acquired vehicle data in the vehicle data storage unit 600. At this time, it is also possible to store a portion of the acquired vehicle data based on the priority and tag information added in step S115. The partial storage of vehicle data may be performed, for example, when the capacity of the area allocated as the vehicle data storage unit 600 falls below a predetermined threshold.

[0039] Furthermore, vehicle data with a high priority can be stored in a memory area reserved on a non-volatile storage medium so that it can be used in later tests, while vehicle data with a low priority can be stored on a volatile storage device and used only in tests performed during the current run (step S125).

[0040] The vehicle data acquisition unit 100 determines whether the acquisition of vehicle data is complete according to the acquisition rules. If the acquisition rules specify a termination condition and the termination condition is met, the unit determines that the acquisition of vehicle data is complete and terminates the process. If the termination condition specified in the acquisition rules is not met, the vehicle data acquisition unit 100 returns to the process in step S105 and continues to acquire vehicle data at the next acquisition timing (step S130).

[0041] Figure 4 is a schematic diagram showing an example of vehicle data stored in the vehicle data storage unit 600. For the sake of explanation, the vehicle data 6000 is shown as table data in the figure, but other arbitrary data formats such as list format may be used.

[0042] The vehicle data 6000 stored in the vehicle data storage unit 600 includes a timestamp indicating the time when the data was acquired, a label name for identifying the vehicle data, a data value of the vehicle data, and a priority and tags added by the vehicle data acquisition unit 100. As the label name, information that can identify the type of the vehicle data and what kind of data it is is attached. For example, when the communication bus 20 from which the vehicle data is acquired conforms to the CAN standard, a label name can be assigned corresponding to the CAN ID, and when it conforms to Ethernet, the source address, etc.

[0043] FIG. 5 is a flowchart showing the flow of the test case information generation process executed by the test case synthesis unit 200.

[0044] The test case synthesis unit 200 first acquires information regarding the usage rate of the calculation resources of the ECU 10. Information regarding the usage rate of the calculation resources can use information that can grasp the processing load status of the ECU 10, such as the usage rate of the CPU of the ECU 10 and the usage rate of the memory, etc. (step S200).

[0045] The test case synthesis unit 200 determines whether it is possible to execute the test case information generation process based on the acquired information regarding the usage rate of the calculation resources. When the usage rate of the calculation resources is high and the processing load of the ECU 10 is high, it is determined that the execution of the test case information generation process cannot be performed because it may affect the operation of the first control software. The usage rate serving as the determination threshold value may be obtained empirically or obtained by calculation or the like from the processing contents such as the first control software 11 and the second control software 12 and the processing capacity of the CPU and the memory capacity of the ECU 10 (step S203).

[0046] When it is determined that the generation process of test case information is possible, the test case synthesis unit 200 reads a synthesis rule from the design test case information storage unit 610. The synthesis rule is distributed from the cloud 1 and stored in the design test case information storage unit 610. The synthesis rule can be, for example, one that specifies a method for synthesizing the distributed design test case information and the vehicle data stored in the vehicle data storage unit 600. The synthesis rule may include a plurality of rules, and priorities may be assigned to these plurality of rules so that test case information can be appropriately generated from the test case information generated by the rule with the highest priority (step S205).

[0047] Next, the test case synthesis unit 200 reads the vehicle information stored in the vehicle data storage unit 600 and the design test case information stored in the design test case information storage unit 610. The reading of the vehicle data from the vehicle data storage unit 600 may be in the storage order (first in first out) or in the order from the last stored (last in first out). Also, according to the synthesis rule, specific data can be acquired.

[0048] An example of the design test case information is shown in FIG. 6. The design test case information 6100 is given as a set including a serial number for identifying the design test case information 6100, one or more types of test inputs for identifying vehicle data that is input data to the second control software, one or more design input values that are values of the test inputs used in the test, and an expected value that is expected to be output when the design input value is input. The test input is information for identifying the input data of the control software for which the test is to be performed, and is information corresponding to the label name of the vehicle data 6000 stored in the vehicle data storage unit 600. In FIG. 6, only one set of design test case information is included, but the design test case information may include a plurality of sets of design test case information (step S210).

[0049] The test case synthesis unit 200 generates new test case information from vehicle data 6000 and design test case information 6100 according to synthesis rules. Examples of methods for generating test case information include the following:

[0050] Figure 7 is a schematic diagram showing a first method for generating test cases. In the first method, the vehicle data corresponding to the test inputs included in the design test case information 6100 is masked from the acquired vehicle data 6000, and the unmasked data values ​​in the vehicle data 6000 and the design input values ​​of each test input in the design test case information 6100 are combined to generate new test case information 6200. Note that in Figure 7, the priority and tag information added to the vehicle data 6000 by the vehicle data acquisition unit 100 is omitted.

[0051] For example, in the example shown in Figure 7, the design input values ​​for "CAN_A" and "ETH_B" are given as test inputs to the second control software, so the vehicle data with the corresponding label names is masked. In the figure, the fact that this vehicle data is masked is indicated by the dashed arrow B. As shown by the solid arrow A, the test case synthesis unit 200 uses the design input values ​​of the design test case information 6100 for "CAN_A" and "ETH_B", and the data values ​​of the vehicle data 6000 for the other data, and synthesizes these to obtain the test input values ​​for the test case information 6200. As shown by the solid arrow C, the expected value of the test case information 6200 is set to the expected value set in the design test case information 6100.

[0052] In this case, the synthesis rule may also include specifying vehicle data that has some of the unmasked label names, and synthesizing the vehicle data of the type identified by the specified label names with the test inputs set in the design test case information 6100 to generate new test case information 6200. By using such a synthesis rule, it is possible to intentionally combine the data specified by the design input values ​​with vehicle data of any type to generate test case information that is appropriate for the purpose of the test.

[0053] Although not shown in the diagram, the newly generated test case information 6200 may include a flag corresponding to the test input, indicating whether the input value is a design input value given in the design test case information 6100 or a data value obtained as vehicle data 6000. It may also include priority and tags that were added when the vehicle data 6000 was acquired.

[0054] This method makes it possible to generate test case information that combines unmasked vehicle data as noise in various vehicle states with the input values ​​specified in the design test case information 6100. By testing the second control software using such test case information, it is possible to verify whether the output (expected value) set in the design test case information 6100 is maintained under various conditions. By analyzing test results that show values ​​different from the expected value and investigating the cause, it is possible to detect whether there are any potential defects in the second control software 12.

[0055] Furthermore, by using the first method, it is possible to define failure conditions and design input values ​​that deviate from specifications as design test case information 6100, thereby generating test case information from the perspective of injecting failure conditions and security, and verifying the stability of the second control software 12.

[0056] Furthermore, as design test case information 6100, vehicle data related to the type of ECU sending vehicle data, its supplier type, communication type, communication that triggers security calculations, and interrupts related to engine and motor control may be set as design input values. By intentionally combining these design input values ​​with unmasked vehicle data to generate test case information 6200, it becomes possible to perform targeted tests.

[0057] Figure 8 is a schematic diagram illustrating a second method for generating test cases. In the second method, all data from the vehicle data 6000 except for the data values ​​of the label names corresponding to the test inputs included in the design test case information 6100 are masked, and the test case information 6210 is generated using the data values ​​of the label names corresponding to the test inputs. Note that, as in Figure 7, the priority and tag information added to the vehicle data 6000 by the vehicle data acquisition unit 100 is omitted in Figure 8.

[0058] For example, in the example shown in Figure 8, the design test case information 6100 has "CAN_A" and "ETH_B" set as test inputs, so the test case synthesis unit 200 generates test case information 6200 using the data values ​​of the vehicle data with the corresponding label names, indicated by the solid arrow D. Vehicle data other than the label names corresponding to the test inputs "CAN_A" and "ETH_B" are masked, as indicated by the dashed arrow E, and are not used in the generation of test case information 6200. In the second method as well, the expected value of the test case information 6200 is set to the expected value set in the design test case information 6100, as indicated by the solid arrow F.

[0059] Although not shown in the diagram, similar to the first method described above, the test case information may include a flag corresponding to the test input, indicating whether the input value is a design input value given in the design test case information 6100 or a data value obtained as vehicle data 6000. It may also include priority and tag information added when the vehicle data 6000 was acquired.

[0060] According to the second method, test case information 6210 can be generated that is similar to the design test case information 6110 but with different input values. In general verification tests, the input values ​​are limited by boundary value analysis or equivalence analysis within the specified range of input values, making it difficult to perform tests with comprehensive input values. By generating similar test case information from vehicle data as in this method, it becomes possible to expand the comprehensiveness of test cases using vehicle data that may occur during actual driving, thereby improving test case coverage.

[0061] The method by which test case information is generated from these multiple generation methods can be specified by a synthesis rule (step S215).

[0062] After generating test case information, the test case synthesis unit 200 determines whether there is sufficient free space in the test case information storage unit 620 to save the generated test cases. For example, if the memory space allocated to the test case information storage unit 620 is running low and the size of the generated test case information is larger, it is determined that it is not possible to save new test case information. If the vehicle data acquisition unit 100 assigns priorities and tags to the vehicle data, and there is information that can be excluded from the test case information based on that information, the unit may determine whether it is possible to save some of the test case information excluding that information.

[0063] If it is determined that the test case information cannot be saved, the test case synthesis unit 200 discards the generated test case information and terminates the process (step S220).

[0064] In step S220, if it is determined that the generated test case information can be saved in the test case information storage unit 620, the test case synthesis unit 220 saves the test case information in the test case information storage unit 620 and terminates the process (step S225).

[0065] Figure 9 shows the flow of processing performed by the execution control unit 300.

[0066] First, the execution control unit 300, similar to the test case synthesis unit 200, acquires information regarding the utilization rate of the computing resources of the ECU 10 (step S300).

[0067] The execution control unit 300 determines, based on the acquired information regarding the utilization rate of computing resources, whether there is sufficient free computing resources and whether the second control software 12 can be executed. Similar to the test case generation process, if the utilization rate of computing resources is not sufficiently low and there are not enough free resources, it is determined that the second control software 12 cannot be executed because it may affect the operation of the first control software 11. The utilization rate that serves as the threshold for this determination may be obtained empirically, as in the case of the test case generation process, or it may be obtained by calculations based on the content of the processing of the first control software 11, the second control software 12, etc., and the processing power and memory capacity of the CPU equipped with the ECU 10. If there is not sufficient free computing resources, the execution control unit 300 terminates the process (step S305).

[0068] In step S305, if it is determined that the ECU 10 has sufficient computing resources, the execution control unit 300 starts executing the second control software 12. While the second control software 12 is running, the execution control unit 300 reads data set as design input values ​​from the test case information stored in the test case information storage unit 620 as appropriate, and inputs it into the second control software 12 as vehicle data to be used in its processing (step S310).

[0069] While the second control software 12 is running, the execution control unit 300 acquires the computing resource usage rate as appropriate (step S315). The execution control unit 300 then determines whether the usage rate exceeds a predetermined threshold, and if the computing resource usage rate is below the threshold, it returns to step S310 and continues the execution of the second control software. The threshold for computing resource usage rate used here may be the same as the threshold used in step S305, obtained empirically, or obtained by calculations based on the processing content of the first control software 11, the second control software 12, etc., and the processing power and memory capacity of the CPU equipped with the ECU 10. The threshold used here will be higher than the threshold used in step S305 (step S320).

[0070] If the utilization rate of computing resources exceeds a threshold in step S320, the execution control unit 300 stops the execution of the second control software 12 and terminates the process (step S325).

[0071] While the second control software 12 is running, the evaluation unit 400 acquires the input test case information 6200 and the output data output by the second control software 12 and evaluates the execution results. Specifically, the evaluation unit 400 compares the output data of the second control software with the expected value set in the test case information used for processing to determine whether the test case is a "success" or a "failure". If the result of the determination is a "failure", the evaluation unit 400 saves the test case information as test result information in the test result storage unit 630.

[0072] Figure 10 is a schematic diagram showing an example of the structure of test result information.

[0073] The test result information 6300 includes, as information obtained from the executed test case information 6200, a serial number identifying the test case information 6200, the type of one or more test inputs identifying the vehicle data used as input data to the second control software 12, one or more design input values ​​which are the values ​​of the test inputs used in the test, and an expected value which is the expected output value. In addition, if the test case information includes priority or tags, that information may also be included in the test result information 6300. Furthermore, the test result information 6300 may include a data flag that distinguishes whether the vehicle data used as test input is a design input value set in the test case information 6200 or vehicle data acquired by the vehicle data acquisition unit 100.

[0074] The test result information 6300 further includes an output value which is the value of the output data output from the second control software 12 as the result of the test execution, the time when the test was executed, and a test result which is the result of the evaluation by the evaluation unit 400.

[0075] The test result information 6300 stored in the test result storage unit 630 is read by the upload unit 500 and sent to the cloud 1 via the gateway 50. The upload of test result information to the cloud 1 can be performed when a certain amount of test result information has been accumulated and the processing load and communication load of the ECU are low, or periodically at predetermined time intervals. Alternatively, the upload of test result information to the cloud 1 may be performed in real time, for example, when the processing capacity of the ECU is high and there is sufficient communication capacity.

[0076] The processes described above, executed by each part of the test execution control software 13, are repeatedly performed during vehicle operation, and vehicle data collection, test case information generation, and testing of the second control software are continuously carried out.

[0077] According to this embodiment, since the control software is tested using vehicle data generated while the vehicle is actually running, it becomes possible to efficiently verify the operation of the control software under various inputs. Testing the control software using vehicles can be performed simultaneously on multiple vehicles with different driving conditions, such as region and environment, and it is possible to perform comprehensive testing more efficiently compared to testing in a development environment with limited resources. For example, the developer of the control software can analyze the test results sent to the cloud to verify whether the control software is operating as desired and whether there are any defects.

[0078] In this embodiment, the second control software is tested on the ECU running the first control software that actually controls the vehicle. However, it is also possible to install a test ECU in the vehicle, separate from the vehicle control ECU, and perform the test using the test ECU. In this case, the above-mentioned test execution control software is run on the test ECU, and the control software to be tested is executed under its management to perform the test. This method minimizes the impact on the operation of the control software that actually controls the vehicle, and allows testing to be performed continuously or at arbitrary timings.

[0079] Figure 11 is a schematic block diagram showing the configuration of the control software test environment in the second embodiment. In the first embodiment described above, the control software test is performed on the ECU mounted on the vehicle, and the test results are sent to the cloud. In contrast, in this embodiment, the ECU mounted on the vehicle collects vehicle data, and test cases are generated and the control software is tested on the cloud. In Figure 11, parts having the same function and configuration as those described in the first embodiment are given the same reference numerals as in Figure 2, and their detailed explanations are omitted below.

[0080] In this embodiment, the ECU 10a mounted on the vehicle executes not only the first control software 11 that actually controls the vehicle, but also data acquisition software 15 that acquires vehicle data. The ECU 10a also has a storage unit 14a used by the data acquisition software 15. The storage unit 14a, like the storage unit 14 in the first embodiment, can be implemented as a storage device such as a hard disk drive or flash memory device that is built into the ECU 10a or can be connected to the outside of the ECU 10a. Furthermore, from the viewpoint of processing speed, a storage device using a high-speed volatile memory element such as DRAM may be temporarily used for the storage unit 14.

[0081] The storage unit 14a includes a vehicle data storage unit 600 and an acquisition rule storage unit 650 that stores acquisition rules indicating the acquisition conditions for vehicle data, which were held in the design test case information storage unit 610 in the first embodiment.

[0082] The data acquisition software 15 includes a vehicle data acquisition unit 100 and an upload unit 500a. The vehicle data acquisition unit 100, similar to the vehicle data acquisition unit in the first embodiment, collects vehicle data according to acquisition rules and stores it in the vehicle data storage unit 600. The upload unit 500a reads the vehicle data stored in the vehicle data storage unit 600 and uploads it to the cloud 1 via the gateway 50.

[0083] The test execution environment 1000 is constructed using computing resources on Cloud 1 and provides an environment for testing control software using uploaded vehicle data. The test execution environment 1000 includes test execution control software 13a that runs on computing resources provided by Cloud 1, a second control software 12 to be tested, and a storage unit 14b used by the test execution control software 13a.

[0084] The test execution control software 13a corresponds to the test execution control software in the first embodiment and includes a vehicle data acquisition / vehicle identification unit 1100, a vehicle data adjustment unit 1200, a test case synthesis unit 200, an execution control unit 300a, and an evaluation unit 400. The test case synthesis unit 200 and the evaluation unit 400 have the same functions as the test case synthesis unit 200 and evaluation unit 400 of the test execution control software 13 of the ECU 10 in the first embodiment.

[0085] The execution control unit 300a controls the execution of the second control software 12 and tests the second control software, similar to the execution control unit 300 of the first embodiment. However, in this embodiment, since there is no need to consider the impact on the first control software 11, the execution control of the second control software is not controlled according to the usage rate of computing resources. If the test execution environment 1000 is constructed with different computing resources, such as a processor, than the ECU on which the second control software 12 actually runs, the second control software cannot be directly executed on the computer provided by the test execution environment 1000. In such a case, the execution control unit 300a may have the function of an emulator that simulates the computing environment of the ECU on which the second control software actually runs.

[0086] The vehicle data acquisition / vehicle identification unit 1100 acquires vehicle data uploaded from the ECU 10a and stores it in the vehicle data storage unit 1600. The vehicle data acquisition / vehicle identification unit 1100 may also be configured to instruct the upload unit 500a of the ECU 10a of a vehicle that matches the conditions for collecting vehicle data, according to a collection rule in which conditions for identifying the vehicle from which vehicle data is to be collected are set, and to collect vehicle data from that specific vehicle. The collection rule may include, for example, conditions for identifying vehicles traveling in a specific area or on a specific road. By setting the collection rule based on the conditions of the test to be conducted, it becomes possible to efficiently perform tests using vehicle data that matches the test conditions.

[0087] The vehicle data adjustment unit 1200 converts the vehicle data stored in the vehicle data storage unit 1600 into data that can be acquired by an ECU of a different vehicle type than the one from which the vehicle data was acquired. The conversion of vehicle data can be performed based on information provided by, for example, a pre-prepared conversion table. By performing the conversion of vehicle data using the vehicle data adjustment unit 1200, for example, data collected from an existing vehicle can be converted to be compatible with new control software for a vehicle that is under development and not yet in production, and the new control software can be tested using the vehicle data collected from the existing vehicle.

[0088] The storage unit 14b includes a vehicle data storage unit 1600, a design test case information storage unit 610, a test case information storage unit 620, and a test result storage unit 630. The design test case information storage unit 610, the test case information storage unit 620, and the test result storage unit 630 hold the same information as the corresponding storage units in the first embodiment. The vehicle data storage unit 1600 stores vehicle information uploaded from the ECU 10a installed in the vehicle. The vehicle data storage unit 1600 stores vehicle data collected from multiple vehicles. For this reason, it is desirable that the vehicle data storage unit 1600 has a large storage area. Each storage unit does not necessarily have to be provided in a single storage device; different storage devices may be used for some or all of the storage units.

[0089] The test case synthesis unit 200, similar to the first embodiment, generates new test case information from vehicle data collected from multiple vehicles stored in the vehicle data storage unit 1600, or from data converted by the vehicle data adjustment unit 1200, according to synthesis rules. The generated test case information is stored in the test case information storage unit 620.

[0090] The execution control unit 300a operates the second control software 12 to sequentially read the test case information stored in the test case information storage unit 620, provides the set input values ​​to the second control software 12, and executes the process. The execution results are evaluated by the evaluation unit 400, and the test case information for which the test case result was "failure" is stored as test result information in the test result storage unit 630.

[0091] In this embodiment as well, the developer can analyze the test result information stored in the test result storage unit 630 to verify whether the control software is operating as desired, whether there are any defects, and so on.

[0092] According to this embodiment, since the execution of the control software under test is performed on the cloud side, the processing performed by the ECU installed in the vehicle can be simplified, and the load on the ECU can be reduced. In addition, it becomes possible to perform more comprehensive tests using vehicle data from vehicles running under different conditions.

[0093] In the embodiments described above, only one test is performed at a time. However, for example, multiple different design test case information can be prepared in advance, multiple new test case information can be generated using the multiple design test case information, and multiple instances of the second control software can be started in parallel. Multiple tests can then be performed in parallel using the multiple new test case information generated from the different design test case information. Alternatively, multiple control software programs of different versions, capable of using the same test case information, can be started in parallel as the second control software, and tests of these multiple control software programs can be performed in parallel using a single generated test case information.

[0094] Although the present invention has been described above using representative embodiments as examples, the present invention is not limited thereto and can be implemented in various ways without departing from the spirit of the invention as described in the claims. Furthermore, the above embodiments have been described in detail for the purpose of clearly explaining the present invention and are not necessarily limited to those having all the configurations described.

[0095] 1, 1a: Cloud, 10, 10a: Electronic Control Unit (ECU), 11: First control software, 12: Second control software, 13, 13b: Test execution control software, 13a: Data acquisition software, 14, 14a, 14b: Storage device, 20: Communication bus, 30: Sensor, 40: Actuator, 50: Gateway, 100: Vehicle data acquisition unit, 200: Test case synthesis unit, 300: Execution control unit, 400: Evaluation unit, 500: Upload unit, 600: Vehicle data storage unit, 610: Test case information storage unit, 620: Test case storage unit, 630: Test result storage unit

Claims

1. A method for testing control software that performs the functions of a vehicle control device mounted on a vehicle, comprising: preparing design test case information relating to a combination of input data and expected output data that has been created in advance and used in the test; collecting vehicle data generated during the actual driving of the vehicle; obtaining a new set of input data obtained by masking a specific type of data based on a given synthesis rule from the input data of the design test case information and the collected vehicle data; generating new test case information from the new set of input data and the expected output data; and operating the control software to be tested to perform a test of the control software based on the new test case information.

2. A method for testing control software according to claim 1, comprising evaluating the output data obtained as a result of the execution of the test based on the expected value of the output data.

3. The method for testing control software according to claim 2, wherein the collection of vehicle data, the generation of new test case information, and the execution of the test are performed by a control device mounted on the vehicle.

4. The method for testing control software according to claim 3, further comprising transmitting the results of the evaluation to a development environment built on the cloud.

5. The method for testing control software according to claim 4, further comprising downloading the control software and the test case information from the cloud to the control device prior to collecting the vehicle data.

6. The control software testing method according to claim 2, wherein the specific type of data is vehicle data of a different type from the input data included in the design test case information, and the generation of the new test case information is performed by generating the new test case information using vehicle data of the same type as the input data included in the design test case information as the new input data set.

7. The method for testing control software according to claim 2, wherein the specific type of data is vehicle data of the same type as the input data included in the design test case information, and the new test case information is generated by combining the input data included in the design test case information and vehicle data of a different type from the input data included in the design test case information as a set of new input data.

8. A method for testing control software according to claim 7, wherein the input data of the design test case information includes input data relating to the failure state of the vehicle.

9. The method for testing control software according to claim 7, wherein the synthesis rules include rules for combining vehicle data of a certain type from among types different from the input data included in the design test case information with the input data included in the design test case information.

10. In preparing the design test case information, a second design test case information different from the first design test case information is prepared; in generating the new test case information, a second new test case information different from the first new test case information is generated based on the second design test case information; and in executing the test, multiple instances of the control software are operated in parallel to execute two tests using the first and second new test case information in parallel, as described in claim 2.

11. The method for testing control software according to claim 2, wherein the test is performed by running a second control software in parallel with the first control software, which is the control software, and which is capable of performing tests using the same test case information as the first control software, and performing tests on the first control software and the second control software in parallel using the new test case information.

12. The method for testing control software according to claim 2, wherein the collection of vehicle data includes a process of updating the vehicle data collected by a control device mounted on the vehicle to the cloud, and the generation of new test case information and the execution of the test are performed using a test environment built on the cloud.

Images