Evaluation data creation device, evaluation data creation method, and evaluation data creation program

Abstract

To prepare more evaluation data used for evaluating performance of driving support with a small processing amount.SOLUTION: An evaluation data preparation device for preparing evaluation data for evaluating a driving support function of a vehicle includes: a scenario setting unit for setting a scenario for reproducing travelling of a vehicle; a sensor data preparation unit for processing the scenario set by the scenario setting unit with a sensor model which is modelled after a sensor of the vehicle and preparing sensor data, which is the information detected by the sensor model; and an actual data conversion unit for associating the sensor data prepared by the sensor data preparation unit with actual data, which is the data acquired by the sensor of the vehicle, converting the sensor data on the basis of the associated actual data, and preparing conversion actual data, which is the evaluation data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] , , , , , , ,

[0006] , , , ,

[0005] , , , , , ,

[0001] The present disclosure relates to an evaluation data creation device, an evaluation data creation method, and an evaluation data creation program.

Background Art

[0002] In order to assist a driver who drives a vehicle, driving support functions such as automatic driving and collision avoidance have been developed. The driving support function makes a determination of the support to be executed based on the acquired data of various sensors mounted on the vehicle. The driving support function needs to be compatible with various environments of the traveling vehicle.

[0003] An evaluation method for evaluating the performance of the driving support function has also been proposed. Patent Document 1 describes that the driving support function is evaluated based on the behavior executed by reproducing the driving environments of a plurality of vehicles by simulation and inputting the reproduced information into each vehicle.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] On the other hand, by driving actual vehicles, it is possible to obtain actual driving data rather than simulated data. In this case, in order to obtain the reliability suitable for evaluating driver assistance performance, it is necessary to conduct tests under various conditions and situations. However, there are limitations to conducting driving tests under various conditions and situations using actual vehicles. Therefore, in reality, it is difficult to obtain data suitable for evaluating driver assistance performance from actual vehicle driving tests.

[0008] This disclosure aims to provide an evaluation data creation device, an evaluation data creation method, and an evaluation data creation program that can efficiently create reliable evaluation data suitable for evaluating driver assistance performance, in order to solve the above-mentioned problems. [Means for solving the problem]

[0009] This disclosure provides an evaluation data creation device for creating evaluation data to evaluate at least one of the autonomous driving and driving assistance functions of a vehicle, comprising: a scenario setting unit that sets a scenario to reproduce the driving of the vehicle; a sensor data creation unit that uses a sensor model that models the sensors of the vehicle to create sensor data acquired when the vehicle drives according to the scenario set by the scenario setting unit; and a real data conversion unit that associates real data, which is data acquired by the sensors of the vehicle, with the sensor data created by the sensor data creation unit, converts the sensor data based on the associated real data, and creates converted real data which is evaluation data.

[0010] Furthermore, this disclosure provides a method for creating evaluation data to evaluate at least one of the autonomous driving and driving assistance functions of a vehicle, the method comprising: setting a scenario to reproduce the driving of the vehicle; creating sensor data acquired when the vehicle drives the scenario set by the scenario setting unit using a sensor model that models the sensors of the vehicle; and associating the created sensor data with actual data, which is data acquired by the sensors of the vehicle, and converting the sensor data based on the associated actual data to create converted actual data which is evaluation data.

[0011] Furthermore, this disclosure provides an evaluation data creation program for creating evaluation data to evaluate at least one of the autonomous driving and driving assistance functions of a vehicle, the program comprising the steps of: setting a scenario to reproduce the driving of the vehicle; creating sensor data to be acquired when the vehicle drives according to the scenario set by the scenario setting unit, using a sensor model that models the sensors of the vehicle; and associating the created sensor data with actual data, which is data acquired by the sensors of the vehicle, and converting the sensor data based on the associated actual data to create converted actual data which is evaluation data. [Effects of the Invention]

[0012] This configuration allows for the creation of more evaluation data used to assess the performance of driver assistance systems with less processing power. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a block diagram showing an example of a data generation device for evaluation. [Figure 2] Figure 2 is a flowchart showing an example of the processing performed by the evaluation data creation device. [Figure 3] Figure 3 is a flowchart showing an example of how the machine learning unit operates. [Figure 4] Figure 4 is a flowchart showing an example of the processing of the evaluation data creation device. [Figure 5] Figure 5 is a flowchart showing an example of the processing of the evaluation data creation device. [Figure 6] Figure 6 is a flowchart showing an example of the processing of the evaluation data creation device. [Figure 7] Figure 7 is a flowchart showing an example of the processing of the evaluation data creation device. [Figure 8] Figure 8 is a block diagram showing an example of the evaluation data creation device. [Figure 9] Figure 9 is a flowchart showing an example of the operation of the machine learning unit. [Figure 10] Figure 10 is a flowchart showing an example of the processing of the evaluation data creation device.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments according to the present disclosure will be described in detail based on the drawings. Note that the present invention is not limited by these embodiments. Also, the constituent elements in the following embodiments include those that can be replaced by those skilled in the art and are easy to replace, or those that are substantially the same. Furthermore, the constituent elements described below can be combined as appropriate, and when there are multiple embodiments, the embodiments can also be combined.

[0015] <Evaluation Data Creation Device> Figure 1 is a block diagram showing an example of the evaluation data creation device. The evaluation data creation device 10 according to the present embodiment creates evaluation data used for evaluating the driving support function of the ECU (Electronic Control Unit) 6.

[0016] The ECU 6 is mounted on a vehicle and executes a driving support function based on information acquired by sensors mounted on the vehicle. The driving support executed by the ECU 6 is not particularly limited. Examples of driving support include traveling speed control, brake control, steering control, automatic driving control, warning control to the driver, warning control to other vehicles and surrounding pedestrians, etc. That is, at least one of the vehicle's automatic driving function and driving support function is targeted. The sensors mounted on the vehicle also include various information, such as information acquired by distance sensors such as LiDAR, temperature sensors, cameras, information acquired from surrounding vehicles, roadside units, and the cloud via communication, operation information of the accelerator, brake, steering, etc. input to the vehicle, and information of sensors arranged in the drive mechanism.

[0017] When evaluating the ECU 6, the evaluation data created by the evaluation data creation device 10 is input to the ECU 6. When the ECU 6 receives the data as sensor data, it processes the input evaluation data and outputs the processing result as instruction information for driving support to the evaluation device 8.

[0018] The evaluation device 8 evaluates the driving support function of the ECU 6. The evaluation device 8 has an arithmetic processing function such as a CPU and a storage function such as a ROM and a RAM, and is a device that executes an evaluation function with a program. The evaluation device 8 acquires the evaluation data input by the evaluation data creation device 10 and the conditions of the evaluation data, and acquires the instruction information for driving support output by processing the evaluation data by the ECU 6. The evaluation device 8 determines whether the timing and content of the driving support executed by the ECU 6 are appropriate for the situation of the scenario of the evaluation data. By evaluating the ECU 6 using the evaluation data, the performance of the driving support of the ECU 6 can be evaluated. By evaluating the ECU 6 using the evaluation data and adjusting the function of the ECU 6, the performance of the driving support can be improved. Also, the evaluation data can be used as learning data for the ECU 6.

[0019] Next, the evaluation data creation device 10 will be described. The evaluation data creation device 10 simulates the driving conditions of the vehicle under evaluation and creates evaluation data that simulates the data input to the ECU 6 when the vehicle under evaluation is in operation.

[0020] The evaluation data creation device 10 includes an input unit 12, an output unit 14, a communication unit 15, a calculation unit 16, and a storage unit 18. The input unit 12 includes input devices such as a keyboard and mouse, a touch panel, or a microphone for collecting speech from an operator, and outputs signals corresponding to operations performed by the operator on the input devices to the calculation unit 16. The output unit 14 includes a display device such as a display and displays a screen containing various information such as processing results and images of the processing target based on the display signals output from the calculation unit 16. The output unit 14 may also include a recording device that outputs data to a recording medium. The communication unit 15 transmits data using a communication interface. The communication unit 15 communicates with the ECU 6 and the evaluation device 8 to send and receive data. The communication unit 15 communicates with external devices and sends various data and programs acquired to the storage unit 16 for storage. The communication unit may be connected to external devices via a wired communication line or a wireless communication line.

[0021] The arithmetic unit 16 includes an integrated circuit (processor) such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), and a memory that serves as a work area. It performs various processes by executing various programs using these hardware resources. Specifically, the arithmetic unit 16 reads programs stored in the storage unit 18, expands them into memory, and performs various processes by having the processor execute the instructions contained in the programs expanded into memory. The arithmetic unit 16 includes a scenario setting unit 22, a sensor data creation unit 24, a real data conversion unit 26, a correspondence processing unit 28, and a machine learning unit 30. Before describing each part of the arithmetic unit 16, the storage unit 18 will be described.

[0022] The memory unit 18 consists of a non-volatile memory device such as a magnetic memory device or a semiconductor memory device, and stores various programs and data. The memory unit 18 includes a data creation program 32, a learning program 34, a scenario model 36, an environmental object model 37, a weather model 38, a light source model 40, a vehicle model 42, a sensor model 44, a correspondence table 46, actual data 48, and a learned model 50.

[0023] The data stored in the memory unit 18 includes a scenario model 36, an environmental object model 37, a weather model 38, a light source model 40, a vehicle model 42, a sensor model 44, a mapping table 46, actual data 48, and a trained model 50. Multiple copies of each of the scenario model 36, environmental object model 37, weather model 38, light source model 40, vehicle model 42, sensor model 44, mapping table 46, actual data 48, and trained model 50 are available, allowing the user to select the model, data, and table to use.

[0024] Scenario model 36 is a model that sets the conditions for simulating vehicle driving. Scenario model 36 is a combination of models selected from the environment object model 37, weather model 38, light source model 40, and vehicle model 42. Scenario model 36 may be used as a model representing a single moment in time without time progression to evaluate the processing of ECU6 at a specific moment, or as a model with time progression to evaluate the processing of ECU6 while driving over a predetermined section.

[0025] The environmental object model 37 is a model that includes information about the shape of the area around the vehicle. The environmental object model 37 includes information about the shape of the road on which the vehicle is traveling (straight lines, curves, intersections, presence or absence of guardrails, presence or absence of traffic lights), information about surrounding vehicles, information about surrounding pedestrians, etc. In other words, the environmental object model 37 includes a model of the three-dimensional shape around the vehicle.

[0026] Weather model 38 is a model that reproduces sunny, cloudy, rainy, stormy, snowy, sleet, and vehicle weather conditions. Weather model 38 is a model in which the reproduced conditions change depending on cloud cover, rainfall, snowfall, temperature, wind speed, etc.

[0027] Light source model 40 is a model of objects that emit light around a vehicle, such as the sun, lighting towers, buildings, and oncoming vehicles.

[0028] Vehicle model 42 is a model that reproduces the vehicle being analyzed and surrounding vehicles. Vehicle model 42 includes information such as the vehicle's shape, driving performance, and installed sensors.

[0029] Sensor model 44 is a model of a sensor installed on the vehicle being analyzed. Sensor model 44 is a model in which the type of information output is configured according to the surrounding information.

[0030] The mapping table 46 is a model that includes information on the correspondence between sensor data, which is data created in the simulation, and actual data obtained in advance by actually driving the vehicle. The mapping table 46 contains information on the conditions for performing the mapping. When the mapping table 46 associates actual data based on the scenario conditions and pre-set distribution conditions based on the acquired sensor data, the mapping table 46 includes a table of conditions for associating the actual data. When a trained model 50 is used for the mapping, processing conditions for inputting sensor data into the trained model 50 are associated with it.

[0031] Actual data 48 is data acquired in advance by actually driving the vehicle. Actual data 48 is data detected by sensors mounted on the vehicle. In addition, actual data 48 includes information about when the vehicle was actually driven, information about environmental objects, weather information, light source information, and vehicle information.

[0032] The pre-trained model 50 is a model created by the machine learning unit 30 using machine learning. When sensor data is input to the pre-trained model 50, it outputs the actual data corresponding to that sensor data.

[0033] The programs stored in the memory unit 18 include a data creation program 32 and a learning program 34.

[0034] The data creation program 32 is a program that creates evaluation data (evaluation data creation program). The data creation program 32 is a program that implements the functions of the scenario setting unit 22, the sensor data creation unit 24, the actual data conversion unit 26, and the correspondence processing unit 28.

[0035] The learning program 34 is a program that uses machine learning to create a trained model used in part of the processing of the data creation program 32. The learning program 34 is a program that implements the functions of the machine learning unit 30. The learning program 34 takes sensor data created by simulation based on a scenario as input and uses training data, which is real data corresponding to the sensor data, as output, to perform deep learning processing and create a trained model 50. As the deep learning model, a Generative Adversarial Network (GAN) or the like can be used. Note that the learning model and machine learning method are not particularly limited, as long as the sensor data can be converted into real data.

[0036] The memory unit 18 may install the data creation program 32 and the learning program 34 by reading them from the recording medium, or it may install them by reading them from the network.

[0037] The functions of each part of the calculation unit 16 will now be explained. Each part of the calculation unit 16 can be executed by executing a program stored in the memory unit 18. The scenario setting unit 22 obtains the conditions for the simulation to be performed and selects a model based on the conditions from the environment object model 37, the weather model 38, the light source model 40, and the vehicle model 42 to create a scenario model, which is the simulation model. The conditions for the simulation to be performed are set based on the user's input results. If a scenario model 36 corresponding to the conditions for the simulation to be performed is stored in the memory unit 18, the scenario setting unit 22 reads the stored scenario model 36.

[0038] The sensor data creation unit 24 uses a sensor model that models the vehicle's sensors to create sensor data for when the vehicle is driven according to the scenario set in the scenario setting unit 22. The sensor data creation unit 24 executes a simulation based on the scenario model created in the scenario setting unit 22 and creates sensor data, which is the data detected by the sensors when the vehicle is driven according to the scenario model.

[0039] The actual data conversion unit 26 performs mapping on the sensor data created by the sensor data creation unit 24 using the mapping processing unit 28, and then converts it into actual data using the result of the mapping.

[0040] The mapping processing unit 28 maps the sensor data input from the real data conversion unit 26 to real data. The mapping processing unit 28 uses the mapping table 46 to identify the real data corresponding to the sensor data. Also, if the pre-trained model 50 is used, the mapping processing unit 28 inputs the sensor data to the machine learning unit 30 and obtains the output real data.

[0041] The machine learning unit 30 takes sensor data created through simulation based on a scenario as input and uses training data, which consists of real data corresponding to that sensor data, as output, to perform deep learning processing and create a trained model 50. The machine learning unit 30 also performs processing using the trained model 50. Specifically, the machine learning unit 30 inputs the input sensor data into the trained model 50 and outputs the corresponding real data.

[0042] <Method> Next, we will explain the method for creating evaluation data using Figure 2. Figure 2 is a flowchart showing an example of the processing of the evaluation data creation device. The processing shown in Figure 2 is a correspondence process that does not use machine learning.

[0043] The calculation unit 16 determines the simulation scenario to be executed in the scenario setting unit 22 (step S12). The scenario setting unit 22 determines the scenario to be executed based on conditions entered by the user, pre-set conditions, etc.

[0044] The calculation unit 16 determines the environment object model, weather model, light source model, and vehicle model to be used based on the scenario in the scenario setting unit 22, and creates a scenario model (step S14).

[0045] The calculation unit 16 inputs the determined scenario model into the sensor model and creates sensor data (step S16). The calculation unit 16 then uses the sensor data creation unit 24 to create a sensor model based on the vehicle model 42. The sensor data creation unit 24 inputs the scenario model into the sensor model and executes a simulation to create sensor data detected by the sensor model, that is, data detected by the sensor when driving according to the scenario model.

[0046] The calculation unit 16 determines the actual data corresponding to the sensor data (step S18). The calculation unit 16 processes the sensor data using the mapping table 46 in the actual data conversion unit 26 and the mapping processing unit 28 to determine the actual data to be mapped to the sensor data. In this embodiment, based on the sensor data and the scenario model, the actual data that best matches the conditions is selected from the actual data mapped in the mapping table 46.

[0047] The calculation unit 16 performs a conversion process to actual data (step S20). The calculation unit 16 converts the sensor data to actual data in the actual data conversion unit 26 based on the correspondence determined by the correspondence processing unit 28.

[0048] The calculation unit 16 saves the created actual data (step S22). The actual data becomes the evaluation data.

[0049] As described above, this embodiment identifies real data for sensor data created in simulation and converts the sensor data into real data, thereby enabling evaluation data to be used as real data.

[0050] Here, the scenario model is created by, for example, forming the shape of an object with polygons, setting the object's properties (reflectance, transmittance, texture, etc.) as materials, and combining these to create a three-dimensional model. If the scenario model is created using a simple model, for example, only using the information initially set in the model creation software, then in this embodiment, the evaluation data supplied to the ECU6 can be real data by converting the sensor data into real data.

[0051] This allows for the creation of evaluation data that closely resembles actual driving conditions without having to create a sophisticated model to output values ​​close to real data from the sensor model. Furthermore, by using a simple model as the scenario model, it is possible to create evaluation data that enables highly accurate evaluation, thus reducing the time required to create evaluation data and simplifying the process. In addition, by correlating it with real data, it is possible to suppress discrepancies between the evaluation data and the actual sensor detections.

[0052] In the above embodiment, the correspondence processing unit 28 determines one real data 48 corresponding to the sensor data, and the determined real data is used as evaluation table data, but the device is not limited to this. The evaluation data creation device 10 may create a trained model in the machine learning unit and perform the conversion from sensor data to real data using the trained model.

[0053] <How to create a pre-trained model> Figure 3 is a flowchart illustrating an example of the operation of the machine learning unit. The processes shown in Figure 3 are executed by the machine learning unit. The machine learning unit 30 acquires real data (step S32). The machine learning unit 30 may acquire the real data stored in the memory unit 18, or it may acquire the real data via the communication unit 15.

[0054] The machine learning unit 30 creates sensor data (step S34). The machine learning unit 30 executes processing in the scenario setting unit 22 and the sensor data creation unit 24 to create sensor data corresponding to the actual data. For example, the machine learning unit 30 obtains the driving conditions of the actual data and creates sensor data using the created conditions.

[0055] The machine learning unit 30 performs the matching of sensor data with real data (step S36). In other words, the machine learning unit 30 matches the real data acquired in step S32 with the sensor data created in step S34.

[0056] The machine learning unit 30 determines whether the creation of training data is complete (step S38). For example, the machine learning unit 30 uses the criterion of whether there are more than a set number of combinations of sensor data and real data.

[0057] If the machine learning unit 30 determines that the creation of training data is not complete (No in step S38), it returns to step S32 and creates further combinations of sensor data and real data.

[0058] If the machine learning unit 30 determines that the creation of training data is complete (Yes in step S38), it takes sensor data as input and the corresponding real data as output to perform training on the learning model (step S40). For example, the machine learning unit 30 uses a portion of the combination of sensor data and real data that will become the training data as training data and the remainder as validation data, performs training with the training data, and validates with the validation data.

[0059] The machine learning unit 30, through the above processing, creates a trained model that outputs the corresponding real data when sensor data is input. Note that the method for creating the trained model is just one example and is not limited to this. In the above embodiment, supervised learning was described, but it is also possible to prepare a trained model and real data and identify the real data for the sensor data using unsupervised learning.

[0060] <Other examples of methods for creating evaluation data> Figure 4 is a flowchart showing an example of the processing performed by the evaluation data creation device. Among the processes shown in Figure 4, those that are the same as those shown in Figure 2 are given the same reference numerals, and detailed explanations are omitted.

[0061] The calculation unit 16 determines the simulation scenario to be executed in the scenario setting unit 22 (step S12). Based on the scenario, the calculation unit 16 determines the environmental object model, weather model, light source model, and vehicle model to be used in the scenario setting unit 22 and creates a scenario model (step S14). The calculation unit 16 inputs the determined scenario model into the sensor model and creates sensor data (step S16).

[0062] The calculation unit 16 inputs the sensor data into the trained model and converts it into real data (step S52).

[0063] The calculation unit 16 saves the created actual data (step S22). The actual data becomes the evaluation data.

[0064] As shown in Figure 4, the evaluation data creation device 10 can convert sensor data into real data using a trained model created with machine learning (deep learning), thereby enabling data conversion without the need to set various conditions.

[0065] Here, the evaluation data creation device 10 may use multiple pre-trained models. When using multiple pre-trained models, classification conditions are set, and training data is created by associating sensor data that satisfies each classification condition with real data. The machine learning unit 30 creates a pre-trained model for each classification condition by performing machine learning for each training data.

[0066] Figure 5 is a flowchart showing an example of the processing performed by the evaluation data creation device. Processes in Figure 5 that are the same as those in Figure 4 are denoted by the same reference numerals, and detailed explanations are omitted. Figure 5 shows a case where a trained model is created for each weather and light source condition.

[0067] The calculation unit 16 determines the simulation scenario to be executed in the scenario setting unit 22 (step S12). Based on the scenario, the calculation unit 16 determines the environmental object model, weather model, light source model, and vehicle model to be used in the scenario setting unit 22 and creates a scenario model (step S14). The calculation unit 16 inputs the determined scenario model into the sensor model and creates sensor data (step S16).

[0068] The calculation unit 16 uses the real data conversion unit 26 to determine the trained model to be used based on the weather model and the light source model (step S54). The real data conversion unit 26 acquires the weather and light source conditions of the sensor data based on the scenario model information and determines the trained model corresponding to the acquired conditions. The calculation unit 16 inputs the sensor data into the trained model and converts it into real data (step S56).

[0069] The calculation unit 16 saves the created actual data (step S22). The actual data becomes the evaluation data.

[0070] In this way, by setting pre-trained data for each condition, the accuracy of identifying real data can be increased. It is possible to associate real data that closely resembles the real data obtained when actually driving under the conditions of the scenario model.

[0071] As in this embodiment, by using weather and light source as classification conditions, it is possible to identify conditions that are difficult to determine using sensor data created from a scenario model. For example, in the case of images, it is possible to suppress the mixing of learning results regarding whether it is raining or snowing, whether it is cloudy and dark, dark in the evening, or dark indoors. This makes it possible to improve the accuracy of converting real data based on sensor data.

[0072] The evaluation data generation device may determine the pre-trained data to be used based on user input. Figure 6 is a flowchart showing an example of the processing of the evaluation data generation device. Among the processes shown in Figure 6, processes that are the same as those shown in Figure 5 are denoted by the same reference numerals and detailed explanations are omitted.

[0073] The calculation unit 16 determines the simulation scenario to be executed in the scenario setting unit 22 (step S12). Based on the scenario, the calculation unit 16 determines the environmental object model, weather model, light source model, and vehicle model to be used in the scenario setting unit 22 and creates a scenario model (step S14). The calculation unit 16 inputs the determined scenario model into the sensor model and creates sensor data (step S16).

[0074] The calculation unit 16 uses the real data conversion unit 26 to determine the trained model to use based on the input (step S62). The real data conversion unit 26 determines the corresponding trained model based on the weather and light source information input to the input unit 12. The calculation unit 16 inputs the sensor data into the trained model and converts it into real data (step S56).

[0075] The calculation unit 16 saves the created actual data (step S22). The actual data becomes the evaluation data.

[0076] The evaluation data creation device 10 can convert sensor data into actual weather and light source conditions required by the user. In this embodiment, the data is input by the user, but it may also be information input from another database.

[0077] The evaluation data creation device 10 may use the weather and light source settings of the scenario determined in step S12 as a single standard setting. This reduces the burden of model creation. Furthermore, since the process involves selecting a trained model based on arbitrary settings by the user, the actual weather and light source data used for evaluation can be converted to actual weather and light source data that the user requires.

[0078] The evaluation data generation device may classify the trained models according to the conditions of the environmental object model. Figure 7 is a flowchart showing an example of the processing of the evaluation data generation device. Among the processes shown in Figure 7, processes that are the same as those shown in Figure 5 are given the same reference numerals and detailed explanations are omitted.

[0079] The calculation unit 16 determines the simulation scenario to be executed in the scenario setting unit 22 (step S12). Based on the scenario, the calculation unit 16 determines the environmental object model, weather model, light source model, and vehicle model to be used in the scenario setting unit 22 and creates a scenario model (step S14). The calculation unit 16 inputs the determined scenario model into the sensor model and creates sensor data (step S16).

[0080] The calculation unit 16 uses the real data conversion unit 26 to determine the trained model to be used based on the environmental object model (step S64). The real data conversion unit 26 acquires weather and light source conditions for the sensor data based on the scenario model information and determines the trained model corresponding to the acquired conditions. The calculation unit 16 inputs the sensor data into the trained model and converts it into real data (step S56).

[0081] The calculation unit 16 saves the created actual data (step S22). The actual data becomes the evaluation data.

[0082] The evaluation data creation device 10 classifies environmental object models according to conditions and improves the accuracy of the trained model by training under similar conditions for people, buildings, trees, etc. Furthermore, it can suppress computational divergence during training and reduce the burden of model creation.

[0083] Furthermore, it is preferable that the evaluation data creation device 10 also inputs the environmental object model as input information during training. This allows the system to process and correlate conditions in the environmental physical model even when the detection state changes due to weather or light source (day / night). This can lead to higher detection accuracy.

[0084] <Other examples of evaluation data generation devices> Figure 8 is a block diagram showing an example of an evaluation data creation device. The evaluation data creation device 10a shown in Figure 8 includes, in addition to the configuration of the evaluation data creation device 10, a weather light source conversion unit 29 and a second trained model 52. The trained model of the evaluation data creation device 10 becomes the first trained model 50. The evaluation data creation device 10 performs data conversion processing twice.

[0085] The evaluation data creation device 10a has two pre-trained models: a first pre-trained model 50 and a second pre-trained model 52. The first pre-trained model 50 is a pre-trained model used in the process of converting sensor data into real data, as described above. The first pre-trained model 50 is created by training a sensor model with standard conditions of weather and light source, and real data.

[0086] The second pre-trained model 52 is a pre-trained model used to transform real data into real data with different weather and light source conditions. The method for creating the second pre-trained model 52 will be described later.

[0087] The weather-light source conversion unit 29 uses the second trained model 52 to create second real data by changing the weather and light source conditions of the real data (first real data) created by the real data conversion unit 26. The weather-light source conversion unit 29 changes the weather and light source conditions based on the conditions input in the input unit 12.

[0088] Figure 9 is a flowchart illustrating an example of the operation of the machine learning unit. The machine learning unit executes the operation using Figure 9. The machine learning unit 30 acquires real data under standard conditions for light source and weather (step S72). The machine learning unit 30 may acquire the real data stored in the memory unit 18 or acquire the real data via the communication unit 15.

[0089] The machine learning unit 30 acquires real data where the light source and weather conditions differ from standard conditions (step S74). The machine learning unit 30 may acquire the real data stored in the memory unit 18 or acquire the real data via the communication unit 15.

[0090] The machine learning unit 30 performs the matching of real data sets that are the same except for the light source and weather conditions (step S76). In other words, the machine learning unit 30 matches the real data acquired in step S72 with the sensor data created in step S74.

[0091] The machine learning unit 30 determines whether the creation of training data is complete (step S78). For example, the machine learning unit 30 uses the criterion of whether there are more than a set number of combinations of real data.

[0092] If the machine learning unit 30 determines that the creation of training data is not complete (No in step S78), it returns to step S72 and creates further combinations of sensor data and real data.

[0093] If the machine learning unit 30 determines that the creation of training data is complete (Yes in step S78), it takes real data under standard conditions as input and the corresponding real data as output to perform training on the second learning model (step S80).

[0094] The machine learning unit 30, through the above processing, creates a second trained model that converts real data under standard conditions into real data under different weather and light source conditions.

[0095] Figure 10 is a flowchart showing an example of the processing of the evaluation data creation device. The calculation unit 16 determines the simulation scenario to be executed in the scenario setting unit 22 (step S12). Based on the scenario, the calculation unit 16 determines the environmental object model and vehicle model to be used in the scenario setting unit 22 and creates a scenario model (step S90). The weather model and light source model are standard condition models. The calculation unit 16 inputs the determined scenario model into the sensor model and creates sensor data (step S16).

[0096] The calculation unit 16 determines the first trained model (step S92). The real data conversion unit 26 determines the trained model corresponding to the acquired conditions based on the scenario model information and input. The first trained model may be a single trained model. The calculation unit 16 inputs the sensor data into the first trained model and converts it into first real data (step S94).

[0097] The calculation unit 16 uses the weather-light source conversion unit 29 to determine the second pre-trained model to use based on the input (step S96). The weather-light source conversion unit 29 determines the corresponding second pre-trained model based on the weather and light source information input to the input unit 12. The calculation unit 16 inputs the first real data into the second pre-trained model and converts it into the second real data (step S98).

[0098] The calculation unit 16 saves the created second real data (step S99). The second real data becomes the evaluation data.

[0099] The evaluation data creation device 10a can use weather and light source data as reference conditions among the sensor data scenarios created by the sensor data creation unit 24 by performing conversion processing twice. This reduces the load of creating sensor data. It also reduces the load of creating the first trained model.

[0100] The evaluation data creation device 10a can convert the first real data converted from sensor data into second real data with desired weather and light source conditions by changing the weather and light source conditions in the weather and light source conversion unit 29.

[0101] Furthermore, the evaluation data creation device 10a may be a machine learning model that performs the process of selecting corresponding real data as the second trained model, but it is preferable to use a machine learning model that learns the process of converting the first real data to the second real data based on the training data. By using the second trained model 52 that can convert from a reference state to desired weather and light source, the evaluation data creation device 10a can convert and correct the effects of weather and light source with higher accuracy. As a result, various second real data with changed weather and light source can be created, and more evaluation data can be created.

[0102] Furthermore, in the above embodiment, the weather and light source were modified in the actual data, but the modifications are not limited to weather and light source. Based on various conditions set in the scenario, the actual data may be modified, that is, the actual data may be corrected. This makes the actual data closer to the scenario. In addition, the ability to perform correction processing makes it possible to create more evaluation data with different conditions.

[0103] This disclosure discloses the following inventions, but is not limited to those described below. (1) An evaluation data creation device for creating evaluation data to evaluate at least one of the autonomous driving and driving assistance functions of a vehicle, comprising: a scenario setting unit for setting a scenario to reproduce the driving of the vehicle; a sensor data creation unit for creating sensor data acquired when the vehicle drives according to the scenario set by the scenario setting unit, using a sensor model that models the sensors of the vehicle; and a real data conversion unit for creating converted real data which is evaluation data, by associating the sensor data created by the sensor data creation unit with real data which is data acquired by the sensors of the vehicle, and converting the sensor data based on the associated real data.

[0104] This allows for the creation of evaluation data that closely resembles actual driving conditions, without having to create a sophisticated model to output values ​​close to real-world data from the sensor model. Furthermore, because it is possible to create evaluation data that enables highly accurate evaluation while using a simple scenario model, the time required to create evaluation data can be reduced, simplifying the process.

[0105] (2) An evaluation data creation device according to (1), having a storage unit for storing the actual data, and a correspondence processing unit that compares the sensor data acquired by the actual data conversion unit with the actual data stored in the storage unit and determines the actual data to be associated with the sensor data. By associating the actual data, it is possible to suppress the deviation of the evaluation data from the actual sensor detection.

[0106] (3) An evaluation data creation device according to (1), which has a mapping processing unit that processes the sensor data using a trained model that has been trained using the sensor data as input and the corresponding real data as output, and determines the real data to be associated with the sensor data, using the trained model. By using a trained model, the load of the mapping processing can be reduced.

[0107] (4) The matching processing unit has multiple trained models classified by the conditions of the scenario, and determines which trained model to use based on the scenario set by the scenario setting unit. The evaluation data creation device described in (3). This makes it possible to increase the accuracy of the trained model. In addition, it is possible to suppress computational divergence during training and reduce the burden of creating the model.

[0108] (5) The matching processing unit determines the trained model to be used based on at least one of the weather and ambient light source of the scenario, as described in (4) for creating evaluation data. By using weather and light source as classification conditions, it is possible to identify conditions that are difficult to determine using sensor data created from the scenario model. For example, in the case of images, it is possible to suppress the mixing of training results regarding whether it is raining or snowing, whether it is cloudy and dark, whether it is dark in the evening or dark indoors, etc. This makes it possible to improve the accuracy of the conversion of real data based on sensor data.

[0109] (6) An evaluation data creation device according to (4) or (5), having an input unit for detecting input, wherein the correspondence processing unit determines the trained model to be used based on the input detected by the input unit. This allows the data to be converted into actual data corresponding to the conditions required by the user.

[0110] (7) The matching processing unit determines the trained model to be used based on the surrounding environment of the vehicle. This is an evaluation data creation device according to any one of (4) to (6). This makes it possible to improve the accuracy of the trained model. In addition, it is possible to suppress divergence in calculations during training and reduce the burden of creating the model.

[0111] (8) An evaluation data creation device according to any one of (1) to (7), further comprising a weather / light source conversion unit that performs a process to convert at least one of the weather and ambient light sources from the scenario to the converted actual data created by the actual data conversion unit, thereby creating second converted actual data. This reduces the burden of creating sensor data and allows for the creation of more evaluation data.

[0112] (9) The weather / light source conversion unit is an evaluation data creation device as described in (8), in which real data of weather and ambient light sources that differ in at least one of them are associated with the weather, takes real data of weather and ambient light sources of a scenario created by the scenario creation unit as input, and converts the converted real data to the second converted real data using a trained model that has been trained to take associated real data that differs in at least one of the weather and ambient light sources as output. This makes it possible to obtain real data that is closer to the scenario. In addition, correction processing can be performed, so that more evaluation data with different conditions can be created.

[0113] (10) A method for creating evaluation data to evaluate at least one of the autonomous driving and driving assistance functions of a vehicle, the method comprising: setting a scenario to reproduce the driving of the vehicle; creating sensor data to be acquired when the vehicle drives the scenario set by the scenario setting unit, using a sensor model that models the sensors of the vehicle; and associating the created sensor data with actual data which is data acquired by the sensors of the vehicle, and converting the sensor data based on the associated actual data to create converted actual data which is evaluation data.

[0114] This allows for the creation of evaluation data that closely resembles actual driving conditions, without having to create a sophisticated model to output values ​​close to real-world data from the sensor model. Furthermore, because it is possible to create evaluation data that enables highly accurate evaluation while using a simple scenario model, the time required to create evaluation data can be reduced, simplifying the process.

[0115] (11) An evaluation data creation program for creating evaluation data for evaluating at least one of the autonomous driving and driving assistance functions of a vehicle, the program comprising the steps of: setting a scenario that reproduces the driving of the vehicle; creating sensor data to be acquired when the vehicle drives the set scenario using a sensor model that models the sensors of the vehicle; and associating the created sensor data with actual data which is data acquired by the sensors of the vehicle, and converting the sensor data based on the associated actual data to create converted actual data which is evaluation data.

[0116] This allows for the creation of evaluation data that closely resembles actual driving conditions, without having to create a sophisticated model to output values ​​close to real-world data from the sensor model. Furthermore, because it is possible to create evaluation data that enables highly accurate evaluation while using a simple scenario model, the time required to create evaluation data can be reduced, simplifying the process. [Explanation of Symbols]

[0117] 6 ECU 8. Evaluation device 10. Evaluation data creation device 12 Input section 14 Output section 15 Communications Department 16 Arithmetic section 18 Memory section 22 Scenario Setting Department 24 Sensor Data Creation Unit 26 Actual Data Conversion Unit 28. Correspondence Processing Unit 30 Machine Learning Department 32 Data Creation Program 34 Learning Programs 36 Scenario Models 37. Environmental Object Models 38 Weather Models 40 Light Source Models 42 Vehicle Models 44 Sensor Models 46 Correspondence Table 48 Actual Data 50 pre-trained models

Claims

1. An evaluation data creation device that creates evaluation data for evaluating at least one of the vehicle's autonomous driving and vehicle's driver assistance functions, A scenario setting unit sets a scenario for reproducing the driving of the aforementioned vehicle, A sensor data creation unit creates sensor data to be acquired when the vehicle travels according to a scenario set in the scenario setting unit, using a sensor model that models the sensors of the vehicle. A mapping processing unit processes the sensor data and determines the corresponding real data using a trained model that has been trained using the sensor data as input and the corresponding real data as output, based on training data in which the sensor data and real data acquired by the vehicle's sensors are associated. A real data conversion unit associates the sensor data created by the sensor data creation unit with the real data, converts the sensor data based on the associated real data, and creates converted real data which is evaluation data. A device for creating evaluation data, including the data itself.

2. The evaluation data creation device according to claim 1, wherein the correspondence processing unit has a plurality of trained models classified by the conditions of the scenario, and determines the trained model to be used based on the scenario set by the scenario setting unit.

3. The evaluation data creation apparatus according to claim 2, wherein the matching processing unit determines the trained model to be used based on at least one of the weather of the scenario and the ambient light source.

4. It has an input unit that detects input, The matching processing unit determines the trained model to be used based on the input detected by the input unit, as described in claim 2.

5. The matching processing unit determines the trained model to be used based on the surrounding environment of the vehicle, as described in claim 2, for evaluation data creation device.

6. An evaluation data creation device according to any one of claims 1 to 5, further comprising a weather / light source conversion unit that performs a process to convert at least one of the weather and ambient light sources from the scenario to the converted actual data created by the actual data conversion unit, thereby creating second converted actual data.

7. The evaluation data creation device according to claim 6, wherein the weather / light source conversion unit has associated real data in which at least one of the weather and ambient light sources differs, and takes the real data of the weather and ambient light sources of a scenario created by the scenario setting unit as input, and a trained model which has learned associated real data in which at least one of the weather and ambient light sources differs as output, converts the converted real data to the second converted real data.

8. A method for creating evaluation data to evaluate at least one of the autonomous driving and driver assistance functions of a vehicle, the driver assistance function, The steps include setting up a scenario to reproduce the driving of the aforementioned vehicle, The steps include: creating sensor data to be acquired when the vehicle drives according to a set scenario using a sensor model that models the vehicle's sensors; The steps include: using training data in which the aforementioned sensor data and real data acquired by the vehicle's sensors are associated, processing the sensor data with a trained model that takes the sensor data as input and the corresponding real data as output, and determining the real data to associate with the sensor data; A method for creating evaluation data, comprising the steps of: associating the created sensor data with the actual data; and converting the sensor data based on the associated actual data to create converted actual data which is evaluation data.

9. A program for creating evaluation data to evaluate at least one of the vehicle's autonomous driving and vehicle driver assistance functions, the driver assistance function, The steps include setting up a scenario to reproduce the driving of the aforementioned vehicle, The steps include: creating sensor data to be acquired when the vehicle drives according to a set scenario using a sensor model that models the vehicle's sensors; The steps include: using training data in which the aforementioned sensor data and real data acquired by the vehicle's sensors are associated, processing the sensor data with a trained model that takes the sensor data as input and the corresponding real data as output, and determining the real data to associate with the sensor data; An evaluation data creation program that performs a process including the steps of associating the created sensor data with the actual data, and converting the sensor data based on the associated actual data to create converted actual data which is evaluation data.

Images