Method and system for evaluating autonomous driving systems

Abstract

The similarity between driving results obtained using a probabilistic model and driving results based on human driving behavior is presented as an intuitively understandable numerical value. [Solution] An autonomous driving system for a vehicle that outputs non-deterministic signals or parameters using a probabilistic model that has learned human driving behavior through machine learning, comprising: a first step of using multiple actual driving data 25 from when the vehicle is driven by a human, and multiple model driving data 26 from when the vehicle is driven based on the output results of the probabilistic model, and calculating the difference between two data selected from the multiple actual driving data 25; a second step of calculating the difference between one data selected from the multiple actual driving data 25 and one data selected from the multiple model driving data 26; and an evaluation step of comparing the difference obtained in the first step and the difference obtained in the second step to evaluate the similarity of the trajectories.

Description

Technical Field

[0001] The present disclosure relates to a method and system for evaluating an autonomous driving system of a vehicle that outputs non-deterministic signals or parameters by means of a probability model obtained by machine learning of human driving behavior, and more particularly to a method and system for evaluating the performance of the probability model.

Background Art

[0002] In recent years, vehicles such as automobiles equipped with a system for automatically or semi-automatically controlling the vehicle by a computer are known. For example, there are known automobiles equipped with an advanced driver assistance system (ADAS) that grasps information around the vehicle using various sensors and controls the vehicle based on this information to assist driving.

[0003] In a general advanced driver assistance system, rules for vehicle control are preset, and a control operation is determined by comparing these rules with the vehicle state (a model for generating such a control operation is hereinafter referred to as a "rule-based model"). However, it is known that human driving is probabilistic, and it is difficult to imitate human driving with preset rules. Therefore, the vehicle control operation in the rule-based model may be far from human driving. It is also conceivable that the comfort of the passengers may be lost or that a control beyond human prediction may be performed, causing a sense of danger, due to a control different from human driving.

[0004] To address these problems, research is being conducted on techniques that collect data from actual vehicle driving and generate models using machine learning (these models are referred to here as "probabilistic models"). Probabilistic models aim to achieve driving that mimics human driving behavior and output parameters used to control the vehicle in a non-deterministic manner (i.e., the same input will not produce the same output). Using such probabilistic models, it may be possible to achieve driving behavior that closely resembles human driving.

[0005] By the way, when vendors develop probabilistic models like the ones described above, it is necessary to evaluate the probabilistic model by determining how well it can mimic human driving.

[0006] In this regard, Patent Document 1 discloses a method for evaluating driving data of an autonomous vehicle. Specifically, Patent Document 1 proposes a computer-based method for evaluating the performance of a dynamic simulator used to simulate the behavior of an autonomous vehicle (ADV), the method comprising: receiving from the dynamic simulator multiple states of the ADV at multiple points in time and multiple control commands given to the ADV at multiple points in time; generating from the dynamic simulator multiple predicted positions of the simulated trajectory of the ADV based on the multiple states and the multiple control commands; receiving multiple actual positions of the ground truth trajectory of the ADV, which are generated by giving the control commands to the ADV at multiple points in time; and generating evaluation metrics that measure the similarity between the multiple predicted positions of the simulated trajectory and the multiple actual positions of the ground truth trajectory at multiple points in time. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2022-58566 [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] Conventional methods for evaluating driving trajectories, such as those described in Patent Document 1, define a single driving trajectory that represents the correct answer (ground truth), and compare the trajectory of this ground truth with the trajectory of the system's execution result. While such conventional methods are easy to calculate and understand, they have the following problems as a way to evaluate whether the probabilistic model realizes behavior similar to human driving.

[0009] Firstly, the human running data used as training data for the probabilistic model is variable, making it difficult to define it as a single ground truth. In other words, even if a single ground truth were to be established, there is a problem in that the validity of that ground truth itself cannot be guaranteed. For this reason, even if the running trajectory obtained from the probabilistic model is similar to a single ground truth, the reliability of the evaluation is uncertain.

[0010] Secondly, because probabilistic models generate control signals or parameters non-deterministically based on probability distributions, the trajectories obtained from probabilistic models inherently contain random variability. Even if an ideal probabilistic model could be created, the trajectories obtained from that model would rarely match the trajectories of ground truth. Therefore, simply comparing the two individually, as in conventional methods, has limitations in evaluation.

[0011] Thirdly, there is the problem of how to evaluate the degree of variability. That is, if there is a lot of variability in human driving data, the probabilistic model that mimics it should also have a lot of variability, and this can be said to be a good imitation of human driving behavior. However, conventional methods cannot evaluate the degree of this variability.

[0012] Given the current situation described above, this disclosure aims to enable the evaluation of the similarity between driving results obtained using a probabilistic model and driving results based on human driving behavior using an intuitively understandable numerical value. [Means for solving the problem]

[0013] One aspect of the present disclosure is an autonomous driving system for a vehicle that outputs non-deterministic signals or parameters using a probabilistic model that has learned human driving behavior through machine learning, and includes a first step of using a plurality of actual driving data from when the vehicle is driven by a human and a plurality of model driving data from when the vehicle is driven based on the output results of the probabilistic model, and calculating the difference between two data selected from the plurality of actual driving data; a second step of calculating the difference between one data selected from the plurality of actual driving data and one data selected from the plurality of model driving data; and an evaluation step of comparing the difference obtained in the first step and the difference obtained in the second step to evaluate the similarity of the trajectories. [Effects of the Invention]

[0014] According to this disclosure, the similarity between driving results obtained using a probabilistic model and driving results based on human driving behavior can be presented as an intuitively understandable numerical value. In other words, non-deterministic outputs can be quantitatively evaluated. [Brief explanation of the drawing]

[0015] [Figure 1] This is a block diagram showing the outline configuration of a vehicle equipped with an autonomous driving system. [Figure 2] This is a block diagram showing the outline configuration of the evaluation system for autonomous driving systems. [Figure 3] This diagram illustrates the steps performed by the evaluation system. [Figure 4] This is a flowchart illustrating an example of the process performed by the evaluation system. [Figure 5] (a) A table showing actual driving data numerically, and (b) An image diagram plotting the actual driving data. [Figure 6] (a) A diagram illustrating an example of the first step of calculating the difference between two data points selected from multiple actual driving data; (b) A conceptual diagram of the difference obtained by performing the first step multiple times; (c) A conceptual diagram of the mean and standard deviation of the difference. [Figure 7] (a) A diagram illustrating an example of the second step, which involves calculating the difference between one data point selected from multiple real-world driving data points and one data point selected from multiple model driving data points; (b) A conceptual diagram of the difference obtained by performing the second step multiple times; (c) A conceptual diagram of the mean and standard deviation of the difference. [Figure 8] This table explains the numerical values ​​obtained using the evaluation method. [Modes for carrying out the invention]

[0016] Embodiments of the present invention will be described with reference to the figures. The evaluation system 200 according to this embodiment is for evaluating an autonomous driving system for a vehicle using a probabilistic model. The probabilistic model is constructed by machine learning human driving behavior and outputs non-deterministic signals or parameters (the same input does not produce the same output). The autonomous driving system is configured to be able to drive autonomously using the output of this probabilistic model. As for the vehicle, for example, a car vehicle 10 as shown in Figure 1 can be considered. However, the vehicle is not limited to a car, but may be other vehicles (for example, a ship or an airplane).

[0017] The vehicle 10 shown in FIG. 1 is equipped with an autonomous driving system. The autonomous driving system grasps the information around the vehicle 10 with various sensors and the like, and controls the vehicle 10 based on this information to assist in driving. It is a so-called advanced driver-assistance system (ADAS). This autonomous driving system includes a driver model that defines parameters for generating control operations of the vehicle 10. The driver model is software that acquires information necessary for operating the vehicle 10 (such as information around the vehicle 10) and generates control operations of the vehicle 10 such as steering and acceleration based on the acquired information. The control operations of the vehicle 10 generated by the driver model are, for example, a list of output signals or parameters, and control signals are issued from the ECU 12 of the control device 11 to various devices (such as the steering 31, accelerator 32, and brake 33) according to this list. Thereby, the angle of the steering 31, the pressure of the accelerator pedal 32, the pressure of the brake pedal 33, etc. are controlled by the ECU 12.

[0018] The autonomous driving system according to this embodiment employs a hybrid driver model as shown in FIG. 1. The hybrid driver model is a computing model that combines a plurality of driver models, and these plurality of driver models are automatically switched and used during driving. Specifically, the autonomous driving system selects an optimal driver model according to the situation of the vehicle, and the control operations of the vehicle 10 are generated by the selected driver model.

[0019] The hybrid driver model according to this embodiment includes at least two types: a rule-based model and a probabilistic model. Note that the number of driver models included in the hybrid driver model does not necessarily have to be two, and it may include three or more driver models.

[0020] First, a rule-based model is a driver model that generates control actions based on predetermined conditions without using probability functions. This rule-based model is commonly used in advanced autonomous driving systems. In a rule-based model, the rules used to control the vehicle 10 are predetermined by the designer (e.g., the automobile manufacturer), and the ECU 12 compares these rules with the state of the vehicle 10 to determine the control action. For example, by pre-setting a speed limit as a rule, it is possible to generate a control action that decelerates the vehicle when its speed reaches the speed limit. Also, by pre-setting a distance between vehicles as a rule, it is possible to generate control that decelerates the vehicle when the distance between vehicles falls below a predetermined value. In this way, a deterministic approach is taken in rule-based models, and the operation of the vehicle 10 is determined based on predetermined explicit rules.

[0021] The advantage of this rule-based model is that it can determine control actions based on predetermined conditions, thus enabling definitive responses to events that can be predicted in advance. Because it is predetermined how vehicle 10 will be controlled under different circumstances, it is possible to generate definitive actions that guarantee safe operation, thereby enhancing the safety of vehicle 10.

[0022] On the other hand, this rule-based model has the disadvantage that its control actions are uniform and not human-like. Human driving is known to be probabilistic, and it is difficult to imitate human driving with pre-set rules. Therefore, the control actions of vehicle 10 in a rule-based driver model may be far removed from human driving. If the control is different from human driving, it may result in a loss of comfort or a feeling of danger due to control that exceeds human prediction.

[0023] On the other hand, the probabilistic model is a probability-based driver model that relies on information (training data) collected from the driver operating the vehicle 10. The probabilistic model is constructed using machine learning techniques that utilize driving data (multiple actual driving data 25 described later) from when a human driver operates the vehicle 10 as training data. For example, information from when the driver actually drove the vehicle 10 (such as the timing of when the accelerator 32 and brake 33 were actually operated) is collected, and the probabilistic model is constructed or updated based on this information. This ensures that the output of the probabilistic model closely resembles the driver's actual driving actions. The information collected from the driver is formalized as mathematical formulas or numerical values, and these formulas or numerical values ​​are applied to the probabilistic model.

[0024] These formulas and numerical values ​​may include probability functions and random variables. For example, a probability function may be created (updated) based on information collected from the driver, and this probability function may be used to generate the control actions of vehicle 10. Alternatively, random variables representing the driver's personality may be prepared in advance within the probability model, and the driver's steering profile may be defined by selecting and discarding random variables based on information collected from the driver. Note that the design of such probability functions and random variables is merely an example. The probability model only needs to determine control actions probabilistically based on information collected from the driver, and the probability functions and random variables can be freely designed. Thus, a probability model includes a probability function based on the training data, and this probability function is used to generate control actions.

[0025] The advantage of this probabilistic model is that it learns from information collected from the driver to determine the control settings, thus reproducing actions that closely resemble the driver's actual operation. This improves comfort. Another advantage is that because the control selected is similar to the driver's own operation, the driver can more easily predict the actions of the automatic control system.

[0026] However, this probabilistic model also has its drawbacks. First, because driving conditions are highly heterogeneous and involve a vast number of variables, it is extremely difficult to train a model that covers all driving conditions. Second, because the control mechanisms are uncertain, it is difficult to guarantee safe behavior.

[0027] Thus, rule-based models and probabilistic models each have their own advantages and disadvantages, and they complement each other. For this reason, the autonomous driving system according to this embodiment employs a hybrid driver model that combines this rule-based model and probabilistic model, and is configured to take advantage of the benefits of both by automatically switching between these driver models during driving.

[0028] As shown in Figure 1, this autonomous driving system is centered around a control device 11 mounted on the vehicle 10. This control device 11 is centered around an ECU 12 and includes an on-board storage device 20. The ECU 12 is configured to control various steering devices 30 and environmental acquisition devices 40 by reading programs and data stored in the on-board storage device 20.

[0029] The in-vehicle storage device 20 can be of any type accessible from the ECU 12, and is not limited to, for example, a hard disk drive, solid state drive, memory card, ROM, RAM, or computer-readable media. A combination of these is also acceptable.

[0030] This in-vehicle storage device 20 stores rule-based model data 21, probability model data 22, state determination program 23, steering control program 24, actual driving data 25, model driving data 26, and more. In addition to the above, this in-vehicle storage device 20 also stores various other data and programs.

[0031] The rule-based model data 21 is data for realizing the rule-based model. This rule-based model data 21 includes mathematical formulas and numerical values ​​for generating control actions according to the state of the vehicle 10. This rule-based model data 21 also includes data related to safety boundaries, which are used by the state determination program 23 (described later) when determining a safe state.

[0032] The probability model data 22 is data for realizing the probability model. The probability model data 22 according to this embodiment includes a probability function that is formulated as a mathematical formula based on information collected from the driver operating the vehicle 10.

[0033] The state determination program 23 is a program for determining whether or not the vehicle 10 is in a safe state. This state determination program 23 uses data related to safety boundaries included in the rule-based model data 21 to determine the state of the vehicle 10. This safety state determination is performed at a short interval that is sufficient to guarantee the safety of the vehicle 10, and in this embodiment, it is performed every 50 milliseconds. By performing the safety state determination at a fixed interval in this way, the state of the vehicle 10 can be constantly monitored.

[0034] Furthermore, the data related to the safety boundary has several conditions pre-set for when vehicle 10 is in an unsafe state. For example, these conditions include "the rate of change of speed is above a predetermined value (vehicle 10 may have suddenly broken down)," "the time until collision with the preceding vehicle is less than a predetermined value," "the distance to the preceding vehicle is less than a predetermined value," and "the vehicle is out of its lane." The state determination program 23 checks whether the current state of vehicle 10 meets any of these conditions and determines whether vehicle 10 is in a safe state.

[0035] It should be noted that just because the state determination program 23 determines that the vehicle 10 is "unsafe" does not necessarily mean that the vehicle 10 is in a dangerous state. In other words, the determination made by the state determination program 23 is solely for the purpose of ensuring safety, and there may be cases where the safety boundary is strictly set to prevent accidents from occurring. For example, if a rare event occurs where the behavior is unstable due to insufficient training data for the probability model, the program may determine that the vehicle is "unsafe."

[0036] The determination result from this state determination program 23 is used to select a driver model. In this embodiment, if the state determination program 23 determines that the vehicle is "unsafe" while driving using the probabilistic model, a process is executed to switch the driver model to a rule-based model. Also, if the state determination program 23 determines that the vehicle 10 is in a safe state for a predetermined period (for example, 3 seconds) while driving using the rule-based model, a process is executed to switch the driver model to a probabilistic model.

[0037] The steering control program 24 is a program that controls the vehicle 10 using the driver model described above. By continuously using sensor data, the steering control program 24 can identify traffic signs, people, other vehicles, obstacles, driving lanes, etc., and autonomously control the vehicle 10. In addition, it can use location data such as map data and GPS to assist in driving to the destination.

[0038] The steering control program 24 acquires data on the state of the vehicle 10 from various devices described later, generates control operations for the steering device 30 based on the acquired data, and outputs control signals for the steering device 30 based on these control operations. The steering control program 24 also selects the driver model to be used according to the determination result of the state determination program 23. When the steering control program 24 generates control operations, either the rule-based model data 21 or the probability model data 22 is used according to the selected driver model. Therefore, the generated control operations may differ depending on which driver model is selected.

[0039] The actual driving data 25 is data that records the driving trajectory when vehicle 10 is driven by a human. In other words, the actual driving data 25 is data that records the driving trajectory when vehicle 10 is driven without using the autonomous driving system. This actual driving data 25 is used as training data for the probabilistic model. This actual driving data 25 is also used for evaluation by the evaluation system 200, which will be described later. This actual driving data 25 will be explained in more detail later.

[0040] The model driving data 26 is data that records the driving trajectory of vehicle 10 when it travels based on the output results of the probabilistic model. This model driving data 26 is used for evaluation by the evaluation system 200 described later.

[0041] The control device 11 described above receives input signals from, for example, the steering device 30 and the environment acquisition device 40 as shown in Figure 1, and thereby understands the current state of the vehicle 10. Then, using the current driver model, it generates control signals or parameters appropriate to the current state of the vehicle 10. The control device 11 outputs control signals to the steering device 30 according to these control signals or parameters to support the operation of the vehicle 10.

[0042] The control system 30 is a group of devices for controlling the vehicle 10. As shown in Figure 3, the control system 30 includes a steering wheel 31 for controlling the direction of travel of the vehicle 10, an accelerator 32 for adjusting the speed of the vehicle 10, a brake 33 for decelerating or stopping the vehicle 10, and a transmission 34 for changing rotational speed and torque. The output of these control devices 30 can be controlled by control signals from the control device 11, as well as by operation by the driver. For example, the angle of the steering wheel 31 can be controlled by the ECU 12, as well as by the driver operating the steering wheel. Similarly, the accelerator 32 and brake 33 can be controlled by the ECU 12, as well as by the driver operating the pedals, etc. The control system 30 is not limited to these, and other control devices 30 may also be included.

[0043] Furthermore, the environmental acquisition device 40 is a group of devices for acquiring the status of the vehicle 10, and includes various sensors and cameras. This environmental acquisition device 40 allows the control device 11 to understand the vehicle 10's driving status and the external environment. Examples of environmental acquisition devices 40 include a speed detection device 41, a position detection device 42, an object detection device 43, a lane detection device 44, a passenger detection device 45, and an image detection device 46. However, the environmental acquisition device 40 is not limited to these, and other environmental acquisition devices 40 may also be included.

[0044] The speed detection device 41 is a device that detects the speed of the vehicle 10 (e.g., a speedometer, accelerometer, gyroscope, etc.). The position detection device 42 is a device that detects the position of the vehicle 10 (e.g., a GPS receiver, etc.). The object detection device 43 is a device that detects objects around the vehicle 10 (e.g., a laser, radar, camera, sonar, etc.). The lane detection device 44 is a device that detects the driving lane (e.g., a laser that detects the amount of reflected light from the lane lines, etc.). The passenger detection device 45 is a device that detects the driver and passengers inside the passenger car (e.g., a seat weight counter, etc.). The image detection device 46 is a device that detects images of the area around the vehicle 10 (e.g., a camera used to identify signs, etc.). The data detected by these devices may be used as raw data, or calculated values ​​from the respective detection results may be used (e.g., the rate of change in speed, the difference in speed from the preceding vehicle, the time until collision with the preceding vehicle, etc. may be calculated and used).

[0045] Incidentally, the autonomous driving system described above aims to achieve operation similar to human driving by outputting non-deterministic signals or parameters using a probabilistic model that has been machine-learned to replicate human driving behavior. However, since the human driving behavior used as training data is diverse and the output of the probabilistic model is non-deterministic, it is difficult to quantitatively evaluate whether operation similar to human driving is being reproduced. In this regard, by using the evaluation system 200 and evaluation method according to this embodiment, the similarity between the driving results when using the probabilistic model and the driving results based on human driving behavior can be evaluated as an intuitively understandable numerical value. As a result, developers and vendors of autonomous driving systems can use this evaluation method during the development and verification stages to evaluate the performance of the probabilistic model.

[0046] As shown in Figure 2, the evaluation system 200 according to this embodiment comprises a computer 210 and a storage device 220. The evaluation system 200 according to this embodiment runs on a server independent of the vehicle 10. After the vehicle 10 has completed its run, the evaluation system 200 can acquire the driving data (actual driving data 25 and model driving data 26) recorded in the vehicle 10 and perform an offline evaluation.

[0047] The configuration of the evaluation system 200 is not limited to those described above, and all or part of the evaluation system 200 may be mounted on the vehicle 10 (for example, the ECU 12 may be used as the computer 210 of the evaluation system 200, and the on-board storage device 20 may be used as the storage device 220 of the evaluation system 200). In this case, the evaluation system 200 may be operated on the vehicle 10. Furthermore, even if the evaluation system 200 is composed of hardware independent of the vehicle 10, the evaluation system 200 and the vehicle 10 may be connected wirelessly to perform the evaluation online. In addition, the computer 210 and storage device 220 of the evaluation system 200 do not necessarily have to be in the same location, and each may be composed of multiple hardware components (for example, they may be configured using distributed computing or cloud storage).

[0048] The storage device 220 in this embodiment stores actual driving data 230, model driving data 240, evaluation programs 250, and the like. In addition, various other data and programs are stored in this storage device 220.

[0049] The actual driving data set 230 includes multiple actual driving data sets 25 collected as a result of the vehicle 10's driving. The actual driving data sets 25 are data obtained when a predetermined route is driven by a human based on a predetermined driving scenario (driving route and driving environment), and are represented by numerical values ​​such as those shown in Figure 5(a). In Figure 5(a), the state of the vehicle 10 is represented by its position in 3D space (x, y, z) and the orientation of the vehicle 10 represented by four numerical values ​​(qx, qy, qz, qw). This state of the vehicle 10 is recorded at regular intervals (for example, every 0.10 seconds) during driving. When the vehicle 10 travels along the trajectory shown in Figure 5(b), the ECU 12 automatically performs the following processes: recording the state of the vehicle 10 at the timing of "10.00", recording the state of the vehicle 10 at the timing of "10.10", recording the state of the vehicle 10 at the timing of "10.20", and so on. The actual driving data 25 collected in this manner is initially recorded in the vehicle's onboard storage device 20 and then transferred to the evaluation system's storage device 220 at any time. This actual driving data 25 is acquired multiple times based on the same driving scenario (essentially the same driving route and driving environment). Multiple such accumulated actual driving data sets 25 constitute the actual driving data group 230. In order to appropriately evaluate the variability of human driving, it is desirable to collect as many actual driving data sets 25 as possible.

[0050] Furthermore, the actual driving data 25 is not limited to data obtained when a person actually drives the vehicle 10; data obtained when a person drives using a driving simulator may also be used. In addition, as a method for obtaining multiple actual driving data 25, multiple experiments may be conducted using the same driving scenario, or the driving trajectory from a single experiment may be divided into multiple actual driving data 25 at regular intervals and used.

[0051] The model driving data set 240 includes multiple model driving data sets 26. The model driving data set 26 is data obtained when driving based on signals or parameters output by the probabilistic model in the predetermined driving scenario described above (a driving scenario that is substantially the same as when the actual driving data set 25 was obtained), and is represented by numerical values ​​as shown in Figure 5(a), similar to the actual driving data set 25. This model driving data set 26 can be obtained, for example, by actually driving the vehicle 10 using the probabilistic model. This model driving data set 26 is obtained multiple times in the same driving scenario (substantially the same driving route and driving environment). Multiple model driving data sets 26 accumulated in this way constitute the model driving data set 240. Since the output of the probabilistic model is non-deterministic, just like that of a human, it is desirable to collect as many model driving data sets 26 as possible in order to appropriately evaluate the variability of driving by the probabilistic model.

[0052] Furthermore, the model driving data 26 is not limited to data obtained when the vehicle 10 is actually driven; it may also be obtained through simulations using a probabilistic model. In addition, multiple model driving data 26 can be obtained by conducting multiple experiments with the same driving scenario, or by dividing the driving trajectory from a single experiment into multiple model driving data 26 at regular intervals.

[0053] The evaluation program 250 is a program for evaluating a probabilistic model using the actual driving data set 230 and the model driving data set 240. This evaluation program 250 performs roughly three steps shown in Figure 3 to evaluate whether the probabilistic model can reproduce actions that are close to human driving. The first step is to calculate the difference (a numerical value that serves as the baseline for evaluation) between two actual driving data 25 selected from the actual driving data set 230. The second step is to calculate the difference (the difference between human and probabilistic model = the numerical value to be evaluated) between one actual driving data 25 selected from the actual driving data set 230 and one model driving data 26 selected from the model driving data set 240. The third step is an evaluation step that compares the difference obtained in the first step with the difference obtained in the second step to evaluate the similarity of the trajectories. Note that the execution order of the first and second steps does not matter. Either can be executed first, or both can be executed simultaneously or in parallel. Furthermore, the number of times the first and second steps are executed can be set arbitrarily. For example, the first and second steps may be executed multiple times, and the results of those multiple executions may be compared in the third step. The evaluation method using evaluation program 250 will be explained below, referring to the flow chart shown in Figure 4.

[0054] First, in step S100 shown in Figure 4, the evaluation system 200 acquires multiple real-world driving data 25 from a human. For example, it receives the real-world driving data 25 stored in the vehicle's onboard storage device 20 (or simulator) via some means of communication. As a result, the group of real-world driving data 230 is stored in the storage device 220 and becomes available for use by the computer 210. Then, the process proceeds to step S105.

[0055] In step S105, the evaluation system 200 acquires multiple model driving data sets 26 using a probabilistic model. For example, it receives model driving data 26 stored in the vehicle's onboard storage device 20 (or simulator) via some means of communication. As a result, the model driving data set 240 is stored in the storage device 220 and becomes available for use by the computer 210. Then, the process proceeds to step S110.

[0056] In step S110, the computer 210 randomly selects two actual driving data 25 from the actual driving data group 230 and calculates the difference between these two actual driving data 25 (the first step shown in Figure 3). In this embodiment, the difference refers to a numerical value that represents the difference between two driving trajectories (driving data) with a single numerical value, and can be considered as a representative value of the difference between the two driving trajectories, or as the difference over the entire two driving trajectories. Specifically, as shown in Figure 6(a), two actual driving data 25 are randomly selected, and the root mean square error (RMSE) of these two actual driving data 25 is calculated. Note that the method of calculating the difference is not necessarily limited to RMSE. For example, the arithmetic mean (the value obtained by dividing the sum of each numerical value by the number of numerical values) may be used, or the median may be used. However, using RMSE can increase the influence of data variability. Then, the process proceeds to step S115.

[0057] In step S115, it is checked whether step S110 has been performed K times (where K is a natural number greater than or equal to 2). If it has been performed less than K times, the process returns to step S110 and the difference between the two newly selected data points is calculated. On the other hand, if K times have been reached, the process proceeds to step S120. If K times have been reached, K RMSEs have been obtained, as shown in Figure 6(b).

[0058] In step S120, the computer 210 randomly selects one actual driving data 25 from the actual driving data group 230 and one model driving data 26 from the model driving data group 240. Then, it calculates the difference between these two data (the second step shown in Figure 3). The same method as in step S110 is used to calculate the difference. In this embodiment, the root mean square error (RMSE) of the two selected data is calculated. Then, the process proceeds to step S125.

[0059] In step S125, it is checked whether step S120 has been performed K times (where K is a natural number greater than or equal to 2). If it has been performed less than K times, the process returns to step S120 and the difference between the two newly selected data points is calculated. On the other hand, if K times have been reached, the process proceeds to step S130. If K times have been reached, K RMSEs have been obtained, as shown in Figure 7(b).

[0060] In step S130, the computer 210 compares the difference shown in Figure 6(b) with the difference shown in Figure 7(b) to evaluate the similarity of the driving trajectories (evaluation step shown in Figure 3). For example, the similarity of the driving trajectories is evaluated using the criteria shown in Figure 8.

[0061] One example of an evaluation criterion is the use of average values. Specifically, computer 210 evaluates the similarity of the trajectories by comparing the average value of the differences obtained by performing the first step multiple times (see Figure 6(c)) with the average value of the differences obtained by performing the second step multiple times (see Figure 7(c)). For example, by calculating the difference between the two average values ​​using the formula shown in Figure 8, the average difference between the running trajectory of a human and the running trajectory of a probabilistic model can be determined. The smaller this value, the closer the probabilistic model is to human behavior.

[0062] The second example of an evaluation criterion uses standard deviation. That is, computer 210 evaluates the similarity of the trajectories by comparing the standard deviation of the difference obtained by performing the first step multiple times (see Figure 6(c)) with the standard deviation of the difference obtained by performing the second step multiple times (see Figure 7(c)). For example, by calculating the rate of change of the two standard deviations using the formula shown in Figure 8, the variability of the trajectory of the probabilistic model relative to the variability of human running trajectories can be determined. The closer this value is to 1, the closer the probabilistic model is considered to be to human behavior.

[0063] The third example of an evaluation criterion involves determining the proportion (number) of data points within a predetermined range from the mean of the differences obtained by performing the first step multiple times, based on the difference data obtained by performing the second step multiple times (see Figure 7(b)). For example, using the calculation formula shown in Figure 8, it is possible to define a range of twice the standard deviation shown in Figure 6, based on the mean of the differences shown in Figure 6, and calculate the proportion of the RMSE values ​​shown in Figure 7 that fall within this range. This proportion (number) indicates the proportion (number) of the trajectories of the probabilistic model that are close to the average human trajectory, and the larger this proportion (number), the closer the probabilistic model is to human behavior.

[0064] Once the calculation of the multiple evaluation values ​​described above is complete, the processing of the evaluation method according to this embodiment is finished. The person performing the evaluation can quantitatively evaluate the probability model based on the evaluation values ​​obtained in step S130.

[0065] For example, suppose a vendor developing a lane-keeping system for highways wants to evaluate the performance of a probabilistic model they have developed. The probabilistic model is expected to behave similarly to the real-world driving data 25 used for training, and the developer wants to express the similarity of the behavior using quantitative indicators to make a well-founded judgment on whether the probabilistic model is good or bad. Here, suppose the vendor has developed a probabilistic model a that models the driving behavior of human A, and similarly, a probabilistic model b that models the driving behavior of human B. In this case, if probabilistic model a and probabilistic model b can be evaluated using the same evaluation criteria, a clear and objective evaluation can be made.

[0066] In this respect, this evaluation method makes it possible to set uniform evaluation criteria, such as whether or not the criteria set by the developer for passing are met. For example, if the developer sets the criteria for passing as "the difference in mean values ​​calculated using the formula shown in Figure 8 is within 0.1m" and "the rate of change in the standard deviation calculated using the formula shown in Figure 8 is 10% or less", then the performance of the probabilistic model (evaluation of whether it can reproduce actions close to human driving) can be performed by whether or not such objective criteria are met. For example, objective judgments such as "probabilistic model a passes because it meets the above criteria" and "probabilistic model b fails because it does not meet the above criteria" become possible.

[0067] Furthermore, since the evaluation method of this embodiment evaluates the model driving data 26 using the variability of actual human driving data 25 as a baseline, it is possible to perform evaluations that take into account the variability of human driving data. In addition, since the similarity between the driving results when using the probabilistic model and the driving results based on human driving behavior can be expressed numerically, the evaluation results can be presented in an intuitive and easy-to-understand manner. Such evaluations can then be used to improve the probabilistic model, contributing to the construction of an autonomous driving system that is closer to human driving behavior.

[0068] In the flowchart shown in Figure 4 above, the first and second steps are performed an equal number of times (K times). While having the same number of steps allows for more reliable statistical comparisons, evaluation is still possible even if the numbers are different, so the number of steps can be different.

[0069] Furthermore, the reasons for randomly sampling data in the first and second steps are (1) to ensure statistically reliable evaluation even when the number of actual driving data 25 and the number of model driving data 26 are not equal, and (2) to limit processing time by specifying the number of processing iterations. If these problems do not occur, the actual driving data 25 and model driving data 26 used in the first and second steps may not be sampled, and all collected data may be used for evaluation. [Explanation of Symbols]

[0070] 10 vehicles 11 Control device 12 ECU 20 In-vehicle storage device 21 Rule-based model data 22 Probability Model Data 23 State Determination Program 24. Flight Control Program 25 Actual Driving Data 26 Model Driving Data 30 Control devices 31 Steering 32 Accelerator 33 Brake 34 Transmission 40 Environment acquisition device 41 Speed ​​detection device 42 Position detection device 43 Object detection device 44-lane detection device 45 Passenger detection device 46. ​​Image detection device 200 Evaluation System 210 Computers 220 Storage device 230 Actual Driving Data Set 240 Model Driving Data Set 250 Evaluation Programs

Claims

1. In an autonomous driving system for a vehicle that outputs non-deterministic signals or parameters using a probabilistic model that has learned human driving behavior through machine learning, Multiple sets of actual driving data from human drivers and model driving data from driving based on the output results of the aforementioned probability model are used. The first step is for the computer to calculate the difference between two data points selected from multiple actual driving data, The second step involves the computer calculating the difference between one data point selected from multiple real-world driving data points and one data point selected from multiple model driving data points. An evaluation step in which a computer compares the difference obtained in the first step with the difference obtained in the second step to evaluate the similarity of the trajectories, Having, Evaluation methods for autonomous driving systems.

2. The aforementioned probability model was constructed using machine learning with the aforementioned multiple real-world driving data. A method for evaluating an autonomous driving system according to claim 1.

3. In the first and second steps described above, the RMSE (Root Mean Square Error) of the two selected data is calculated. A method for evaluating an autonomous driving system according to claim 1.

4. In the evaluation step, the degree of similarity of the trajectories is evaluated by comparing the average value of the differences obtained by performing the first step multiple times with the average value of the differences obtained by performing the second step multiple times. A method for evaluating an autonomous driving system according to claim 1.

5. In the evaluation step, the degree of similarity of the trajectories is evaluated by comparing the standard deviation of the difference obtained by performing the first step multiple times with the standard deviation of the difference obtained by performing the second step multiple times. A method for evaluating an autonomous driving system according to claim 1.

6. In an autonomous driving system for a vehicle that outputs non-deterministic signals or parameters using a probabilistic model that has learned human driving behavior through machine learning, Multiple sets of actual driving data from human drivers and model driving data from driving based on the output results of the aforementioned probability model are used. By running the program, The first step is to calculate the difference between two data points selected from multiple actual driving data, The second step involves calculating the difference between one data point selected from multiple real-world driving data points and one data point selected from multiple model driving data points. An evaluation step to evaluate the similarity of the trajectories by comparing the difference obtained in the first step with the difference obtained in the second step, Execute An evaluation system for autonomous driving systems.

7. The aforementioned probability model was constructed using machine learning with the aforementioned multiple real-world driving data. An evaluation system for an autonomous driving system according to claim 6.

8. In the first and second steps described above, the RMSE (Root Mean Square Error) of the two selected data is calculated. An evaluation system for an autonomous driving system according to claim 6.

9. In the evaluation step, the degree of similarity of the trajectories is evaluated by comparing the average value of the differences obtained by performing the first step multiple times with the average value of the differences obtained by performing the second step multiple times. An evaluation system for an autonomous driving system according to claim 6.

10. In the evaluation step, the degree of similarity of the trajectories is evaluated by comparing the standard deviation of the difference obtained by performing the first step multiple times with the standard deviation of the difference obtained by performing the second step multiple times. An evaluation system for an autonomous driving system according to claim 6.

Images