Vehicle travelling way evaluation device

Abstract

To provide a vehicle travelling state estimation device that can properly evaluate vehicle travelling states in various situations.SOLUTION: A vehicle travelling state evaluation device, which evaluates a travelling state of an object vehicle by a simulator in enhanced learning about travelling of the object vehicle using the simulator, is configured to have: a reference selection unit (S3) that has an evaluation reference about the travelling state of the object vehicle under a specific situation with respect to each of a plurality of situations representable by the simulator set, and selects an evaluation reference corresponding to a situation represented by the simulator in a process of the enhanced learning from a plurality of evaluation references set to a plurality of situations; and an evaluation determination unit (S4-1 to S4-n) that determines an evaluation point representing the evaluation of the travelling state of the object vehicle under the situation represented by the simulator in the process of the enhanced learning on the basis of the evaluation reference selected by the reference selection unit.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a vehicle driving behavior evaluation device that evaluates the driving behavior of a target vehicle in a simulator in reinforcement learning related to the driving of the target vehicle using the simulator. [Background technology]

[0002] Conventionally, a learning device that performs reinforcement learning on the behavior (driving mode) of a vehicle (target vehicle) is known (see, for example, Patent Document 1). Generally, reinforcement learning involves learning "behavior that maximizes value" in a certain environment through trial and error. Specifically, an agent (a controller of a behavioral entity) determines its behavior in a certain environment based on a policy. The behavior affects the environment, and the behavior is evaluated based on the changed environment influenced by the behavior to determine whether the behavior was good or not. The evaluation result is given to the agent as a reward. The policy is then updated based on the evaluation result (reward). Thereafter, the process of determining the agent's behavior based on the policy, evaluating the behavior in the environment influenced by the behavior (reward), and updating the policy based on the evaluation result is repeated in sequence, and the policy is updated (learned) in sequence so as to maximize the final reward (evaluation). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-35222 Summary of the Invention [Problem to be solved by the invention]

[0004] In reinforcement learning for vehicle driving behavior to realize autonomous driving of a vehicle, it is important to appropriately evaluate the vehicle driving behavior in various situations as a reward. Therefore, the present invention has been made in consideration of such circumstances, and provides a vehicle driving behavior evaluation device that can appropriately evaluate the vehicle driving behavior in various situations in reinforcement learning. [Means for solving the problem]

[0005] The vehicle driving behavior evaluation device of the present invention is a vehicle driving behavior evaluation device that evaluates the driving behavior of a target vehicle in a simulator during reinforcement learning regarding the driving of the target vehicle using the simulator, and is configured to have evaluation criteria set for the driving behavior of the target vehicle under each of a plurality of situations that can be represented by the simulator, and to have the following configuration: a criterion selection unit that selects an evaluation criterion corresponding to the situation represented by the simulator during the reinforcement learning process from the plurality of evaluation criteria set for the plurality of situations, and an evaluation point determination unit that determines an evaluation point that represents an evaluation of the driving behavior of the target vehicle under the situation represented by the simulator during the reinforcement learning process based on the evaluation criterion selected by the criterion selection unit.

[0006] With this configuration, in the process of reinforcement learning regarding the driving of a target vehicle using a simulator, an evaluation criterion corresponding to a situation represented by the simulator is selected, and in the process of reinforcement learning, an evaluation score representing an evaluation of the driving behavior of the target vehicle in the situation represented by the simulator is determined based on the selected evaluation criterion. For example, a reward based on the evaluation score can be provided to an agent that determines an action (driving behavior) in an environment including the situation represented by the simulator in the reinforcement learning.

[0007] In the vehicle driving behavior evaluation device according to the present invention, the conditions under which the driving behavior of the target vehicle in a situation represented by the simulator is the behavior recommended in the reinforcement learning are set as the evaluation criteria, and the evaluation score determination unit can be configured to include: a judgment unit that determines whether the driving behavior of the target vehicle in a situation represented by the simulator during the reinforcement learning process satisfies the conditions for the behavior to be the recommended behavior in that situation; and a value determination unit that determines the evaluation score to a predetermined value when the judgment unit determines that the conditions for the behavior to be the recommended behavior are satisfied, and determines the evaluation score to a value that represents an evaluation lower than the evaluation represented by the predetermined value when the judgment unit determines that the conditions for the behavior to be the recommended behavior are not satisfied.

[0008] With this configuration, it is determined whether the driving behavior of the target vehicle in a situation represented by the simulator during the reinforcement learning process satisfies the conditions for the recommended behavior in that situation. If it is determined that the driving behavior of the target vehicle satisfies the conditions for the recommended behavior, the evaluation score is set to a predetermined value. On the other hand, if it is determined that the driving behavior of the target vehicle does not satisfy the conditions for the recommended behavior, the evaluation score is set to a value representing an evaluation lower than the evaluation represented by the predetermined value. As a result, if the driving behavior of the target vehicle in a situation represented by the simulator during the reinforcement learning process is an behavior recommended for that situation in the reinforcement learning, an evaluation score representing an evaluation higher than when the driving behavior is not an behavior recommended for that situation in the reinforcement learning can be obtained.

[0009] In the vehicle driving behavior evaluation device according to the present invention, the multiple situations that can be represented by the simulator can include a situation in which a route on a road on which the target vehicle should travel is set in the simulator, and an evaluation criterion is set for that situation according to the distance by which the target vehicle deviates from the route.

[0010] With this configuration, in the process of reinforcement learning regarding the driving of a target vehicle using a simulator, in a situation where a route to be driven by the target vehicle is set on a road in the simulator, an evaluation score representing an evaluation of the driving behavior of the target vehicle is determined based on an evaluation criterion according to the distance by which the target vehicle deviates from the route. As a result, for example, in the process of reinforcement learning, in a situation where a route to be driven by the target vehicle is set on a road in the simulator, the evaluation score representing the evaluation of the driving behavior of the target vehicle can be determined to a value representing a higher evaluation the shorter the distance by which the target vehicle deviates from the route.

[0011] In the vehicle driving behavior evaluation device according to the present invention, the multiple situations that can be represented by the simulator can include a situation in which the target vehicle is driving at a constant speed in the simulator, and an evaluation criterion is set for that situation according to the driving speed of the target vehicle.

[0012] With this configuration, in the process of reinforcement learning regarding the driving of a target vehicle using a simulator, when the target vehicle is driving at a constant speed in the simulator, an evaluation score representing an evaluation of the driving behavior of the target vehicle is determined based on an evaluation criterion based on the driving speed of the target vehicle. As a result, for example, the evaluation score representing the evaluation of the driving behavior of the target vehicle under the condition that the target vehicle is driving at a constant speed in the simulator during the reinforcement learning process can be determined to be a higher value the closer the driving speed of the target vehicle is to a certain speed (constant speed).

[0013] In the vehicle driving behavior evaluation device according to the present invention, the multiple situations that can be represented by the simulator can include a situation in which an obstacle is present on the road along which the target vehicle is to travel in the simulator, and an evaluation criterion is set for that situation according to the distance between the target vehicle and the obstacle.

[0014] With this configuration, in the process of reinforcement learning regarding the driving of a target vehicle using a simulator, in a situation in which an obstacle is present on a road on which the target vehicle is to travel in the simulator, an evaluation score representing an evaluation of the driving behavior of the target vehicle is determined based on an evaluation criterion according to the distance between the target vehicle and the obstacle. As a result, for example, in the process of reinforcement learning, in a situation in which an obstacle is present on a road on which the target vehicle is to travel in the simulator, the evaluation score representing an evaluation of the driving behavior of the target vehicle can be determined to be a higher value the greater the distance between the target vehicle and the obstacle.

[0015] In the vehicle driving behavior evaluation device according to the present invention, the multiple situations that can be represented by the simulator can be configured to include a situation in which the simulator has a location where the target vehicle should stop based on traffic rules, and an evaluation criterion is set for that situation according to the distance between the target vehicle and the location where it should stop.

[0016] With this configuration, in the process of reinforcement learning regarding the driving of a target vehicle using a simulator, when the simulator finds a place where the target vehicle must stop based on traffic rules, an evaluation score representing an evaluation of the driving behavior of the target vehicle can be determined based on evaluation criteria according to the positional relationship between the target vehicle and the place where the target vehicle must stop.When the positional relationship between the target vehicle and the place where the target vehicle must stop is such that the target vehicle has not yet reached the place where the target vehicle must stop but is relatively close to it, an evaluation criterion can be set according to, for example, the degree of deceleration of the target vehicle.Furthermore, when the target vehicle is located at the place where the target vehicle must stop, an evaluation criterion can be set according to, for example, whether the target vehicle will stop.

[0017] The vehicle driving behavior evaluation device according to the present invention may be configured to include a learning progress evaluation point determination unit that determines, during the reinforcement learning process, an evaluation point representing an evaluation of the driving behavior of the target vehicle based on the driving history of the target vehicle in the simulator up to a certain point.

[0018] With this configuration, during the reinforcement learning process, an evaluation score representing an evaluation of the driving behavior of the target vehicle in the simulator can be determined based on the driving history of the target vehicle up to a certain point, along with evaluation criteria corresponding to the situation represented by the simulator.

[0019] The vehicle driving behavior evaluation device of the present invention is a vehicle driving behavior evaluation device that evaluates the driving behavior of a target vehicle in a simulator during reinforcement learning regarding the driving of the target vehicle using the simulator, and is configured to define a plurality of driving behaviors that the target vehicle can take when driving in the simulator, and to set evaluation criteria for each of the defined plurality of driving behaviors, and to include a driving behavior determination unit that determines which of the defined plurality of driving behaviors the driving behavior of the target vehicle when driving in the simulator during the reinforcement learning process is, and an evaluation point determination unit that determines an evaluation point that represents an evaluation of the driving behavior of the target vehicle based on the evaluation criteria corresponding to the driving behavior related to the determination result of the driving behavior determination unit.

[0020] With this configuration, in the process of reinforcement learning regarding the driving of a target vehicle using a simulator, it is determined whether the driving behavior of the target vehicle is one of a plurality of defined driving behaviors. An evaluation score representing an evaluation of the driving behavior of the target vehicle is determined based on an evaluation criterion corresponding to the driving behavior related to the determination result. For example, a reward based on the evaluation score can be provided to an agent that determines the behavior (driving behavior) of the target vehicle in the simulator during the reinforcement learning.

[0021] In the vehicle driving behavior evaluation device according to the present invention, the defined plurality of driving behaviors may include a constant speed driving behavior, an accelerating driving behavior, a decelerating driving behavior, and a stopped driving behavior.

[0022] With this configuration, in the process of reinforcement learning regarding the driving of a target vehicle using a simulator, when the target vehicle is driving in any of the driving modes of constant speed driving, accelerating driving, decelerating driving, and stopped driving, an evaluation score representing the evaluation of the corresponding driving mode of the target vehicle is determined based on one of the evaluation criteria corresponding to the constant speed driving mode, the evaluation criteria corresponding to the accelerating driving mode, the evaluation criteria corresponding to the decelerating driving mode, and the evaluation criteria corresponding to the stopped driving mode.

[0023] In the vehicle driving behavior evaluation device according to the present invention, a state transition of the vehicle driving behavior may be defined for each of the plurality of defined driving behaviors.

[0024] With this configuration, in the process of reinforcement learning regarding the driving of a target vehicle using a simulator, it is determined whether the driving behavior of the target vehicle is one of a plurality of state transitions of the defined driving behaviors. Based on the evaluation criteria corresponding to the state transitions related to the determination result, an evaluation score representing an evaluation of the driving behavior of the target vehicle is determined.

[0025] In the vehicle driving behavior evaluation device according to the present invention, conditions for each of the defined driving behaviors to be a recommended behavior in the reinforcement learning are set as evaluation criteria corresponding to the defined driving behavior, and the evaluation point determination unit can be configured to include: a determination unit that determines whether or not a condition as the evaluation criterion corresponding to a driving behavior related to a determination result by the driving behavior determination unit is satisfied during the reinforcement learning process; and a value determination unit that determines the evaluation point to a predetermined value when the condition determination unit determines that the condition is satisfied, and determines the evaluation point to a value representing an evaluation lower than the predetermined value when the condition determination unit determines that the condition is not satisfied.

[0026] With this configuration, during reinforcement learning related to the driving of a target vehicle using a simulator, it is determined whether the driving behavior of the target vehicle is one of a plurality of defined driving behaviors. Then, it is determined whether the driving behavior of the target vehicle satisfies a condition for recommended behavior in the reinforcement learning, which is an evaluation criterion corresponding to the driving behavior related to the result of the determination. If it is determined that the driving behavior of the target vehicle satisfies the condition for recommended behavior, the evaluation score is set to a predetermined value. On the other hand, if it is determined that the driving behavior of the target vehicle does not satisfy the condition for recommended behavior, the evaluation score is set to a value representing an evaluation lower than the evaluation represented by the predetermined value. As a result, if the driving behavior of the target vehicle in the simulator during the reinforcement learning is an behavior recommended in the reinforcement learning, an evaluation score representing an evaluation higher than when the driving behavior is not an behavior recommended in the reinforcement learning can be obtained.

[0027] The vehicle driving behavior evaluation device according to the present invention may be configured to have a learning progress evaluation point determination unit that determines an evaluation point based on the driving history of the target vehicle in the simulator up to a certain point during the reinforcement learning process.

[0028] With this configuration, in the process of reinforcement learning regarding the driving of a target vehicle using a simulator, it is possible to determine the driving mode of the target vehicle based on the driving history of the target vehicle up to a certain point, along with an evaluation criterion corresponding to the driving mode related to the judgment result of whether the driving mode of the driving target vehicle is one of multiple defined driving modes. [Effects of the Invention]

[0029] According to the vehicle driving behavior evaluation device of the present invention, it becomes possible to appropriately evaluate the driving behavior of a vehicle in various situations in reinforcement learning. [Brief explanation of the drawings]

[0030] [Figure 1]FIG. 1 is a block diagram showing a reinforcement learning device to which a vehicle driving behavior evaluation device according to an embodiment of the present invention is applied. [Figure 2] FIG. 2 is a diagram showing an example of a simulation image generated by a simulator, including a road on which a grid (squares) is set, and a target vehicle moving (traveling) on ​​the road. [Figure 3] FIG. 3 is a diagram showing an example of the movement of a target vehicle on a road in a simulation image. [Figure 4] FIG. 4 is a functional block diagram showing functions of the evaluation device (vehicle driving behavior evaluation device) according to the first embodiment included in the reinforcement learning device shown in FIG. [Figure 5] FIG. 5 is a diagram showing an example of a situation (environment) represented in a simulation image including a target vehicle. [Figure 6] FIG. 6 is a diagram showing a collision distance range CD and a close distance range AD set around an obstacle in a simulation image. [Figure 7A] FIG. 7A is a diagram showing an evaluation criterion (part 1) used by the evaluation device in reinforcement learning under the situation (environment) shown in FIG. [Figure 7B] FIG. 7B is a diagram showing a second evaluation criterion used by the evaluation device in reinforcement learning under the situation (environment) shown in FIG. [Figure 7C] FIG. 7C is a diagram showing an evaluation criterion (part 3) used by the evaluation device in reinforcement learning under the situation (environment) shown in FIG. [Figure 8A] FIG. 8A is a diagram showing an evaluation score table for determining evaluation scores based on the evaluation criteria shown in FIGS. 7A and 7C. [Figure 8B] FIG. 8B is a diagram showing an evaluation score table for determining evaluation scores based on the evaluation criteria shown in FIG. 7B. [Figure 9] FIG. 9 is a diagram showing an example of an evaluation score given to the target vehicle when the vehicle finishes traveling in reinforcement learning under the situation (environment) shown in FIG. [Figure 10]FIG. 10 is a diagram showing an example of a travel trajectory of a target vehicle that comes to a stop without achieving a goal during the reinforcement learning process under the situation (environment) shown in FIG. [Figure 11] FIG. 11 is a diagram showing an example of a travel trajectory of a target vehicle moving toward achieving a goal in the process of reinforcement learning under the situation (environment) shown in FIG. [Figure 12] FIG. 12 is a diagram showing another example of a situation (environment) represented in a simulation image including a target vehicle. [Figure 13A] FIG. 13A is a diagram showing an evaluation criterion (part 1) used by the evaluation device in reinforcement learning under the situation (environment) shown in FIG. [Figure 13B] FIG. 13B is a diagram showing a second evaluation criterion used by the evaluation device in reinforcement learning under the situation (environment) shown in FIG. [Figure 13C] FIG. 13C is a diagram showing an evaluation criterion (part 3) used by the evaluation device in reinforcement learning under the situation (environment) shown in FIG. [Figure 13D] FIG. 13D is a diagram showing an evaluation criterion (part 4) used by the evaluation device in reinforcement learning under the situation (environment) shown in FIG. [Figure 14A] FIG. 14A is a diagram showing an evaluation score table for determining evaluation scores based on the evaluation criteria shown in FIG. 13A. [Figure 14B] FIG. 14B is a diagram showing an evaluation score table for determining evaluation scores based on the evaluation criteria shown in FIG. 13B. [Figure 14C] FIG. 14C is a diagram showing an evaluation score table for determining evaluation scores based on the evaluation criteria shown in FIG. 13C. [Figure 14D] FIG. 14D is a diagram showing an evaluation score table for determining evaluation scores based on the evaluation criteria shown in FIG. 13D. [Figure 15] FIG. 15 is a diagram showing an example of an evaluation score given to the target vehicle at the end of travel in reinforcement learning under the situation (environment) shown in FIG. [Figure 16]FIG. 16 is a diagram showing an example of a travel trajectory of a target vehicle that comes to a stop without achieving a goal during the reinforcement learning process under the situation (environment) shown in FIG. [Figure 17] FIG. 17 is a diagram showing an example of a travel trajectory of a target vehicle moving toward achieving a goal in the process of reinforcement learning under the situation (environment) shown in FIG. [Figure 18] FIG. 18 is a functional block diagram showing functions of an evaluation device (vehicle driving behavior evaluation device) according to the second embodiment, which is included in the reinforcement learning device shown in FIG. [Figure 19] FIG. 19 is a diagram showing state transitions (part 1) of a target vehicle that can be represented by a simulator. [Figure 20] FIG. 20 is a diagram showing an example of a condition for recommending a state transition of the target vehicle shown in FIG. 19 in reinforcement learning as an evaluation criterion for the state transition. [Figure 21] FIG. 21 is a diagram showing state transitions (part 2) of a target vehicle displayed in the simulator. [Figure 22A] FIG. 22A is a diagram showing an evaluation score table for determining evaluation scores based on the evaluation criteria for each state transition shown in FIG. 19 (conditions for recommending that state transition in reinforcement learning). [Figure 22B] FIG. 22B is a diagram showing an evaluation score table for determining evaluation scores based on the evaluation criteria for each state transition shown in FIG. 21 (conditions for recommending that state transition in reinforcement learning). [Figure 23] FIG. 23 is a diagram showing an evaluation score table for determining evaluation scores based on the evaluation criteria for each state transition shown in FIG. 19 and the evaluation criteria for each state transition shown in FIG. [Figure 24A] FIG. 24A is a diagram showing an example of an evaluation score table showing evaluation criteria corresponding to driving modes and evaluation scores determined based on the evaluation criteria. [Figure 24B] FIG. 24B is a diagram showing another example of an evaluation score table showing evaluation criteria corresponding to driving modes and evaluation scores determined based on the evaluation criteria. DETAILED DESCRIPTION OF THE INVENTION

[0031] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0032] FIG. 1 is a block diagram showing the configuration of a reinforcement learning device to which a vehicle driving behavior evaluation device according to an embodiment of the present invention is applied.

[0033] 1, reinforcement learning device 10 is configured by a computer system including various hardware and software, and has a main control unit 11, a controller 12, a learning control unit 13, an evaluation device 14 (a vehicle driving behavior evaluation device according to an embodiment of the present invention), and a simulator 20. Simulator 20 includes a simulation control unit 21 and an image processing unit 22, and generates a simulation image including symbols (figures, signs, characters, marks, etc.) representing a target vehicle that is the subject of reinforcement learning of driving behavior.

[0034] In addition to a symbol representing the target vehicle, the simulation image may include symbols representing abstracted roads on which the target vehicle and other moving objects travel, moving objects other than the target vehicle (vehicles, people, motorcycles, emergency vehicles), traffic lights (red, green, and yellow), stationary objects (obstacles, other structures, etc.), regulatory signs, guide signs, and directional signs. Furthermore, a simulation image including a symbol representing road R has a grid 50 formed thereon, as shown in FIG. 2, which is composed of a plurality of squares. The actual vertical and horizontal distances represented by each square can be set to, for example, approximately 30 centimeters. Furthermore, in the simulation image, the movement (travel) of the target vehicle E can be represented as movement in units of squares on the grid 50, as shown in FIG. 3. Specifically, the traveling behavior of the target vehicle E can be represented by specifying the position of each square on the grid 100 at predetermined time intervals (for example, 0.5 seconds).

[0035] The reinforcement learning device 10 performs reinforcement learning regarding the traveling of the target vehicle E using the simulator 20 described above. As described above, the simulator 20 generates a simulation image in which the road R, the target vehicle E, other moving objects, structures, etc. are simply represented by figures, symbols, etc. However, the simulator 20 is not limited to this, and can also generate a simulation image in which the target vehicle E and the environment (roads, moving objects, stationary objects, traffic lights, signs, etc.) are represented by other representation methods such as three-dimensional images. Hereinafter, in the simulation image, each symbol and figure will be described by the name of the object it represents (road, target vehicle, obstacle, stop line, etc.).

[0036] Returning to FIG. 1 , the library 30 manages scenarios and the corresponding moving targets. The main control unit 11 controls the entire reinforcement learning device 10, and provides information representing a scenario, among the scenarios and moving targets managed in the library 20, to the simulation control unit 21 of the simulator 20, and provides information representing the moving targets corresponding to the scenario to the controller 12. The image processing unit 22 of the simulator 20 generates image information representing a simulation image including the road R, the target vehicle E, etc., as described above. The simulation control unit 21 controls the image processing unit 22 so that the target vehicle E moves along the road R in the simulation image in accordance with a movement instruction from the controller 12, described later, based on the moving targets, under the environment (situation) on the road R represented according to the scenario.

[0037] The controller 12 receives as input information environmental information representing the environment on road R (situation: the presence of a travel route set on road R, the presence of obstacles, etc.) represented according to the scenario in a simulation image generated by the image processing unit 22 of the simulator 20, and behavioral information representing the travel mode (movement mode) of the target vehicle E, and determines the travel mode of the target vehicle E based on a policy so that the movement goal is achieved. Then, the controller 12 instructs the simulator 20 (simulation control unit 21) to move the target vehicle E on the road R in the determined travel mode (movement instruction). Based on the instruction from the control unit 12, the simulation control unit 22 controls the image information generated by the image processing unit 22 so that the target vehicle E moves to a specified next position (a square on the grid 50) (see FIG. 3 ) in the environment on road R represented according to the scenario.

[0038] The controller 12 described above has a neural network NN (comprised of an input layer, an intermediate layer (hidden layer), and an output layer). The weighting coefficients between each node of this neural network NN correspond to a policy for determining the driving mode of the target vehicle E (for example, its next position (a square on the grid 50)). In the controller 12, environmental information (road R, obstacles, etc.) in the simulation image generated by the image control unit 22 of the simulator 20, the behavior information representing the driving mode of the target vehicle E, and information on moving targets provided by the main control unit 21 are input to the input layer of the neural network NN. Then, information determined based on the weighting coefficients (policy) between each node from the information input to the input layer is output from the output layer of the neural network NN as information representing the next position (square) of the target vehicle E on the road R (grid 50). This new position is provided to the simulation control unit 21 of the simulator 20 as a movement instruction for the target vehicle E.

[0039] The learning control unit 13 receives as input information behavior information representing the driving behavior (movement behavior) of the target vehicle E in a simulation image generated by the simulator 20 (simulation control unit 21, image processing unit 22) based on a movement instruction from the controller 12, and environmental information representing the environment (road R, driving route, obstacles, etc.) in the simulation image, and evaluates the driving behavior of the target vehicle E in the environment represented by the environmental information according to a predetermined reinforcement learning algorithm (e.g., DeepQ-Network, MuZero, etc.). Then, based on the evaluation result, the learning control unit 13 updates the weight coefficients (corresponding to a policy used to determine the driving behavior of the target vehicle E) between each node of the neural network NN in the controller 12 that determined the driving behavior of the target vehicle E. Here, the evaluation of the driving behavior of the target vehicle E is performed using a reward value (evaluation points) assigned to the driving behavior. This reward value (evaluation points) is provided from an evaluation device 14, which will be described later.

[0040] In the evaluation device 14, evaluation criteria are set for the driving behavior of the target vehicle E under each of a plurality of situations (environments) that can be represented by a simulation image generated by the simulator 20 (simulation control unit 21, image processing unit 22). Specifically, information representing the evaluation criteria is stored. The plurality of situations (environments) that can be represented by the simulation image can include, for example, "a situation in which a route (represented by a series of waypoints WP set on the road R) that the target vehicle E should travel is set on the road R in the simulation image," "a situation in which the target vehicle E travels at a constant speed on the road R in the simulation image," "a situation in which an obstacle is present on the road R that the target vehicle E should travel," and "a situation in which there is a place in the simulation image where the target vehicle E should stop based on the Road Traffic Act (traffic rules)."

[0041] The evaluation device 14 has the functions shown in FIG. 4 as the vehicle driving behavior evaluation device according to the first embodiment of the present invention.

[0042] Similarly to the learning control unit 13, the evaluation device 14 receives input of behavioral information representing the driving behavior of the target vehicle E in the simulation image (S1: behavior input) and input of environmental information representing the environment (situation) in the simulation image (S2: situation determination). Then, the evaluation device 14 selects an evaluation criterion corresponding to the environment (situation) represented by the input environmental information from a plurality of evaluation criteria set for the plurality of situations (environments) (S3: evaluation formula selection / criterion selection unit). Based on the selected evaluation criterion, the evaluation device 14 determines an evaluation score representing an evaluation of the driving behavior of the target vehicle E represented by the input behavioral information under the situation represented by the input environmental information (S4-1 to S4-n: evaluation; S5: evaluation score addition / normalization: evaluation score determination unit). Here, when the input environmental information represents a plurality of situations (i), a plurality of evaluation criteria (i: i≦n) corresponding to the plurality of situations are selected from the plurality of evaluation criteria (n) set. Then, multiple evaluation points determined based on these i evaluation criteria (for example, S4-1 to S4-i: Evaluation 1 to Evaluation i) are added together and normalized, for example, so that the maximum value becomes "1" (S5: Evaluation point addition / normalization).

[0043] For example, if the input environmental information represents two situations, namely, "a situation in which a route (for example, represented by a series of waypoints WP set on road R) that the target vehicle E should travel is set on road R in the simulation image" (situation 1) and "a situation in which an obstacle exists on road R that the target vehicle E should travel" (situation 2), two evaluation criteria, that is, an evaluation criterion corresponding to situation 1 and an evaluation criterion corresponding to situation 2, are selected from the multiple (n) evaluation criteria that have been set. Then, two evaluation points determined based on the two evaluation criteria are added together and normalized so that the maximum value becomes "1." In this case, an evaluation point of the maximum value "1" may be determined for the traveling mode in which "the target vehicle E travels on the set route and travels while avoiding obstacles" (details will be described later).

[0044] The evaluation device 14 provides the determined evaluation score as a reward value to the learning control unit 13 (S6: reward value output). Then, while the reinforcement learning process is being performed, the evaluation device 14 repeatedly executes the above-mentioned processes (S1 to S6) at a predetermined cycle, and each time a driving mode (position of a square on the grid 50) of the target vehicle E in the environment represented by the simulation image is obtained (S1), the evaluation device 14 provides the determined evaluation score to the learning control unit 13 as a reward value.

[0045] 4, the evaluation device 14 also has a function (S11 to S13: learning progress evaluation point determination unit) of determining an evaluation score representing an evaluation of the driving behavior of the target vehicle E based on the driving history (episode) of the target vehicle E up to a certain point (for example, a point of goal achievement) in the simulator 20 during the reinforcement learning process. Details of this function will be described later. Then, the sum of the evaluation points (S2 to S5) determined based on the above-mentioned evaluation criteria and the evaluation points (S11 to S13: learning progress evaluation point determination unit) determined based on the driving history of the target vehicle is provided to the learning control unit 13 as a reward value (S6).

[0046] The operation of the evaluation device 14 will now be described in more detail.

[0047] For example, as shown in Figure 5, a route (represented by a series of waypoints WP; the same applies below) that the target vehicle E should travel on road R is set, the target vehicle E should travel at a certain constant speed, and there is an obstacle x on that route. In this situation, we will explain the case where the target vehicle E learns (reinforcement learning) the traveling behavior of the target vehicle E so that the target vehicle E can travel at a constant speed while avoiding the obstacle x while maintaining its traveling position on the route as much as possible.

[0048] In this case, in FIG. Area A1 (normal driving area) of road R is The situation (environment) is such that "a route that the target vehicle E should travel is set on the road R" and "the target vehicle E should travel at a constant speed." Area A2 (obstacle avoidance area) of road R is The situation is that "an obstacle x exists on the route that the target vehicle E should travel" and "the target vehicle E should travel at a constant speed." Area A3 (route return area) of road R is The situation is that "a route for the target vehicle E to return to is set on road R" and "the target vehicle E should travel at a constant speed." Area A4 (normal driving area) of road R is As in the area A1, the situation is one in which "a route for the target vehicle E to travel is set on the road R" and "the target vehicle E should travel at a constant speed."

[0049] As shown in Fig. 5 and Fig. 6, a collision distance range CD and a proximity distance range AD are set around an obstacle x present in area A2 (obstacle avoidance area). The collision distance range CD is a range within which a target vehicle E is deemed to have collided with obstacle x when it enters, and is, for example, a rectangular range of 2 grids x 2 grids centered on obstacle x. The proximity distance range AD, which is set outside the collision distance range CD, is a range within which the target vehicle E is deemed to have approached obstacle x when it enters, and is, for example, a rectangular range of 4 grids x 4 grids centered on obstacle x. Furthermore, the distance from the start point of area A2 (obstacle avoidance area) (the end point of area A1) to the collision distance range CD is set to, for example, the distance of one side of the proximity distance range AD (for example, 4 grids).

[0050] Conditions for the driving behavior of the target vehicle E to be a recommended behavior in reinforcement learning in each of a plurality of situations that can be represented in the simulation image generated by the simulator 20 are set as evaluation criteria corresponding to the situation. A recommended behavior in reinforcement learning is a driving behavior (behavior) that the target vehicle E aims for in reinforcement learning in each of various situations (environments).

[0051] Each of the regions A1 and A4 (normal driving regions) shown in FIG. 5 corresponds to a situation where "a route on road R is set for the target vehicle E to travel," and under such circumstances, reinforcement learning aims to have the target vehicle E travel without deviating significantly from the set route. Therefore, for example, as shown in the WP distance item in Evaluation Criteria 1 of FIG. 7A , one evaluation criterion may be set for the target vehicle E, i.e., that the distance from the route (WP) of the target vehicle E is N1 grids (number of squares: e.g., 2 grids) or less (≦N1). Furthermore, each of the regions A1 and A4 (normal driving regions) corresponds to a situation where "the target vehicle E should travel at a constant speed," and under such circumstances, reinforcement learning aims to have the target vehicle E travel at a certain constant speed. Therefore, for example, as shown in the constant speed item in Evaluation Criteria 1 of FIG. 7A , another evaluation criterion may be set for the target vehicle E, i.e., that the target vehicle E travels at a "constant speed equal to or greater than vehicle speed Vb (≧Vb)."

[0052] Region A2 (obstacle avoidance region) shown in FIG. 5 corresponds to a situation where "obstacle x exists on the path along which the target vehicle E should travel," and under such circumstances, reinforcement learning aims for the target vehicle E to travel without colliding with the obstacle x. Therefore, for example, as shown in the item "safe distance" in evaluation criterion 2 of FIG. 7B, one evaluation criterion may be set for region A2, in which "the distance between the target vehicle E and the obstacle x is maintained at a safe distance greater than (>AD) the proximity distance AD ​​(see FIGS. 5 and 6)." Furthermore, region A2 corresponds to a situation where "the target vehicle E should travel at a constant speed," and under such circumstances, reinforcement learning aims for the target vehicle E to travel at a certain constant speed. Therefore, for example, as shown in the item "constant speed" in evaluation criterion 2 of FIG. 7B, another evaluation criterion may be set for region A2, in which "the target vehicle E should travel at a constant speed equal to or less than vehicle speed Vc1 (≦Vc1)."

[0053] Region A3 (path return region) shown in FIG. 5 corresponds to a situation where "a path to which the target vehicle E should return is set on road R." Under such a situation, reinforcement learning aims to have the target vehicle E, having avoided obstacle x, gradually return to the set path. Therefore, in region A3, as shown in the WP distance item in Evaluation Criteria 3 of FIG. 7C, a condition that the target vehicle E's distance from the path (WP) is N2 grids (e.g., 3 grids) or less (≦N2) can be set as one evaluation criterion. Also, region A3 corresponds to a situation where "the target vehicle E should travel at a constant speed." Under such a situation, reinforcement learning aims to have the target vehicle E travel at a certain constant speed. Therefore, in region A3, as shown in the constant speed item of FIG. 7C, a condition that the target vehicle E travels at a "constant speed of Vc2 or less (≦Vc2)" can be set as another evaluation criterion.

[0054] In the area A1 and area A4 (normal driving area) to which the evaluation standard 1 shown in FIG. 7A is applied, the evaluation score is determined based on each evaluation standard 1, and therefore the evaluation score table shown in FIG. 8A is used.

[0055] When the target vehicle E travels in either area A1 or area A4 (normal travel area), the evaluation device 14 (S4-x in FIG. 4 (x is a numerical value between 1 and n; the same applies below): corresponding to the WP distance item) determines whether the distance between the travel position (position of a square in the grid 50) of the target vehicle E and the set route is equal to or less than N1 grids (e.g., 2 grids) (see FIG. 7A), which is a condition as an evaluation criterion (determination unit). If the condition is met, the travel mode of the target vehicle E is determined to be an action recommended in reinforcement learning, and the evaluation score is set to a predetermined value, for example, "1" point (marked with a circle in FIG. 8A). If the condition is not met, the evaluation score is set to a value lower than "1", for example, "0" point (marked with an X in FIG. 8A) (value determination unit).

[0056] Furthermore, the evaluation device 14 (corresponding to item S4-x: constant speed in FIG. 4) determines whether or not a change in the traveling position of the subject vehicle E (traveling manner: change in the position of the squares of the grid 50: see FIG. 3) corresponds to a constant speed equal to or greater than the speed Vb (see FIG. 7A), which is a condition as an evaluation criterion (determination unit). If the condition is met, the traveling manner of the subject vehicle E is determined to be an action recommended in reinforcement learning, and the evaluation score is set to "1" (circle in FIG. 8A), and if the condition is not met, the evaluation score is set to "0" (cross in FIG. 8A), which is a value lower than "1" (value determination unit).

[0057] Then, the evaluation device 14 (S5 in FIG. 4) adds up the evaluation points determined for these two items (WP distance item, constant speed item) and normalizes the sum so that the maximum value is 1 point. As a result, in each of the areas A1 and A2 (normal driving areas), as shown in FIG. 8A, the evaluation point to be provided to the learning control unit 13 as a reward value is determined to be 1 point, 0.5 point, or 0 point, depending on the evaluation points (circle, cross) for the two items (WP distance item, constant speed item).

[0058] In the area A2 (obstacle avoidance area) to which evaluation criterion 2 as shown in FIG. 7B is applied, an evaluation score is determined based on that evaluation criterion 2, and an evaluation score table as shown in FIG. 8B is used.

[0059] When the target vehicle E travels in the area A2 (obstacle avoidance area), the evaluation device 14 (corresponding to the item S4-x: safety distance in FIG. 4) determines whether the distance between the target vehicle E and the obstacle x is maintained at a safe distance exceeding the proximity distance AD ​​(>AD: for example, 4 grids), which is a condition as an evaluation criterion (determination unit) (see FIG. 7B). If the condition is met, the traveling manner of the target vehicle E is determined to be an action recommended in reinforcement learning, and the evaluation score is set to "1" (circle in FIG. 8B), and if the condition is not met, the evaluation score is set to "0" (cross in FIG. 8B) (value determination unit).

[0060] Furthermore, the evaluation device 14 (corresponding to item S4-x: constant speed in FIG. 4) determines whether or not a change in the traveling position of the subject vehicle E (traveling manner: change in the position of the squares of the grid 50) corresponds to a constant speed equal to or less than Vc1 (see FIG. 7B), which is a condition as an evaluation criterion (determination unit). If the condition is met, the traveling manner of the subject vehicle E is determined to be an action recommended in reinforcement learning, and the evaluation score is set to "1" (circle in FIG. 8B), and if the condition is not met, the evaluation score is set to "0" (cross in FIG. 8B) (value determination unit).

[0061] Then, the evaluation device 14 (S5 in FIG. 4) adds up the evaluation scores determined for these two items (safety distance item and constant speed item) and normalizes them so that the maximum value is 1 point. As a result, in the area A2 (obstacle avoidance area), as shown in FIG. 8B, the evaluation score to be provided to the learning control unit 13 as a reward value is determined to be 1 point, 0.5 point, or 0 point, depending on the evaluation scores (circle and cross marks) for the two items (safety distance item and constant speed item).

[0062] Furthermore, in area A3 (route return area) where evaluation criterion 3 as shown in Figure 7C is provided, the evaluation score is determined based on that evaluation criterion 3, so an evaluation score table such as that shown in Figure 8A is used, as in areas A1 and A2 (normal driving areas).

[0063] When the target vehicle E travels through area A3 (route return area), the evaluation device 14 (corresponding to the item S4-x:WP distance in FIG. 4) determines whether the distance between the travel position of the target vehicle E and the set route is equal to or less than N2 grids (e.g., 3 grids) (see FIG. 7C), which is a condition as an evaluation criterion (determination unit). If the condition is met, the travel mode of the target vehicle E is determined to be an action recommended in reinforcement learning, and the evaluation score is set to "1" (circle in FIG. 8A), and if the condition is not met, the evaluation score is set to "0" (cross in FIG. 8A) (value determination unit).

[0064] Furthermore, the evaluation device 14 (corresponding to S4-x: constant speed item in FIG. 4) determines whether or not the change in the traveling position of the subject vehicle E corresponds to a constant speed equal to or less than Vc2 (see FIG. 7C), which is a condition as an evaluation criterion (determination unit). If the condition is met, the traveling mode of the subject vehicle E is determined to be an action recommended in reinforcement learning, and the evaluation score is set to "1" (circle in FIG. 8A), and if the condition is not met, the evaluation score is set to "0" (cross in FIG. 8A) (value determination unit).

[0065] Then, in the evaluation device 14 (S5 in Figure 4), the evaluation points determined for the two items are added together and normalized to the maximum value of "1", just as when the target vehicle E travels through areas A1, A2, and A4, and the evaluation point to be provided to the learning control unit 13 as a reward value is determined to be either "1", "0.5", or "0" (see Figure 8A).

[0066] The evaluation device 14 further determines an evaluation point for the end of movement of the target vehicle E in the reinforcement learning process, for example, as shown in Fig. 9, and provides the evaluation point as a reward value to the learning control unit 13. In Fig. 9, if the target vehicle E deviates from the road and stops moving, the evaluation point is assigned "-5" points; if the target vehicle E deviates significantly from the set route (waypoint WP) and stops moving, the evaluation point is assigned "-5" points; if the target vehicle E collides with obstacle x (enters the collision distance range CD) and stops moving, the evaluation point is assigned "-5" points; and if the target vehicle E traveling along the set route achieves its goal and stops moving, the evaluation point is assigned "+10" points.

[0067] In the reinforcement learning device 10, a route along a road R to be traveled by a target vehicle E is set as shown in Fig. 5, the target vehicle E should travel at a constant speed, and an obstacle x is present on the route. In this situation, trial and error is repeated to achieve the goal of the target vehicle E traveling at a constant speed while avoiding the obstacle x while maintaining the traveling position of the target vehicle E on the route as much as possible. As a result, for example, as shown in Fig. 10, the target vehicle E may move (travel) along various trajectories (TT1, TT2, TT3, TT4, TT5, etc.) and come to a stop. Then, for the traveling behavior of the target vehicle E along each of these various movement trajectories, the evaluation device 14 provides the learning control unit 13 with an evaluation score determined based on the evaluation criteria as described above as a reward value.

[0068] As the above-described processing is repeated in the reinforcement learning device 10, the policy (weighting coefficients of the neural network NN) of the controller 12 is sequentially updated under the control of the learning control unit 13 so as to maximize the reward value (evaluation point) finally obtained. Then, in an environment (situation) in which an obstacle x exists on the route of the road R as shown in Fig. 5 described above, for example, as shown in Fig. 11, a policy (policy: weighting coefficients of the neural network NN) of the controller 12 is obtained that can provide the simulator 20 (simulation control unit 21) with movement instructions that enable the target vehicle E to travel along a movement trajectory TTo that can achieve the objective of traveling at a constant speed so as to avoid the obstacle x while maintaining the traveling position of the target vehicle E on the route as much as possible.

[0069] Next, for example, as shown in Figure 12, in a situation where a route (WP) that the target vehicle E should travel is set on road R, the target vehicle E should travel at a certain constant speed, and a "stop position SP" where the target vehicle E should stop temporarily as it crosses the route is set on road R, we will describe a case where the target vehicle E's travel mode is learned (reinforcement learning) so that the target vehicle E's travel position is maintained on the route as much as possible, stops temporarily at the "stop position SP", and then travels at a constant speed.

[0070] The "stop position SP" is a position where the target vehicle E must stop according to traffic regulations, and may be formed not only by a white line WL, but also by other objects such as an intersection, sign, railroad crossing, traffic light, etc.

[0071] In this case, in FIG. Area A1 (normal driving area) of road R is The situation (environment) is such that "a route that the target vehicle E should travel is set on the road R" and "the target vehicle E should travel at a constant speed." Area A2 (decelerating driving area) of road R is The situation is that "a route for the target vehicle E to travel is set on the road R" and "the target vehicle E should travel at a reduced speed." The area A3 (stop area) immediately before the "stop position SP" on road R is The situation is that "a route that the target vehicle E should travel is set on road R" and "the target vehicle E should temporarily stop." Area A4 (speed recovery area) of road R is This is a situation in which "a route that the target vehicle E should travel is set on the road R" and "the target vehicle E should travel at an accelerated speed." Area A5 (normal driving area) of the current route R is similar to area A1. The situation is that "a route that the target vehicle E should travel is set on the road R" and "the target vehicle E should travel at a constant speed."

[0072] In each of all areas shown in Figure 12, namely area A1 (normal driving area), area A2 (deceleration driving area), area A3 (stop area), area A4 (speed recovery area), and area 5 (normal area), where "a route for the target vehicle E to travel is set on road R," the condition that "the distance from the route (WP) of the target vehicle E is N1 grids (e.g., 2 grids) or less (≦N1)" can be set as one evaluation criterion, as in the case of area A1 and area A4 shown in Figure 5 described above, as shown in the WP distance item in each of evaluation criterion 1 in Figure 13A, evaluation criterion 2 in Figure 13B, evaluation criterion 3 in Figure 13C, and evaluation criterion 4 in Figure 13D.

[0073] Also, in each of region A1 and region A5 (normal driving regions) where the situation is that "the target vehicle E should travel at a constant speed", as shown in the constant speed item in evaluation criterion 1 of FIG. 13A, the condition that the target vehicle E travels at "a constant speed of vehicle speed Vb or higher (≧Vb)" can be set as another evaluation criterion, similar to the cases of region A1 and region A4 shown in FIG. 5 described above.

[0074] The region A2 (deceleration driving region) shown in FIG. 12 is in a situation where "the target vehicle E should decelerate and travel". Under such a situation, in reinforcement learning, it is aimed that the target vehicle E travels with a certain deceleration. Therefore, in the region A2, as shown in the deceleration item in evaluation criterion 2 of FIG. 13B, the condition that the target vehicle E travels at "a speed lower than vehicle speed Vb, with an acceleration a greater than -a1 and less than 0 (-a < a < 0)" can be set as another evaluation criterion.

[0075] In the region A3 (stop region) shown in FIG. 12, the situation is that "the target vehicle E should temporarily stop". Under such a situation, in reinforcement learning, it is aimed that the target vehicle E temporarily stops. Therefore, in the region A3, as shown in the stop item in evaluation criterion 3 of FIG. 13C, the condition that the target vehicle E is stopped (vehicle speed V = 0) during a time t satisfying t1 ≦ t < t2 can be set as another evaluation criterion.

[0076] In the region A4 (speed recovery region) shown in FIG. 12, the situation is that "the target vehicle E should accelerate and travel". Under this situation, in reinforcement learning In it is aimed that the target vehicle E travels with a certain acceleration. Therefore, in the region A4, as shown in the acceleration item in evaluation criterion 4 of FIG. 13D, the condition that the target vehicle E travels at "a speed lower than vehicle speed Vb, with an acceleration a less than a2 (0 < a < a2)" can be set as another evaluation criterion.

[0077] In the areas A1 and A5 (normal driving areas) to which the evaluation standard 1 shown in FIG. 13A is applied, the evaluation score is determined based on each evaluation standard 1, and therefore the evaluation score table shown in FIG. 14A is used.

[0078] When the target vehicle E travels in either area A1 or area A5 (normal travel area), the evaluation device 14 (corresponding to the item S4-x:WP distance in FIG. 4) determines whether the distance between the travel position (position of a square in the grid 50) of the target vehicle E and the set route is equal to or less than N1 grids (e.g., 2 grids) (see FIG. 13A), which is a condition as an evaluation criterion (determination unit). If the condition is met, the travel mode of the target vehicle E is determined to be an action recommended in reinforcement learning, and the evaluation score is set to a predetermined value, for example, "1" point (circle in FIG. 14A). If the condition is not met, the evaluation score is set to a value lower than "1", for example, "0" point (cross in FIG. 14A) (value determination unit).

[0079] Furthermore, the evaluation device 14 (corresponding to item S4-x: constant speed in FIG. 4) determines whether or not a change in the traveling position of the subject vehicle E (traveling manner: change in the position of the squares of the grid 50: see FIG. 3) corresponds to a constant speed equal to or greater than the speed Vb (see FIG. 13A), which is a condition as an evaluation criterion (determination unit). If the condition is met, the traveling manner of the subject vehicle E is determined to be an action recommended in reinforcement learning, and the evaluation score is set to "1" (circle in FIG. 14A), and if the condition is not met, the evaluation score is set to "0" (cross in FIG. 14A), which is a value lower than "1" (value determination unit).

[0080] Then, the evaluation device 14 (S5 in FIG. 4) adds up the evaluation points determined for these two items (WP distance item, constant speed item) and normalizes the result so that the maximum value is 1 point. As a result, in either area A1 or area A5 (normal driving area), as shown in FIG. 14A, the evaluation point to be provided to the learning control unit 13 as a reward value is determined to be 1 point, 0.5 point, or 0 point, depending on the evaluation points (circle, cross) for the two items (WP distance item, constant speed item).

[0081] In the region A2 (deceleration driving region) where the evaluation criterion 2 as shown in FIG. 13B is applied, in order to determine the evaluation points based on the evaluation criterion 2, an evaluation point table as shown in FIG. 14B is used.

[0082] When the target vehicle E travels in the region 2 (deceleration driving region), similar to the case when traveling in the regions A1 and A5, in the evaluation device 14 (corresponding to the item of S4-x: WP distance in FIG. 4), it is determined whether the distance between the traveling position of the target vehicle E and the set route is equal to or less than N1 grid (see FIG. 13B) which is a condition as an evaluation criterion (determination unit). If the condition is satisfied, the evaluation point is determined to be "1" point (〇 mark in FIG. 14B), and if the condition is not satisfied, the evaluation point is determined to be "0" point (× mark in FIG. 14B) respectively (value determination unit).

[0083] Also, in the evaluation device 14 (corresponding to the item of S4-x: deceleration in FIG. 4), it is determined whether the change in the traveling position of the target vehicle E (traveling mode: change in the position of the grid 50 squares: see FIG. 3) is less than the speed Vb and the change in its speed a (acceleration) is greater than -a and less than 0 (-a < a < 0: see FIG. 13B), which is a condition as an evaluation criterion (determination unit). If the condition is satisfied, it is considered that the traveling mode of the target vehicle E is an action recommended in reinforcement learning, and the evaluation point is determined to be "1" point (〇 mark in FIG. 14B), and if the condition is not satisfied, it is considered that the target vehicle E is not decelerating, and the evaluation point is determined to be "0" point (× mark in FIG. 14B) respectively (value determination unit).

[0084] Then, in the evaluation device 14 (S5 in FIG. 4), the evaluation points determined for these two items (the item of WP distance, the item of deceleration) are added and normalized so that the maximum value becomes "1" point. As a result, in the region 2 (deceleration driving region), as shown in FIG. 14B, according to the evaluation points (〇 mark, × mark) for the two items (the item of WP distance, the item of deceleration), the evaluation point to be provided to the learning control unit 13 as a reward value is determined to be either "1" point, "0.5" point, or "0" point.

[0085] In the area A3 (stop area) to which the evaluation criterion 3 shown in FIG. 13C is applied, the evaluation score is determined based on the evaluation criterion 3, and therefore the evaluation score table shown in FIG. 14C is used.

[0086] When the target vehicle E is in area 3 (stop area), similarly to when traveling in areas A1 and A5, the evaluation device 14 (corresponding to the item S4-x:WP distance in FIG. 4) determines whether the distance between the position of the target vehicle E and the set route is equal to or less than N1 grids (see FIG. 13C), which is a condition as an evaluation criterion (determination unit). If the condition is met, the evaluation score is set to "1" (circle in FIG. 14C), and if the condition is not met, the evaluation score is set to "0" (cross in FIG. 14C) (value determination unit).

[0087] Furthermore, the evaluation device 14 (corresponding to S4-x: Stop item in FIG. 4) determines whether the traveling position of the target vehicle E maintains a speed V=0 (see FIG. 13C) for a time t that is equal to or greater than time t1 and less than time t2, which is a condition as an evaluation criterion (determination unit). If the condition is met, the traveling mode of the target vehicle E is determined to be an action recommended in reinforcement learning, and the evaluation score is set to "1" (circle in FIG. 14C). If the condition is not met, the target vehicle E is determined not to have stopped, and the evaluation score is set to "0" (cross in FIG. 14C) (value determination unit).

[0088] Then, the evaluation device 14 (S5 in FIG. 4) adds up the evaluation scores determined for these two items (WP distance item and stopping item) and normalizes the sum so that the maximum value is 1 point. As a result, in region 2 (deceleration traveling region), as shown in FIG. 14C, the evaluation score to be provided to the learning control unit 13 as a reward value is determined to be 1 point, 0.5 point, or 0 point, depending on the evaluation scores (circle, cross) for the two items (WP distance item and stopping item).

[0089] Furthermore, if the target vehicle E is in area 3, as shown in Figure 14C, if the target vehicle E does not stop (the stop item is marked with an X), the evaluation score will be determined to be "0" regardless of the positional relationship between the target vehicle E and the route.

[0090] In the region A4 (speed recovery region) to which the evaluation criterion 4 shown in FIG. 13D is applied, the evaluation score is determined based on the evaluation criterion 3, and therefore the evaluation score table shown in FIG. 14D is used.

[0091] When the target vehicle E travels through area A4 (speed recovery area), similarly to when it travels through areas A1 and A5, the evaluation device 14 (corresponding to the item S4-x:WP distance in FIG. 4) determines whether the distance between the travel position of the target vehicle E and the set route is equal to or less than N1 grids (see FIG. 13D), which is a condition as an evaluation criterion (determination unit). If the condition is met, the evaluation score is set to "1" (circle in FIG. 14D), and if the condition is not met, the evaluation score is set to "0" (cross in FIG. 14D) (value determination unit).

[0092] Furthermore, the evaluation device 14 (corresponding to S4-x: acceleration item in FIG. 4) determines whether or not the change in the position of the subject vehicle (driving manner: change in the position of the squares of the grid 50: see FIG. 3) satisfies the conditions as an evaluation criterion, that is, whether the speed is equal to or less than Vb, and whether the change in speed a (acceleration) is greater than 0 and less than a2 (see FIG. 13D) (determination unit). If the conditions are met, the driving manner of the subject vehicle E is determined to be an action recommended in reinforcement learning, and the evaluation score is set to "1" (circle in FIG. 14D); if the conditions are not met, the subject vehicle E is determined to be not accelerating appropriately, and the evaluation score is set to "0" (cross in FIG. 14D) (value determination unit).

[0093] Then, the evaluation device 14 (S5 in FIG. 4) adds up the evaluation scores determined for these two items (WP distance item and acceleration item) and normalizes the sum so that the maximum value is 1 point. As a result, in region 4 (speed recovery region), as shown in FIG. 14D, the evaluation score to be provided to the learning control unit 13 as a reward value is determined to be 1 point, 0.5 point, or 0 point, depending on the evaluation scores (circle, cross) for the two items (WP distance item and acceleration item).

[0094] The evaluation device 14 further determines an evaluation score, for example, as shown in Fig. 15, for the end of movement of the target vehicle E in the reinforcement learning process, and provides the evaluation score to the learning control unit 13 as a reward value. In Fig. 15, if the target vehicle E deviates from the road and stops, the evaluation score is set to "-5" points; if the target vehicle E significantly deviates from the set route (waypoint WP) and stops, the evaluation score is set to "-5" points; if the target vehicle E traveling along the set route achieves its goal and stops, the evaluation score is set to "+10" points; if the target vehicle E stops in area A3 (between the stop position SP (white line WL)) for a time less than time t1 or for a time greater than time t2, the evaluation score is set to "-5" points; and if the target vehicle E passes through area A3 (between the stop position SP (white line WL)) without stopping, the evaluation score is set to "-5" points.

[0095] In the reinforcement learning device 10, a route along a road R on which a target vehicle E should travel is set as shown in FIG. 12 , the target vehicle E should travel at a certain constant speed, and a “stop position SP” where the target vehicle E should temporarily stop is set on the road R so as to cross the route. In this situation, trial and error is repeated to achieve the goal of the target vehicle E temporarily stopping at the “stop position SP” and then traveling at a constant speed while maintaining the traveling position of the target vehicle E on the route as much as possible. As a result, for example, as shown in FIG. 16 , the target vehicle E may travel (travel) along various trajectories (TT1, TT2, TT3, TT4, TT5, etc.) and eventually stop. Then, for the traveling behavior of the target vehicle E along each of these various movement trajectories, the evaluation device 14 provides the learning control unit 13 with an evaluation score determined based on the evaluation criteria described above as a reward value.

[0096] As the processing in the reinforcement learning device 10 is repeated as described above, the policy (weighting coefficients of the neural network NN) of the controller 12 is sequentially updated under the control of the learning control unit 13 so as to maximize the reward value (evaluation point) finally obtained. Then, in an environment (situation) in which the "stop position SP" as shown in Fig. 12 exists, for example, as shown in Fig. 17, a policy (policy: weighting coefficients of the neural network NN) of the controller 12 can be obtained that can provide the simulator 20 (simulation control unit 21) with movement instructions that enable the target vehicle E to travel along a movement trajectory TTo that can achieve the objective of temporarily stopping at the "stop position SP" and then traveling at a constant speed while maintaining the traveling position of the target vehicle E on the route as much as possible.

[0097] According to the evaluation device 14 (the vehicle driving mode evaluation device according to the first embodiment of the present invention) as described above, in the process of reinforcement learning regarding the driving of the target vehicle E using the simulator 20, an evaluation criterion corresponding to the situation represented by the simulator 20 is selected from a plurality of evaluation criteria corresponding to various situations, and an evaluation score representing an evaluation of the driving mode of the target vehicle E under the situation represented by the simulator 20 is determined based on the selected evaluation criterion, so that the vehicle driving mode under various situations can be appropriately evaluated in reinforcement learning. As a result, a controller 12 (neural network NN) with good performance can be constructed by the reinforcement learning.

[0098] Furthermore, in the evaluation device 14, in the process of reinforcement learning regarding the driving of the target vehicle E using the simulator 20, when a plurality of evaluation criteria (for example, a WP distance item, a constant speed item) corresponding to a plurality of situations that can be represented by the simulator 20 are selected, a plurality of evaluation points based on the plurality of evaluation criteria for the driving behavior of the target vehicle E under the plurality of situations represented by the simulator 20 during the reinforcement learning process are normalized in accordance with the number of the evaluation criteria to determine a normalized evaluation point. As a result, the evaluation of the driving behavior of the target vehicle under the various situations can be expressed with a unified numerical sense based on the normalized evaluation points.

[0099] As described above, the evaluation device 14 has a function of determining an evaluation score (see S11 to S13 in FIG. 4) that represents an evaluation of the driving behavior of the target vehicle E based on the driving history (episode) of the target vehicle E up to a certain point in the simulator 20 during the reinforcement learning process. For example, during the reinforcement learning process, the quality of the movement trajectory (behavior history: S1) of the target vehicle E until the movement goal is achieved (until the goal achievement point) or up to a certain point (a point set as a subtask completion point such as an intersection) is judged according to certain evaluation criteria (number of collisions with obstacles, number of road departures, number of traffic rule violations, etc.), and an evaluation score can be temporarily determined according to the quality (S11: episode evaluation and commentary). In addition, the degree of shortness of the travel time (behavioral history: S1) from the start of the movement of the target vehicle E until the movement goal is achieved (until the goal is reached) or until the vehicle reaches a certain point (a point set as the subtask completion point) is judged, and the evaluation score can be determined as temporary depending on the degree of shortness of the time (S12: episode evaluation - time).

[0100] Furthermore, the timing for distributing each of the two temporary evaluation points (such as when the moving target is reached or when the subtask is completed) is determined (S13). Then, at that distribution timing, the temporary evaluation points are added to the evaluation points (functions of S4-1 to S4-n and S5) that are routinely determined based on the evaluation criteria corresponding to the situation (environment) represented by the simulator 20, and the final evaluation points to be provided to the learning control unit 13 as reward values ​​are determined.

[0101] In this way, in the evaluation device 14, an evaluation score representing an evaluation of the driving behavior of the target vehicle E in the simulator 20 during the reinforcement learning process is determined based on the evaluation criteria corresponding to the situation (environment) displayed in the simulator 20, as well as the history of the driving of the target vehicle E repeated until a certain purpose of the driving of the target vehicle E is achieved, so that the driving behavior of the vehicle in various situations can be more appropriately evaluated in reinforcement learning. As a result, a controller 12 (neural network NN) with even better performance can be constructed through reinforcement learning.

[0102] The evaluation device 14 may have functions as shown in FIG. 18 as a vehicle driving behavior evaluation device according to a second embodiment of the present invention, instead of the functions as shown in FIG. 4 (vehicle driving behavior evaluation device according to the first embodiment of the present invention).

[0103] 18 defines a plurality of driving modes that the target vehicle E traveling in the simulation image generated by the simulator 20 can take, and an evaluation criterion is set for each of the plurality of driving modes. Here, the driving mode of the target vehicle E refers to the manner in which the target vehicle E travels, and may include a constant speed driving mode, an accelerating driving mode, a decelerating driving mode, a stopped mode, etc. Furthermore, in an environment (situation) in which a route that the target vehicle E should travel (represented by, for example, a series of waypoints WP) is set, the driving mode of the target vehicle may include a mode of traveling while following the route, a mode of traveling off the route, a mode of traveling so as to return to the route, etc.

[0104] Furthermore, the driving behavior of the target vehicle E may also include the concept of a state transition, which is a transition of the driving state. For example, the concept of driving behavior may include a state transition from a stopped behavior to an accelerating behavior, a state transition from an accelerating behavior to a constant speed driving behavior, a state transition from a constant speed driving behavior to a decelerating behavior, and a state transition from a decelerating behavior to a stopped behavior. Furthermore, maintaining a driving behavior for a certain period of time may be included in the concept of a state transition as a transition from a certain driving behavior to the same driving behavior.

[0105] Hereinafter, a case will be described in which a state transition is defined as the driving mode to which the evaluation criterion is associated.

[0106] In the evaluation device 14, for example, state transitions as shown in Fig. 19 and Fig. 20 are defined. Fig. 19 shows specific state transitions defined in the evaluation device 14, and Fig. 20 shows examples of multiple (four) state transitions that can occur between a "constant speed" driving mode and a "deceleration" driving mode, and specific conditions for each state transition to be a recommended action in reinforcement learning. In other words, each condition means that if the condition is met, the actual driving of the target vehicle E related to the state transition is a recommended action in reinforcement learning (one that should obtain a high reward value).

[0107] 19, multiple state transitions are defined between multiple (seven) driving modes, including a deceleration mode (hereinafter referred to as "deceleration"), a constant speed mode (hereinafter referred to as "constant speed"), an acceleration mode (hereinafter referred to as "acceleration"), a temporary stop mode (hereinafter referred to as "temporary stop"), a stopped mode (hereinafter referred to as "stopped"), a deadlock mode (hereinafter referred to as "deadlock"), and a mode in which a deadlock resolution action is being taken (hereinafter referred to as "resolving action being taken"). Note that a "deadlock" is a situation in which there is another moving body (such as a vehicle opposing the target vehicle E) that affects the behavior (travel) of the target vehicle E, and the target vehicle E and the other moving body have no choice of a normal action (driving forward in compliance with the rules) and both continue to stop (a situation in which the deadlock cannot be resolved unless either the target vehicle E or the other moving body takes an action that is unusual or that should not be taken, such as moving backward or off the road or designated route).

[0108] As described above, the condition for each state transition to be a recommended action in reinforcement learning means that, if the condition is met, the traveling of the target vehicle E associated with the state transition is a recommended action in reinforcement learning (should obtain a high reward value). For this reason, for example, the condition x1 for the state transition from "deceleration" to "constant speed" to be a recommended action in reinforcement learning is set as the evaluation criterion for the state transition from "deceleration" to "constant speed", the condition x2 for the state transition from "constant speed" to "constant speed" (maintaining the state) to be a recommended action in reinforcement learning is set as the evaluation criterion for the state transition from "constant speed" to "constant speed", the condition x3 for the state transition from "constant speed" to "deceleration" to be a recommended action in reinforcement learning is set as the evaluation criterion for the state transition from "constant speed" to "deceleration", and the condition x4 for the state transition from "deceleration" to "deceleration" (maintaining the state) to be a recommended action in reinforcement learning is set as the evaluation criterion for the state transition from "deceleration" to "deceleration".

[0109] The above conditions x1 to x4 will be specifically described with reference to FIG.

[0110] The condition x1 for the actual driving of the target vehicle E related to the state transition from "deceleration" to "constant speed" to be a recommended action in reinforcement learning is: TTC (Time To Colligion) > 3 seconds, and Lateral G<0.5G It includes at least one of the following. Here, TTC is the time until the target vehicle E collides with (reaches) an object. Objects that are the subject of TTC may include static obstacles, dynamic obstacles (pedestrians, other vehicles), objects indicating stopping places such as stop lines, stopping places that occur depending on the situation such as traffic lights, and stopping places that may occur due to the movement of emergency vehicles (fire engines, ambulances, patrol cars). Furthermore, lateral G is the lateral acceleration of the target vehicle E, and may represent a measure of an appropriate speed while traveling (cornering) on ​​a curved road. This lateral G can be used as information for considering the ride comfort of the target vehicle E. Condition x1 means that if this condition x1 (for example, TTC > 3 seconds) is met, the actual driving of the target vehicle E related to the state transition from "deceleration" to "constant speed" is an action that is recommended in reinforcement learning (should obtain a higher reward value), and this condition x1 is set as the evaluation criterion for the state transition from "deceleration" to "constant speed".

[0111] The condition x2 for the actual driving of the target vehicle E related to the state transition from "constant speed" to "constant speed" to be a recommended action in reinforcement learning is: TCC (Time to Colligion) > 3 seconds, and "Specified speed" It includes at least one of the following. "Specified speed" means that the traveling speed of the target vehicle E is within a specified speed range (traveling speed="specified speed"). Condition x2 means that if this condition x2 is met (TTC>3 seconds, etc.), the actual driving of the target vehicle E in relation to the state transition from "constant speed" to "constant speed" (driving maintaining "constant speed") is an action that is recommended in reinforcement learning (should obtain a higher reward value), and this condition 2 is set as the evaluation criterion for the state transition from "constant speed" to "constant speed".

[0112] The condition x3 for the actual driving of the target vehicle E related to the state transition from "constant speed" to "deceleration" to be a recommended behavior in reinforcement learning is: TCC(Time To Colligion)<3 seconds, DIST(Distance)<2m, Lateral G>0.5G, Running speed > specified speed, and Approaching emergency vehicle Contains at least one of the following: Here, DIST (Distance) represents the distance between the target vehicle E and the object that may cause the target vehicle to collide (stop). Condition x3 means that if this condition x3 is met (if TCC<3 seconds, if an emergency vehicle is approaching, etc.), the actual driving of the target vehicle E regarding the state transition from "constant speed" to "deceleration" is an action that is recommended in reinforcement learning (should obtain a higher reward value), and this condition x3 is set as the evaluation criterion for the state transition from "constant speed" to "deceleration".

[0113] The condition x4 for the actual driving of the target vehicle E related to the state transition from "Decelerate" to "Decelerate" to be a recommended action in reinforcement learning is: TCC (Time To Colligion) < 3 seconds, and DIST(Distance)<2m It includes at least one of the following. Condition x4 means that if this condition x4 is met (e.g., if DIST<2m), the actual driving of the target vehicle E in relation to the state transition from "Decelerate" to "Decelerate" (driving while maintaining "Decelerate") is an action that is recommended in reinforcement learning (should obtain a higher reward value), and this condition x4 is set as the evaluation criterion for the state transition from "Decelerate" to "Decelerate".

[0114] As specifically described for conditions x1 to x4, conditions x1 to x19 for the actual driving of the target vehicle E relating to state transitions between multiple (seven) driving modes, namely, "deceleration," "constant speed," "acceleration," "temporarily stopped," "stopped," "deadlock," and "taking resolution action," to be recommended actions in reinforcement learning are set as evaluation criteria for the state transitions.

[0115] In addition, in an environment (situation) in which a route (for example, represented by a series of waypoints WP) that the target vehicle E should travel is set in the evaluation device 14, for example, state transitions such as those shown in FIG. 21 are also defined.

[0116] 21, multiple state transitions (such as a state transition from "route following" to "off-route", a state transition from "off-route" to "returning", etc.) are defined between multiple (five) driving modes: a mode of driving while following a route (hereinafter referred to as "route following"), a mode of driving off a route (hereinafter referred to as "off-route"), a mode of driving to return to a route (hereinafter referred to as "returning"), a "deadlock", and "taking action to resolve" (resolving a "deadlock"). Similarly to the cases shown in FIGS. 19 and 20, conditions y1 to y12 for the actual driving of the target vehicle E related to the state transitions between these multiple (five) driving modes to be actions recommended in reinforcement learning are set as evaluation criteria for the state transitions.

[0117] For example, the condition y2 for the actual driving of the target vehicle E in the state transition from "route following" to "off-route" to be a recommended behavior in reinforcement learning may include the condition "TTC<α (which may mean that an obstacle is present in the immediate vicinity on the route of the target vehicle E)." In this case, the condition y2 including the condition "TTC<α" is the condition for the actual driving of the target vehicle E in the state transition from "route following" to "off-route" to be a recommended behavior in reinforcement learning, and if the condition "TTC<α seconds" is satisfied, it can be said that the actual driving of the target vehicle E in the state transition from "route following" to "off-route" is a recommended behavior (that should obtain a higher reward value) in reinforcement learning. Then, the condition y2 including the condition "TTC<α seconds" may be set as the evaluation criterion for the state transition from "route following" to "off-route."

[0118] Furthermore, for example, the condition y4 for the actual driving of the target vehicle E in the state transition from "off route" to "returning" to be a recommended behavior in reinforcement learning may include the condition "after completing the avoidance of a stopping obstacle, the TTC > β seconds on the route (which may mean that there are no obstacles immediately adjacent to the target vehicle E on the route)." In this case, the condition y4 including the condition "after completing the avoidance of a stopping obstacle, the TTC > β seconds on the route" is a condition for the actual driving of the target vehicle E in the state transition from "off route" to "returning" to be a recommended behavior in reinforcement learning, and if the condition "after completing the avoidance of a stopping obstacle, the TTC > β seconds on the route" is satisfied, the actual driving of the target vehicle E in the state transition from "off route" to "returning" to be a recommended behavior (that should obtain a higher reward value) in reinforcement learning. Then, the condition y4 including the condition "after completing the avoidance of a stopping obstacle, the TTC > β seconds on the route" may be set as an evaluation criterion for the state transition from "off route" to "returning."

[0119] 21, the state transition from "taking resolution action" to "taking resolution action" (maintenance of "taking resolution action"), which resolves a deadlock in which the target vehicle E is unable to move forward or stop in accordance with traffic rules, includes a state in which the target vehicle E is traveling backward in the wrong direction. The condition y12 for the behavior recommended in reinforcement learning for actual driving related to such a state in which the target vehicle E is traveling backward in the wrong direction, becomes the evaluation criterion for the state transition from "taking resolution action" to "taking resolution action".

[0120] 18 showing the function of the evaluation device 14 (a vehicle driving behavior evaluation device according to a second embodiment of the present invention), the evaluation device 14 acquires behavior information (position of squares in the grid 100: see FIG. 3) representing the driving behavior of the target vehicle E in the simulation image at a predetermined period (e.g., every 0.5 seconds) (S21: behavior input). Then, the evaluation device 14 determines which of a plurality of predefined state transitions (see FIGS. 19 and 21) a transition (Action) of the driving behavior (change in position of the target vehicle E on the grid 100) over a certain period (e.g., a period of three periods) corresponds to (S22: driving behavior determination unit).

[0121] The evaluation device 14 determines an evaluation score for the actual driving behavior of the target vehicle E based on the evaluation criteria set as described above for the state transitions related to the results of the determination (any of S23-1 to S23-n: evaluation, S24: evaluation point addition / normalization: evaluation point determination unit). Here, when it is determined that the transition of the actual driving behavior of the target vehicle E corresponds to two or more of the plurality of state transitions, for example, the state transition from "constant speed" to "constant speed" shown in Figures 19 and 20 and the state transition from "route following" to "route following" shown in Figure 21 (S22), two or more evaluation points (S23-i, ...) determined based on two or more evaluation criteria for those two or more state transitions are added together and further normalized so that the maximum value becomes "1" (S24: evaluation point addition / normalization), thereby determining a final evaluation score.

[0122] The evaluation device 14 provides the determined evaluation point as a reward value to the learning control unit 13 (S25: output reward value). Then, while the reinforcement learning process is being performed, the evaluation device 14 repeatedly executes the above-mentioned processes (S21 to S25) at a predetermined cycle, and each time a state transition is obtained from the traveling mode of the target vehicle E in the environment represented by the simulation image (change in the position of the squares on the grid 50) (S21, S22), it provides the determined evaluation point to the learning control unit 13 as a reward value.

[0123] 18, similarly to the first embodiment (see FIG. 4), the evaluation device 14 also has a function (S31 to S33: learning progress evaluation point determination unit) of determining an evaluation score representing an evaluation of the driving behavior of the target vehicle E based on the history (episodes) of repeated driving of the target vehicle E until the movement objective of the target vehicle E in the simulator 20 is achieved in reinforcement learning and until the subtask is completed. Then, the sum of the evaluation score (S22 to S24) determined based on each of the evaluation criteria described above and the evaluation score (S31 to S33: learning progress evaluation point determination unit) determined based on the driving history of the target vehicle E is provided to the learning control unit 13 as a reward value (S25).

[0124] A more specific operation of the evaluation device 14 will be described using, for example, four state transitions between two driving modes, "deceleration" and "constant speed," as shown in Fig. 20. Here, the evaluation score table shown in Fig. 22A is used to determine the evaluation score based on the condition xi (i = 1 to 19) as the judgment criterion for each state transition (state transition A) shown in Fig. 20 together with Fig. 19.

[0125] 20, for example, if the position change (Action) of the target vehicle E on the grid 100 for three periods transitions from 3 grids to 2 grids to 2 grids (one grid is, for example, 30 centimeters), the transition of the target vehicle E's traveling mode is determined to be a state transition from "deceleration" (Action1: 3 grids to 2 grids) to "constant speed" (Action2: 2 grids to 2 grids) (S21, S22 in FIG. 18: traveling mode determination unit). Then, based on the evaluation criterion (condition x1) set for the state transition from "deceleration" to "constant speed", an evaluation score for the transition of the actual traveling mode of the target vehicle E (Action: 3 grids to 2 grids to 2 grids) is determined (S22-x in FIG. 18: corresponding to the item of the state transition from "deceleration" to "constant speed").

[0126] Specifically, it is determined whether or not a condition x1 (for example, TCC>3 seconds) serving as an evaluation criterion is satisfied (determination unit). If the condition x1 is satisfied, for example, if the distance relationship between the target vehicle E and the obstacle is TCC>3 seconds, the evaluation score is set to a predetermined value, for example, "1" point (marked with a circle in FIG. 22A). If the condition x1 is not satisfied, for example, if the distance relationship between the target vehicle E and the obstacle is not TCC>3 seconds, the evaluation score is set to a value lower than "1", for example, "0" point (marked with an x ​​in FIG. 22A) (value determination unit).

[0127] 20, for example, if the position change (Action) of the target vehicle E on the grid 100 for three cycles transitions from 3 grids → 3 grids → 3 grids, the transition of the target vehicle E's traveling mode is determined to be a state transition from "constant speed" (Action1: 3 grids → 3 grids) to "constant speed" (Action2: 3 grids → 3 grids) (S21, S22 in FIG. 18). Then, based on the evaluation criterion (condition x2) set for the state transition from "constant speed" to "constant speed", an evaluation score is determined for the transition of the actual traveling mode of the target vehicle E (Action: 3 grids → 3 grids → 3 grids) (S22-x in FIG. 18: corresponds to the item of the state transition from "constant speed" to "constant speed").

[0128] Specifically, it is determined whether or not a condition x2 (for example, TCC>3 seconds) serving as an evaluation criterion is satisfied (determination unit). If the condition x2 is satisfied, for example, if the distance relationship between the target vehicle E and the obstacle is TCC>3 seconds, the evaluation score is set to "1" (marked with a circle in FIG. 22A), and if the condition x2 is not satisfied, for example, if the distance relationship between the target vehicle E and the obstacle is not TCC>3 seconds, the evaluation score is set to "0" (marked with an x ​​in FIG. 22A) (value determination unit).

[0129] 20, for example, if the position change (Action) of the target vehicle E on the grid 100 for three cycles transitions from 3 grids to 3 grids to 2 grids, the transition of the target vehicle E's traveling mode is determined to be a state transition from "constant speed" (Action1: 3 grids to 3 grids) to "deceleration" (Action2: 3 grids to 2 grids) (S21, S22 in FIG. 18). Then, based on the evaluation criterion (condition x3) set for the state transition from "constant speed" to "deceleration", an evaluation score is determined for the transition of the actual traveling mode of the target vehicle E (Action: 3 grids to 3 grids to 2 grids) (S22-x in FIG. 18: corresponds to the item of the state transition from "constant speed" to "deceleration").

[0130] Specifically, it is determined whether or not a condition x3 (for example, TTC<3 seconds) serving as an evaluation criterion is satisfied (determination unit). If the condition x3 is satisfied, for example, if the distance relationship between the target vehicle E and an obstacle is TCC<3 seconds, or if an emergency vehicle is approaching, etc., the evaluation score is set to "1" (circle in FIG. 22A). If the condition x3 is not satisfied, for example, if the distance relationship between the target vehicle E and an obstacle is not TCC<3 seconds, or if an emergency vehicle is not approaching, etc., the evaluation score is set to "0" (cross in FIG. 22A) (value determination unit).

[0131] 20, for example, if the position change (Action) of the target vehicle E on the grid 100 for three cycles transitions from 3 grids to 2 grids to 1 grid, the transition of the target vehicle E's traveling mode is determined to be a state transition from "deceleration" (Action1: 3 grids to 2 grids) to "deceleration" (Action2: 2 grids to 1 grid) (S21, S22 in FIG. 18). Then, based on the evaluation criterion (condition x4) set for the state transition from "deceleration" to "deceleration", an evaluation score is determined for the transition of the actual traveling mode of the target vehicle E (Action: 3 grids to 2 grids to 1 grid) (S22-x in FIG. 18: corresponds to the item of the state transition from "deceleration" to "deceleration").

[0132] Specifically, it is determined whether or not condition x4 (for example, TCC<3 seconds) as an evaluation criterion is satisfied (determination unit). If condition x2 is satisfied, for example, if the distance relationship between the target vehicle E and the obstacle is TCC<3 seconds, the evaluation score is set to "1" point (marked with a circle in FIG. 22A), and if condition x4 is not satisfied, for example, if the distance relationship between the target vehicle E and the obstacle is not TCC<3 seconds, the evaluation score is set to "0" point (marked with an x ​​in FIG. 22A) (value determination unit).

[0133] Furthermore, as described above, in an environment (situation) in which a route to be traveled by the target vehicle E is set, 12 state transitions (state transition B) are defined between five driving modes as shown in Fig. 21. In this case, the evaluation score table shown in Fig. 22B is used to determine the evaluation score based on the condition yi (i = 1 to 12) as the evaluation criterion for those state transitions.

[0134] 21, as described above, for example, for a state transition from "route following" to "off-route," a condition y2 of "TTC<α seconds (an obstacle may be present in the immediate vicinity on the route of the target vehicle E)" may be set as the evaluation criterion. Here, for example, if the distance between the traveling position of the target vehicle E and the route transitions (Action) from a state within two grids to a state where the distance exceeds two grids within a predetermined time interval, the transition of the target vehicle E's traveling mode is determined to be from "route following" to "off-route" (S21, S22 in FIG. 18). Then, an evaluation score for the transition (Action) of the actual traveling mode of the target vehicle E is determined based on the evaluation criterion (condition y2) set for the state transition from "route following" to "off-route" (S22-x in FIG. 18: corresponding to the item of the state transition from "route following" to "off-route").

[0135] Specifically, it is determined whether or not a condition y2 (for example, TTC<α seconds) serving as an evaluation criterion is satisfied (determination unit). If the condition y2 is satisfied, for example, if the positional relationship between the target vehicle E and an obstacle is TCC<α seconds (the obstacle may be present in the immediate vicinity of the target vehicle E), the evaluation score is set to "1" (marked with a circle in FIG. 22B). If the condition y2 is not satisfied, for example, if the positional relationship between the target vehicle E and the obstacle is not TCC<α seconds, the evaluation score is set to "0" (marked with an x ​​in FIG. 22B) (value determination unit).

[0136] 21, as described above, for example, for a state transition from "off route" to "returning," a condition y4 of "TTC>β seconds (no obstacles are present in the immediate vicinity on the route of the target vehicle E)" may be set as the evaluation criterion. Here, for example, if the distance between the traveling position of the target vehicle E and the route transitions (Action) from a state where it exceeds three grids to a state where the distance is within three grids within a predetermined time interval, the transition of the target vehicle E's traveling mode is determined to be from "off route" to "returning" (S21, S22 in FIG. 18). Then, an evaluation score for the transition (Action) of the actual traveling mode of the target vehicle E is determined based on the evaluation criterion (condition y4) set for the state transition from "off route" to "returning" (S22-x in FIG. 18: corresponding to the item of the state transition from "off route" to "returning").

[0137] Specifically, it is determined whether or not a condition y4 (for example, TTC>β seconds) serving as an evaluation criterion is satisfied (determination unit). If the condition y4 is satisfied, for example, if the positional relationship between the target vehicle E and an obstacle is TCC>β seconds (no obstacle exists in the immediate vicinity on the route of the target vehicle E), the evaluation score is set to "1" (marked with a circle in FIG. 22B). If the condition y4 is not satisfied, for example, if the positional relationship between the target vehicle E and an obstacle is not TCC>α seconds, the evaluation score is set to "0" (marked with an x ​​in FIG. 22B) (value determination unit).

[0138] Furthermore, in the evaluation device 14, as shown in the evaluation point table shown in Figure 23, the evaluation points (see Figure 22A) determined for the item of the state transition between driving modes (state transition A) shown in Figure 19 (Figure 20) and the evaluation point (see Figure 22B) determined for the item of the state transition between driving modes (state transition B) shown in Figure 21 are added together and normalized so that the maximum value becomes "1" point.As a result, depending on the evaluation points (circle, cross) for the two items, the evaluation reduction to be provided to the learning control unit 13 as a reward value is determined to be either "1" point, "0.5" point, or "0" point.

[0139] In addition, in the evaluation device 14 of the vehicle driving behavior evaluation device according to the second embodiment of the present invention, which has the function shown in FIG. 18, as in the first embodiment, an evaluation point can be determined, for example, as shown in FIG. 9 or FIG. 15, for the end of movement of the target vehicle E in the reinforcement learning process.

[0140] In the reinforcement learning device 10, in a situation where a route that the target vehicle E should travel on the road R is set, trial and error is repeated to achieve the goal of the target vehicle E traveling along the route as much as possible and reaching the destination. As a result, in the simulation image, the target vehicle E moves along various trajectories and comes to a stop. Then, with respect to the transition of the target vehicle E's traveling mode along each of these various movement trajectories, the evaluation device 14 provides the learning control unit 13 with an evaluation score determined based on the evaluation criteria set for the state transitions as described above as a reward value.

[0141] In the process of repeating the above-described processing in reinforcement learning, the policy (weighting coefficients of the neural network NN) of the controller 12 is successively updated under the control of the learning control unit 13 so as to maximize the reward value (evaluation point) finally obtained. Then, in an environment (situation) in which a route to be traveled by the target vehicle E is set on the road R, a policy (policy: weighting coefficients of the neural network NN) of the controller 12 is obtained that can provide the simulator 20 with movement instructions that enable the target vehicle E to travel along a movement trajectory that can reach the destination while maintaining the travel position of the target vehicle E on the route as much as possible.

[0142] According to the evaluation device 14 (the vehicle driving mode evaluation device according to the second embodiment of the present invention) as described above, in the process of reinforcement learning regarding the driving of the target vehicle E using the simulator 20, the evaluation points representing the transition of the driving mode of the target vehicle E are determined based on the evaluation criteria set in advance for various state transitions corresponding to the transition of the driving mode. Therefore, it becomes possible to appropriately evaluate the driving modes of vehicles in various situations in reinforcement learning. As a result, a controller 12 (neural network NN) with good performance can be constructed by reinforcement learning.

[0143] Also, similar to the first embodiment, the evaluation points for a plurality of items are added and normalized to obtain the final evaluation points. Therefore, the evaluation of the driving mode of the target vehicle under the various situations can be represented with a unified numerical sense.

[0144] In the evaluation device 14 according to the second embodiment, the state transition, which is the transition of the driving mode, is adopted as the concept of the driving mode. However, it is of course possible to associate evaluation criteria with driving modes such as constant-speed driving, accelerating driving, decelerating driving, stopping, driving on the route, driving off the route, etc. In this case, for example, the evaluation points for the actual driving mode of the target vehicle E can be determined based on the evaluation point table shown in FIGS. 24A and 24B.

[0145] In the case shown in FIG. 24A, a stop mode (hereinafter referred to as "stop"), a deceleration mode (hereinafter referred to as "deceleration"), and a normal driving mode (hereinafter referred to as "driving") are defined.

[0146] For each of the driving modes of "stop" and "deceleration", the following two evaluation criteria are set. First criterion: 1 second < TTC ≤ 3 seconds or Distance ≤ 0.6 m Second criterion: 3 seconds < TTC ≤ 6 seconds or Distance ≤ 0.9 m are included. Here, the first criterion is a condition for recommending "stopping" of the target vehicle E in reinforcement learning, and the second criterion is a condition for recommending "deceleration" of the target vehicle in reinforcement learning.

[0147] When the actual state of the target vehicle E (≦0.1 km / h) is determined to be "stopped," first, it is determined whether or not the first criterion is met (determination unit). If the first criterion is met, the evaluation score a for the actual driving state of the target vehicle E is determined to be "1" (value determination unit). On the other hand, when the first criterion is not met, the evaluation score a is determined to be a value representing an evaluation lower than the evaluation represented by "1" (determination unit). Specifically, if the first criterion is not met but the second criterion is met (determination unit), the evaluation score a is determined to be "0.5," and if neither the first criterion nor the second criterion is met (determination unit: else), the evaluation score a is determined to be "0" (value determination unit).

[0148] When the actual driving behavior of the target vehicle E (≦4 km / h) is determined to be "deceleration," first, it is determined whether or not the second criterion is satisfied (determination unit). If the second criterion is satisfied, the evaluation score a for the actual driving behavior of the target vehicle E is determined to be "1" (value determination unit). On the other hand, when the second criterion is not satisfied, the evaluation score a is determined to be a value representing an evaluation lower than the evaluation represented by "1." Specifically, if the second criterion is not satisfied but the first criterion is satisfied (determination unit), the evaluation score a is determined to be "0.5" points, and if neither the second criterion nor the first criterion is satisfied (determination unit: else), the evaluation score a is determined to be "0" points (value determination unit).

[0149] For "normal driving" driving mode, TTC>6 seconds and Distance>0.9m is set as the evaluation criterion. This evaluation criterion is the condition for recommending "normal driving" of the target vehicle E in reinforcement learning.

[0150] If the actual driving behavior of the target vehicle E (>4 km / h) is determined to be "normal driving," it is determined whether or not the above evaluation criteria (conditions) are met (determination unit). If the evaluation criteria are met, the evaluation score a for the actual driving behavior of the target vehicle E is determined to be "1," and if it is determined that the above evaluation criteria are not met (determination unit: else), the evaluation score a is determined to be "0," which is lower than "1."

[0151] 24B shows an environment (situation) in which a route is set for the target vehicle E. In this case, a state in which the target vehicle E travels on the route (hereinafter referred to as "on the route") and a state in which the target vehicle E travels off the route (hereinafter referred to as "off the route") are defined.

[0152] For each of "on route" and "off route", There is an obstacle on the path is set as the evaluation criterion. Here, this evaluation criterion is a condition for recommending "off-route" for the target vehicle E in reinforcement learning. Note that the condition (evaluation criterion) that contradicts this evaluation criterion, "there are no obstacles on the route," is a condition for recommending "on-route" for the target vehicle E in reinforcement learning.

[0153] At a predetermined interval (e.g., 5 seconds), it is determined whether the target vehicle E is on the set route (whether it is within a predetermined range (e.g., 2 grids) from the route). During this process, if the actual driving mode of the target vehicle E is determined to be "on the route," and an evaluation criterion that contradicts the above evaluation criterion, "there are no obstacles on the route," is met (determination unit: no), the evaluation score b of the actual driving mode of the target vehicle E traveling on the route is determined to be "1" points. On the other hand, if an evaluation criterion that contradicts the above evaluation criterion is not met, that is, if the above evaluation criterion is met (determination unit: yes), the evaluation score b is determined to be a value that represents an evaluation lower than the evaluation represented by "1" points, for example, "0.5" points (value determination unit).

[0154] When the actual driving manner of the target vehicle E is determined to be "off-route", if the above evaluation criteria are met (determination unit: yes), the evaluation score b of the actual driving manner of the target vehicle E traveling off-route is determined to be "1". On the other hand, when the above evaluation criteria are met (determination unit: no), the evaluation score b is determined to be "0", which represents an evaluation lower than the evaluation score represented by "1".

[0155] Using the evaluation points a and b determined based on the evaluation point tables shown in FIGS. 24A and 24B, the final evaluation point is calculated by the following method, instead of the above-mentioned addition and normalization: Evaluation score = base score x evaluation score a x evaluation score b is determined in accordance with The base score is an evaluation score component that can be determined by factors other than the driving mode of the target vehicle in the simulation image (corresponding to evaluation score a) and the route setting situation (corresponding to evaluation score b), and can be determined by various conditions in reinforcement learning, etc. Also, this base score may be a constant.

[0156] The method for determining the final evaluation score from the evaluation scores for multiple items may be other than the above-mentioned addition and normalization method or the method according to the above formula, as long as appropriate reinforcement learning is possible.

[0157] As described above, when evaluation criteria are set for driving behavior rather than state transition, in the process of reinforcement learning regarding the driving of the target vehicle E using the simulator 20, an evaluation score representing an evaluation of the driving behavior of the target vehicle E is determined based on the evaluation criteria set for that driving behavior among the evaluation criteria set in advance for various driving behaviors, so that the driving behavior of the vehicle in various situations can be appropriately evaluated in reinforcement learning. As a result, a controller 12 (neural network NN) with good performance can be constructed by reinforcement learning.

[0158] Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. The novel embodiments described above can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. [Industrial Applicability]

[0159] The vehicle driving behavior evaluation device according to the present invention has the advantage of being able to appropriately evaluate the driving behavior of a vehicle in various situations, and is useful as a vehicle driving behavior evaluation device that evaluates the driving behavior of a target vehicle in a simulator in reinforcement learning regarding the driving of the target vehicle using the simulator. [Explanation of symbols]

[0160] 10 Reinforcement learning device 11 Main control unit 12 Controller 13 Learning control unit 14 Evaluation equipment 20 Simulator 21 Simulation control section 22 Image processing section 50 road

Claims

1. A vehicle driving behavior evaluation device that evaluates a driving behavior of a target vehicle in a simulator in reinforcement learning regarding the driving of the target vehicle using the simulator, an evaluation standard is set for each of a plurality of situations that can be represented by the simulator, regarding the driving behavior of the target vehicle under the situation; a criterion selection unit that selects an evaluation criterion corresponding to a situation represented by the simulator during the reinforcement learning process from the plurality of evaluation criteria set for the plurality of situations; and an evaluation point determination unit that determines an evaluation point representing an evaluation of the driving behavior of the target vehicle under the situation represented by the simulator during the reinforcement learning process, based on the evaluation criteria selected by the criteria selection unit.

2. a condition for the driving mode of the target vehicle under each of a plurality of situations that can be represented by the simulator to be an action recommended in the reinforcement learning is set as the evaluation criterion; The evaluation point determination unit a determination unit that determines whether or not a driving behavior of the target vehicle in a situation represented by the simulator during the reinforcement learning process satisfies a condition for the target vehicle to be a recommended behavior in the situation; 2. The vehicle driving behavior evaluation device according to claim 1, further comprising: a value determination unit that, when the determination unit determines that the conditions for the recommended behavior are satisfied, determines the evaluation score to a predetermined value; and, when the determination unit determines that the conditions for the recommended behavior are not satisfied, determines the evaluation score to a value representing an evaluation lower than the evaluation represented by the predetermined value.

3. 2. The vehicle driving behavior evaluation device according to claim 1, wherein the plurality of situations that can be represented by the simulator include a situation in which a route on which the target vehicle should travel is set on a road in the simulator, and an evaluation criterion is set for that situation according to a distance by which the target vehicle deviates from the route.

4. 2. The vehicle driving behavior evaluation device according to claim 1, wherein the plurality of situations that can be represented by the simulator include a situation in which the target vehicle is traveling at a constant speed in the simulator, and an evaluation criterion is set for that situation according to the traveling speed of the target vehicle.

5. 2. The vehicle driving behavior evaluation device according to claim 1, wherein the plurality of situations that can be represented by the simulator include a situation in which an obstacle is present on a road along which the target vehicle is to travel in the simulator, and an evaluation criterion is set for that situation according to a distance between the target vehicle and the obstacle.

6. 2. The vehicle driving behavior evaluation device according to claim 1, wherein the plurality of situations that can be represented by the simulator include a situation in which the target vehicle has to stop at a place in the simulator based on traffic rules, and an evaluation criterion is set for the situation according to a positional relationship between the target vehicle and the place where the target vehicle has to stop.

7. 2. The vehicle driving behavior evaluation device according to claim 1, further comprising a learning progress evaluation point determination unit that determines an evaluation point representing an evaluation of the driving behavior of the target vehicle based on the driving history of the target vehicle in the simulator up to a certain point during the reinforcement learning process.

8. A vehicle driving behavior evaluation device that evaluates a driving behavior of a target vehicle in a simulator in reinforcement learning regarding the driving of the target vehicle using the simulator, A plurality of driving modes that the target vehicle traveling in the simulator can take are defined, and an evaluation criterion is set for each of the defined plurality of driving modes; a driving mode determination unit that determines which of the plurality of defined driving modes the driving mode of the target vehicle traveling in the simulator during the reinforcement learning process is; and an evaluation score determination unit that determines an evaluation score representing an evaluation of the driving behavior of the target vehicle based on an evaluation criterion corresponding to the driving behavior related to the judgment result by the driving behavior judgment unit.

9. 9. The vehicle driving behavior evaluation device according to claim 8, wherein the defined driving behaviors include a constant speed driving behavior, an accelerating driving behavior, a decelerating driving behavior, and a stopped driving behavior.

10. 9. The vehicle driving behavior evaluation device according to claim 8, wherein a state transition of the vehicle driving behavior is defined for each of the plurality of defined driving behaviors.

11. a condition for each of the defined driving behaviors to be a recommended behavior in the reinforcement learning is set as an evaluation criterion corresponding to the defined driving behavior; The evaluation point determination unit a determination unit that determines whether or not a condition serving as the evaluation criterion corresponding to a driving mode related to a determination result by the driving mode determination unit is satisfied during the reinforcement learning process; and 9. The vehicle driving behavior evaluation device according to claim 8, further comprising: a value determination unit that determines the evaluation score to a predetermined value when the condition determination unit determines that the condition is satisfied, and that determines the evaluation score to a value representing an evaluation lower than the predetermined value when the condition determination unit determines that the condition is not satisfied.

12. 9. The vehicle driving behavior evaluation device according to claim 8, further comprising a learning progress evaluation point determination unit that determines an evaluation point based on a driving history of the target vehicle in the simulator up to a certain point during the reinforcement learning process.

Images