Vehicle control device, reinforcement learning method, reinforcement learning device, and program

Abstract

To provide a vehicle control device that can perform optimal merging control in response to changing conditions. [Solution] The vehicle control device 1 that performs merging control of vehicle 2 includes an ambient environment recognition unit 41 that recognizes the ambient environment around vehicle 2, a vehicle recognition unit 42 that recognizes the state of the vehicle 2, a driving plan unit 43 that sequentially acquires the target merging position by sequentially inputting the ambient environment and the state of the vehicle to a learned model 45 that outputs a target merging position in response to inputs of the ambient environment and the state of the vehicle, and creates a driving plan for vehicle 2 based on the latest value of the target merging position, and a driving control unit 44 that controls the acceleration, deceleration and steering of vehicle 2 based on the driving plan without relying on the operation of the occupants.

Description

[Technical Field]

[0001] The present invention relates to a vehicle control device and a reinforcement learning method. [Background technology]

[0002] In recent years, efforts to provide sustainable transportation systems that take into account vulnerable groups among transportation users have become increasingly active. To achieve this, research and development are being conducted on driver assistance technologies and autonomous driving technologies to further improve the safety and convenience of transportation.

[0003] Patent Document 1 discloses a vehicle control device for smoothly merging vehicles traveling on a secondary track into a merging area where a secondary track merges with a main track. The vehicle control device acquires the positions of multiple vehicles traveling on the main track, sets a target merging position based on the position of each vehicle, and automatically controls the acceleration and deceleration of the vehicles toward the target merging position. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2017-165197 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, vehicles traveling on the main line can sometimes make unexpected movements. Therefore, a target merging position set at a certain point in time may become unsuitable for merging a few seconds later.

[0006] One aspect of the present invention, in view of the above background, aims to provide a vehicle control device that can perform optimal merging control in response to changing circumstances. It also provides a reinforcement learning method for a trained model used in the vehicle control device. Through this, the present invention aims to contribute to the development of sustainable transportation systems. [Means for solving the problem]

[0007] To solve the above problems, one aspect of the present invention provides a vehicle control device for merging vehicles, comprising: an ambient environment recognition unit that recognizes the surrounding environment of the vehicle; a vehicle recognition unit that recognizes the state of the vehicle; a driving plan unit that sequentially acquires the target merging position by sequentially inputting the ambient environment and the state of the vehicle into a learned model that outputs a target merging position in response to inputs of the ambient environment and the state of the vehicle, and creates a driving plan for the vehicle based on the latest value of the target merging position; and a driving control unit that controls the acceleration, deceleration, and steering of the vehicle based on the driving plan without relying on the operation of the occupants.

[0008] Another aspect of the present invention is a reinforcement learning method performed by a computer to generate a trained model that outputs a target merging position in response to inputs including the surrounding environment of a vehicle and the vehicle state, the method comprising: acquiring state information including the surrounding environment and the vehicle state from a simulator; generating an action policy in a neural network using the state information as input; performing an action based on the action policy; receiving a reward and the next state information; updating the parameters of the neural network to adjust the action policy based on the reward and the state information; and determining the reward based on a plurality of auxiliary rewards set based on a plurality of different objectives.

[0009] Another aspect of the present invention is a reinforcement learning device that generates a trained model that outputs a target merging position in response to inputs including the surrounding environment of a vehicle and the vehicle state, the device comprising: a simulator that outputs state information including the surrounding environment and the vehicle state; and an agent that takes the state information as input, generates an action policy using a neural network, performs an action based on the action policy, receives a reward and the next state information, and updates the parameters of the neural network and adjusts the action policy based on the reward and the state information, the reward being determined based on a plurality of auxiliary rewards set based on a plurality of different objectives.

[0010] Another aspect of the present invention is a program for causing a computer to execute a reinforcement learning method that generates a trained model that outputs a target merging position in response to inputs including the surrounding environment of a vehicle and the vehicle state, wherein the program causes a computer to acquire state information including the surrounding environment and the vehicle state from a simulator, generates an action plan using a neural network as input, executes an action based on the action plan, receives a reward and the next state information, updates the parameters of the neural network and adjusts the action plan based on the reward and the state information, and the reward is determined based on a plurality of auxiliary rewards set based on a plurality of different objectives. [Effects of the Invention]

[0011] According to the above embodiments, a vehicle control device capable of performing optimal merging control in response to changes in circumstances can be provided. Furthermore, a reinforcement learning device, a reinforcement learning method, and a program for learning a trained model used in the vehicle control device can be provided. [Brief explanation of the drawing]

[0012] [Figure 1] Configuration diagram of a vehicle control device according to this embodiment [Figure 2] Diagram explaining the merging area [Figure 3]Flowchart of merging control executed by the vehicle control device [Figure 4] Configuration diagram of the learning device according to the embodiment [Figure 5] Diagram conceptually showing the model structure of DQN [Figure 6] Graph showing the first reward [Figure 7] Graph showing the third reward

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments of a vehicle control device, a reinforcement learning method, a reinforcement learning device, and a program will be described with reference to the drawings.

[0014] As shown in FIG. 1, the vehicle control device 1 is provided in the vehicle 2. The vehicle 2 is preferably, for example, a four-wheeled automobile. The vehicle 2 is an autonomous driving vehicle or a vehicle with a driving support function.

[0015] The vehicle 2 has a propulsion device 3, a braking device 4, and a steering device 5. The propulsion device 3 is a device that imparts a driving force to the vehicle 2 and includes, for example, a power source and a transmission. The power source has at least one of an internal combustion engine such as a gasoline engine or a diesel engine and an electric motor. The braking device 4 is a device that imparts a braking force to the vehicle 2 and includes, for example, a brake caliper that presses a pad against a brake rotor and an electric cylinder that supplies hydraulic pressure to the brake caliper. The steering device 5 is a device for changing the steering angle of the wheels and has, for example, a rack and pinion mechanism for steering the wheels and an electric motor for driving the rack and pinion mechanism. The propulsion device 3, the braking device 4, and the steering device 5 are controlled by the vehicle control device 1.

[0016] The vehicle 2 has an external recognition device 7. The external recognition device 7 is a device that detects an object outside the vehicle. The external recognition device 7 is a sensor that captures electromagnetic waves or light from the surroundings of the vehicle 2 to detect an object outside the vehicle. The external recognition device 7 includes, for example, a radar 8, a lidar 9 (LIDAR), and an external camera 10.

[0017] Vehicle 2 has a vehicle sensor 12. The vehicle sensor 12 includes a vehicle speed sensor 13 for detecting the speed of vehicle 2 and an acceleration sensor 14 for detecting acceleration. The vehicle sensor 12 may also include a yaw rate sensor for detecting angular velocity around the vertical axis, an orientation sensor for detecting the orientation of vehicle 2, and the like.

[0018] Vehicle 2 is equipped with a communication device 15, a navigation device 16, a driving control device 17, and an HMI 19 (Human Machine Interface). The communication device 15 mediates communication between the vehicle control device 1 and the navigation device 16 and surrounding vehicles 200 (see Figure 2) located outside the vehicle and a server.

[0019] The navigation device 16 acquires the current position of the vehicle 2 and provides route guidance to the destination. The navigation device 16 may include a GNSS receiver 26, a map storage unit 27, a navigation interface 28, and a route determination unit 29. The GNSS receiver 26 determines the position (latitude and longitude) of the vehicle 2 based on signals received from artificial satellites (positioning satellites). The map storage unit 27 is composed of a known storage device such as flash memory or a hard disk and stores map information. The navigation interface 28 accepts input such as destinations from the occupants and presents various information to the occupants by display and voice. The navigation interface 28 may be, for example, a touch panel display.

[0020] The driving control device 17 receives input operations performed by the occupant (driver) to control the vehicle 2. The driving control device 17 includes a steering wheel 21, an accelerator pedal 22, and a brake pedal 23. The driving control device 17 may also include a shift lever, a parking brake lever, etc. Each driving control device 17 is equipped with a sensor to detect the amount of operation. The driving control device 17 outputs a signal indicating the amount of operation to the vehicle control device 1.

[0021] The HMI19 provides the crew with various information through displays and voice prompts, and also accepts input operations from the crew. The HMI19 may be a touch panel display including liquid crystal or organic EL displays.

[0022] The vehicle control device 1 is a computer having a processor 31 and a memory 32 that is communicatively connected to the processor 31. The processor 31 may include at least one of the following as its core: a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a RISC (Reduced Instruction Set Computer). The memory 32 stores control programs executed by the processor 31 and various data. The memory 32 may include at least one of volatile memory and non-volatile memory. The volatile memory may be, for example, DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The non-volatile memory may be an SSD (Solid State Drive), flash memory, magnetic disk storage device, or optical disk storage device. At least a part of the vehicle control device 1 may be implemented by hardware such as an LSI (Large Scale Integration), ASIC (application specific integrated circuit), or FPGA (field-programmable gate array), or by a combination of software and hardware. The vehicle control device 1 may be composed of a single piece of hardware, or it may be composed of multiple pieces of hardware that can communicate with each other. Part of the vehicle control device 1 may be composed of an external server located outside the vehicle 2.

[0023] The processor 31 implements various applications by executing programs stored in memory 32. Programs may be stored on removable recordable media such as DVDs or CD-ROMs, and installed in memory 32 when the recordable media is read by a reader. Alternatively, programs may be downloaded to and installed in memory 32 via a communication network such as the internet.

[0024] The processor 31 functions as an ambient environment recognition unit 41, a vehicle recognition unit 42, a driving plan unit 43, and a driving control unit 44 by executing programs stored in the memory 32.

[0025] The surrounding environment recognition unit 41 recognizes the surrounding environment of the vehicle 2. Based on the detection results of the external environment recognition device 7, the surrounding environment recognition unit 41 recognizes the surrounding environment (external world), including obstacles located around the vehicle 2, the shape of the road, the presence or absence of sidewalks, road markings, etc. Obstacles include, for example, guardrails, utility poles, surrounding vehicles 200, and people such as pedestrians. The surrounding environment recognition unit 41 can acquire the position, speed, acceleration, and other states of the surrounding vehicles 200 from the detection results of the external environment recognition device 7. In the merging area 100 shown in Figure 2, the surrounding environment recognition unit 41 recognizes the merging area 102C and the positions and speeds of the multiple surrounding vehicles 200 as part of the surrounding environment.

[0026] As shown in Figure 2, the merging area 100 has a main lane 101 and a merging lane 102 that merges with the main lane 101. The main lane 101 may be the end lane of a main line 104 that includes multiple lanes. Note that the main line 104 may consist only of the main lane 101. In the main lane 101 and the merging lane 102, the direction of travel of vehicle 2 is forward. The main lane 101 may extend in a straight line or curve. The merging area 100 may constitute part of a highway.

[0027] The merging lane 102 has, in order, a first section 102A, a second section 102B, and a merging area 102C facing forward. The first section 102A is separated from the main lane 101 by a hard nose 105. The first section 102A may be positioned away from the main lane 101. The first section 102A may also be inclined with respect to the main lane 101. The side of the front end of the first section 102A may be connected to the side of the main lane 101. The hard nose 105 may be formed by a structure such as a wall or a guardrail.

[0028] The second section 102B extends along the main lane 101. The road surface of the second section 102B is preferably connected to the road surface of the main lane 101 in the lateral direction. A restrictor 107 is provided at the boundary between the second section 102B of the merging lane 102 and the main lane 101. The restrictor 107 restricts the movement of vehicles 2 from the merging lane 102 to the main lane 101. The restrictor 107 may be continuous along the boundary between the main lane 101 and the merging lane 102, or it may be provided intermittently. The restrictor 107 may be composed of structures such as multiple poles, pylons, road studs, and curbs. Guard ropes may be stretched between multiple poles. The restrictor 107 is also called a soft nose. The front end of the restrictor 107 is called the restrictor end 107A.

[0029] The merging area 102C extends along the main lane 101. The merging area 102C constitutes the end of the merging lane 102. Vehicle 2 can change lanes from the merging lane 102 to the main lane 101, i.e., merge, in the merging area 102C. The beginning of the merging area 102C is preferably the end of the regulatory body 107A. The end of the merging area 102C is preferably the position where the width of the merging lane 102 begins to narrow.

[0030] The surrounding environment recognition unit 41 acquires the positions of the start and end of the merging area 102C, and the positions and speeds of each of the multiple surrounding vehicles 200 traveling in the main lane 101. The position of each surrounding vehicle 200 is preferably a position relative to the start of the merging area 102C. However, the reference position for each surrounding vehicle 200 is not limited to the start of the merging area 102C and may be changed as desired. For example, the reference position may be the tip of the hard nose 105, the end of the merging area 102C, or the midpoint between the start and end of the merging area 102C. The surrounding environment recognition unit 41 recognizes all surrounding vehicles 200 located within a predetermined range in front of and behind vehicle 2.

[0031] The vehicle recognition unit 42 recognizes the vehicle status, which is the state of vehicle 2 (the vehicle itself). The vehicle status includes the position of vehicle 2 and the speed of vehicle 2. The position of vehicle 2 may be a position relative to the beginning of the merging area 102C. The vehicle recognition unit 42 may acquire the speed of vehicle 2 based on the signal from the vehicle speed sensor 13. The vehicle recognition unit 42 may recognize the position of the restrictor end 107A based on the detection result of the external environment recognition device 7, and recognize the position of vehicle 2 relative to the beginning of the merging area 102C based on the position of the restrictor end 107A. Alternatively, the vehicle recognition unit 42 may acquire the position of vehicle 2 based on the GNSS signal received by the GNSS receiver 26 and the position of vehicle 2 relative to the beginning of the merging area 102C based on map information.

[0032] The driving plan unit 43 creates a driving plan for vehicle 2. The driving plan unit 43 sequentially creates a driving plan for automatically driving vehicle 2 along the route. More specifically, the driving plan unit 43 first determines the automated driving events necessary for vehicle 2 to travel along the target lane determined by the route determination unit 29 without coming into contact with any obstacles. Based on the determined events, the driving plan unit 43 generates a target trajectory that vehicle 2 should travel in the future. The target trajectory is a sequence of trajectory points that vehicle 2 should reach at each time point. The driving plan unit 43 may generate a target trajectory, target speed, and target acceleration for each event. Automated driving events include constant speed driving events, follow driving events, lane change events, branching events, merging events, overtaking events, etc.

[0033] The driving plan unit 43 generates a merging event when vehicle 2 is traveling in the merging lane 102. The driving plan unit 43 may determine that vehicle 2 is traveling in the merging lane 102 based on the vehicle 2's position and map information.

[0034] The driving plan unit 43 sequentially inputs the surrounding environment and the vehicle's state into a trained model 45 that outputs a target merging position in response to inputs of the surrounding environment and the vehicle's state during a merging event. Based on the latest value of the target merging position, the driving plan unit 43 creates a driving plan for vehicle 2. The trained model 45 is a model trained using reinforcement learning, a type of machine learning.

[0035] The trained model 45 outputs a target merging position for inputs that include the surrounding environment and the state of the vehicle itself. The target merging position is the position where vehicle 2 traveling in the merging lane 102 begins to change lanes to the main lane 101. The target merging position is preferably a position relative to the beginning of the merging area 102C. The inputs that include the surrounding environment and the state of the vehicle itself preferably include at least the position of vehicle 2 (first input data), the length of the merging area 102C (second input data), the speed of vehicle 2 (third input data), the positions of each surrounding vehicle 200 (fourth input data), the speeds of each surrounding vehicle 200 (fifth input data), and the previous target merging position (sixth input data).

[0036] The position of vehicle 2 in the first input data should be a position relative to the beginning of the merging area 102C. The position of vehicle 2 in the first input data should be calculated based on the position of the regulatory body end 107A (the beginning of the merging area 102C) acquired by the surrounding environment recognition unit 41 and the position of vehicle 2 acquired by the vehicle recognition unit 42.

[0037] The length of the confluence area 102C of the second input data should be normalized based on the assumed maximum confluence area length. The length of the confluence area 102C should be calculated based on the following equation (1).

number

[0038] The speed of vehicle 2 of the third input data may be normalized based on the merge lane speed limit. The normalized speed of vehicle 2 may be calculated based on the following formula (2).

Number

[0039] The positions of the surrounding vehicles 200 of the fourth input data include the positions of a plurality of surrounding vehicles 200. The position of each surrounding vehicle 200 may be normalized by the following formula (3).

Number

[0040] The speed of each surrounding vehicle 200 in the fifth input data should be normalized based on the main lane speed limit. The normalized speed of each surrounding vehicle 200 should be calculated based on the following equation (4).

number

[0041] For the sixth input data, the previous target confluence position is the one output from the trained model 45.

[0042] The driving plan unit 43 may create first to fifth input data to input into the trained model 45 based on the information acquired by the surrounding environment recognition unit 41 and the vehicle recognition unit 42. The fourth input data may be created as an array in which the positions of multiple surrounding vehicles 200 are arranged in order from the front. The fourth and fifth input data may be created as array data in which the position and speed of each surrounding vehicle 200 are arranged in order from the front. For example, the fourth and fifth input data may be represented as [position of the first surrounding vehicle 200 from the front, speed of the first surrounding vehicle 200 from the front, position of the second surrounding vehicle 200 from the front, speed of the second surrounding vehicle 200 from the front, ...]. The length of the array may be set to a fixed length. If the number of surrounding vehicles 200 is less than the length of the array, 0 may be set in the part where there is no data.

[0043] The trained model 45 is generated by reinforcement learning, using the surrounding environment and the vehicle's state as input data, to maximize the reward-based value. The reward is determined based on multiple auxiliary rewards set for different objectives.

[0044] The multiple auxiliary rewards include a first reward r1 which increases as the time to collision (TTC) between vehicle 2 and surrounding vehicles 200 increases, and a second reward r2 which increases as the deceleration of vehicle 2 decreases. The multiple auxiliary rewards may further include a third reward r3 which increases as the target merging position is closer to the beginning of the merging area 102C. The multiple auxiliary rewards may further include a fourth reward r4 which increases as the difference between the current value of the target merging position and the previous value of the target merging position decreases. The training method for the trained model 45 will be described later.

[0045] The trained model 45 receives the latest first to sixth input data sequentially at predetermined time intervals, such as 0.1 seconds. The trained model 45 sequentially outputs the target merging position corresponding to each input data.

[0046] The driving plan unit 43 sequentially creates a driving plan, including the target trajectory and target speed of vehicle 2, for vehicle 2 to merge at the target merging position, based on the latest value of the target merging position output sequentially from the learned model 45. The driving plan unit 43 updates the driving plan, including the target trajectory and target speed of vehicle 2, at predetermined time intervals.

[0047] The driving control unit 44 controls the acceleration, deceleration, and steering of the vehicle 2 based on the driving plan, without relying on the occupants' input. Specifically, the driving control unit 44 controls the propulsion system 3, braking system 4, and steering system 5 based on the driving plan. As a result, the vehicle 2 travels along the latest target trajectory at the target speed. When the driving plan is updated, the driving control unit 44 controls the acceleration, deceleration, and steering of the vehicle 2 based on the updated driving plan, without relying on the occupants' input. As a result, the vehicle 2 travels along the latest target trajectory at the latest target speed.

[0048] The vehicle control device 1 may control the vehicle 2 based on the merging control procedure shown in Figure 3. When a merging event starts, the driving plan unit 43 first generates the first to sixth input data (ST1). The first to fifth input data may be obtained based on information acquired from the surrounding environment recognition unit 41 and the vehicle recognition unit 42. At the start of a merging event, a predetermined initial value may be set for the sixth input data. The previous target merging position of the sixth input data may be set, for example, to the midpoint of the merging area 102C.

[0049] Next, the driving plan unit 43 inputs the first to sixth input data into the trained model 45 and obtains the target merging position output from the trained model 45 (ST2). Subsequently, the driving plan unit 43 creates a driving plan based on the target merging position, including the target trajectory and target speed of vehicle 2 for merging at the target merging position (ST3).

[0050] Next, the driving control unit 44 controls the propulsion system 3, braking system 4, and steering system 5 of the vehicle 2 based on a driving plan that includes the target trajectory and target speed of the vehicle 2 (ST4). In other words, the driving control unit 44 controls the driving of the vehicle 2 based on the driving plan.

[0051] Next, the driving plan unit 43 determines whether the position of vehicle 2, obtained from the vehicle recognition unit 42, has reached the target merging position (ST5). If the position of vehicle 2 has reached the target merging position (ST5: Yes), the process proceeds to the end and stops updating the target merging position. As a result, the driving control unit 44 controls the propulsion unit 3, braking unit 4, and steering unit 5 of vehicle 2 based on the target trajectory and target speed of vehicle 2 set based on the latest target merging position, and causes vehicle 2 to merge. If the position of vehicle 2 has not reached the target merging position (ST5: No), the process returns to ST1 and repeats updating the target merging position.

[0052] With the vehicle control device 1 described above, the driving plan unit 43 outputs target merging positions sequentially at predetermined time intervals based on the learned model 45, so that an appropriate target merging position can be set according to the movements of multiple surrounding vehicles 200 traveling in the main lane 101. In other words, even if multiple surrounding vehicles 200 make unexpected movements, the vehicle control device 1 can update the target merging position and allow the vehicle 2 to smoothly merge into the main lane 101.

[0053] The reward used when generating the trained model 45 by reinforcement learning includes a first reward r1 that increases as the collision margin between vehicle 2 and surrounding vehicles 200 increases. This sets the target merging position so that there is sufficient collision margin between vehicle 2 and surrounding vehicles 200 at the target merging position. As a result, the safety of vehicle 2 at merging is improved.

[0054] The reward used when generating the trained model 45 by reinforcement learning includes a second reward r2 which is set to increase as the deceleration of vehicle 2 decreases. This sets the target merging position so that deceleration is suppressed as vehicle 2 travels toward the target merging position. As a result, deceleration of vehicle 2 is suppressed as it travels toward the target merging position, improving the ride comfort of vehicle 2.

[0055] The reward used when generating the trained model 45 by reinforcement learning may include a third reward r3, which is set to increase as the target merging position gets closer to the beginning of the merging area 102C. If the third reward r3 is included, the target merging position is set to be closer to the beginning of the merging area 102C. As a result, the merging is completed earlier, reducing the psychological burden on the occupants of vehicle 2.

[0056] The reward used when generating the trained model 45 by reinforcement learning may include a fourth reward r4, which is set to increase as the difference between the current value of the target merging position and the previous value of the target merging position decreases. When the fourth reward r4 is included, the fluctuation of the updated target merging position becomes smaller, and the driving plan set based on the target merging position becomes more stable. As a result, the behavior of vehicle 2 at merging becomes more stable.

[0057] The following describes a reinforcement learning method for creating a trained model 45, a reinforcement learning device 50 for executing the reinforcement learning method, and a program for causing the reinforcement learning device 50 to execute the reinforcement learning method.

[0058] The reinforcement learning method is executed by the reinforcement learning device 50. As shown in Figure 4, the reinforcement learning device 50 is a computer having a processor 51 and a memory 52 that is communicatively connected to the processor 51. The processor 51 may include at least one of the following as its core: a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a RISC (Reduced Instruction Set Computer). The memory 52 stores the control program executed by the processor 51 and various data. The memory 52 may include at least one of volatile memory and non-volatile memory. The volatile memory may be, for example, DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The non-volatile memory may be an SSD (Solid State Drive), flash memory, magnetic disk storage device, or optical disk storage device. At least a portion of the reinforcement learning device 50 may be implemented by hardware such as LSI (Large Scale Integration), ASIC (application-specific integrated circuit), FPGA (field-programmable gate array), or by a combination of software and hardware. The reinforcement learning device 50 may consist of a single piece of hardware, or it may consist of multiple pieces of hardware that can communicate with each other. A portion of the reinforcement learning device 50 may consist of an external server located outside the device.

[0059] The processor 51 executes the reinforcement learning method by running a control program stored in memory 52. ​​The control program may be stored on a removable recordable medium such as a DVD or CD-ROM, and installed in memory 52 when the recordable medium is read by a reader. Alternatively, the program may be downloaded to and installed in memory 52 via a communication network such as the Internet.

[0060] The reinforcement learning method according to this embodiment can use various known reinforcement learning algorithms. Examples of such algorithms include Q-learning, SARSA, DQN (Deep Q Network), Actor-Critic method, and DDPG (Deep Deterministic Policy Gradient). In this embodiment, as an example, the case using DQN, a type of deep reinforcement learning, will be described.

[0061] As shown in Figure 4, the processor 51 functions as the environment 61 and agent 62 by executing a program stored in memory 52. ​​Agent 62 selects an action based on information from the environment 61 and learns based on the reward obtained in response to that action. Agent 62 receives state information provided by the environment 61, decides what action to take based on the obtained state information, and learns to optimize its action based on the experiential data (state, action, reward, next state) obtained from interaction with the environment 61.

[0062] Environment 61 is comprised of a simulator that simulates the real world. Environment 61 feeds back the results of Agent 62's actions to Agent 62. Environment 61 includes a state generation unit 67 that generates the next state based on the actions input from Agent 62, and a reward generation unit 68 that generates a reward based on the state. The state generation unit 67 generates a state that includes the surrounding environment of Vehicle 2 and the state of the Vehicle itself. Specifically, the state may include at least the position of Vehicle 2 (first input data), the length of the merging area 102C (second input data), the speed of Vehicle 2 (third input data), the position of each surrounding Vehicle 200 (fourth input data), the speed of each surrounding Vehicle 200 (fifth input data), and the previous target merging position (sixth input data).

[0063] The reward generation unit 68 determines the reward based on the state. The reward is determined based on multiple auxiliary rewards set based on multiple different objectives. The auxiliary rewards include the first to fourth rewards r1 to r4.

[0064] The first reward r1 is set to increase as the collision margin between vehicle 2 and surrounding vehicles 200 increases. The collision margin is the value when vehicle 2 is at the target merging position. If there are multiple surrounding vehicles 200 around vehicle 2, the smallest collision margin among the collision margins between vehicle 2 and each surrounding vehicle 200 should be selected. The collision margin should be calculated based on the position and speed of vehicle 2 and the position and speed of each surrounding vehicle 200 when vehicle 2 is at the target merging position.

[0065] The first reward r1 is set using the first reward function shown in Figure 6. The first reward function is defined by the collision margin T. C For each input, a first reward r1 is output. The first reward r1 should be a value between 0 and 1 (inclusive). The first reward function can be, for example, a sigmoid function or a logistic function. The first reward function can be expressed, for example, by the following equation (5).

number

[0066] The first reward, r1, is awarded when Vehicle 2 is at the target merging position, i.e., at the end of the episode.

[0067] The second reward r2 is given in each state, i.e., each step in an episode. The second reward r2 is a negative reward, and its value should increase in the negative direction as the deceleration of vehicle 2 increases. When the deceleration is 0, the second reward r2 should be set to 0. The second reward function outputs the second reward r2 in response to the input of the deceleration D of vehicle 2. The second reward function can be expressed, for example, by the following equation (6).

number

[0068] The third reward r3 is set to increase as the target confluence position gets closer to the beginning of the confluence area 102C. The third reward r3 is set using a third reward function. The third reward function outputs the third reward r3 for the input of the target confluence position. The third reward r3 should be a value greater than 0. The third reward function may be set based on, for example, a sigmoid function or a logistic function. The third reward function may be expressed by, for example, the following equation (7).

number

[0069] The fourth reward r4 is given in each state, i.e., each step in an episode. The fourth reward r4 is a negative reward, and its value should increase in the negative direction as the difference between the current and previous values ​​of the target confluence position increases. When the difference between the current and previous values ​​of the target confluence position is 0, the fourth reward r4 should be set to 0. The fourth reward function outputs the fourth reward r4 in response to the inputs of the current and previous values ​​of the target confluence position. The fourth reward function can be expressed, for example, by the following equation (8).

number

[0070] A reward of r2 + r4 is given for each state during the episode. Additionally, at the end of the episode, i.e., when vehicle 2 reaches the target merging position, a reward of r1 × r3 is given. Since the first reward r1 and the third reward r3 are product of each other, if the first reward r1 given for the collision buffer time is 0, the overall reward will be low regardless of the value of the third reward r3. In other words, collision buffer time is considered an important factor in the process of determining the target merging position.

[0071] Agent 62 has a DQN model 71. Agent 62 takes state information as input and generates an action policy using the DQN model 71. As shown in Figure 5, the DQN model 71 has an input layer 72, an intermediate layer 73, and an output layer 74. The DQN model 71 approximates the Q function using a deep neural network.

[0072] The input layer 72 has multiple nodes 72A. Each node 72A receives different state information. The state information may include the surrounding environment and the state of the vehicle itself. Specifically, the state may include the first to sixth input data described above. The number of nodes 72A in the input layer 72 may correspond to the number of states. The input layer 72 passes the input information to the intermediate layer 73.

[0073] The hidden layer 73 includes multiple layers. Each layer constituting the hidden layer 73 has multiple nodes 73A. The hidden layer 73 compresses the information input to the input layer 72 and extracts the feature quantities of the information.

[0074] The output layer 74 has multiple nodes 74A, each node 74A outputting value information for each action. Here, the action corresponds to the target confluence position. The value information is the expected value of the discounted cumulative reward obtained when a specific action is taken in a specific state, i.e., the state-action value function (Q-value). It is preferable that each node 74A of the output layer 74 corresponds to the number of actions, i.e., the number of target confluence positions.

[0075] In DQN-based learning, the update formula for the state-action-value function is used, as shown in equation (9).

number

[0076] The loss function (error) in updating the Q value can be expressed, for example, by equations (10) and (11), if the loss is considered as the mean squared error.

number

number

[0077] The DQN model 71 with optimized weight coefficients is used as the trained model 45 for the driving plan unit 43 of the vehicle control device 1. The trained model A outputs the target merging position for inputs including the first to sixth input data.

[0078] The embodiments are not limited to the above configuration and can be broadly modified. For example, the trained model 45 may be generated based on the first to fourth rewards r1 to r4 above, as well as other auxiliary rewards set for other purposes. For example, a negative reward may be given when the acceleration of vehicle 2 exceeds a predetermined value. This sets the target merging position so that excessive acceleration of vehicle 2 is suppressed. Alternatively, a negative reward may be given when the speed of vehicle 2 exceeds a predetermined value. This sets the target merging position so that the speed of vehicle 2 is below a predetermined value such as the speed limit.

[0079] The above embodiments may also be described as follows:

[0080] One embodiment is a vehicle control device 1 that performs merging control of a vehicle 2, comprising: an ambient environment recognition unit 41 that recognizes the ambient environment around the vehicle 2; a vehicle recognition unit 42 that recognizes the state of the vehicle 2, which is the state of the vehicle itself; a driving plan unit 43 that sequentially acquires the target merging position by sequentially inputting the ambient environment and the state of the vehicle into a learned model 45 that outputs a target merging position in response to inputs of the ambient environment and the state of the vehicle itself, and creates a driving plan for the vehicle 2 based on the latest value of the target merging position; and a driving control unit 44 that controls the acceleration, deceleration, and steering of the vehicle 2 based on the driving plan, without relying on the operation of the occupants.

[0081] In this embodiment, the driving plan unit 43 outputs target merging positions sequentially at predetermined time intervals based on the learned model 45, so that an appropriate target merging position can be set according to the movements of multiple surrounding vehicles 200 traveling on the main lane 101.

[0082] In the above embodiment, the trained model 45 is reinforced-learned to output the target merging position that maximizes the reward-based value, with the surrounding environment and the state of the vehicle as input data, and the reward may be determined based on a plurality of auxiliary rewards set based on a plurality of different objectives.

[0083] According to this embodiment, the trained model 45 can output a target merging position capable of achieving multiple different objectives. For example, the trained model 45 can output a target merging position capable of achieving things like securing sufficient time to avoid collisions with surrounding vehicles 200 and improving ride comfort.

[0084] In the above embodiment, the plurality of auxiliary rewards may include a first reward r1 which increases as the collision margin time between the vehicle 2 and the surrounding vehicle 200 increases, and a second reward r2 which increases as the deceleration of the vehicle 2 decreases.

[0085] In this embodiment, since the auxiliary reward used when generating the trained model 45 by reinforcement learning includes reward 1, the target merging position is set such that there is sufficient time for collision between vehicle 2 and surrounding vehicles 200 at the target merging position. As a result, the safety of vehicle 2 at merging is improved. Furthermore, since the auxiliary reward includes a second reward r2, the target merging position is set such that deceleration of vehicle 2 is suppressed while vehicle 2 is traveling toward the target merging position. As a result, deceleration of vehicle 2 is suppressed while vehicle 2 is traveling toward the target merging position, and the ride comfort of vehicle 2 is improved.

[0086] In the above embodiment, the plurality of auxiliary rewards may further include a third reward r3 which is set to increase as the target confluence position gets closer to the beginning of the confluence area 102C.

[0087] In this embodiment, the target merging position is set to be close to the beginning of the merging area 102C. As a result, the merging is completed earlier, which reduces the psychological burden on the occupants of vehicle 2.

[0088] In the above embodiment, the plurality of auxiliary rewards may further include a fourth reward r4 which increases as the difference between the current value of the target confluence position and the previous value of the target confluence position becomes smaller.

[0089] According to this embodiment, fluctuations in the updated target merging position are reduced, and the driving plan set based on the target merging position becomes more stable. As a result, the behavior of vehicle 2 during merging becomes more stable.

[0090] Another embodiment is a reinforcement learning method performed by a computer to generate a trained model 45 that outputs a target merging position in response to inputs including the surrounding environment of a vehicle 2 and the vehicle state which is the state of the vehicle 2, wherein state information including the surrounding environment and the vehicle state is acquired from a simulator, the state information is used as input to generate an action policy in a neural network, an action is performed based on the action policy, a reward and the next state information are received, the parameters of the neural network are updated and the action policy is adjusted based on the reward and the state information, and the reward is determined based on a plurality of auxiliary rewards set based on a plurality of different objectives.

[0091] According to this embodiment, the reinforcement learning method can create a trained model 45 that can sequentially output a target merging position based on the surrounding environment and the state of the vehicle itself.

[0092] Another embodiment is a reinforcement learning device 50 that generates a trained model 45 that outputs a target merging position in response to inputs including the surrounding environment of a vehicle 2 and the vehicle state which is the state of the vehicle 2, and comprises a simulator that outputs state information including the surrounding environment and the vehicle state, and an agent 62 that takes the state information as input, generates an action policy using a neural network, executes an action based on the action policy, receives a reward and the next state information, and updates the parameters of the neural network and adjusts the action policy based on the reward and the state information, wherein the reward is determined based on a plurality of auxiliary rewards set based on a plurality of different objectives.

[0093] According to this embodiment, the reinforcement learning device 50 can create a trained model 45 that can sequentially output a target merging position based on the surrounding environment and the state of the vehicle itself.

[0094] Another embodiment is a program for causing a computer to execute a reinforcement learning method that generates a trained model 45 that outputs a target merging position in response to inputs including the surrounding environment of a vehicle 2 and the vehicle state which is the state of the vehicle 2, wherein the program causes a computer to acquire state information including the surrounding environment and the vehicle state from a simulator, uses the state information as input to generate an action policy in a neural network, executes an action based on the action policy, receives a reward and the next state information, updates the parameters of the neural network and adjusts the action policy based on the reward and the state information, and the reward is determined based on a plurality of auxiliary rewards set based on a plurality of different objectives.

[0095] According to this embodiment, the program can cause the computer to perform reinforcement learning that can create a trained model 45 that can sequentially output a target merging position based on the surrounding environment and the state of the vehicle. [Explanation of symbols]

[0096] 1: Vehicle control system 2: Vehicles 7: External world recognition device 13: Vehicle speed sensor 31: Processor 32: Memory 41: Surrounding Environment Recognition Unit 42: Vehicle recognition unit 43: Driving Planning Department 44: Driving Control Unit 45: Pre-trained model 50: Reinforcement Learning Device 51: Processor 52: Memory 100: Merging Area 101: Main lane 102: Merging lane 102C: Area where merging is possible 107: Regulatory body 107A: Regulator end 200: Surrounding vehicles

Claims

1. A vehicle control device that controls the merging of vehicles, The surrounding environment recognition unit recognizes the surrounding environment of the vehicle, A vehicle recognition unit that recognizes the state of the vehicle, which is the state of the vehicle, A driving planning unit sequentially acquires the target merging position by sequentially inputting the surrounding environment and the vehicle's state into a trained model that outputs a target merging position in response to the input of the surrounding environment and the vehicle's state, and creates a driving plan for the vehicle based on the latest value of the target merging position. A vehicle control device having a driving control unit that controls the acceleration, deceleration, and steering of the vehicle based on the aforementioned driving plan, without relying on the operation of the occupants.

2. The trained model is reinforced to output the target merging position that maximizes the reward-based value, using the surrounding environment and the state of the vehicle as input data. The vehicle control device according to claim 1, wherein the aforementioned reward is determined based on a plurality of supplementary rewards set based on a plurality of different objectives.

3. The vehicle control device according to claim 2, wherein the plurality of auxiliary rewards include a first reward set to increase as the collision margin time between the vehicle and surrounding vehicles increases, and a second reward set to increase as the deceleration of the vehicle decreases.

4. The vehicle control device according to claim 3, wherein the plurality of auxiliary rewards further include a third reward set to increase as the target merging position is closer to the beginning of the merging area.

5. The vehicle control device according to claim 3 or 4, further comprising a fourth reward set to increase as the difference between the current value of the target merging position and the previous value of the target merging position becomes smaller.

6. A reinforcement learning method performed by a computer to generate a trained model that outputs a target merging position in response to inputs including the surrounding environment of a vehicle and the state of the vehicle, The simulator acquires state information including the surrounding environment and the state of the vehicle itself. Using the aforementioned state information as input, a neural network generates a course of action. The action is performed based on the aforementioned action plan, and the reward and the next aforementioned state information are received. Based on the reward and the state information, the parameters of the neural network are updated to adjust the action strategy. The aforementioned reward is determined based on a reinforcement learning method that uses multiple auxiliary rewards set based on multiple different objectives.

7. A reinforcement learning device that generates a trained model that outputs a target merging position in response to inputs including the surrounding environment of a vehicle and the state of the vehicle, A simulator that outputs state information including the surrounding environment and the state of the vehicle itself, The agent includes a neural network that takes the aforementioned state information as input, generates an action plan, executes an action based on the action plan, receives a reward and the next aforementioned state information, and updates the parameters of the neural network and adjusts the action plan based on the reward and the aforementioned state information. The aforementioned reward is determined based on a plurality of auxiliary rewards set based on multiple different objectives.

8. A program for causing a computer to execute a reinforcement learning method that generates a trained model that outputs a target merging position in response to inputs including the surrounding environment of a vehicle and the state of the vehicle, The simulator is used to obtain state information including the surrounding environment and the state of the vehicle itself. Using the aforementioned state information as input, a neural network is used to generate a course of action. The action is carried out based on the aforementioned action plan, and the recipient receives a reward and the next aforementioned state information. Based on the aforementioned reward and the aforementioned state information, the parameters of the neural network are updated and the action strategy is adjusted. The aforementioned reward is determined by a program based on multiple supplementary rewards set up for multiple different objectives.

Images