An autonomous driving decision-making system based on multimodal perception and hierarchical actions

Abstract

This invention discloses an autonomous driving decision-making system based on multimodal perception and hierarchical actions, including a data preprocessing module, a state representation module, an actor / critic module, a hybrid reward function module, and a multi-process training module. Parallel training accelerates model training speed and improves model performance. This invention offers greater applicability, utilizing multiple sensors and deep learning algorithms for comprehensive perception of the surrounding environment, enabling autonomous vehicles to better handle unexpected situations. The lane-specific cross-modal aggregation model allows autonomous vehicles to extract useful features faster and more effectively. The hierarchical action-based reinforcement learning model better optimizes the lane-keeping and lane-changing behaviors of autonomous vehicles. This system satisfies the safety, efficiency, and comfort requirements of each decision made by the autonomous vehicle, reduces traffic flow fluctuations caused by abrupt speed changes and lane changes, and improves overall traffic efficiency and stability.

Description

Technical Field

[0001] This invention relates to the field of transportation technology, and in particular to an autonomous driving decision-making system based on multimodal perception and hierarchical actions. Background Technology

[0002] Due to urbanization and the widespread use of private cars, many large cities around the world frequently suffer from traffic congestion. Generally speaking, traffic congestion is mainly caused by road conditions and drivers. While environmental factors such as road construction and merging lanes can easily cause traffic jams, research has found that poor driving behaviors (such as sudden braking and abrupt lane changes) are the primary cause of traffic congestion and even traffic accidents. This is because human driving performance is usually limited by factors such as limited visibility and long reaction times. Therefore, maintaining good driving performance at all times is difficult for human drivers.

[0003] Autonomous driving has garnered widespread social attention in current technologies. Compared to human drivers who use their eyes and brains to perceive their surroundings and make decisions, autonomous vehicles utilize various sensor technologies to acquire information about their environment and employ computer technology to calculate ideal speed and lane-changing decisions. Traditional decision-making algorithms use rule matching and mathematical calculations to determine the speed and lane-changing actions of autonomous vehicles. However, this method requires complex parameter tuning and struggles to adapt to complex traffic scenarios. With the development of deep learning technology, researchers have used neural networks to mimic human driver behavior. However, deep learning models are prone to modeling errors, and human drivers' decisions are often not optimal. Given the workflow of autonomous driving tasks—first acquiring information about the surrounding environment and then making decisions—reinforcement learning appears to be a promising approach to solving autonomous driving problems. Recently, many researchers have used reinforcement learning algorithms to optimize the decisions of autonomous vehicles and achieved better decision-making performance than traditional algorithms and deep learning algorithms. However, existing reinforcement learning decision-making frameworks have two drawbacks: poor environmental adaptability and limited optimization objectives. Regarding the first drawback, most existing algorithms are tested in simple straight-line environments and assume that surrounding traffic information can be accurately acquired. Therefore, these algorithms cannot adapt to complex traffic scenarios and cannot cope with the uncertainties in real-world traffic situations. Recently, some researchers have begun conducting autonomous driving experiments on urban roads, using cameras and LiDAR to perceive complex traffic information. However, simply acquiring image data and point cloud data of the surrounding environment is insufficient for autonomous vehicles to calculate ideal decisions. Autonomous vehicles also need to acquire information about the road ahead and their own dynamics, such as vehicle orientation and distance from lane departure. Regarding this second limitation, existing algorithms primarily optimize the safety, efficiency, and comfort of autonomous vehicles, rarely considering reducing their impact on other vehicles. In high-density scenarios, a poor driving behavior can trigger a chain reaction, spreading like a domino effect in traffic flow, ultimately leading to traffic congestion. Recently, a tree search-based decision framework has been developed to search for decisions that minimize the impact on other vehicles using decision tree algorithms. However, this method ideally divides lane-changing behavior into left lane change, lane keeping, and right lane change, ignoring intermediate states. This means that an autonomous vehicle cannot reverse a lane change midway through, even if it would cause safety issues. For example, if an autonomous vehicle notices a vehicle behind it suddenly accelerating while changing lanes, continuing to change lanes could lead to a collision with that vehicle. Summary of the Invention

[0004] The purpose of this invention is to provide an autonomous driving decision-making system based on multimodal perception and hierarchical actions.

[0005] To achieve the above objectives, the present invention is implemented according to the following technical solution:

[0006] This invention includes a data preprocessing module, a state representation module, an actor / critic module, a hybrid reward function module, and a multi-process training module;

[0007] Data processing module: Acquires status information of the surrounding environment through high-precision maps and multiple sensors;

[0008] State representation module: Extracts useful state information through neural networks and learns the latent representations of this information;

[0009] Actor / Critic Module: This module uses an actor network model and a critic network model to generate and evaluate the autonomous vehicle's decisions, respectively.

[0010] Hybrid reward function module: guides reinforcement learning models to make good decision-making performance;

[0011] Multi-process training module: Accelerates model training speed and improves model training performance through parallel training.

[0012] The beneficial effects of this invention are:

[0013] This invention is an autonomous driving decision-making system based on multimodal perception and hierarchical actions. Compared with existing technologies, this invention has the following advantages:

[0014] (1) The present invention has better applicability. It uses a variety of sensors and deep learning algorithms to comprehensively perceive the surrounding environment information, so that autonomous vehicles can better cope with some emergencies, such as sudden braking and forced cutting in by surrounding vehicles.

[0015] (2) This invention proposes a lane-segmented cross-modal aggregation model, which enables autonomous vehicles to extract useful features faster and better.

[0016] (3) This invention proposes a reinforcement learning model based on hierarchical actions to better optimize the lane keeping and lane changing behaviors of autonomous vehicles.

[0017] (4) The present invention satisfies the safety, efficiency and comfort of every decision made by the autonomous vehicle, and greatly reduces traffic flow fluctuations caused by the rapid speed change and lane change of the autonomous vehicle, thereby improving the traffic efficiency and traffic flow stability of the entire traffic flow. Attached Figure Description

[0018] Figure 1 This is a diagram of the overall system architecture of the present invention;

[0019] Figure 2This is a network structure diagram of the state representation module of the present invention;

[0020] Figure 3 This is a structural diagram of the layered actions in the actor critic module of this invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0022] like Figure 1 As shown, this invention is divided into five parts: a data preprocessing module, a state representation module, an actor / critic module, a hybrid reward function module, and a multi-process training module.

[0023] The data preprocessing module acquires information about the surrounding environment using high-precision maps and various sensors. Specifically, it uses cameras to acquire traffic light information, LiDAR to acquire information about surrounding vehicles, high-precision maps to acquire road topology and lane information, and localization technology and inertial measurement units to acquire information such as the autonomous vehicle's own position and speed. Finally, this information is combined to form the autonomous vehicle's state information and input into the reinforcement learning model.

[0024] The state representation module extracts useful state information through neural networks and learns the latent representations of this information. For example... Figure 2 As shown, the multimodal information (lanes, regular vehicles, and traffic lights) surrounding the autonomous vehicle is first organized into three star-shaped graphs (G) centered on the autonomous vehicle. k-1 G k G k+1 Each star diagram corresponds to multimodal information from one lane. To aggregate this multimodal information, a cross-modal attention mechanism is used to combine information from different modalities. Next, we will use the multimodal features of the current lane (lane k) of the autonomous vehicle as an example to introduce the method for calculating the attention score.

[0025]

[0026] Where e A The encoding of the feature vector of an autonomous vehicle, e l The encoding of the feature vector representing the structural features of lane k, e tl The encoding of the feature vector of the traffic light belonging to lane k, e v α represents the sum of the encoded feature vectors of regular vehicles located in lane k. k The attention score represents the feature value for each modality, W1 and W2 represent the parameters of the linear layer encoding these feature vectors, and D represents the feature value for each modality. fLet represent the dimension of the feature vector, and Softmax represent the activation function before obtaining the attention scores. Then, these attention scores α are... k Its corresponding eigenvector (e l e tl e v Multiplying them again yields the state representation E of the multimodal information on lane k. k :

[0027] E k =α k (W3[e l e tl e v ])

[0028] Here, W3 represents the parameters of the linear layer that aggregates these multimodal information.

[0029] Next, the state representations of the multimodal features on each lane are concatenated and a linear layer is used to compute the representation E that includes multimodal information from all lanes. s :

[0030]

[0031] Where W4 represents the parameters of the linear layer, E k-1 E represents the representation of multimodal information on lane k-1. k E represents the representation of multimodal information on lane k. k+1 This represents the multimodal information on lane k+1.

[0032] The actor / critic module uses an actor network model and a critic network model to generate and evaluate the autonomous vehicle's decisions, respectively. Because lane following and lane changing decisions both require controlling the autonomous vehicle's steering angle, a hierarchical structure is designed to decouple the different decisions. Figure 3 As shown, the autonomous vehicle's decisions are first divided into three high-level decisions: changing lanes to the left, maintaining lane position, and changing lanes to the right. Each high-level decision then requires calculating an independent action value, including steering angle, throttle, and braking. In the actor network, a linear layer is used to simultaneously calculate the action value X for the three high-level decisions:

[0033] X = TAnh(W5E) s )

[0034] Where W5 represents the parameters of the linear layer, E s The state representation is shown, and TAnh represents the activation function of the linear layer. After the actor network computes the action values, the critic network is used to compute the Q-value for each action.

[0035] In reinforcement learning, the Q-value represents the expected future reward of an action. A higher Q-value indicates better performance for that action. Specifically, a linear layer is used to calculate the Q-values ​​for three action values:

[0036]

[0037] Where W6 represents the parameters of the linear layer, E s E represents the state representation. a This represents the action value X. After obtaining the action value and Q-value for each decision, the action value of the decision with the largest Q-value is selected as the final action to be executed by the autonomous vehicle.

[0038] The hybrid reward function module aims to guide reinforcement learning models to make better decisions. For autonomous vehicles, an ideal decision must first meet the requirements of safety, efficiency, and comfort. Secondly, the decision should minimize the impact on surrounding vehicles. Therefore, the hybrid reward function consists of four parts: safety, efficiency, comfort, and impact on surrounding vehicles.

[0039] The safety-related bonus is calculated using the autonomous vehicle's collision risk and distance from the lane centerline. The formula is as follows:

[0040]

[0041] The TTC indicator indicates how long it would take for the autonomous vehicle to collide with the car in front if it maintains its current speed. This is a threshold used by TTC to assess collision risk. Dis is the distance between the autonomous vehicle and the center line of the lane, and d refers to half the width of the lane. If the autonomous vehicle is too close to the vehicle in front or too far off the lane, it will receive a small safety bonus.

[0042] The efficiency-related bonus is calculated using the autonomous vehicle's speed and lane-change choices. The formula is as follows:

[0043]

[0044] Where v represents the speed of the autonomous vehicle, v m D represents the maximum speed of the autonomous vehicle, D′ represents the distance between the autonomous vehicle and the vehicle in front after changing lanes, and D represents the distance between the autonomous vehicle and the vehicle in front before changing lanes. This reward is only calculated when the autonomous vehicle changes lanes. When the autonomous vehicle changes from a congested lane to an empty lane, the reward will be a positive number. Conversely, the reward will be a negative number.

[0045] The reward value for comfort is calculated using the change in the acceleration of the autonomous vehicle, and the formula is as follows:

[0046]

[0047] Where acc′ and acc represent the acceleration values ​​of the autonomous vehicle at the current and previous moments, respectively. m This represents the threshold for the acceleration of an autonomous vehicle.

[0048] The reward value for impact is calculated using the speed change of the vehicle behind the autonomous vehicle, and the calculation formula is as follows:

[0049]

[0050] Where Δv represents the change in speed of the vehicle behind the autonomous vehicle, and acc m The threshold value represents the acceleration, and Δt represents the time interval between two time steps.

[0051] Additionally, if an autonomous vehicle runs a red light, collides with another vehicle, or leaves its lane, it will receive a reward of -20.

[0052] The multi-process training module accelerates model training and improves model performance through parallel training. Due to the slow rendering of traffic environments and the insufficient exploration capabilities of a single model, multiple processes will be used to collect exploration experience in parallel. In this invention, each exploration experience includes the autonomous driving's state information, action value, reward value, and state information for the next time step. Furthermore, one process will continuously sample the explored experience and then update the parameters of the network model in our framework based on this.

[0053] The technical solutions of the present invention are not limited to the specific embodiments described above. Any technical modifications made in accordance with the technical solutions of the present invention fall within the protection scope of the present invention.

Claims

1. An autonomous driving decision-making system based on multimodal perception and hierarchical actions, characterized in that: It includes a data preprocessing module, a state representation module, an actor / critic module, a hybrid reward function module, and a multi-process training module; Data preprocessing module: Acquires environmental status information through high-precision maps and multiple sensors; State representation module: Extracts useful state information through neural networks and learns the latent representations of this information; Actor / Critic Module: This module uses an actor network model and a critic network model to generate and evaluate the autonomous vehicle's decisions, respectively. Hybrid reward function module: guides reinforcement learning models to make good decision-making performance; Multi-process training module: Accelerates model training speed and improves model training performance through parallel training; The state representation module first organizes the multimodal information around the autonomous vehicle into three star-shaped diagrams centered on the autonomous vehicle. , , Each star diagram corresponds to multimodal information for one lane; a cross-modal attention mechanism is used to aggregate information from different modalities to determine the current lane (lane) of the autonomous vehicle. Taking multimodal features on a given surface as an example, the attention score is calculated as follows: in Encoding of the feature vector of an autonomous vehicle. lane Encoding of the feature vectors of structural features, Indicates that it belongs to the lane Encoding of the feature vectors of traffic lights, Indicates that it is located in the lane The sum of the encodings of the feature vectors of conventional vehicles on the road. The attention score represents the feature of each modality. and This represents the parameters of the linear layer that encodes these feature vectors. The dimension of the feature vector. This represents the activation function before obtaining the attention scores; afterwards, these attention scores are... Its corresponding feature vector ( , , Multiply again to get the lane State representation of multimodal information : in, The parameters represent the linear layer that aggregates this multimodal information; Next, the state representations of the multimodal features on each lane are concatenated and a linear layer is used to compute a representation that includes multimodal information from all lanes. : in, This represents the parameters of the linear layer. lane Representation of multimodal information on the surface lane Representation of multimodal information on the surface, lane Representation of multimodal information on the surface.

2. The autonomous driving decision-making system based on multimodal perception and hierarchical action according to claim 1, characterized in that: The data preprocessing module uses a camera to acquire traffic light information, uses a lidar to acquire surrounding vehicle information, uses a high-precision map to acquire road topology and lane information, and uses positioning technology and an inertial measurement unit to acquire the autonomous vehicle's own position and speed information. Finally, it combines this information into the autonomous vehicle's state information and inputs it into the reinforcement learning model.

3. The autonomous driving decision-making system based on multimodal perception and hierarchical action according to claim 1, characterized in that: The actor critic module first divides the autonomous vehicle's decisions into three high-level decisions: changing lanes to the left, maintaining lane position, and changing lanes to the right. Each high-level decision then requires the calculation of an independent action value, including steering angle, throttle, and braking values. In the actor network, a linear layer is used to simultaneously calculate the action values ​​for all three high-level decisions. : in This represents the parameters of the linear layer. Representing the state characterization, This represents the activation function of the linear layer; after the actor network has calculated the motion values, the critic network will be used to calculate the motion values ​​for each action. Value; in reinforcement learning, The value represents the expected future reward of an action; A higher value indicates better performance for the action; a linear layer is used to calculate the values ​​corresponding to the three action values. value: in This represents the parameters of the linear layer. Representing the state characterization, Indicates action value The representation; in obtaining the action value of each decision and After that, the one with the largest value will be selected. The action value under the decision is the final action executed by the autonomous vehicle.

4. The autonomous driving decision-making system based on multimodal perception and hierarchical action according to claim 1, characterized in that: The hybrid reward function module consists of four parts: safety, efficiency, comfort, and impact on surrounding vehicles. The safety-related reward value is calculated using the collision risk of the autonomous vehicle and its distance from the lane centerline, using the following formula: in, This indicates how long it will take for the self-driving car to collide with the vehicle in front if it maintains its current speed. yes A threshold used to assess collision risk. It is the distance between the autonomous vehicle and the center line of the lane. It refers to half the width of the lane; The reward value for efficiency is calculated using the speed and lane-change choices of the autonomous vehicle, and the formula is as follows: in Indicates the speed of the self-driving car. Indicates the maximum speed of the autonomous vehicle. This indicates the distance between the autonomous vehicle and the vehicle in front after the vehicle changes lanes. This indicates the distance between the autonomous vehicle and the vehicle in front before the vehicle changes lanes; The reward value for comfort is calculated using the change in the acceleration of the autonomous vehicle, and the formula is as follows: in, and These represent the acceleration values ​​of the autonomous vehicle at the current moment and the previous moment, respectively. A threshold representing the acceleration of an autonomous vehicle; The reward value for the impact on surrounding vehicles is calculated using the speed change of the vehicles behind the autonomous vehicle, and the calculation formula is as follows: in, This indicates the change in speed of the vehicle behind the autonomous vehicle. The threshold representing acceleration. This represents the time interval between two time steps.

5. The autonomous driving decision-making system based on multimodal perception and hierarchical action according to claim 1, characterized in that: The multi-process training module uses multiple processes to collect exploration experience in parallel. Each exploration experience includes the autonomous driving's state information, action value, reward value, and state information at the next moment.

Images