An automatic driving centralized decision-making method based on a hybrid layered reinforcement learning

Abstract

The application discloses an automatic driving centralized decision-making method based on a hybrid layered reinforcement learning, comprising the following steps: constructing a deep reinforcement learning network and performing hybrid network layering on the deep reinforcement learning network to obtain an upper network and a lower network; performing horizontal division on the upper network to obtain a horizontal global strategy network and a vertical global strategy network; training the horizontal global strategy network and the vertical global strategy network through a discrete DRL algorithm, wherein the horizontal global strategy network outputs a lane-changing instruction, and the vertical global strategy network outputs a vertical control strategy; dividing the lower network into a vertical lower network and a horizontal lower control network, training the vertical lower network and the horizontal lower control network through a continuous DRL algorithm, wherein the vertical lower network selects and activates a corresponding sub-strategy network according to the network output of the vertical global strategy and outputs continuous acceleration; and the horizontal lower control network changes lanes based on an optimal control rule to complete centralized decision-making of automatic driving. The application further improves the accuracy, comprehensiveness and effectiveness of decision-making.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of Internet of Vehicles and automatic driving, and particularly relates to an automatic driving centralized decision-making method based on a hybrid hierarchical reinforcement learning. BACKGROUND

[0002] An unmanned driving decision-making system needs to consider multiple factors, such as road conditions, traffic rules, passenger demands, etc., to make the best decision. The unmanned driving decision-making system can not only improve traffic efficiency, reduce traffic congestion and emissions, but also provide humans with a more comfortable, safe and convenient travel experience. Therefore, the research and development of unmanned driving decision-making is an important development direction in the future transportation field.

[0003] Deep reinforcement learning (DRL) can learn the best strategy by simulating the environment, taking actions and obtaining rewards. In unmanned driving decision-making, deep reinforcement learning can learn to determine the best driving strategy in various traffic scenarios based on vehicle sensor data and map information, such as acceleration, turning, lane changing and parking, etc. In addition, deep reinforcement learning can also learn to actively avoid other vehicles, pedestrians or obstacles, optimize route planning and control vehicle speed, so as to achieve a safer and more efficient driving experience.

[0004] In unmanned driving decision-making, the state space is usually very large, containing multiple factors such as vehicle speed, road conditions, traffic conditions, etc. The efficiency and accuracy of directly applying traditional reinforcement learning algorithms to calculate the optimal solution in such a state space will be greatly challenged. Hierarchical reinforcement learning provides an effective method to solve this problem. Hierarchical reinforcement learning can decompose unmanned driving decision-making into multiple sub-tasks, and train a sub-policy network for each sub-task to complete the corresponding task. For example, one sub-task can be a single operation such as acceleration, braking or turning, while another sub-task may involve complex path planning and obstacle avoidance operations. This decomposition makes the state space smaller, making the reinforcement learning search more efficient. Therefore, the application of hierarchical reinforcement learning in unmanned driving decision-making has great significance, as it can improve the intelligence and autonomy of vehicles, providing a more reliable and intelligent solution for safe, efficient and convenient travel. SUMMARY

[0005] Existing deep reinforcement learning forces the function to fit the human brain's decision-making, but due to the existence of neural network error propagation, dimension explosion, etc., it cannot meet the needs of the actual application of the decision-making system. To solve this problem, the present application proposes an automatic driving centralized decision-making method based on a hybrid hierarchical reinforcement learning, based on the decision-making hierarchy of the human brain.

[0006] To achieve the object, the application provides the following scheme: an automatic driving centralized decision-making method based on a hybrid layered reinforcement learning, comprising:

[0007] constructing a deep reinforcement learning network and hybrid network layering the deep reinforcement learning network to obtain an upper network and a lower network;

[0008] horizontally dividing the upper network to obtain a horizontal global policy network and a vertical global policy network; training through a discrete DRL algorithm, the horizontal global policy network outputs a lane-changing instruction, and the vertical global policy network outputs a vertical control strategy;

[0009] dividing the lower network into a vertical lower network and a horizontal lower control network, training through a continuous DRL algorithm, the vertical lower network selects an activated corresponding sub-policy network according to the network output of the vertical global policy and outputs continuous acceleration, and the horizontal lower control network changes lanes based on an optimal control rule to complete centralized decision-making of automatic driving.

[0010] Preferably, the process of training the horizontal global policy network to output a lane-changing instruction and the vertical global policy network to output a vertical control strategy through a discrete DRL algorithm comprises,

[0011] defining that M unmanned vehicles in N vehicles in an environment interact with the environment, and the centralized decision-making layer obtains a state information matrix S from the environment t M×F F is the total number of vehicle characteristics;

[0012] designing a state space according to the functions of different neural networks, respectively inputting the horizontal global policy network and the vertical global policy network into the designed horizontal global state space and the vertical global state space FH and FL are respectively the number of characteristics designed for the corresponding network;

[0013] the horizontal global policy network and the vertical global policy network obtain their own information matrix and adopting a greedy strategy to respectively output lane-changing centralized decision-making actions through a graph convolutional neural network and a multi-layer fully connected layer wherein -1 represents activating a left lane-changing rule, 0 represents activating a lane-maintaining rule, and 1 represents activating a right lane-changing rule; and outputting a vertical centralized decision-making action wherein -1 represents that the unmanned vehicle adopts the output of a deceleration sub-policy network, 0 represents that the unmanned vehicle adopts the output of an adjustment sub-policy network, and 1 represents that the unmanned vehicle adopts the output of an acceleration sub-policy network, wherein the horizontal global policy network and the vertical global policy network output every Z time steps and are updated.

[0014] Preferably, the lateral global state space includes a node feature matrix N t H and an adjacency matrix A t H , the node feature matrix N t H and the adjacency matrix A t H is input into a lateral global policy network, and the topological structure features of the vehicles in the scene are extracted by using a graph convolutional neural network;

[0015] The node feature matrix N t H includes a self-vehicle speed V i , a self-vehicle lateral position X i , a self-vehicle longitudinal position Y i , a standard safety distance coefficient K of a vehicle in front of and behind a neighboring lane of the unmanned vehicle ij , j = 1, 3, 4, 6, are coefficients of the right front, left front, right rear and left rear vehicles of the unmanned vehicle, a road segment R i , a lane L i , a self-vehicle category I i ;

[0016] The node feature matrix N t H is expressed as:

[0017]

[0018] The adjacency matrix A t H is used to represent the interaction between the vehicles as information sharing between the vehicles, and is expressed as:

[0019]

[0020] Preferably, the reward function of the lateral global policy network is defined as a function based on safety after lane changing, recommended lane and lane changing comfort:

[0021] wherein, E defines an upper limit of the penalty and can be regarded as a collision;

[0022]

[0023] If frequent lane changing is performed within a specified time

[0024] Each time step Stored to experience replay area and trained with discrete DRL algorithm.

[0025] Preferably, the longitudinal global state space Comprises a node feature matrix N t L With an adjacency matrix A t L , the adjacency matrix A t L Same as the adjacency matrix A t H ; the node feature matrix N t L With an adjacency matrix A t L Input into a longitudinal global policy network, and the topological structure features of vehicles in the scene are extracted by using a graph convolutional neural network;

[0026] The node feature matrix N t L Comprises a vehicle speed V i , a vehicle lateral position X i , a vehicle longitudinal position Y i , a standard safety distance coefficient κ ij of the front and rear vehicles of the unmanned vehicle, j=2,5, respectively, the coefficients of the front and rear vehicles of the unmanned vehicle, a road segment R i , a lane L i , and a vehicle category Ii to which the vehicle belongs;

[0027] The node feature matrix N t L The expression of the node feature matrix N

[0028]

[0029] Preferably, the reward function of the longitudinal global policy network is defined as a function based on the safety of the front and rear vehicles and the recommended speed function of the corresponding lane:

[0030] Wherein, E defines the upper limit of the penalty and can be regarded as a collision;

[0031]

[0032] Each time step Stored to experience replay area and trained with discrete DRL algorithm.

[0033] Preferably, the process of centralized decision-making of autonomous driving completed by training through a continuous DRL algorithm comprises,

[0034] The acceleration sub-policy network, the adjustment sub-policy network and the deceleration sub-policy network of the longitudinal lower-layer network are input into the designed longitudinal local state space Fl is the number of features designed by the sub-network;

[0035] The acceleration sub-policy network, the adjustment sub-policy network and the deceleration sub-policy network output through the shared longitudinal local state space through multiple fully connected layers The specific acceleration value, wherein the acceleration sub-policy network, the adjustment sub-policy network and the deceleration sub-policy network output and update at each time step.

[0036] Preferably, the acceleration sub-policy network, the adjustment sub-policy network and the deceleration sub-policy network of the longitudinal lower-layer network share the same local state space The unmanned vehicle feature matrix N The unmanned vehicle feature matrix N t L is input into the longitudinal sub-policy network.

[0037] The unmanned vehicle feature matrix N t l includes the speed V i of the ego vehicle, the lateral position X i of the ego vehicle, the longitudinal position Y i of the ego vehicle, the relative speed ΔV 1f of the front vehicle, the longitudinal relative displacement ΔY 1f of the front vehicle, the relative speed ΔV 1r of the rear vehicle, the longitudinal relative displacement ΔY 1r of the rear vehicle, the standard safety distance coefficient κ ij of the front and rear vehicles of the unmanned vehicle, the road segment R i on which the ego vehicle is located, the lane L i ,

[0038] The unmanned vehicle feature matrix N t l The expression of the unmanned vehicle feature matrix N

[0039]

[0040] Preferably, the reward function of the longitudinal lower-layer network is defined based on the safety of the front and rear vehicles, the corresponding lane recommended speed function and the comfort:

[0041] Particularly, the reward function calculation of the acceleration sub-policy network, the adjustment sub-policy network and the deceleration sub-policy network of the longitudinal lower-layer network is based on the calculation of the unmanned vehicle using the corresponding sub-network after the decision of the longitudinal global policy network for Z time steps in the future;

[0042] wherein, E' defines the upper limit of the penalty and can be regarded as a collision, and the value of E' is smaller than E of the longitudinal global policy network;

[0043]

[0044]

[0045] Energy is the total energy consumption of the unmanned vehicle controlled by the current sub-network at each time step;

[0046] The replay experience of the acceleration sub-policy network, the adjustment sub-policy network and the deceleration sub-policy network is stored into the experience replay area at each time step, and a continuous DRL algorithm is used for training to output continuous acceleration;

[0047] wherein, respectively represent the experience of the acceleration sub-policy network, the adjustment sub-policy network and the deceleration sub-policy network.

[0048] Compared with the prior art, the present application has the following advantages and technical effects:

[0049] The automatic driving decision method based on the hybrid series type hierarchical reinforcement learning of the present application trains the global policy network to output discrete high-level decisions of lane changing, acceleration and deceleration for Z time steps, and trains the sub-policy network to output specific acceleration for a single time step, so that the hierarchical decision is similar to the human brain decision, the internal transparency of the unmanned vehicle decision calculation is improved, the cooperation between the layers is promoted through the long-term goal and the unit execution action, and the possibility of the unmanned vehicle decision landing is increased.

[0050] The present application designs the state space and the reward function for different policy networks. Global vehicle information in the traffic environment is input to the global policy network, and local vehicle information is input to the sub-policy network. In addition, the state space and the reward function of the two global policy networks are also designed and optimized. Through the analysis of data and information at different levels, the multiple global and local networks can more comprehensively understand various aspects of the decision problem, so as to make more accurate, more comprehensive and more effective decisions.

[0051] The upper and lower networks of the application adopt different neural network structures. The two global policy networks adopt a graph convolution network, the purpose of which is to converge the interactive vehicle feature information through the graph convolution network and improve the learning ability of the overall model. In the local state space, the lower network adopts a multi-layer fully connected layer to achieve the training effect and output the final acceleration. The autonomous driving decision problem is divided into a local small problem under the control of the horizontal and vertical long-term goals, so that the complex driving decision problem becomes more manageable and easy to handle. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application, and of the description of these embodiments, are used to explain the application and are not intended to limit the application. In the drawings:

[0053] Figure 1 A framework diagram of the autonomous driving decision method of the embodiment of the application;

[0054] Figure 2 A high-speed road entry and exit scene diagram of the embodiment of the application;

[0055] Figure 3 A horizontal / longitudinal global policy network diagram of the embodiment of the application;

[0056] Figure 4 A reward graph about different lanes under the longitudinal global policy network of the embodiment of the application;

[0057] Figure 5 A reward graph about different lanes under the longitudinal sub-policy network of the embodiment of the application;

[0058] Figure 6 A sub-longitudinal policy network structure diagram of the embodiment of the application. DETAILED DESCRIPTION

[0059] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0060] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0061] As Figure 1 shown, the autonomous driving centralized decision method based on the hybrid series hierarchical reinforcement learning provided by the application comprises,

[0062] The deep reinforcement learning network is constructed and the deep reinforcement learning network is mixed and connected to obtain an upper network and a lower network; the upper network is divided into a horizontal global policy network and a vertical global policy network in the horizontal direction, and is trained by using a discrete DRL algorithm. The horizontal global policy network outputs a lane changing instruction, and the vertical global policy network outputs a vertical control strategy. The vertical lower network is divided into three vertical sub-policy networks, which are trained by using a continuous DRL algorithm, and according to the output of the vertical global policy network, the corresponding sub-policy network is selected and activated to output a continuous acceleration. The horizontal lower control includes an optimal control, a left lane changing module, a right lane changing module and a lane maintaining module generated based on rules.

[0063] Further optimization scheme, M unmanned vehicles in N vehicles in the environment obtain state information matrix S from the environment through interaction with the environment, and the centralized decision layer t M×F F is the total number of vehicle characteristics. According to the different functions of the neural network, the state space is designed respectively, and the horizontal global state space and the vertical global state space FH and FL are the designed feature quantities of the corresponding networks. The acceleration sub-policy network, the adjustment sub-policy network and the deceleration sub-policy network input the designed vertical local state space Fl is the designed feature quantity of the sub-network.

[0064] The horizontal global policy network and the vertical global policy network obtain their own information matrix and Through the graph convolutional neural network and the multi-layer fully connected layer, the greedy strategy is adopted to output the lane changing centralized decision action Where -1 represents activating the left lane changing rule, 0 represents activating the lane maintaining rule, and 1 represents activating the right lane changing rule; in addition, the vertical centralized decision action is output Where -1 represents that the unmanned vehicle adopts the output of the deceleration sub-policy network, 0 represents that the unmanned vehicle adopts the output of the adjustment sub-policy network, and 1 represents that the unmanned vehicle adopts the output of the acceleration sub-policy network; the three sub-policy networks output the specific acceleration value of through the shared vertical local state space and the multi-layer fully connected layer respectively.

[0065] The two global policy networks output and update once every Z time steps. The three sub-policy networks output and update every time step.

[0066] Further, the horizontal global policy network global state space includes a node feature matrix N t H and an adjacency matrix A tH Node feature matrix N t H including the ego vehicle speed V i , ego vehicle lateral position X i , ego vehicle longitudinal position Y i , standard safety distance coefficient K of the adjacent lane vehicles in front and behind the ego vehicle ij , j = 1, 3, 4, 6, which are the coefficients of the right front, left front, right rear and left rear vehicles of the ego vehicle, respectively, the road segment R i , the lane L i , the category I i to which the ego vehicle belongs, and the specific expression is:

[0067]

[0068] The standard safety distance coefficient K is defined as the ratio of the standard safety distance D s between two vehicles to the difference in longitudinal displacement D between two vehicles. The standard safety distance D is defined as the current vehicle speed v e , the speed of the vehicle in front v f , the maximum braking speed a, the communication and mechanical delay time t0, and the sum of the vehicle length and the vehicle spacing length D0. If the vehicle is a background vehicle, K is specified as 1.

[0069] Adjacency matrix A t H The interaction between vehicles is represented as information sharing between vehicles. The calculation is based on five assumptions: a) all ego vehicles within a specified range can share information in the constructed scenario, for example, the i-th ego vehicle shares information with the j-th vehicle within 100 m, represented as a ij = 1; b) information cannot be shared between background vehicles; c) all ego vehicles can share the information of background vehicles within their perception range; d) vehicles can share information with themselves, represented as a ii = 1, and represented as

[0070]

[0071] The node feature matrix N t H and the adjacency matrix A t H are input into the lateral global policy network, which includes fully connected layers, graph convolution layers, aggregation layers and activation layers. The graph convolution neural network is used to extract the topological structure features of the vehicles in the scene.

[0072] The reward function of the lateral global policy network is defined as a function based on safety after lane changing, recommended lane and lane changing comfort:

[0073] E defines the upper limit of the penalty and can be considered as a collision

[0074]

[0075] If frequent lane changes are made within a specified time

[0076] Based on the above design, each time step Stored to the experience replay area and trained with discrete DRL algorithm.

[0077] Further, the longitudinal global policy network global state space Including node feature matrix N t L With adjacency matrix A t L , adjacency matrix A t L Same as A t H Node feature matrix N t L Including the speed of the ego vehicle V i , the lateral position of the ego vehicle X i , the longitudinal position of the ego vehicle Y i , the standard safety distance coefficient κ of the current lane vehicle in front of the unmanned vehicle ij , j = 2, 5, which are the coefficients of the front and rear vehicles of the unmanned vehicle, the road segment R i , the lane L i , the category Ii of the ego vehicle, the lane changing behavior C taken by the unmanned vehicle at the current time i The specific expression is:

[0078]

[0079] The node feature matrix N t L With adjacency matrix A t L Input to the longitudinal global policy network, which includes: fully connected layer, graph convolution layer, aggregation layer and activation layer. Among them, the graph convolution neural network is used to extract the topological structure features of the vehicles in the scene.

[0080] The reward function of the longitudinal global policy network is defined as the safety of the front and rear vehicles and the corresponding lane recommended speed function:

[0081] E defines the upper limit of the penalty and can be considered as a collision

[0082]

[0083] Based on the above design, each time step Stored in the experience playback area and trained using the discrete DRL algorithm.

[0084] Furthermore, the vertical sub-policy network shares the same local state space. Including the autonomous vehicle feature matrix N t l Autonomous vehicle feature matrix N t l Including the vehicle's own speed V i The lateral position of the vehicle itself X i The longitudinal position Y of the vehicle itself i The relative speed ΔV of the vehicle in front 1f The longitudinal relative displacement ΔY of the front vehicle 1f The relative speed ΔV of the following vehicle 1r The longitudinal relative displacement ΔY of the rear vehicle 1r The standard safety distance coefficient κ between vehicles in front and behind the driverless vehicle in the current lane ij j = 2, 5, where j represents the coefficients of the vehicles in front of and behind the autonomous vehicle, respectively, and j represents the road segment R in which the autonomous vehicle is located. i The lane L where the vehicle is located i The lane-changing behavior C of the autonomous vehicle at the current moment. i The specific expression is:

[0085]

[0086] The feature matrix N of the autonomous vehicle t L The input is fed into a vertical sub-policy network, which includes fully connected layers and activation layers.

[0087] The reward function of the longitudinal sub-policy network is defined based on the safety of the vehicles in front and behind, the suggested speed function for the corresponding lane, and comfort: Specifically, the reward function calculation for each sub-network is based on the autonomous vehicle using the corresponding sub-network for the next Z time steps after the vertical global policy network decision. For example, if the i-th autonomous vehicle uses the deceleration sub-policy network for the next Z time steps, then its reward function will only be fed back to the retrieval sub-policy network during this time period.

[0088] E′ specifies the upper limit of the penalty and can be considered as a collision. The value of E′ is smaller than E of the vertical global policy network. The main reason for designing it as a collision is that the global policy network judges based on safety factors, and its decision-making is more focused on safety.

[0089]

[0090]

[0091] Energy is the total energy consumed by the autonomous vehicle controlled by the current sub-network at each time step.

[0092] Based on the above design, the playback experience of the three networks is used at each time step. The experiences of the deceleration, adjustment, and acceleration sub-policy networks are respectively stored in the experience replay area and trained using the continuous DRL algorithm to output continuous acceleration.

[0093] Example 1

[0094] like Figure 2 The illustration shows an embodiment of this application at highway entrances and exits. Five unmanned vehicles out of N vehicles in the environment interact with the environment, and the centralized decision-making layer obtains the state information matrix S from the environment. t M×F F represents the total number of vehicle features. The state spaces of different neural networks are designed separately according to their respective functional objectives. The horizontal global policy network and the vertical global policy network are input into the designed horizontal global state space. With vertical global state space The acceleration sub-policy network, adjustment sub-policy network, and deceleration sub-policy network are inputs to the designed longitudinal local state space. Fl represents the number of features designed for the subnetwork.

[0095] The horizontal global policy network and the vertical global policy network acquire their respective information matrices. and By employing a graph convolutional neural network and multiple fully connected layers, and adopting a greedy strategy, the centralized decision-making actions for lane changing are output separately. -1 represents activating the left lane change rule, 0 represents activating the lane-keeping rule, and 1 represents activating the right lane change rule; additionally, the vertical centralized decision-making action is output. Where -1 represents the output of the deceleration sub-policy network, 0 represents the output of the adjustment sub-policy network, and 1 represents the output of the acceleration sub-policy network; the three sub-policy networks share a vertical local state space and output their respective values ​​through multiple fully connected layers. The specific acceleration value.

[0096] Two global policy networks output and update every 5 time steps. Three sub-policy networks output and update at every time step. Each time step is defined as 0.1 seconds.

[0097] Horizontal Global Policy Network Global State Space Including node feature matrix Nt H with the adjacency matrix A t H . The node feature matrix N t H including the ego vehicle speed V i , ego vehicle lateral position X i , ego vehicle longitudinal position Y i , the standard safety distance coefficient K of the adjacent lane vehicles in front and behind the ego vehicle ij , j = 1, 3, 4, 6, which are the coefficients of the ego vehicle's right front, left front, right rear and left rear vehicles, respectively, the road segment R i = road segment number, the lane L i = 0, 1, 2, the ego vehicle's category I i = 0, 1, respectively representing the ego vehicle and the background vehicle, and the specific expression is:

[0098]

[0099] The standard safety distance coefficient K is defined as the ratio of the standard safety distance D s between two vehicles and the difference in longitudinal displacement D between the two vehicles. The standard safety distance is defined as a function of the current vehicle speed ve, the front vehicle speed v f , the maximum braking speed a = -6 m·s -2 , the communication and mechanical delay time t0=0.5s, and the sum of the vehicle length and the vehicle spacing length D0=5m. If the vehicle is a background vehicle, K is defined as 1.

[0100] with the adjacency matrix A t H The interaction between vehicles is represented as information sharing between vehicles. Its calculation is based on five assumptions: a) all ego vehicles within a specified range can share information in the constructed scenario, for example, the ithego vehicle shares information with the jthvehicle within 100m, represented as a ij = 1; b) information cannot be shared between background vehicles; c) all ego vehicles can share information of background vehicles within their perception range; d) vehicles can share information with themselves, represented as a ii = 1, and represented as

[0101]

[0102] The node feature matrix N t H with the adjacency matrix A t HThe input is fed into the lateral global policy network, which includes fully connected layers, graph convolution layers, aggregation layers, and activation layers. The graph convolutional neural network is used to extract the topological structure features of the vehicles in the scene. The network structure is consistent with that of the lateral global policy network, as shown in Figure 3 .

[0103] The reward function of the lateral global policy network is defined as a function based on safety after lane changing, recommended lane, and lane changing comfort:

[0104] The upper limit of the penalty is defined and can be considered as a collision

[0105]

[0106] If the number of lane changes m ≥ 2 within 5s

[0107] Based on the above design, each time step is stored in the experience replay area and trained using the D3QN algorithm.

[0108] Further, the longitudinal global policy network global state space includes the node feature matrix N t L and the adjacency matrix A t L , the adjacency matrix A t L is the same as A t H . The node feature matrix N t L includes the ego vehicle speed V i , the ego vehicle lateral position X i , the ego vehicle longitudinal position Y i , the standard safety distance coefficient κ of the ego vehicle's front and rear vehicles ij , j = 2, 5, which are the coefficients of the ego vehicle's front and rear vehicles, the road segment R i = road segment number, the lane L i = 0, 1, 2, the ego vehicle's category I i = 0, 1, representing the ego vehicle and the background vehicle, respectively, the lane changing behavior C i = -1, 0, 1, representing left lane changing, no lane changing, and right lane changing, respectively. The specific expression is:

[0109]

[0110] The node feature matrix N t L is combined with the adjacency matrix At L The input to the longitudinal global policy network includes fully connected layers, graph convolution layers, aggregation layers, and activation layers. The graph convolutional neural network is used to extract the topological structure features of the vehicles in the scene.

[0111] The reward function of the longitudinal global policy network is defined based on the safety of the front and rear vehicles and the corresponding lane speed function: As shown in Figure 4 .

[0112] E defines the upper limit of the penalty and can be considered as a collision

[0113] Based on the above design, each time step is stored in the experience replay area and trained using the D3QN algorithm.

[0114] Further, as shown in Figure 6 , the longitudinal sub-policy network shares the same local state space including the unmanned vehicle feature matrix N t l . The unmanned vehicle feature matrix N t l includes the vehicle speed V i , the vehicle lateral position X i , the vehicle longitudinal position Y i , the front vehicle relative speed ΔV 1f , the front vehicle longitudinal relative displacement ΔY 1f , the rear vehicle relative speed ΔV 1r , the rear vehicle longitudinal relative displacement ΔY 1r , the unmanned vehicle current lane front and rear vehicle standard safety distance coefficient κ ij , j = 2, 5, which are the coefficients of the unmanned vehicle front and rear vehicles, the road segment R i = road segment number, the lane L i = 0, 1, 2, the vehicle class I i = 0, 1, representing the unmanned vehicle and the background vehicle, respectively, the lane change behavior C i = -1, 0, 1, representing left lane change, no lane change, and right lane change, respectively. The specific expression is:

[0115]

[0116] The unmanned vehicle feature matrix N t L is input into the longitudinal sub-policy network, which includes fully connected layers and activation layers.

[0117] The reward function of the longitudinal sub-policy network is defined as follows, based on the safety of the front and back vehicles, the corresponding lane speed function, and comfort: In particular, the reward function calculation of each sub-network is based on the calculation of the corresponding sub-network of the autonomous vehicle after the decision of the longitudinal global policy network for the next 5 time steps. For example: the ith autonomous vehicle uses the deceleration sub-policy network for the next 5 time steps, and the reward function generated by it only feeds back to the retrieval sub-policy network during this period.

[0118] E defines the upper limit of the penalty and can be considered as a collision

[0119] As shown in Figure 5 The speed reward function of the sub-network is greater than the gradient of the positive reward of the global network, so that the autonomous vehicle can learn the appropriate speed faster.

[0120]

[0121]

[0122] Based on the above design, the replay experience of the three networks at each time step is representing the experience of the deceleration, adjustment, and acceleration sub-policy networks, respectively, is stored in the experience replay area and trained using the TD3 algorithm to output continuous acceleration.

[0123] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An automatic driving centralized decision-making method based on a hybrid layered reinforcement learning, characterized in that, The method comprises the following steps: a deep reinforcement learning network is constructed and a hybrid network layering is performed on the deep reinforcement learning network to obtain an upper layer network and a lower layer network; the upper layer network is horizontally divided to obtain a horizontal global policy network and a vertical global policy network; the horizontal global policy network outputs a lane-changing instruction and the vertical global policy network outputs a longitudinal control strategy through a discrete DRL algorithm; the lower layer network is divided into a vertical lower layer network and a horizontal lower layer control network, and the vertical lower layer network selects an activated corresponding sub-policy network and outputs a continuous acceleration according to the network output of the vertical global policy through a continuous DRL algorithm; the horizontal lower layer control network changes lanes based on an optimal control rule to complete centralized decision-making of automatic driving; the process of outputting a lane-changing instruction by the horizontal global policy network and outputting a longitudinal control strategy by the vertical global policy network through a discrete DRL algorithm comprises, Define M number of unmanned vehicles in N number of vehicles in the environment, through interaction with the environment, the centralized decision layer obtains the state information matrix from the environment , is the total number of vehicle features; According to the function of different neural networks, a state space is designed, and the lateral global strategy network and the longitudinal global strategy network are respectively input into the designed lateral global state space and the longitudinal global state space , and are respectively the designed feature numbers of the corresponding networks. The lateral global policy network and the longitudinal global policy network obtain respective information matrices With Through the graph convolutional neural network and the multi-layer fully connected layer, a greedy strategy is adopted to respectively output lane-changing centralized decision actions Wherein -1 represents activating a left lane-changing rule, 0 represents activating a lane-maintaining rule, and 1 represents activating a right lane-changing rule; and a longitudinal centralized decision action is output Wherein -1 represents that the unmanned vehicle adopts an output of a deceleration sub-policy network, 0 represents that the unmanned vehicle adopts an output of an adjustment sub-policy network, and 1 represents that the unmanned vehicle adopts an output of an acceleration sub-policy network, wherein the lateral global policy network and the longitudinal global policy network output and update once every Z time steps The lateral global state space The node feature matrix And the adjacency matrix The node feature matrix And the adjacency matrix Is input into the lateral global policy network, and the topological structure features of the vehicles in the scene are extracted by using the graph convolutional neural network. The node feature matrix Including the speed of the ego vehicle , the lateral position of the ego vehicle , the longitudinal position of the ego vehicle , the standard safety distance coefficient of the front and rear vehicles adjacent to the ego vehicle , respectively, the coefficients of the right front, left front, right rear and left rear vehicles of the ego vehicle, the section where the ego vehicle is located , the lane where the ego vehicle is located , the category to which the ego vehicle belongs ; the node feature matrix The expression is: The adjacency matrix For representing the interaction between vehicles as information sharing between vehicles, the expression is: the longitudinal global state space comprising a node feature matrix and an adjacency matrix , the adjacency matrix the adjacency matrix ; inputting the node feature matrix and the adjacency matrix to the longitudinal global policy network, and extracting topological structure features of vehicles in the scene by using a graph convolutional neural network The node feature matrix Including the speed of the ego vehicle , the lateral position of the ego vehicle , the longitudinal position of the ego vehicle , the current lane front and rear vehicle standard safety distance coefficient of the unmanned vehicle , respectively, the coefficient of the front and rear vehicles of the unmanned vehicle, the section where the ego vehicle is located , the lane where the ego vehicle is located , the category to which the ego vehicle belongs ; the node feature matrix The expression is: the process of completing centralized decision-making of automatic driving through a continuous DRL algorithm comprises, The acceleration sub-policy network, the adjustment sub-policy network and the deceleration sub-policy network of the longitudinal lower-layer network are respectively input into the designed longitudinal local state space , The number of features designed for the sub-network The acceleration sub-policy network, the adjustment sub-policy network and the deceleration sub-policy network respectively output specific acceleration values of the longitudinal local state space through a plurality of fully connected layers , , by sharing, wherein the acceleration sub-policy network, the adjustment sub-policy network and the deceleration sub-policy network output and update at each time step.

2. The centralized decision-making method for automatic driving based on hybrid layering reinforcement learning according to claim 1, characterized in that, The reward function of the lateral global policy network is defined as a function based on safety after changing lanes, recommended lane, and comfort of changing lanes: ; wherein, ； Each time step (t) is stored to an experience replay area and trained with a discrete DRL algorithm. ) is stored to an experience replay area and trained with a discrete DRL algorithm.

3. The centralized decision-making method for automatic driving based on hybrid layering reinforcement learning according to claim 1, characterized in that, The reward function of the longitudinal global policy network is defined as based on front-back vehicle safety, corresponding lane suggested speed function: ; wherein, ； Each time step (t) is stored to an experience replay area and trained with a discrete DRL algorithm. ) is stored to an experience replay area and trained with a discrete DRL algorithm.

4. The centralized decision-making method for automatic driving based on hybrid layering reinforcement learning according to claim 1, characterized in that, The acceleration sub-policy network of the longitudinal lower-level network shares the same local state space with the adjustment sub-policy network and the deceleration sub-policy network including an unmanned vehicle feature matrix and inputting the unmanned vehicle feature matrix to a longitudinal sub-policy network The unmanned vehicle feature matrix Including the speed of the ego vehicle The lateral position of the ego vehicle The longitudinal position of the ego vehicle The relative speed of the front vehicle The longitudinal relative displacement of the front vehicle The relative speed of the rear vehicle The longitudinal relative displacement of the rear vehicle The current vehicle standard safety distance coefficient of the unmanned vehicle The coefficients of the front and rear vehicles of the unmanned vehicle, respectively, the road section where the ego vehicle is located The lane where the ego vehicle is located , The unmanned vehicle feature matrix The expression is: 。 5. The centralized decision-making method for automatic driving based on hybrid layering reinforcement learning according to claim 1, characterized in that, the reward function of the vertical lower layer network is defined based on front and rear vehicle safety, a corresponding lane speed function and comfort: ； In particular, the reward function calculation of the acceleration sub-policy network, the adjustment sub-policy network and the deceleration sub-policy network of the vertical lower layer network is based on the calculation of the unmanned vehicle using the corresponding sub-network in the future Z time steps after the decision of the vertical global policy network; wherein, , the value of the lateral global policy network is to be smaller than the value of the longitudinal global policy network ; , total energy consumed by the ego vehicle controlled by the current subnetwork for each time step; the replay experience of the acceleration sub-policy network, the adjustment sub-policy network and the deceleration sub-policy network is stored in an experience replay area at each time step, and a continuous DRL algorithm is used for training to output a continuous acceleration; wherein, , respectively, represent experience of the accelerating sub-policy network, the adjusting sub-policy network, and the decelerating sub-policy network.

Images