An Efficient Reinforcement Learning Method for Multi-Vehicle Intersection Coordination Decision-Making and Control

Abstract

The present invention discloses an efficient reinforcement learning method for multi-vehicle intersection coordination decision-making and control, including: off-policy training, extracting features of the collected high-dimensional samples to obtain approximately linearly independent sub-samples, and using the sub-samples to construct basis functions to obtain an approximation structure of the action-state value function; in the online deployment control: according to the observed real-time state quantities of the unmanned vehicles, deploying a decision-making strategy to obtain a utility function after the unmanned vehicles adopt actions, and obtaining a local joint action reward function according to the collaborative relationship between the unmanned vehicles, so as to obtain the corresponding global value function and decision-making actions, and synchronously updating the control strategies and current states of the unmanned vehicles. The present invention is applied to the field of multi-unmanned vehicle intersection collaborative decision-making and control, and has the advantages of strong value function representation ability and high online calculation efficiency, and can improve the timeliness and safety of multi-unmanned vehicle intersection passing.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-unmanned vehicle collaborative control, and specifically to an efficient reinforcement learning method for multi-vehicle intersection coordination decision-making and control. Background Art

[0002] In intelligent logistics systems, unmanned vehicles play an important role in scenarios such as material stacking, sorting, and handling. Among them, logistics unmanned vehicles, also known as Automatic Guided Vehicles (AGVs), are mainly responsible for material transportation work. In intelligent transportation systems, most research focuses on the safe passage of individual autonomous unmanned vehicles, and further research is still needed on the safe and efficient decision-making control problem of multi-unmanned vehicle systems at intersections. In the current existing technologies, there are proposals to predict the future path of autonomous unmanned vehicles through digital maps, identify potential threat and collision areas, and use Bayesian inference and time window filtering methods for action decision-making. There are also proposals for a hierarchical decision-making and planning method based on generalized key turning points. The upper-level planner extracts parameterized models to generate behavior-oriented paths, and the lower-level planner conducts real-time two-dimensional planning. Currently, most of the control of intelligent logistics systems only focuses on the safety and effectiveness of the passage of a single autonomous unmanned vehicle, but the coordination mechanism for the passage of multiple unmanned vehicles at intersections remains to be studied. Based on the above analysis, there is little research on the decision-making problem of coordinated passage of multiple unmanned vehicles at signal-less intersections in the logistics system. The present invention combines a collaborative graph to study a collaborative decision-making and control mechanism for multi-unmanned vehicle systems based on reinforcement learning, and realizes the safe and efficient passage of multiple unmanned vehicles. Summary of the Invention

[0003] Aiming at the problem of low passing efficiency of multiple unmanned vehicles at signal-less intersections in the existing intelligent logistics system, the present invention provides an efficient reinforcement learning method for multi-vehicle intersection coordination decision-making and control, which can enhance the representation ability of the reinforcement learning value function of multi-unmanned vehicles, improve the efficiency of collaborative decision-making, and thus realize the safe and efficient passage of multiple unmanned vehicles at signal-less intersections.

[0004] To achieve the above object, the present invention provides an efficient reinforcement learning method for multi-vehicle intersection coordination decision-making and control, including three stages: sample data collection, offline policy training, and online deployment control;

[0005] In the sample data collection:

[0006] According to the Markov decision process, a sample set for each unmanned vehicle is generated based on random sampling, where the size of the sample set for each unmanned vehicle is and the sample tuple at each time step contains representing the current state at time Under the following circumstances, a random decision is made on the action space to obtain an action . This action will drive the autonomous vehicle to update its state to obtain . The reward function is ;

[0007] In the off-policy training:

[0008] Based on the sample set of each autonomous vehicle, the kernel sparsification method is used to extract the features of the collected high-dimensional samples to obtain approximately linearly independent sub-samples, and the sub-samples are used to construct the basis functions corresponding to each original sample point to obtain the approximation structure of the action-state value function. Then, with the goal of minimizing the temporal difference error of reinforcement learning, the network weight vector in the approximation structure of the action-state value function is updated;

[0009] In the online deployment control:

[0010] According to the observed real-time state variables of the autonomous vehicle, the decision-making strategy is deployed to obtain the action-state value function corresponding to each action, that is, the utility function after the autonomous vehicle adopts the action, and the local joint action reward function is obtained according to the cooperation relationship between autonomous vehicles. Based on the utility function and the local joint action reward function, the global value function of the autonomous vehicle is obtained, and the decision-making action of the autonomous vehicle is obtained according to the global value function. Finally, according to the decision-making action of the autonomous vehicle, the control strategy of the autonomous vehicle is learned to update the state of the autonomous vehicle.

[0011] Compared with the prior art, the present invention has the following beneficial technical effects:

[0012] The present invention includes three stages: sample data collection, off-policy training, and online deployment control. The off-policy training adopts the least squares policy iteration reinforcement learning method based on sparse kernels to construct high-dimensional sample features and learn approximate optimal policies; in the online deployment control, the autonomous vehicle obtains the local action-behavior value function under different decisions according to the learned policy, and at the same time establishes a cooperative edge with neighboring autonomous vehicles, introduces a reward function representing the performance of joint actions, and multiple autonomous vehicles solve the optimized joint decision through iterative message propagation on the cooperative edge. Taking the decision result as the expected value, each autonomous vehicle uses the rolling horizon reinforcement learning method for trajectory tracking control, which has obvious advantages in the cooperative passing efficiency and safety of multiple autonomous vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on the structures shown in these drawings.

[0014] Figure 1 It is a flowchart of an efficient reinforcement learning method for multi-vehicle intersection coordination decision-making and control in an embodiment of the present invention;

[0015] Figure 2 It is a schematic diagram of multi-unmanned vehicle reinforcement learning in an embodiment of the present invention;

[0016] Figure 3 It is a message propagation diagram between unmanned vehicles in an embodiment of the present invention;

[0017] Figure 4 It is a schematic diagram of the scenario of a simulation example in an embodiment of the present invention;

[0018] Figure 5 It is a schematic diagram of the collaborative decision-making of the unmanned vehicle system at 0.1 s in a simulation example in an embodiment of the present invention;

[0019] Figure 6 It is a schematic diagram of the collaborative decision-making of the unmanned vehicle system at 2.5 s in a simulation example in an embodiment of the present invention;

[0020] Figure 7 It is a schematic diagram of the collaborative decision-making of the unmanned vehicle system at 3.4 s in a simulation example in an embodiment of the present invention;

[0021] Figure 8 It is a schematic diagram of the collaborative decision-making of the unmanned vehicle system at 4.2 s in a simulation example in an embodiment of the present invention;

[0022] Figure 9 It is a schematic diagram of the collaborative decision-making of the unmanned vehicle system at 5 s in a simulation example in an embodiment of the present invention;

[0023] Figure 10 It is a schematic diagram of the collaborative decision-making of the unmanned vehicle system at 5.7 s in a simulation example in an embodiment of the present invention;

[0024] Figure 11 It is a schematic diagram of the collaborative decision-making of the unmanned vehicle system at 6.1 s in a simulation example in an embodiment of the present invention;

[0025] Figure 12 It is a schematic diagram of the collaborative decision-making of the unmanned vehicle system at 7.5 s in a simulation example in an embodiment of the present invention;

[0026] Figure 13This is a schematic diagram of the collaborative decision-making of the unmanned vehicle system at 8.2 s in the simulation example of the embodiment of the present invention.

[0027] The realization, functional features and advantages of the present invention will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0028] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0029] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement them. When the combination of technical solutions appears to be contradictory or unable to be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0030] As Figure 1 shown, a high-efficiency reinforcement learning method for multi-vehicle intersection coordination decision-making and control disclosed in this embodiment includes three stages: sample data collection, offline policy training, and online deployment control;

[0031] In sample data collection:

[0032] According to the Markov decision process, a sample set of each unmanned vehicle is generated based on random sampling;

[0033] In offline policy training:

[0034] Based on the sample set of each unmanned vehicle, a kernel sparsification method is used to extract the features of the collected high-dimensional samples, obtaining approximately linearly independent sub-samples, and using the sub-samples to construct the basis functions corresponding to each original sample point, obtaining an approximation structure of the action-state value function, and then updating the network weight vector in the approximation structure of the action-state value function with the goal of minimizing the temporal difference error of reinforcement learning;

[0035] In online deployment control:

[0036] According to the observed real-time state quantities of the unmanned vehicles, a decision-making strategy is deployed to obtain the action-state value functions corresponding to each action, that is, the utility function after the unmanned vehicle adopts the action, and a local joint action return function is obtained according to the collaborative relationship between the unmanned vehicles. Based on the utility function and the local joint action return function, the global value function of the unmanned vehicle is obtained, and the decision-making action of the unmanned vehicle is obtained according to the global value function. Finally, the control strategy of the unmanned vehicle is learned according to the decision-making action of the unmanned vehicle, and the state of the unmanned vehicle is updated.

[0037] Reference Figure 2 , in this embodiment, the multi-unmanned vehicle collaborative decision-making problem is modeled as a Decentralized-Partially Observable Markov Decision Process (Dec-POMDP). Dec-POMDP can effectively handle the serial decision-making problem of multi-unmanned vehicles and is specifically described by the following tuple:

[0038] ;

[0039] Among them, is the state space of is the state of the th unmanned vehicle at time ; is the action space of the th unmanned vehicle, is the action decided by the th unmanned vehicle at time ; is the probability of state transition, that is , ; is the reward function, indicating that the unmanned vehicle at state , after taking the joint action , the return obtained at state , that is , is the discount factor, is the observation space of the th unmanned vehicle.

[0040] In the specific implementation process, the reward functions of each unmanned vehicle can be different, generally set according to specific task requirements. When the multi-unmanned vehicles are in a cooperative relationship, the same reward function is set. When the multi-unmanned vehicles are in a competitive and adversarial relationship, the two competing parties have opposite reward functions. When the reward function is between the two, it indicates that the multi-unmanned vehicles are in a mixed relationship.

[0041] In the stage of sample data collection, by randomly initializing the distance of the unmanned vehicle within the range of the distance from the road , and randomly initializing the speed of the unmanned vehicle within the speed range , is the maximum speed of the unmanned vehicle, and each unmanned vehicle collects groups of data as a sample set. Among them, for the unmanned vehicle , the sample tuple at each time step contains , representing the unmanned vehicle at the current state at moment , under which, a random decision is made for an action in the action space , and this decision-making action will drive the unmanned vehicle to update its state to obtain , and the reward function is . Among them, the state of the unmanned vehicle at moment is , the execution policy is , , is the speed of the unmanned vehicle , is the speed of the unmanned vehicle and the nearest unmanned vehicle in front, is the distance between the unmanned vehicle and the nearest unmanned vehicle in front, is the speed of the unmanned vehicle and the nearest unmanned vehicle behind, is the speed of the unmanned vehicle and the nearest unmanned vehicle behind.

[0042] The Probabilistic Graphical Model (PGM) combines probability theory and graph theory, including the representation, inference, and learning problems of directed graph models (also known as Bayesian networks) or undirected graph models (also known as Markov random fields). From the perspective of graph theory, PGM is a graph that contains nodes and edges. Nodes can be divided into two categories: hidden nodes and observed nodes, and edges can be directed or undirected. From the perspective of probability theory, PGM is a probability distribution. The nodes in the graph correspond to random variables, and the edges correspond to the relationships between random variables. In the multi-unmanned vehicle system of this embodiment, the nodes represent unmanned vehicles, and the edges with connection relationships describe the local performance of adjacent unmanned vehicles taking joint actions, that is, inferring the posterior distribution of joint actions based on the current unmanned vehicle information.

[0043] The multi-unmanned vehicle reinforcement learning algorithm based on value function decomposition usually adopts a sequence decision-making framework of centralized training and decentralized execution, decomposing the global action-behavior value function into the sum of value functions of each unmanned vehicle , that is , but does not consider the local cooperation performance after neighboring unmanned vehicles adopt different actions , in the observation space. And this embodiment introduces a collaborative graph through a collaborative reinforcement learning method to establish connection edges between unmanned vehicles with collaborative relationships, and the messages transmitted on the edges can fully describe the coordination relationship among multiple unmanned vehicles, that is Figure 3 as shown. Combining the cooperation graph with the Variable Elimination (VE) algorithm or the Max-plus algorithm can improve the representation ability of the value function and effectively solve the decision-making problem of the multi-unmanned vehicle system. At this time, the global value function is expressed as:

[0044] ;

[0045] where is the utility function of the unmanned vehicle after adopting the action ; is the local joint action return function of the adjacent unmanned vehicles and the unmanned vehicle after respectively adopting the actions , ; is the set of unmanned vehicles;

[0046] The unmanned vehicle sends a message to its neighbor unmanned vehicle . This message indicates that after the unmanned vehicle adopts the action , the unmanned vehicle can obtain the maximum return. In this embodiment is specifically:

[0047] ;

[0048] where represents belongs to the set of unmanned vehicles in the neighbor set of the unmanned vehicle excluding the unmanned vehicle

[0049] Before the message of the unmanned vehicle converges, sum the messages received from the neighbor nodes, but there is no need to enumerate all the joint action spaces of the neighbor nodes. After several iterations of sending and updating messages between neighbor unmanned vehicles, it will converge to a fixed point. At each iteration, the global value function of the unmanned vehicle is expressed as:

[0050] ;

[0051] where is the unmanned vehicle after adopting the action The global value function after for the driverless vehicle is the message sent to neighboring driverless vehicles, for the driverless vehicle is the set of neighbors, that is, the global value function of the driverless vehicle is composed of the utility function of the driverless vehicle and the message values of different subtrees rooted at neighboring driverless vehicles ;

[0052] After obtaining the global value function of the driverless vehicle , the optimal decision-making action of the driverless vehicle can be obtained, which is:

[0053] ;

[0054] Among them, is the decision-making action of the driverless vehicle.

[0055] In this embodiment, after obtaining a sufficient number of samples in the stage of sample data collection, the Kernel-based Least-squares Policy Iteration (KLSPI) reinforcement learning method is used to construct the utility function of the driverless vehicle after adopting the action , and its specific implementation process is as follows: First, the kernel sparsification method is used to extract the features of the collected high-dimensional samples to obtain approximately linearly independent sub-samples. When the adopted kernel function

[0056] satisfies the Mercer kernel condition, there exists a mapping from the sample space to the Hilbert space, that is:

[0057] ;

[0058] Among them, , are the th and th training samples, is the inner product operation in the Hilbert space, indicating that all inner product operations in the Hilbert space can be calculated through the kernel function without knowing the specific mapping form ;

[0059] Then, the basis functions corresponding to each original sample point are constructed using the sub-samples, which are:

[0060] ;

[0061] Among them, is the basis function corresponding to the original sample point, is the kernel function, 、 、 、 are to the training samples of the th driverless vehicle at different times, is the transpose of the matrix, is dimensional feature space;

[0062] After that, construct the approximation structure of the action-state value function, which is:

[0063] ;

[0064] Among them, is the approximation structure of the action-state value function, is the network weight vector, is the weight matrix;

[0065] Then, update the network weight vector in the approximation structure of the action-state value function with the goal of minimizing the temporal difference error of reinforcement learning, where the temporal difference error of reinforcement learning is:

[0066] ;

[0067] ;

[0068] Among them, is the temporal difference error of reinforcement learning, is the target action-state value function, is the candidate decision-making action, 、 are positive definite weight matrices, is the action-state value function after state transition;

[0069] After that, combine the approximation structure of the action-state value function in formula (8) and rewrite the temporal difference error of reinforcement learning in formula (9) as:

[0070] ;

[0071] Then, multiply both sides of the rewritten temporal difference error of reinforcement learning by the basis function , and let the intermediate parameters 、 be:

[0072] ;

[0073] ;

[0074] Finally, since the intermediate parameter is full rank, the least squares solution of the network weight vector is obtained, that is .

[0075] Thus, by adopting the method based on kernel features, the decision-making strategy of the driverless vehicle is pre-learned from the collected high-dimensional samples, and then the decision-making strategy is deployed online. According to the observed real-time state variables, the action-state value function corresponding to each action is obtained by deploying the decision-making strategy, that is, the utility function adopted by the driverless vehicle is obtained. .

[0076] In addition, according to the basic theory of the cooperation graph, in the multi-driverless vehicle collaborative decision-making problem of the intelligent logistics system, the global utility function based on the cooperation graph is expressed as:

[0077] ;

[0078] Equation (14) shows that by iteratively interacting the local joint action rewards on the edges with cooperation relationships, the optimal decisions of all driverless vehicles can be solved without solving the decision-making actions of multiple driverless vehicles according to the overall high-dimensional joint action space;

[0079] After the driverless vehicle obtains the neighbors with cooperation relationships, the local joint action reward function on the edge is further defined . At the current moment, the state variables of the driverless vehicle are expressed as , where , respectively represent the horizontal and vertical coordinates of the driverless vehicle , and the cooperative driverless vehicle state variables , . After the driverless vehicle executes the joint action, the cooperative state variables will also be updated to obtain . In this embodiment, the local joint action reward function defined on the cooperative edge of adjacent driverless vehicles in the current state is:

[0080] ;

[0081] where is the local joint action reward function after the adjacent driverless vehicles and adopt the actions , respectively, For the predicted driverless vehicle the distance to the target position, For the predicted driverless vehicle the distance to the target position;

[0082] Specifically:

[0083] ;

[0084] Among them, , , is the position of the target point.

[0085] Characterizes the influence of the joint actions among driverless vehicles with a collaborative relationship. According to the constructed dynamic collaboration graph and the joint message iterative propagation mechanism, the driverless vehicle will make a decision on the joint action that can make the predicted distances differ greatly, that is, drive the driverless vehicle closer to the target point at the current moment to reach the target first, and at the same time avoid collisions between driverless vehicles. This is also in line with the actual situation in the intelligent transportation system where, at intersections without traffic signal indications, driverless vehicles in different lanes will make a decision to let the driverless vehicle closer to the intersection pass first.

[0086] When constructing the driverless vehicle using the kernel-based least squares policy iteration reinforcement learning method Adopt action The utility function after , and the local joint action return function obtained according to Equation (15) , you can substitute them into Equations (3), (4), and (5) to obtain the decision-making actions of the driverless vehicle, so as to learn the control strategy of the driverless vehicle system according to the decision-making actions of the driverless vehicle and update the state of the driverless vehicle.

[0087] The driverless vehicle system with non-linear characteristics is expressed as:

[0088] ;

[0089] Among them, , are respectively the state and control input of the driverless vehicle at time, is the non-linear dynamic characteristic of the system state, which is only related to the system state and represents the free movement of the system without external input, is the influence of the input on the state change, reflecting the input coupling characteristic of the system. Assume is Lipschitz continuous on a set containing the origin, and at There exists a control strategy in it to make the system asymptotically stable;

[0090] The desired trajectory tracked by the unmanned vehicle system in Equation (17) is expressed as:

[0091] ;

[0092] where, is the state quantity of the desired trajectory of the unmanned vehicle at time.

[0093] In the collaborative decision-making and control problem of the multi-unmanned vehicle system, the speed obtained from the collaborative decision-making is used as the expected value to perform trajectory tracking control on the unmanned vehicle. The goal of the controller design is to solve an optimized controller so that the state of the unmanned vehicle can track the reference trajectory. Therefore, in this embodiment, the tracking error is defined as . Combining Equation (17) and Equation (18) further gives:

[0094] ;

[0095] For the tracking control problem of the unmanned vehicle system in Equation (17), the finite-time domain performance index is defined as:

[0096] ;

[0097] ;

[0098] where, is the reward function of the unmanned vehicle , is the terminal penalty matrix, , are the preset positive definite weight matrices, is the length of the prediction time domain, , is the terminal domain, is the terminal penalty function.

[0099] The nonlinear system in Equation (19) is expressed as linearized at the origin:

[0100] ;

[0101] where, the intermediate parameter , the intermediate parameter ;

[0102] The terminal penalty matrix can be obtained by solving the Riccati equation of the discrete-time system in Equation (22).

[0103] An optimization strategy for the tracking control of an autonomous vehicle is solved by using a rolling horizon mechanism. At moment, an evaluator-actor reinforcement learning with a prediction horizon length of is performed to obtain a control sequence . The first control quantity is applied to the controlled system, the system state quantity is updated, and this is used as the new initial moment to enter the next prediction horizon. At moment, the feasible solution of the optimization problem is also feasible at moment. By repeating the above steps, online rolling horizon learning control can be achieved.

[0104] At the sampling moment , with the current tracking error of the autonomous vehicle as the initial state, reinforcement learning is carried out within the prediction horizon . The value function within the prediction horizon is defined as:

[0105] ;

[0106] According to the Bellman optimization principle, the optimal value function satisfies the HJB equation of the discrete-time system, that is:

[0107] ;

[0108] The co-state of the value function is defined, where is the target value function within the prediction horizon;

[0109] Then the optimal co-state is:

[0110] ;

[0111] Therefore, the optimal control strategy is:

[0112] ;

[0113] Among them, is the control input of the autonomous vehicle system calculated at when performing the decision-making action at

[0114] It can be seen from Equation (26) that it is difficult to obtain an analytical expression of the control quantity for a complex system. Therefore, in this embodiment, within the prediction horizon, a multi-group evaluator-actor reinforcement learning framework is adopted, and based on the value iteration method, the time-varying value function and the optimized control quantity are approximated by a neural network.

[0115] In the specific implementation process, both the actuator and the evaluator are constructed using a three-layer neural network. Both the actuator and the evaluator include an input layer, a hidden layer, and an output layer.

[0116] In the evaluator, the input of the input layer is the tracking error of the unmanned vehicle , the number of nodes in the hidden layer is , and the output of the output layer is the co-state ;

[0117] The structure of the evaluator approximating the co-state through a neural network is:

[0118] ;

[0119] Among them, is the weight of the evaluator network from the hidden layer to the output layer, is the weight of the evaluator network from the input layer to the hidden layer, is the activation function;

[0120] Within the prediction time domain, the evaluator updates the network based on the time-domain differential error , and the time-domain differential error is:

[0121] ;

[0122] Among them, is the target co-state obtained by substituting the estimated co-state into Equation (25), expressed as:

[0123] ;

[0124] ;

[0125] The goal of the evaluator network is to minimize the error function ;

[0126] Based on the gradient descent method, the weight update rule of the evaluator network is:

[0127] ;

[0128] ;

[0129] ;

[0130] ;

[0131] Among them, is the learning rate of the evaluator network, and .

[0132] In the actuator, the input of the input layer is the tracking error of the driverless vehicle. , the number of nodes in the hidden layer is , and the output of the output layer is the control quantity of the driverless vehicle. ;

[0133] The actuator adopts a three-layer network structure to approximate the control quantity, , which is:

[0134] ;

[0135] Among them, is the weight of the actuator network from the hidden layer to the output layer, is the weight of the actuator network from the input layer to the hidden layer;

[0136] In the prediction time domain, the actuator updates the network based on the time-domain differential error , and the time-domain differential error is:

[0137] ;

[0138] Among them, is the target control quantity obtained by substituting the estimated control quantity into Equation (26), expressed as;

[0139] ;

[0140] The goal of the actuator network is to minimize the error function ;

[0141] Based on the gradient descent method, the weight update rule of the actuator network is:

[0142] ;

[0143] ;

[0144] ;

[0145] ;

[0146] Among them, is the learning rate of the actuator network, and ;

[0147] Due to the real-time requirement of the solution and the limitation of actual computing resources, it is impossible to perform an infinite number of iterations for reinforcement learning in the prediction time domain. Therefore, the termination condition of reinforcement learning in the prediction time domain in this embodiment is:

[0148] ;

[0149] ;

[0150] Among them, is the number of iterations of reinforcement learning within the prediction horizon, is the threshold for the convergence of the weights of the critic network, is the threshold for the convergence of the weights of the actor network;

[0151] When the network weights of two consecutive iterations satisfy the termination condition, it is considered that a convergent policy has been learned, and the first actor network in the optimized solution control sequence at this time is applied to the actual controlled unmanned vehicle system. On the other hand, the maximum number of learning times can also be given. If the policy has not converged within , the network is re-initialized and the actor-critic learning within the prediction horizon is carried out again.

[0152] It should be noted that, under the rolling horizon reinforcement learning mechanism, the length of the prediction horizon is . At , an optimized control sequence is learned within the prediction horizon, and the first control element is applied to the controlled unmanned vehicle system. In two adjacent prediction horizons, the critic network and the policy network are continuous, that is, the last actor networks learned at will be passed to the first prediction steps at . The corresponding critic network will also be passed to the next prediction horizon. That is, the previous experience results are fully utilized during the learning process, which helps to accelerate the convergence speed of learning. The number of learning iterations within subsequent prediction horizons gradually decreases, improving the real-time performance of the optimized policy solution and facilitating transplantation into the actual system.

[0153] The following further illustrates the efficient reinforcement learning method for multi-vehicle intersection coordination decision-making and control in this embodiment with specific simulation examples.

[0154] In Figure 4 the simulation scenario of an intersection without traffic lights shown, there are straight unmanned vehicles, right-turn unmanned vehicles, and left-turn unmanned vehicles. At the same time, there are many social unmanned vehicles in the scenario. The left-turn unmanned vehicle may collide with the right-turn unmanned vehicle, and a cooperative relationship needs to be established so that vehicles in different lanes can make reasonable joint decisions to safely pass through the intersection.

[0155] The analysis of the cooperation process of unmanned vehicles in the simulation scenario is as follows:

[0156] When the straight unmanned vehicle enters the decision-making area, it observes the running speed and relative distance of the unmanned vehicle in the left-turn lane to obtain the state of the straight unmanned vehicle , at the same time, establish a collaborative relationship with the driverless vehicle in the left-turn lane, deploy the learned decision-making strategy to obtain the behavior-action value function under discrete actions, combine the local action reward function, and use the collaborative message passing mechanism to derive the optimized decision-making action;

[0157] Similarly, when the driverless vehicle in the left-turn lane enters the decision-making area, establish collaborative relationships with the driverless vehicles going straight and turning right respectively, observe the states of the driverless vehicles in the relevant lanes, and use the method of this embodiment to obtain the optimized joint decision-making actions. When the driverless vehicle in the right-turn lane enters the decision-making area, establish a neighbor collaborative relationship with the driverless vehicle in the left-turn lane.

[0158] According to the above analysis of the collaborative process of vehicles, deploy the learned strategy in the crossroads passing scenario to test the performance of the method of this embodiment. Figures 5 to 13 Respectively show the decision-making states of the driverless vehicles at some typical moments in a simulation scenario. Among them, Figure 5 is the schematic diagram of the collaborative decision-making of the driverless vehicle system at 0.1 s, Figure 6 is the schematic diagram of the collaborative decision-making of the driverless vehicle system at 2.5 s, Figure 7 is the schematic diagram of the collaborative decision-making of the driverless vehicle system at 3.4 s, Figure 8 is the schematic diagram of the collaborative decision-making of the driverless vehicle system at 4.2 s, Figure 9 is the schematic diagram of the collaborative decision-making of the driverless vehicle system at 5 s, Figure 10 is the schematic diagram of the collaborative decision-making of the driverless vehicle system at 5.7 s, Figure 11 is the schematic diagram of the collaborative decision-making of the driverless vehicle system at 6.1 s, Figure 12 is the schematic diagram of the collaborative decision-making of the driverless vehicle system at 7.5 s, Figure 13 is the schematic diagram of the collaborative decision-making of the driverless vehicle system at 8.2 s. The color identifications of the driverless vehicles c0~c11 in Figures 6 to 13 are the same as those in Figure 5 . At the initial moment, the driverless vehicles on each lane start from the initial positions at a distance of from the intersection. At the same time, the initial speeds of each driverless vehicle are also random. Before entering the decision-making area, the driverless vehicle performs tracking control with an expected speed of 8 m / s.

[0159] As Figure 5 shows, when , it indicates the expected passing direction of the driverless vehicle. The ones marked with black boxes represent the social driverless vehicles on that lane, and due to traffic control and other reasons, the road on that lane is congested, causing the left-turn driverless vehicle to pass to the lane with less traffic flow;

[0160] As Figure 6 shows, when When the straight - running autonomous vehicle c6 observes that both the front and rear are within the safe distance range in the decision area at this moment, it decides to accelerate. At this time, the left - turning autonomous vehicle c8 observes that c0 has just passed in front, so for safety, c8 decides to decelerate;

[0161] As Figure 7 shown, when the left - turning autonomous vehicle c2 enters the decision area, it obtains that the autonomous vehicle c6 is going straight from behind at a certain speed. The method of this embodiment decides the combined action of c2 decelerating and c6 accelerating. At this time, the left - turning autonomous vehicle c8 also enters the decision area, obtains the states of the neighboring autonomous vehicles in front and behind, and further decides the action of c8 accelerating;

[0162] As Figure 9 shown, when the straight - running c7 enters the decision area, it establishes a cooperative relationship with the left - turning c2 and c3. The combined strategy and the current state result in c7 decelerating and waiting, c2 accelerating through the lane intersection point. At this time, c3 adjusts its speed according to the decided action and the autonomous vehicle following control. At this time, the straight - running c1 also enters the decision area, and similarly, the algorithm decides the decelerating action;

[0163] As Figure 10 shown, when the left - turning c2 observes that there is an autonomous vehicle c11 turning right in front. Therefore, c2 will decelerate, c7 will continue to decelerate and wait. The straight - running c1, based on the fact that the previous vehicle c8 has passed and the following vehicle c9 is in a decelerating and waiting state at this time, and c1 maintains a relatively safe distance from c9, thus c1 decides to accelerate and pass. c8 enters a new decision area, establishes a cooperative relationship with the right - turning autonomous vehicle, and according to the learned strategy, obtains the combined action of c5 decelerating and c8 accelerating, which also conforms to the driving rule of yielding to the left - turning vehicle when turning right in daily life;

[0164] As Figure 11 shown, when among the neighboring autonomous vehicles with a cooperative relationship, the autonomous vehicles c2, c5, and c9 located in their respective decision areas obtain the states of the autonomous vehicles in front and behind, and after deploying the strategy, decide the discrete action of continuing to wait;

[0165] As Figure 12 shown, after the autonomous vehicle runs for a period of time, when the phenomenon of heavy traffic of multiple vehicles at the previous intersection has been greatly alleviated, and many autonomous vehicles have traveled to the desired lanes;

[0166] As Figure 13 shown, when it can be seen that the autonomous vehicle has basically passed through the intersection.

[0167] At Figure 4In the cross - road passing scenario without traffic signal indication shown, the current unmanned vehicle establishes a collaborative relationship with neighboring unmanned vehicles, deploys the learned strategy, observes the current state, and infers a reasonable joint decision action by transmitting messages on the collaborative edge to safely pass through the intersection. A total of 100 tests are carried out. In each test, the initial position of the unmanned vehicle is random within the intersection, and the speed of the unmanned vehicle is random within m / s. The passing time, collision rate, failure rate, and comfort index of the unmanned vehicle at the intersection are statistically analyzed. Among them, the passing time refers to the time after all unmanned vehicles pass through the intersection and reach the desired lane. The collision rate refers to the number of collisions between unmanned vehicles in 1000 tests. The failure rate refers to the proportion of the number of times that there are still unmanned vehicles within the intersection and have not completed the intersection passing task within 10 s. Comfort is calculated as the average value of the root - mean - square of the acceleration of the unmanned vehicle. The smaller this value is, the higher the comfort. Comparing the performance of the method of this embodiment with the existing independent learning method, the results are shown in Table 1.

[0168]

[0169] According to Table 1, the average value and standard deviation of the passing time of the independent learning method are both greater than those of the method of this embodiment. The collision rate of independent learning is 26%. The failure rate of independent learning refers to the proportion of the number of times that a passing task has not been completed within 13 s, and the failure rate of the independent learning method is 14 . 5%. The comfort of the method of this embodiment is better than that of the independent learning method. Thus, the decision - making control performance of the method of this embodiment in terms of the passing efficiency, safety, and driving smoothness of multiple unmanned vehicles at the intersection is superior to the independent learning method.

[0170] This is because in the collaborative decision - making process of the multi - unmanned vehicle system, the method of this embodiment decomposes the overall value function of the system into two parts: a single action - state value function and the utility of the joint action between unmanned vehicles, which is different from the independent learning method that only focuses on individual performance. Considering the reward of the joint action while deploying the strategy, the designed joint - action reward function effectively represents the collaborative goal between agents. Through message iteration and propagation among neighboring agents with a collaborative relationship, it converges to an optimized strategy, that is, a reasonable decision action is obtained. Applying the method of this embodiment to the multi - unmanned vehicle passing scenario at an intersection without traffic signal indication can improve the passing efficiency while avoiding collisions and ensuring the safe passing of multiple unmanned vehicles.

[0171] The above - mentioned is only the preferred embodiment of the present invention, and thus does not limit the protection scope of the present invention. Any equivalent structural transformation made under the inventive concept of the present invention by using the content of the specification and drawings of the present invention, or directly / indirectly applied to other related technical fields, is included in the protection scope of the present invention.

Claims

1. An efficient reinforcement learning method for multi-vehicle intersection coordination decision-making and control, characterized in that, It includes three stages: sample data collection, offline policy training, and online deployment control; In the sample data collection: According to the Markov decision process, a sample set of each unmanned vehicle is generated based on random sampling, where the sample set size of each unmanned vehicle is , the sample tuple for each time step contains , indicating that Current state at the moment Next, in the action space A random action is chosen , the action Will drive the unmanned vehicle to update the status , the reward function is ; In the offline policy training: Based on the sample set of each unmanned vehicle, the kernel sparsification method is used to extract the features of the collected high-dimensional samples, obtaining approximately linearly independent sub-samples, and using the sub-samples to construct the basis functions corresponding to each original sample point, obtaining the approximation structure of the action-state value function. Then, with the goal of minimizing the temporal difference error of reinforcement learning, the network weight vector in the approximation structure of the action-state value function is updated. The approximation structure of the action-state value function is: Among them, is the basis function corresponding to the original sample point, is the kernel function, , , , are to the training samples of the th unmanned vehicle at different times, is the transpose of the matrix, is the dimensional feature space, is the approximation structure of the action-state value function, is the network weight vector, is the weight matrix; The updating of the network weight vector in the approximation structure of the action-state value function with the goal of minimizing the temporal difference error of reinforcement learning includes: Obtaining the temporal difference error of reinforcement learning, which is: Among them, is the temporal difference error of reinforcement learning, is the target action-state value function, is the candidate decision-making action, and are positive definite weight matrices, is the discount factor, is the action-state value function after state transition; Based on the approximation structure of the action-state value function, rewriting the temporal difference error of reinforcement learning, which is: Multiply both sides of the temporal difference error rewritten by reinforcement learning by the basis function , and let the intermediate parameter 、 be: Since the intermediate parameter is full rank, the least squares solution of the network weight vector is obtained, that is ; In the online deployment control: According to the observed real-time state variables of the unmanned vehicle, the decision-making strategy is deployed to obtain the action-state value function corresponding to each action, that is, the utility function after the unmanned vehicle adopts the action, and the local joint action reward function is obtained according to the cooperation relationship between unmanned vehicles. Based on the utility function and the local joint action reward function, the global value function of the unmanned vehicle is obtained, and the decision-making action of the unmanned vehicle is obtained according to the global value function. Finally, the control strategy of the unmanned vehicle is learned according to the decision-making action of the unmanned vehicle, and the state of the unmanned vehicle is updated.

2. The efficient reinforcement learning method for multi-vehicle intersection coordination decision-making and control according to claim 1, characterized in that, In the online deployment control, the obtaining of the local joint action reward function is: Among them, is the adjacent driverless vehicle and the driverless vehicle respectively adopt the local joint action reward function after the action , . is the predicted distance of the driverless vehicle to the target position, is the predicted distance of the driverless vehicle to the target position.

3. The efficient reinforcement learning method for multi-vehicle intersection coordination decision-making and control according to claim 2, wherein In the online deployment control, the obtaining of the global value function of the unmanned vehicle based on the utility function and the local joint action reward function is specifically: wherein, is the global value function after the driverless vehicle adopts an action ; is the utility function after the driverless vehicle adopts an action ; is the message sent to neighboring driverless vehicles by the driverless vehicle ; is the set of neighbors of the driverless vehicle . The message sent by the unmanned vehicle to its neighboring unmanned vehicles is defined as: Among them, is an unmanned vehicle is the message sent to neighboring unmanned vehicles and represents the set of unmanned vehicles that belong to the neighboring set of the unmanned vehicle excluding the unmanned vehicle itself.

4. The efficient reinforcement learning method for multi-vehicle intersection coordination decision-making and control according to claim 3, wherein In the online deployment control, the decision-making action of the unmanned vehicle is: Among them, is the decision-making action of the driverless vehicle.

5. The efficient reinforcement learning method for multi-vehicle intersection coordination decision-making and control according to claim 1 or 2 or 3 or 4, characterized in that, In the online deployment control, the learning of the control strategy of the unmanned vehicle according to the decision-making action of the unmanned vehicle and the updating of the state of the unmanned vehicle are specifically: At the sampling moment , taking the current tracking error of the driverless vehicle as the initial state, enter the prediction time domain to perform reinforcement learning, where is the length of the prediction time domain; ​ Define the value function within the prediction horizon as follows: Among them, is the tracking error of the driverless vehicle at moment, is the control input of the driverless vehicle at moment, is the reward function of the driverless vehicle , is the terminal cost of the prediction horizon, is the terminal penalty matrix, is the transpose of the matrix; According to the Bellman optimization principle, the optimal value function satisfies the HJB equation of the discrete-time system, that is: wherein, is the control input of the driverless vehicle ; Define the co-state of the value function , where is the objective value function; Then the optimal co-state is: Among them, is a preset positive definite weight matrix; Therefore, the optimal control strategy is obtained. It is: Among them, is the decision-making action of the driverless vehicle corresponding to the control input, is a preset positive definite weight matrix.

6. The efficient reinforcement learning method for multi-vehicle intersection coordination decision-making and control according to claim 5, characterized in that, In the online deployment control, a multi-group actuator-critic reinforcement learning framework is adopted. Based on the value iteration method, the time-varying value function and the optimal control strategy are approximated through a neural network. Specifically: Both the actuator and the critic are constructed using a three-layer neural network. Both the actuator and the critic include an input layer, a hidden layer, and an output layer; In the evaluator, the input of the input layer is the tracking error of the driverless vehicle , the number of nodes in the hidden layer is , and the output of the output layer is the co-state ; The structure of the evaluator approximating the co-state through a neural network is as follows: Among them, is the weight of the evaluator network from the hidden layer to the output layer, is the weight of the evaluator network from the input layer to the hidden layer, is the activation function; Within the prediction time domain, the evaluator updates the network based on the time-domain differential error to update the network, and the time-domain differential error is as follows: wherein, is the target co-state obtained according to the estimated co-state, expressed as: The goal of the evaluator network is to minimize the error function ; Based on the gradient descent method, the weight update rule of the critic network is: Among them, is the learning rate of the evaluator network, and ; In the actuator, the input of the input layer is the tracking error of the unmanned vehicle , the number of nodes in the hidden layer is , and the output of the output layer is the control quantity of the unmanned vehicle ; The actuator adopts a structure that uses a three-layer network to approximate the control quantity , which is as follows: Among them, is the weight of the actuator network from the hidden layer to the output layer, is the weight of the actuator network from the input layer to the hidden layer; Within the prediction time domain, the actuator updates the network based on the time-domain differential error to update the network, and the time-domain differential error is as follows: Among them, is the target control quantity obtained according to the estimated control quantity, expressed as: The goal of the actuator network is to minimize the error function ; Based on the gradient descent method, the weight update rule of the actuator network is: Among them, is the learning rate of the actuator network, and ; The termination condition of reinforcement learning within the prediction time domain is: wherein, is the number of iterations of reinforcement learning within the prediction time domain, is the threshold for the convergence of the weights of the evaluator network, is the threshold for the convergence of the weights of the actuator network; When the network weights of two consecutive iterations satisfy the termination condition, the first actuator network in the optimized solution control sequence at this time is applied to the actual controlled unmanned vehicle system.

Images