An end-to-end vehicle group collaborative decision-making method based on hybrid deep reinforcement learning in mixed traffic flow environment

Abstract

The present invention relates to the field of intelligent transportation technology, and discloses an end-to-end vehicle group collaborative decision-making method based on hybrid deep reinforcement learning in a mixed traffic flow environment. The method realizes the communication mechanism between traffic participants from the bottom layer by designing a graph structure modeling method for complex traffic scenes. Secondly, the spatiotemporal encoder based on the graph convolution method of the present invention realizes the effective embedding of spatiotemporal information in environmental observation, and realizes the prerequisite for the system to understand the spatiotemporal coupling information in the environment. Finally, the actor-critic architecture is deeply coupled with the Q learning architecture to construct a new type of reinforcement learning method suitable for hybrid action space. After completing the model training based on this method, the model can be deployed by saving and loading the model parameters. The deployed model can output the collaborative decision of the controlled vehicle group by inputting the environmental observation data.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and specifically to an end-to-end vehicle group collaborative decision-making method based on hybrid deep reinforcement learning in a mixed traffic flow environment. Background Art

[0002] Nowadays, the integration of artificial intelligence and Internet of Vehicles technology has not only opened up a new way to improve the safety and stability of the transportation system. With the improvement of relevant policies and regulations and the further maturity of technology, the penetration rate of intelligent connected vehicles in the market will gradually increase. Considering the huge number of traditional cars, road traffic will evolve into a mixed traffic scene with traditional manually driven vehicles and intelligent connected vehicles. Vehicle data, spatiotemporal information, and intelligent body forms in this complex traffic flow will present heterogeneous forms.

[0003] At present, the research on control strategies of intelligent connected vehicles for complex traffic scenarios mainly focuses on the centralized execution decision framework, with the self-vehicle decision control model as the main body, and the surrounding vehicle data obtained by V2V communication to achieve lateral and longitudinal control. However, this type of algorithm focuses on the operation optimization of local traffic flow, and it is difficult to obtain good scalability in multi-agent scenarios, and it is impossible to form multi-dimensional collaborative optimization of safety and efficiency from a global perspective. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the shortcomings of the prior art, the present invention provides an end-to-end vehicle group collaborative decision-making method based on hybrid deep reinforcement learning in a mixed traffic flow environment, which solves the above-mentioned technical problems.

[0006] (II) Technical solution

[0007] To achieve the above object, the present invention provides the following technical solution: an end-to-end vehicle group collaborative decision-making method based on hybrid deep reinforcement learning in a mixed traffic flow environment, comprising the following steps:

[0008] S1. Establish a two-layer network graph structure to represent the communication mechanism of mixed traffic flow, including the communication connection between agents and the communication connection between agents and human-driven vehicles;

[0009] S2. Construct a spatiotemporal encoder to divide the environmental observations generated by the two-layer network graph structure in step S1 into discrete observations and continuous observations, and input the continuous observations into the batch normalization layer and then integrate them into the augmented tensor input long short-term memory layer and graph convolution layer to realize temporal convolution and spatial convolution, and then obtain two dimensionality-reduced tensors after spatiotemporal convolution through global average pooling and drop layer;

[0010] S3, based on the spatiotemporal encoder in step S2 as the basic shallow structure of the deep graph network, and regularizing the output according to whether there are traffic participants in the environment through masks, the specific expression is as follows:

[0011]

[0012] in, represents the output tensor of the output layer, It means that the Hadamard product of the input and mask of the output layer is used to obtain the regularized output. represents the mask matrix, Represents the input tensor of the output layer;

[0013] S4. Construct a multi-agent hybrid deep reinforcement learning model. Construct different basic graph convolutional networks as actor sub-networks for different types of agents. Each actor sub-network outputs the continuous and discrete actions of the corresponding agent. Construct a generalized Q network shared by all agents, which evaluates discrete and continuous actions, so that the multi-agent hybrid deep reinforcement learning model learns the action with the maximum reward.

[0014] S5. Construct a hybrid exploration and exploitation strategy;

[0015] S6. Iterate the multi-agent hybrid deep reinforcement learning model;

[0016] S7. Save the weight parameters of the trained model, and realize the online deployment of the model by overloading the parameters. By inputting environmental observations, the collaborative decision of the vehicle group can be output.

[0017] As a preferred technical solution of the present invention, step S4 includes the following steps:

[0018] S4.1. Use the actor subnetwork to complete action generation. The specific expression is as follows:

[0019]

[0020]

[0021] in, and Respectively represent the continuous action and discrete action at the current moment, and Represent the strategy functions of continuous actions and discrete actions respectively, and Represent the strategy parameters of continuous actions and discrete actions respectively, Indicates the current state The exploration and exploitation strategy of continuous actions obtained by inputting into the continuous policy function, Indicates the current state The exploration and exploitation strategy of discrete actions obtained by inputting into the discrete policy function, represents the state of the current environment feedback, and , represents the state space, where and are the horizontal and vertical coordinates of the vehicle in the Cartesian coordinate system. and are the current speed and acceleration values ​​of the vehicle, represents the current moment strategy exploration noise, Represents exploration noise The mean is 0 and the covariance matrix is The multivariate normal distribution of represents the decay factor, Indicates input in the past The combination of the moment and the action of the agent at that moment The resulting agent is The influence of the operating status at all times, Indicates that in the interval Between To perform integration, Represents a Categorical function, express Activation function, Indicates that according to the current state and discrete strategy parameters ,pass The function obtains the probability distribution of discrete actions and passes Construct a category distribution and sample discrete actions from it ;

[0022] S4.2, the Q value of the action is estimated through the generalized Q network, and the specific expression is as follows:

[0023]

[0024] in, Indicates action at the current moment Under the action, according to the state of environmental feedback The reward value, represents the value judgment function of the generalized Q network, represents the weight parameter of the generalized Q network, represents the reward discount factor, represents the calculation of mathematical expectation, Indicates the current policy parameters The state of the environment feedback at the next moment Execute an action After that, let The biggest move;

[0025] S4.3. Calculate the policy gradient of the mixed action. The specific expression is:

[0026]

[0027] in, Represents the objective function Strategy parameters relative to continuous and discrete actions The gradient of Indicates in status Execute continuous actions and discrete actions The log probability of the joint strategy, Indicates the state of environmental feedback and continuous action Discrete Action The action value function is Indicated in strategy Executed under

[0028] S4.4. Obtain the authenticity of the reward based on batch sampling and calculate the prediction loss:

[0029]

[0030] in, represents the mean square error function, represents the total number of agents, Indicates The true value of the reward for each agent, Indicates The action taken by the agent at the current moment Status of feedback with the environment The network parameters are The estimated Q value is Express The data are summed;

[0031] S4.5. The dual network weight soft update mode is adopted to stabilize the learning process. The update process of the weight parameters is expressed as:

[0032]

[0033] in, represents the soft update weight, Represented as the state of all agents and actions The combined space composed of and Represent the weight parameters of the main network and the secondary network respectively, represents the probability density function of the generalized Q network weight parameter space sampling, Represents integral.

[0034] As a preferred technical solution of the present invention, the specific process of step S5 is as follows:

[0035] S5.1. During the exploration process, for discrete actions, the ε-greedy strategy is used to compare the Q value distribution of the action, and the random number ε is obtained to determine whether the probability domain falls into needs to explore new actions. If new actions need to be explored, actions are randomly selected from the discrete action space. Otherwise, the action corresponding to the maximum Q value is obtained. For continuous actions, normal noise is applied through the Ornstein-Uhlenbeck process to explore the action-reward tuple;

[0036] S5.2. During the utilization process, for discrete actions, the action corresponding to the maximum value in the Q value distribution is selected. For continuous actions, no noise is added and the action of the actor subnetwork is directly used.

[0037] Compared with the prior art, the present invention provides an end-to-end vehicle group collaborative decision-making method based on hybrid deep reinforcement learning in a mixed traffic flow environment, which has the following beneficial effects:

[0038] The present invention firstly designs a graph structure modeling method for complex traffic scenes, and realizes the communication mechanism between traffic participants from the bottom layer. Secondly, the spatiotemporal encoder based on the graph convolution method of the present invention realizes the effective embedding of spatiotemporal information in environmental observations, and realizes the prerequisite for the system to understand the spatiotemporal coupling information in the environment. Then, the actor-critic architecture is deeply coupled with the Q learning architecture to construct a new reinforcement learning method suitable for mixed action space. Furthermore, this method can realize the training process of the model, that is, the optimization process of the model weight parameters, and the online deployment of the model can be realized by saving the parameters and further reloading. Given the observation information of the test environment, the collaborative decision of the controlled vehicle group can be output. Finally, the present invention can realize the "end-to-end" mapping of multiple types of intelligent agents from the sensor input end to the decision output end, and can improve the operation of complex traffic flow by jointly planning and controlling the intelligent networked vehicle group in the test environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the process of the present invention;

[0040] Figure 2A schematic diagram of modeling a mixed traffic flow diagram of the present invention;

[0041] Figure 3 It is a schematic diagram of the structure of the space-time encoder of the present invention;

[0042] Figure 4 This is a schematic diagram of the multi-agent hybrid deep reinforcement learning structure of the present invention;

[0043] Figure 5 Schematic diagram of the simulation environment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] See also Figure 1-Figure 5 , an end-to-end vehicle group collaborative decision-making method based on hybrid deep reinforcement learning in a mixed traffic flow environment, characterized in that it includes the following steps:

[0046] S1. Graph structure modeling of mixed traffic flow: Figure 2 As shown in FIG. 1 , a two-layer network graph structure is established to characterize the communication mechanism of mixed traffic flow, forming a two-layer network graph structure to realize graph structure modeling of mixed traffic flow. Specifically, Figure 2 The white nodes are regarded as human-driven vehicles, the gray nodes are regarded as controlled intelligent networked vehicles, and the black nodes are traffic lights. All controlled intelligent networked vehicles are connected to form a global network connection, and human-driven vehicles are connected to the controlled intelligent networked vehicles with the closest Euclidean distance to form a local network connection. Thus, the double-layer graph structure of the mixed traffic flow can be modeled as a double-layer network graph structure. When sampling at a certain sampling frequency, the above-mentioned double-layer network graph structure has a time dimension, forming a dynamic double-layer network graph structure. In order to further abstract the structure and facilitate model input, the present invention connects the structure through a [T,N,N] It can be expressed as an adjacency tensor of Indicates the total sampling time length, represents the number of all controlled intelligent connected vehicles. The elements in the adjacency tensor are composed of one-hot encoding numbers 0 and 1, where 0 indicates that there is no connection between the two nodes, and 1 indicates that there is a connection between the two nodes. In order to represent the state observation of the environment, based on the above adjacency tensor, the present invention further creates a tensor with size [T,N,F] The feature tensor of is the number of features, i.e. the size of the environment state space plus the discrete category features F=[ g l , g t ] The number of Indicates the lane the vehicle is in. Indicates the type of vehicle (human-driven vehicle or intelligent connected vehicle). This feature tensor reflects the state stack of the environment over a certain period of time. Finally, the adjacency tensor and feature tensor will be input into the model as abstracted environmental observation data;

[0047] S2. Design and implementation of space-time encoder: Figure 3 As shown, the environmental observations generated by the graph structure in step S1 are divided into discrete observations and continuous observations, wherein the continuous observations mainly represent discrete coded data such as vehicle type, while the continuous observations are mainly continuously changing observation values ​​such as speed. Figure 3 Both types of input have a time dimension The continuous observations are input into the batch normalization layer and then integrated into the augmented tensor to input the long short-term memory layer and the graph convolution layer to realize the temporal convolution and spatial convolution. Finally, the two reduced-dimensional tensors after the spatiotemporal convolution are obtained through the global average pooling and the discard layer. At this time, the output size of the encoder is [N,F] , that is, the dimensionality reduction of the time dimension is achieved. It should be noted that the structural design of the spatiotemporal encoder is for mixed actions, that is, it can implement spatiotemporal encoding for discrete actions and continuous actions respectively, so as to obtain high-dimensional feature embedding in the spatiotemporal dimension. In addition, a time wrapper is used in the time encoding to ensure the validity of the tensor size input of the long short-term memory layer;

[0048] S3. Establishment of basic graph convolutional network: Based on the spatiotemporal encoder in step S2 as the basic shallow structure of the deep graph network, different decoding output layers are further designed in the deep layer according to different network structures. In addition, a mask mechanism is designed to regularize the output according to different environments, that is, the output is regularized by masking according to whether there are traffic participants in the environment. The specific expression is as follows:

[0049]

[0050] in, represents the output tensor of the output layer, It means that the Hadamard product of the input and mask of the output layer is used to obtain the regularized output. represents the mask matrix, Represents the input tensor of the output layer;

[0051] S4, such as Figure 4 As shown in the figure, a multi-agent hybrid deep reinforcement learning model is constructed. Different basic graph convolutional networks are constructed for different types of agents as actor sub-networks. Each actor sub-network outputs the continuous and discrete actions of the corresponding agent. A generalized Q network shared by all agents is constructed, which evaluates discrete and continuous actions, so that the multi-agent hybrid deep reinforcement learning model can learn the action with the maximum reward.

[0052] The step S4 comprises the following steps:

[0053] S4.1. Use the actor subnetwork to complete action generation. The specific expression is as follows:

[0054]

[0055]

[0056] in, and Respectively represent the continuous action and discrete action at the current moment, and Represent the strategy functions of continuous actions and discrete actions respectively, and Represent the strategy parameters of continuous actions and discrete actions respectively, Indicates the current state The exploration and exploitation strategy of continuous actions obtained by inputting into the continuous policy function, Indicates the current state The exploration and exploitation strategy of discrete actions obtained by inputting into the discrete policy function, represents the state of the current environment feedback, and , represents the state space, where and are the horizontal and vertical coordinates of the vehicle in the Cartesian coordinate system. and are the current speed and acceleration values ​​of the vehicle, represents the current moment strategy exploration noise, Represents exploration noise The mean is 0 and the covariance matrix is The multivariate normal distribution of represents the decay factor, Indicates input in the past The combination of the moment and the action of the agent at that moment The resulting agent is The influence of the operating status at all times, Indicates that in the interval Between To perform integration, Represents a Categorical function, express Activation function, Indicates that according to the current state and discrete strategy parameters ,pass The function obtains the probability distribution of discrete actions and passes Construct a category distribution and sample discrete actions from it ;

[0057] S4.2, the Q value of the action is estimated through the generalized Q network, and the specific expression is as follows:

[0058]

[0059] in, Indicates action at the current moment Under the action, according to the state of environmental feedback The reward value, represents the value judgment function of the generalized Q network, represents the weight parameter of the generalized Q network, represents the reward discount factor, represents the calculation of mathematical expectation, Indicates the current policy parameters The state of the environment feedback at the next moment Execute an action After that, let The biggest move;

[0060] S4.3. Calculate the policy gradient of the mixed action. The specific expression is:

[0061]

[0062] in, Represents the objective function Strategy parameters relative to continuous and discrete actions The gradient of Indicates in status Execute continuous actions and discrete actions The log probability of the joint strategy, Indicates the state of environmental feedback and continuous action Discrete Action The action value function is Indicated in strategy Executed under

[0063] S4.4. Obtain the authenticity of the reward based on batch sampling and calculate the prediction loss:

[0064]

[0065] in, represents the mean square error function, represents the total number of agents, Indicates The true value of the reward for each agent, Indicates The action taken by the agent at the current moment Status of feedback with the environment The network parameters are The estimated Q value is Express The data are summed;

[0066] S4.5. The dual network weight soft update mode is adopted to stabilize the learning process. The update process of the weight parameters is expressed as:

[0067]

[0068] in, represents the soft update weight, Represented as the state of all agents and actions The combined space composed of and Represent the weight parameters of the main network and the secondary network respectively, represents the probability density function of the generalized Q network weight parameter space sampling, represents integral;

[0069] S5. Construct a hybrid exploration and exploitation strategy;

[0070] S5.1. During the exploration process, for discrete actions, the ε-greedy strategy is used to compare the Q value distribution of the action, and the random number ε is obtained to determine whether the probability domain falls into needs to explore new actions. If new actions need to be explored, actions are randomly selected from the discrete action space. Otherwise, the action corresponding to the maximum Q value is obtained. For continuous actions, normal noise is applied through the Ornstein-Uhlenbeck process to explore the action-reward tuple;

[0071] S5.2. During the utilization process, for discrete actions, the action corresponding to the maximum value in the Q value distribution is selected. For continuous actions, no noise is added and the action of the actor subnetwork is directly used.

[0072] S6, iterate the multi-agent hybrid deep reinforcement learning model, build a reinforcement learning environment and implement the model building based on steps S1 to S5, train the multi-agent deep network and save the trained model weight parameters through continuous interaction and iteration between the model and the environment;

[0073] S7. After obtaining the trained multi-agent model, the model can be deployed online by loading the model weights. Figure 5 The figure shows an environment used for testing, in which intelligent connected vehicles and human-driven vehicles will randomly enter from corresponding entrances, and the controlled vehicle group needs to enter the correct lane at the most appropriate time so that the controlled vehicle group can reach its corresponding destination. Through the online deployment of the model and the input of the observation data of the environment, the collaborative decision output of the controlled vehicle group can be obtained.

[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An end-to-end vehicle group collaborative decision-making method based on hybrid deep reinforcement learning in a mixed traffic flow environment, characterized by: The following steps are involved: S1. Establish a two-layer network graph structure to represent the communication mechanism of mixed traffic flow, including the communication connection between agents and the communication connection between agents and human-driven vehicles; S2. Construct a spatiotemporal encoder to divide the environmental observations generated by the two-layer network graph structure in step S1 into discrete observations and continuous observations, and input the continuous observations into the batch normalization layer and then integrate them into the augmented tensor input long short-term memory layer and graph convolution layer to realize temporal convolution and spatial convolution, and then obtain two dimensionality-reduced tensors after spatiotemporal convolution through global average pooling and drop layer; S3, based on the spatiotemporal encoder in step S2 as the basic shallow structure of the deep graph network, and regularizing the output according to whether there are traffic participants in the environment through masks, the specific expression is as follows: y=f(M⊙x) Where y represents the output tensor of the output layer, f(M⊙x) represents the regularized output obtained by performing the Hadamard product of the input and mask of the output layer, M represents the mask matrix, and x represents the input tensor of the output layer; S4. Construct a multi-agent hybrid deep reinforcement learning model. Construct different basic graph convolutional networks as actor sub-networks for different types of agents. Each actor sub-network outputs the continuous and discrete actions of the corresponding agent. Construct a generalized Q network shared by all agents, which evaluates discrete and continuous actions, so that the multi-agent hybrid deep reinforcement learning model learns the action with the maximum reward. S5. Construct a hybrid exploration and exploitation strategy; The specific process of step S5 is as follows: S5.

1. During the exploration process, for discrete actions, the ε-greedy strategy is used to compare the Q value distribution of the action, and the random number ε is obtained to determine whether the probability domain falls into needs to explore new actions. If new actions need to be explored, actions are randomly selected from the discrete action space. Otherwise, the action corresponding to the maximum Q value is obtained. For continuous actions, normal noise is applied through the Ornstein-Uhlenbeck process to explore the action-reward tuple; S5.

2. During the utilization process, for discrete actions, the action corresponding to the maximum value in the Q value distribution is selected. For continuous actions, no noise is added and the action of the actor subnetwork is directly used. S6. Iteratively train the multi-agent hybrid deep reinforcement learning model; S7. Save the weight parameters of the trained model, and realize the online deployment of the model by overloading the parameters. By inputting environmental observations, the collaborative decision of the vehicle group can be output.

2. According to claim 1, an end-to-end vehicle group collaborative decision-making method based on hybrid deep reinforcement learning in a mixed traffic flow environment is characterized by: The step S4 comprises the following steps: S4.

1. Use the actor subnetwork to complete action generation. The specific expression is as follows: Among them, a t,c and a t,d Respectively represent the continuous action and discrete action at the current moment, and Represent the strategy functions of continuous actions and discrete actions respectively, θ c and θ d Represent the strategy parameters of continuous actions and discrete actions respectively, Indicates that the current state s t The exploration and exploitation strategy of continuous actions obtained by inputting into the continuous policy function, Indicates that the current state s t The exploration and exploitation strategy of discrete actions obtained by inputting into the discrete policy function, s t represents the state of the current environment feedback, and represents the state space, where x t and t are the horizontal and vertical coordinates of the vehicle in the Cartesian coordinate system, respectively. t and a t are the current speed and acceleration of the vehicle, ε t represents the current strategy exploration noise, ε t ~N(0,∑ c ) represents the exploration noise ε t The mean is 0 and the covariance matrix is ​​∑ c The multivariate normal distribution of κ represents the decay factor, h(t-τ,a t-τ ) represents the input of the past τ time and the action combination a of the agent at that time t-τ The obtained influence of the agent's operating state at time t-τ is Indicates that h(t-τ,a) is t-τ ) for integration, Categorical(*) represents Categorical function, softmax(*) represents softmax activation function, Indicates that according to the current state s t and discrete strategy parameter θ d , the probability distribution of discrete actions is obtained through the softmax function, and the category distribution is constructed through Categorical, and the discrete action a is sampled from the category distribution t,d ; S4.2, the Q value of the action is estimated through the generalized Q network, and the specific expression is as follows: Among them, r(s t ,a t ) indicates action at the current moment Under the action, according to the state s of environmental feedback t The reward value, Q(s t ,a t ; φ) represents the value judgment function of the generalized Q network, φ represents the weight parameter of the generalized Q network, γ represents the reward discount factor, E[*] represents the calculation of mathematical expectation, Indicates the state s of the environment feedback at the next moment under the current policy parameter φ t+1 After executing action a', let Q(s t+1 ,a';φ) the largest action; S4.

3. Calculate the policy gradient of the mixed action. The specific expression is: in, Denotes the objective function J(θ c ,θ d ) relative to the policy parameters θ for continuous and discrete actions c ,θ d The gradient of Indicates that in state s t Execute continuous action a t,c and discrete action a t,d The logarithmic probability of the joint strategy, Q π (s t ,a t,c ,a t,d ) represents the state s of environmental feedback t and continuous action a t,c With discrete action a t,d The action value function of π represents the action executed under the strategy π; S4.

4. Obtain the authenticity of the reward based on batch sampling and calculate the prediction loss: Where ζ(φ) represents the mean square error function, N represents the total number of agents, and y i represents the true value of the reward of the ith agent, Q(s t,i ,a t,i ; φ) represents the action a taken by the i-th agent at the current moment t,i The state of feedback with the environment t,i The estimated Q value when the network parameter is φ, * indicates summing N data; S4.

5. The dual network weight soft update mode is adopted to stabilize the learning process. The update process of the weight parameters is expressed as: in, It is represented as the combined space consisting of the states s and actions a of all agents, and Represent the weight parameters of the main network and the secondary network respectively, represents the probability density function of the generalized Q network weight parameter space sampling, Represents integral.

Images