Robust decision-making method for interconnected autonomous vehicles in mixed traffic flow based on deep reinforcement learning

Abstract

The invention discloses a robust decision-making method for an interconnected autonomous vehicle in a mixed traffic flow based on deep reinforcement learning. The method comprises the following steps: step 1, setting a task scene of CAV decision-making in the mixed traffic flow; 2, defining a state space which can be observed by the intelligent agent; 3, determining an overall network structure of the decision model; 4, defining an action space and a reward function of the reinforcement learning agent; defining an action space as a set of lane changing actions, and setting a reward function; 5, training and testing the decision model; step 6, constructing an adversarial observation state detection module and a real observation state reconstruction module, and combining the modules as a risk processing module; 7, integrating a risk processing module in the decision model to construct a robust decision framework; and testing the performance of the deep reinforcement learning decision model combined with risk processing under different traffic conditions. According to the method, the robust decision of the CAV in the mixed traffic flow is realized under the condition that excessive vehicle computing resources are not occupied.

Description

Technical Field

[0001] The present invention belongs to the intersection of intelligent transportation and deep reinforcement learning, and relates to a robust decision-making method for interconnected autonomous driving vehicles in mixed traffic flows based on deep reinforcement learning. Background Art

[0002] Intelligent transportation systems are developing towards a higher level of connectivity and intelligence, and play a vital role in economic and social development. However, in the transition to fully autonomous driving, mixed traffic scenarios consisting of human-driven vehicles (HDVs) and connected autonomous vehicles (CAVs) will exist for a long time. Deep reinforcement learning (DRL) combines the advantages of deep learning and reinforcement learning. It can not only cope with complex traffic environments, but also show high flexibility and adaptability in real-time decision-making. It has demonstrated significant capabilities in mixed autonomous traffic scenarios.

[0003] However, the traffic environment in the real world is extremely complex and uncertain. In practical applications, observation data is often interfered by many factors, including sensor errors, weather changes, terrain differences, and sudden obstacles. These factors may introduce observation noise. When observation noise exists in the form of adversarial disturbances, it may mislead the model to make inappropriate or even dangerous decisions. Therefore, it is crucial to improve the robustness of the DRL-based CAV decision model. From the perspective of data security, risk processing of observation signals can ensure that the information relied on by the decision model is accurate and reliable, which can improve the robustness of the decision. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art and to ensure driving efficiency and safety, the present invention provides a robust decision-making method for connected autonomous vehicles in mixed traffic flows based on deep reinforcement learning. The method improves observation robustness by processing observation states, thereby achieving robust decision-making for connected autonomous vehicles in mixed traffic flows in highway diversion areas. The present invention uses highway diversion areas as application scenarios and can generate robust lane change decisions for connected autonomous vehicles.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A robust decision-making method for connected autonomous vehicles in mixed traffic flow based on deep reinforcement learning, comprising the following steps:

[0007] Step 1: Set up the task scenario of CAV decision-making in mixed traffic flow; build a simulation scenario based on the SUMO platform, set the behavior mode of HDV in mixed traffic flow and the target task of CAV;

[0008] Step 2: Define the state space that the agent can observe; build the adjacency matrix of the vehicle network based on the interactions between vehicles;

[0009] Step 3: Determine the overall network structure of the decision model; construct feature encoder, graph neural network and policy network;

[0010] Step 4: Define the action space and reward function of the reinforcement learning agent; define the action space as a set of lane-changing actions, and set the reward function to guide the agent to learn the strategy that can obtain the optimal action;

[0011] Step 5: Train and test the decision model; use experience replay and target network to train the overall model;

[0012] Step 6: Construct the adversarial observation state detection AOSD module, construct the real observation state reconstruction GTOSR module, and combine the AOSD module and the GTOSR module as a risk processing module;

[0013] Step 7: Integrate the risk management module into the decision model to build a robust decision framework; test the performance of the deep reinforcement learning decision framework combined with risk management under different traffic conditions.

[0014] The technical concept of the present invention is to build a CAV decision framework in mixed traffic flow based on DRL. A risk processing module is built to detect the risk of the observation state based on the inconsistency of adversarial sensitivity, and when the risk is detected, the decision model is guided back to a safe state by reconstructing the real observation state. The reliability of the observation data is improved by adding a risk processing module in the decision-making process, thereby enhancing the robustness of the decision.

[0015] The beneficial effects of the present invention are: using the inconsistency of the adversarial sensitivity of the network before and after distillation to detect risk states, using graphical data to characterize the state of the vehicle network in mixed traffic flow, using the graph attention mechanism to capture the correlation between the current observation state and the historical observation state, and then reconstructing the true observation state through encoding-decoding. The graph neural network under the reinforcement learning framework is used to mine vehicle interaction relationships and state characteristics, and obtain a decision strategy from vehicle state to corresponding action output. The decision framework proposed by the present invention can be deployed in a roadside unit or a cloud computing platform to achieve robust decision-making of CAVs in mixed traffic flow without occupying too many vehicle computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of the highway ramp diversion area scene.

[0017] Figure 2 It is a graphical diagram based on the vehicle relationship network.

[0018] Figure 3 This is a schematic diagram of intention reward.

[0019] Figure 4 It is a schematic diagram of the CAV decision-making process in mixed traffic flow.

[0020] Figure 5 It is a schematic diagram of the risk treatment module.

[0021] Figure 6 It is a schematic diagram of a robust decision-making framework. DETAILED DESCRIPTION

[0022] The present invention is further described below in conjunction with the accompanying drawings.

[0023] Reference Figure 1 to Figure 6 , a robust decision-making method for connected autonomous vehicles in mixed traffic flow based on deep reinforcement learning, comprising the following steps:

[0024] Step 1: Set up the task scenario for CAV decision-making in mixed traffic flow;

[0025] Traffic scenarios in the SUMO simulation platform are as follows: Figure 1 As shown in Figure 1, the traffic scenario includes a three-lane highway with two ramp exits. According to the locations of ramp exit 1 (Ramp1) and ramp exit 2 (Ramp2), the highway is divided into two segments, named Seg1 and Seg2. The simulation involves mixed traffic flows of CAVs and HDVs. All vehicles enter the scenario from the left side of the highway. The HDV flow is f h , CAV flow is f c The speed limit on the road section is v max , where the maximum speed of the CAV is v c , the maximum speed of HDV is v h . Different vehicles have different driving tasks, and they need to cooperate to complete the predetermined driving tasks more efficiently. Specifically, the HDV task is to leave the highway from the right side; the CAV1 task is to leave from ramp 1; and the CAV2 task is to leave from ramp 2. The longitudinal and lateral motion control of the HDV is controlled using SUMO's built-in IDM following model and LC2013 lane change model, respectively, and the longitudinal motion of the CAV is the same as that of the HDV. The DRL model in the present invention outputs lane changing decisions for the CAV, so that the CAV leaves the highway from its intended ramp in a safe and efficient manner.

[0026] Step 2: Define the state space that the agent can observe;

[0027] At each time step t, the state of the agent is s t =(X t ,A t ,M t), including vehicle driving characteristics X t , adjacency matrix A t and the mask matrix M t ;

[0028] Vehicle driving characteristics: The driving characteristics of vehicle i at time step t include position, speed, lane and intention, expressed as And each element is determined as follows: represents the normalized longitudinal position of vehicle i at time step t relative to the road length, represents the normalized longitudinal velocity of vehicle i at time step t relative to the maximum longitudinal velocity, Indicates the lane where vehicle i is located at time step t, using one-hot encoding to describe it, that is, [1, 0, 0], [0, 1, 0], and [0, 0, 1] represent the right lane, middle lane, and left lane, respectively. represents the driving intention of vehicle i at time step t, which is also represented by one-hot encoding, that is, [1, 0, 0], [0, 1, 0], and [0, 0, 1] represent leaving from ramp 1, leaving from ramp 2, and going straight along the main line of the highway, respectively;

[0029] Adjacency matrix: Figure 2 As shown in the figure, the vehicle network is modeled as a graph. Each vehicle in the scene is regarded as a node of the graph. The node features are the vehicle driving features. The adjacency matrix is ​​used to represent the edge relationship between the graph nodes. The adjacency matrix is ​​based on the following assumptions: all vehicles can share information with themselves; all CAVs can share information with each other; all CAVs can share information with HDVs within their perception range. The dimension of the adjacency matrix is ​​N×N, which matches the dynamic number of vehicles in the scene. When the number of vehicles is less than N, or when a vehicle leaves the scene, the corresponding position in the matrix is ​​occupied by 0. Represents the edge value between vehicle i and vehicle j, and the adjacency matrix A t Depend on When vehicle i and vehicle j share information, when When vehicle i and vehicle j do not share any information;

[0030]

[0031] Mask matrix: CAV mask M t It is used to filter out the features of HDV and only retain the features of CAV. t According to each In X t The length of the vector is the number of vehicles N. It is a binary vector with a value of 1 at the index corresponding to CAV and 0 otherwise;

[0032] Mt =[m1,m2,...,m i ,...,m n ] (2)

[0033] Step 3: Determine the overall network structure of the decision model;

[0034] Construct a fully connected network (FCN) encoder φ FCN , at each time step t, the node feature matrix X t First, it is fed to φ FCN , to generate the high-dimensional features H of the node t ;

[0035] H t =φ FCN (X t ) (3)

[0036] Then, H t and the adjacency matrix A t Input into GCN to generate graph convolution features. The graph convolution process is expressed as the following formula:

[0037]

[0038] Among them A t is the adjacency matrix, Is The calculated degree matrix, H t is the input of the graph convolution operation, W GCN is the weight matrix, b GCN is the bias matrix, and ReLU means using Relu as the activation function;

[0039] GCN aggregated graph convolutional features Z t The DQN decision processor is input, and the policy neural network calculates the Q value of each action in a given state and generates a policy for action selection as output. t Filter out the CAV strategy output, and the derivation of the Q value is as follows:

[0040] Q(s t ,a t )=M t ·φ DRL (Z t ,a t ) (5)

[0041] Where Q(s t ,a t ) represents the Q value of the lane-changing action of the CAV, φ DRL represents the policy neural network, a tRepresents all possible actions;

[0042] Step 4: Define the action space and reward function of the reinforcement learning agent. The process is as follows:

[0043] Step 4.1: Define the action space of the agent;

[0044] At each moment, the agent calculates the action that the CAV should perform in the current state based on the current observed state. The action space of the CAV is discrete, including left lane change, keeping the current lane, and right lane change, which can be expressed as: The action space of the agent is the set of CAV actions Where n is the number of CAVs in the scene at the current time step;

[0045] Step 4.2: Define the agent’s reward function.

[0046] The reward function consists of four parts: intention reward, average speed reward, lane change penalty, and collision penalty.

[0047] R=ω1R I +ω2R v -ω3P c -ω4P Lc (6)

[0048] Among them, R I represents the intention reward, which is calculated based on the road segment and is used to ensure that the CAV can leave from its intended ramp. v represents the average speed reward, which encourages CAVs to travel at high speed to improve the efficiency of the overall traffic flow, and P C represents the collision penalty, which is used to ensure the safety of the system, P LC represents the lane change penalty, which encourages the CAV to leave the intended ramp with the least lane change maneuvers, ω i Indicates the weight of each item;

[0049]

[0050] Where N CAV is the number of CAVs in the current scene; r ij (p ij )r ij (p ij ) and road segment Seg j Related to the lane the vehicle is in, to guide CAV2 to give way to CAV1 before entering Ramp1, and then turn to the right lane to leave Ramp2 as soon as possible; Used to indicate whether vehicle k belongs to the ijth th category, by ij (p ij) is set as a function of the longitudinal position of the vehicle, encouraging the agent to make lane change decisions as quickly as possible, r ij (p ij )'s specific settings are as follows Figure 3 As shown;

[0051]

[0052] where v i represents the speed of the i-th vehicle, v max Indicates the maximum speed limit for all vehicles;

[0053] P C =n collision (9)

[0054] Where n collision is the number of collisions at the current time;

[0055] P LC =n LC (10)

[0056] Where n LC is the number of lane changes of the CAV at the current time;

[0057] Step 5: Train and test the overall model;

[0058] The decision-making process of CAV in mixed traffic flow is as follows: Figure 4 As shown, the model relies on the experience replay buffer and the target network to be trained through Q-learning, during which the agent explores the environment by performing actions and obtains rewards. The experience replay buffer B is used to store the transfer (s t ,a t ,r t ,s t+1 ), randomly extract small batches of transfers from the experience replay buffer B at regular intervals. The training goal is to minimize the loss function, which is formulated as follows:

[0059]

[0060] in

[0061] After obtaining the trained model, the performance of the model is tested under different traffic conditions, and the performance of the model is evaluated using the round cumulative reward, the immediate reward of each time step, and the number of driving time steps;

[0062] Step 6: Build a risk processing module. The process is as follows:

[0063] Step 6.1: Build the AOSD module.

[0064] The above model is trained as the HAS module using the standard training process. On this basis, a LAS module with low adversarial sensitivity is obtained through distillation. Compared with the LAS module obtained after distillation, the HAS module has high adversarial sensitivity. Based on the inconsistency of the adversarial sensitivity of the LAS module and the HAS module, the AOSD module is constructed.

[0065] In order to ensure that the Q values ​​of the LAS module and the HAS module under normal observation are as consistent as possible, the main structure of GCQ is retained in the LAS module, and the distillation loss function is defined as follows:

[0066] L=λ1L LAS +λ2L Learn +λ3L KL (12)

[0067] Among them, L LAS is the Q-learning loss function of the LAS module itself; L Learn is the learning loss function, guiding the LAS module to learn knowledge from the HAS module; L KL is the high temperature distillation loss function, which is used to reduce the adversarial sensitivity of the LAS module. i is the coefficient corresponding to each loss function, which is used to balance the degree of knowledge learning;

[0068] L Learn = CrossEntropy Loss(Q LAS ,a HAS ) (13)

[0069] Among them, Q LAS In the observation state s t The Q value list of the LAS module is as follows: HAS In the observation state s t The action of the HAS module;

[0070]

[0071] Among them, Q represents the entire Q value list set, and are the Q-value embeddings of the i-th Q-value list given by the HAS module and the LAS module, respectively, where The jth element in represents the Q value of the jth item in the i-th list. τ is the distillation temperature;

[0072] Given an observation state s, first according to Q LAS and Q HAS The difference between the observed state and the adversarial state is detected, and the JS divergence is used to measure Q LAS and Q HASIf it is adversarial, the risk function is calculated. The risk function of each observed state s is defined as:

[0073]

[0074] According to the risk function, a given state s can be identified as risky or risk-free. Specifically, given an observed state s, the actions corresponding to the HAS module and the LAS module under state s are calculated respectively. When a HAS =a LAS When , state s is considered risk-free; and a HAS ≠a LAS When , the state s is considered risky; when the agent is in a risk-free state, it is considered to be in a safe state and does not need to perform any form of defense. However, if the agent perceives a risky state, it is in an unsafe situation and needs to perform defensive measures.

[0075] Step 6.2: Build the GTOSR module;

[0076] The GTOSR module consists of GAT, GRU encoder and GRU decoder. GAT is based on the traffic state (i.e., vehicle driving characteristics X t ) and traffic scene graph representation (i.e., vehicle adjacency matrix A t ) captures the spatial relationship of traffic entities in the scene and extracts the potential features of historical observation states. The GRU encoder uses the graph feature G output by GAT as input, and its output is used as the input of the decoder. The decoder outputs the reconstructed real observation state. The objective function of the reconstruction process is:

[0077]

[0078] Vehicle driving characteristics and the adjacency matrix A t is the input of the GAT module, of which GAT is the main component. GAT can classify nodes in graph structure data with the help of attention mechanism. In order to pay attention to adjacent nodes in the graph, GAT uses self-attention method to try to retrieve the hidden representation of each node. With the help of attention mechanism, GAT can improve the efficiency and accuracy of information processing;

[0079] GAT applies a shared linear transformation to each node, which is represented by the weight matrix W GAT Parameterize, then use the shared attention mechanism Α to calculate the attention coefficient at time t, and use masked attention to ensure that the operation of the attention mechanism is only performed on neighboring nodes;

[0080]

[0081] Where * is the matrix product, the attention coefficient c ij Indicates the importance of the j-th node vehicle to the i-th node vehicle;

[0082] In order to better distribute weights among different node vehicles, the attention coefficients calculated between the target node vehicle and all neighboring node vehicles are normalized;

[0083]

[0084] Where N i is the number of neighboring vehicles of the i-th node vehicle in the traffic graph (i.e., adjacency matrix);

[0085] In addition, LeakyRelu is applied to the feed-forward layer. Then, the coefficients calculated by the attention mechanism are formulated as:

[0086]

[0087] Among them, T is the transpose of Α, || represents the concatenation operation;

[0088] In order to output the final features of each node vehicle at time t, the normalized attention coefficients are used to obtain the linear combination of the corresponding features;

[0089]

[0090] In order to stabilize the learning process, multi-head attention is used in the GAT block. To this end, K GAT heads are used to perform the attention mechanism. Then, the K outputs from the GAT heads are averaged to obtain the output features. The calculation process is expressed as:

[0091]

[0092] Among them, α is the weight coefficient calculated by the k-th group of attention mechanism;

[0093] Using the GRU encoder, each time dimension in the graph features can produce a matching hidden feature in the following way:

[0094] output,hidden=GRU(G) (22)

[0095] Where GRU is a gated recurrent unit network, output is the output of the last hidden layer, and hidden is the hidden state of the last time step, that is, the hidden feature extracted by the encoder for the graphic feature G output by GAT;

[0096] According to the hidden features extracted by the GRU encoder, the GRU decoder outputs the reconstructed ground truth traffic state. The GRU decoding method is expressed as:

[0097] z t =σ(W z ·[h t-1 ,b t ]) (twenty three)

[0098] r t =σ(W r ·[h t-1 ,b t ]) (twenty four)

[0099]

[0100] In the first decoder step, the input b t is the vehicle driving characteristics observed at time t; h t-1 is the hidden feature of the graph extracted by the encoder; z t and r t are the update gate and reset gate, σ is the sigmoid activation function; W z and W r is the weight matrix for updating and resetting the gating calculation step; W is the generation The weight matrix in the step; (1-z t )*h t-1 represents the update gate z t Used to selectively forget the hidden features of the current input; Indicates selective storage of the current input b t Information. After GRU decoding, the current real observation state can be obtained;

[0101] Set up a normal traffic scene and construct an observation state dataset by online sampling. Use the observation state dataset to train the reconstruction module;

[0102] Combine the AOSD module and the GTOSR module as a risk handling module, such as Figure 5 As shown;

[0103] Step 7: Build a robust decision framework;

[0104] Integrate risk management modules into decision-making models to build a robust decision-making framework. Figure 6 As shown. At each time step, the observed state s t After risk processing, it is passed to the decision model. If the observed state s t If the AOSD module determines that the state is risky, the GTOSR module will reconstruct a state To replace the state t , guide the agent back to a safe state, and then the decision model obtained in the training phase is used to reconstruct the state Make a decision. If the observed state is judged to be a risk-free state, the decision model will make a decision based on the observed state s t Make decisions directly.

[0105] This embodiment uses data from actual experiments, and the implementation process is as follows:

[0106] (1) Select experimental data

[0107] The experimental data is based on the traffic status information generated by the interaction between HDV and CAV and the SUMO simulation environment in the diverging area of ​​the highway.

[0108] (2) Parameter determination

[0109] An experimental scenario is constructed based on the SUMO platform to evaluate the performance of the proposed method. The length of the main highway is 500 meters, and the lengths of Seg1 and Seg2 are both 200 meters. The maximum total number of vehicles N in the scenario is set to 40, while the maximum total number of HDVs and CAVs is 20, and the maximum total number of CAV1 and CAV2 is 10. The maximum speed of CAV is limited to 14 m / s, and the maximum speed of HDV is 10 m / s. At the beginning of the simulation, all vehicles enter the scenario from the left at a set inflow rate and have random positions and speeds at initialization. The default inflow rate of HDV in training is 0.2 vehicles / s, and the inflow rate of CAV is 0.1 vehicles / s. The perception range of CAV is set to 20 meters.

[0110] The deep reinforcement learning decision model of the present invention is built based on the Pytorch environment. The specific parameters of the model are: feature encoder φ FCN The number of layers is 2, the number of hidden layer units in each layer is 32, the number of hidden layer units in the GCN layer is 32, and the policy network φ DRL The number of layers is 2, the number of hidden layer units is 32 and 16 respectively, and the number of output layer neurons is 3. The training parameters of the deep reinforcement learning decision model are: 4 The first step is the random exploration phase. The batch size of data taken from the experience replay pool is 32, and the total number of training rounds is 200. Adam is used as the optimizer for model parameters, and the learning rate starts at 1×10 -4 , the target Q network uses soft update to update parameters, with an update rate of 1×10 -2 .

[0111] The HAS module is composed of the above model. The LAS module is obtained by high-temperature distillation of the HAS module. The coefficients of the distillation loss function are set to λ1:λ2:λ3=6:2:5. The distillation temperature is 5. The number of distillation rounds is 200.

[0112] AOSD module detection threshold d Bound Set to 5×10-9 An offline dataset consisting of normal observation states is constructed to train the GTOSR module. The GTOSR module is trained for 20 epochs. The optimizer Adam is used to optimize the GTOSR module with a learning rate of 5×10 -4 .

[0113] (3) Experimental results

[0114] The present invention aims to control CAV to make robust decisions and complete ramp exit tasks in confrontation scenarios, and experiments are carried out in a highway diversion area simulation environment based on the SUMO platform.

[0115] In order to evaluate the effectiveness of the proposed method, this study simulates observation perturbations through the following adversarial attack methods. The following adversarial attack methods are implemented based on PGD: random attack, which attempts to make the model perform incorrect or suboptimal actions regardless of the specific category of the action; targeted attack, which attempts to make the model perform pre-specified target actions instead of just making the model perform incorrect or suboptimal actions; MAD attack, which attempts to minimize the likelihood of the action with the highest probability; Q-value attack, which attempts to minimize the Q-value of the model under adversarial conditions.

[0116] In order to evaluate the performance of the robust decision-making method integrated with risk handling, 50 rounds of testing were conducted. The performance indicators include the round cumulative reward, the immediate reward for each time step, and the number of driving time steps. Table 1 lists the average values ​​of these indicators. In Table 1, the test results under four different adversarial scenarios (random attack, target attack, MAD attack and Q-value attack) are compared. The results show that the method of the present invention performs well in all four adversarial scenarios. Table 1 shows the round reward, immediate reward, and driving time steps under different adversarial scenarios;

[0117] Random Attack Targeted Attack MAD Attack Q-value attack Round Rewards 2651 2370 2425 2486 Instant Rewards 1.90 1.74 1.82 1.80 Driving time steps 1427 1388 1357 1424

[0118] Table 1

[0119] The performance of the AOSD module and the GTOSR module are evaluated separately. The performance of the AOSD module is shown in Table 2. The low false negative rate and high F1 score indicate that the AOSD module can effectively detect adversarial observation states. The performance of the GTOSR module is shown in Table 3. The low mean square error indicates that the reconstruction results of the GTOSR module are accurate enough to provide reliable observation states for subsequent decisions.

[0120] Table 2 shows the F1 score and false negative rate of the AOSD module in different adversarial scenarios;

[0121]

[0122]

[0123] Table 2

[0124] Table 3 shows the mean square error of the GTOSR module under different adversarial scenarios;

[0125] Random Attack Targeted Attack MAD Attack Q-value attack Mean square error 0.00 0.00 0.00 0.00

[0126] Table 3

[0127] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A robust decision-making method for connected autonomous vehicles in mixed traffic flows based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Set up the task scenario of CAV decision-making in mixed traffic flow; build a simulation scenario based on the SUMO platform, set the behavior mode of HDV in mixed traffic flow and the target task of CAV; Step 2: Define the state space that the agent can observe; establish the adjacency matrix of the vehicle network based on the interactions between vehicles; Step 3: Determine the overall network structure of the decision model; Build feature encoder, graph neural network and policy network; Step 4: Define the action space and reward function of the reinforcement learning agent; define the action space as a set of lane-changing actions, and set the reward function to guide the agent to learn the strategy that can obtain the optimal action; Step 5: Train and test the decision model; use experience replay and target network to train the overall model; Step 6: Construct the adversarial observation state detection AOSD module, construct the real observation state reconstruction GTOSR module, and combine the AOSD module and the GTOSR module as a risk processing module; Step 7: Integrate the risk management module into the decision model to build a robust decision framework; test the performance of the deep reinforcement learning decision model combined with risk management under different traffic conditions.

2. The robust decision-making method for interconnected autonomous vehicles in mixed traffic flow based on deep reinforcement learning as claimed in claim 1, characterized in that: The process of step 1 is as follows: The traffic scenario in the SUMO simulation platform includes a three-lane highway with two ramp exits. According to the locations of ramp exit 1, namely Ramp1, and ramp exit 2, namely Ramp2, the highway is divided into two sections, named Seg1 and Seg2 respectively. The simulation involves mixed traffic flows of CAVs and HDVs. All vehicles enter the scenario from the left side of the highway. The HDV flow is f h , CAV flow is f c ; The speed limit of the road section is v max , where the maximum speed of the CAV is v c , the maximum speed of HDV is v h Different vehicles have different driving tasks, and they need to cooperate to complete the predetermined driving tasks more efficiently; the HDV task is to leave the highway from the right side; the CAV1 task is to leave from ramp 1; the CAV2 task is to leave from ramp 2. The longitudinal and lateral motion control of the HDV is controlled using SUMO's built-in IDM following model and LC2013 lane change model respectively, and the longitudinal motion of the CAV is the same as that of the HDV; the DRL model outputs lane changing decisions for the CAV, so that the CAV can leave the highway from its intended ramp in a safe and efficient manner.

3. The robust decision-making method for interconnected autonomous vehicles in mixed traffic flow based on deep reinforcement learning as claimed in claim 2, characterized in that: The process of step 2 is as follows: At each time step t, the state of the agent is s t =(X t ,A t ,M t ), including vehicle driving characteristics X t , adjacency matrix A t and the mask matrix M t ; Vehicle driving characteristics: The driving characteristics of vehicle i at time step t include position, speed, lane and intention, expressed as And each element is determined as follows: represents the normalized longitudinal position of vehicle i at time step t relative to the road length, represents the normalized longitudinal velocity of vehicle i at time step t relative to the maximum longitudinal velocity, represents the lane where vehicle i is located at time step t, using one-hot encoding to describe it, that is, [1, 0, 0], [0, 1, 0], and [0, 0, 1] represent the right lane, middle lane, and left lane respectively; represents the driving intention of vehicle i at time step t, which is also represented by one-hot encoding, that is, [1, 0, 0], [0, 1, 0], and [0, 0, 1] represent leaving from ramp 1, leaving from ramp 2, and going straight along the main line of the highway, respectively; Adjacency matrix: The vehicle network is modeled as a graph. Each vehicle in the scene is regarded as a node of the graph. The node features are the driving features of the vehicle. The edge relationship between the nodes of the graph is represented by the adjacency matrix. The adjacency matrix is ​​based on the following assumptions: all vehicles can share information with themselves; all CAVs can share information with each other; all CAVs can share information with the HDVs within their perception range. The dimension of the adjacency matrix is ​​N×N, which matches the dynamic number of vehicles in the scene. When the number of vehicles is less than N, or when the vehicle leaves the scene, the corresponding position in the matrix is ​​occupied by 0 and is represented by Represents the edge value between vehicle i and vehicle j, and the adjacency matrix A t Depend on When vehicle i and vehicle j share information, when When vehicle i and vehicle j do not share any information; Mask matrix: CAV mask M t It is used to filter out the features of HDV and only retain the features of CAV. t According to each In X t The length of the vector is the number of vehicles N. It is a binary vector with a value of 1 at the index corresponding to CAV and 0 otherwise; M t =[m1,m2,...,m i ,...,m n ] (2)。 4. The robust decision-making method for interconnected autonomous vehicles in mixed traffic flow based on deep reinforcement learning as claimed in claim 3, characterized in that: The process of step 3 is as follows: Construct a fully connected network FCN encoder φ FCN , at each time step t, the node feature matrix X t First, it is fed to φ FCN , to generate the high-dimensional features H of the node t ; H t =φ FCN (X t ) (3) Then, H t and the adjacency matrix A t Input into GCN to generate graph convolution features. The graph convolution process is expressed as the following formula: Among them A t is the adjacency matrix, Is The calculated degree matrix, H t is the input of the graph convolution operation, W GCN is the weight matrix, b GCN is the bias matrix, ReLU means using Relu as the activation function; GCN aggregated graph convolutional features Z t The DQN decision processor is input, and the policy neural network calculates the Q value of each action in a given state and generates a policy for action selection as output, by applying the mask matrix M t Filter out the CAV strategy output, and the derivation of the Q value is as follows: Q(s t ,a t )=M t ·φ DRL (Z t ,a t ) (5) Where Q(s t ,a t ) represents the Q value of the lane-changing action of the CAV, φ DRL represents the policy neural network, a t Represents all possible actions.

5. The robust decision-making method for interconnected autonomous vehicles in mixed traffic flow based on deep reinforcement learning as claimed in claim 4, characterized in that: The process of step 4 is as follows: Step 4.1: Define the action space of the agent; At each moment, the agent calculates the action that the CAV should perform in the current state based on the current observed state. The action space of the CAV is discrete, including left lane change, keeping the current lane, and right lane change, which can be expressed as: The action space of the agent is the set of CAV actions Where n is the number of CAVs in the scene at the current time step; Step 4.2: Define the agent’s reward function. The reward function consists of four parts: intention reward, average speed reward, lane change penalty, and collision penalty. R=ω1R I +ω2R v -ω3P c -ω4P Lc (6) Among them, R I represents the intention reward, which is calculated based on the road segment and is used to ensure that the CAV can leave from its intended ramp; R v represents the average speed reward, which encourages CAVs to travel at high speeds to improve the efficiency of the overall traffic flow;;P C represents the collision penalty, which is used to ensure the safety of the system; P LC represents the lane change penalty, which encourages the CAV to leave the intended ramp with minimal lane change operations; ω i Indicates the weight of each item; Where N CAV is the number of CAVs in the current scene; r ij (p ij )r ij (p ij ) and road segment Seg j Related to the lane the vehicle is in, to guide CAV2 to give way to CAV1 before entering Ramp1, and then turn to the right lane to leave Ramp2 as soon as possible; Used to indicate whether vehicle k belongs to the ijth th category, by ij (p ij ) is set as a function of the longitudinal position of the vehicle, encouraging the agent to make lane change decisions as quickly as possible; where v i represents the speed of the i-th vehicle, v max Indicates the maximum speed limit for all vehicles; P C =n collision (9) where n collision is the number of collisions at the current time; P LC =n LC (10) where n LC is the number of lane changes of the CAV at the current time.

6. The robust decision-making method for interconnected autonomous vehicles in mixed traffic flow based on deep reinforcement learning as claimed in claim 5, characterized in that: The process of step 5 is as follows: The decision model relies on the experience replay buffer and the target network to be trained through Q-learning. During this process, the agent explores the environment and obtains rewards by performing actions, using the experience replay buffer B to store the transfer (s t ,a t ,r t ,s t+1 ), randomly extract small batches of transfers from the experience replay buffer B at regular intervals, and the training goal is to minimize the loss function, which is formulated as follows: in After obtaining the trained model, the performance of the model is tested under different traffic conditions, and the performance of the model is evaluated using the cumulative reward of the round, the immediate reward of each time step, and the number of driving time steps.

7. The robust decision-making method for interconnected autonomous vehicles in mixed traffic flow based on deep reinforcement learning as claimed in claim 6, characterized in that: The process of step 6 is as follows: Step 6.1: Build AOSD module; The above model is trained as the HAS module using the standard training process. On this basis, a LAS module with low adversarial sensitivity is obtained through distillation. Compared with the LAS module obtained after distillation, the HAS module has high adversarial sensitivity. Based on the inconsistency of the adversarial sensitivity of the LAS module and the HAS module, the AOSD module is constructed. In order to ensure that the Q values ​​of the LAS module and the HAS module under normal observation are as consistent as possible, the main structure of GCQ is retained in the LAS module, and the distillation loss function is defined as follows: L=λ1L LAS +λ2L Learn +λ3L KL (12) Among them, L LAS is the Q-learning loss function of the LAS module itself; L Learn is the learning loss function, guiding the LAS module to learn knowledge from the HAS module; L KL is the high temperature distillation loss function, which is used to reduce the adversarial sensitivity of the LAS module, λ i is the coefficient corresponding to each loss function, which is used to balance the degree of knowledge learning; L Learn =CrossEntropy Loss(Q LAS ,a HAS ) (13) Among them, Q LAS In the observation state s t The Q value list of the LAS module is as follows: HAS In the observation state s t The action of the HAS module; Among them, Q represents the entire Q value list set, and are the Q-value embeddings of the i-th Q-value list given by the HAS module and the LAS module, respectively, where The jth element in represents the Q value of the jth item in the i-th list, and τ is the distillation temperature; Given an observation state s, first according to Q LAS and Q HAS The difference between the observed state and the adversarial state is detected, and the JS divergence is used to measure Q LAS and Q HAS If it is adversarial, the risk function is calculated. The risk function of each observed state s is defined as: According to the risk function, a given state s can be identified as risky or risk-free. Given an observed state s, the actions corresponding to the HAS module and the LAS module under state s are calculated respectively. When a HAS =a LAS When , state s is considered risk-free; and a HAS ≠a LAS When the agent is in a risk-free state, it is considered to be in a safe state and does not need to perform any form of defense. If the agent perceives a risky state, it is in an unsafe situation and needs to perform defensive measures. Step 6.2: Build the GTOSR module; The GTOSR module consists of GAT, GRU encoder and GRU decoder. GAT uses traffic status, i.e., vehicle driving characteristics X t And the traffic scene graph representation, that is, the vehicle adjacency matrix A t Capture the spatial relationship of traffic entities in the scene and extract the potential features of historical observation states. The GRU encoder uses the graph feature G output by GAT as input, and its output is used as the input of the decoder. The decoder outputs the reconstructed real observation state. The objective function of the reconstruction process is: Vehicle driving characteristics and the adjacency matrix A t is the input of the GAT module, of which GAT is the main component. GAT can classify nodes in graph structure data with the help of attention mechanism. In order to pay attention to adjacent nodes in the graph, GAT uses self-attention method to try to retrieve the hidden representation of each node. With the help of attention mechanism, GAT can improve the efficiency and accuracy of information processing; GAT applies a shared linear transformation to each node, which is represented by the weight matrix W GAT Parameterize, then use the shared attention mechanism Α to calculate the attention coefficient at time t, and use masked attention to ensure that the operation of the attention mechanism is only performed on neighboring nodes; Where * is the matrix product, the attention coefficient c ij Indicates the importance of the j-th node vehicle to the i-th node vehicle; In order to better distribute weights among different node vehicles, the attention coefficients calculated between the target node vehicle and all neighboring node vehicles are normalized; Where N i is the traffic graph, that is, the number of neighboring vehicles of the i-th node vehicle in the adjacency matrix; Furthermore, LeakyRelu is applied to the feed-forward layer, and then the coefficients calculated by the attention mechanism are formulated as: Among them, T is the transpose of Α, || represents the concatenation operation; In order to output the final features of each node vehicle at time t, the normalized attention coefficients are used to obtain the linear combination of the corresponding features; In order to stabilize the learning process, multi-head attention is used in the GAT block. To this end, K GAT heads are used to perform the attention mechanism. Then, the K outputs from the GAT heads are averaged to obtain the output features. The calculation process is expressed as: Among them, α is the weight coefficient calculated by the k-th group of attention mechanism; Using the GRU encoder, each time dimension in the graph features can produce a matching hidden feature in the following way: output,hidden=GRU(G) (22) Where GRU is a gated recurrent unit network, output is the output of the last hidden layer, and hidden is the hidden state of the last time step, that is, the hidden feature extracted by the encoder for the graphic feature G output by GAT; According to the hidden features extracted by the GRU encoder, the GRU decoder outputs the reconstructed ground truth traffic state. The GRU decoding method is expressed as: z t =σ(W z ·[h t-1 ,b t ]) (23) r t =σ(W r ·[h t-1 ,b t ]) (24) In the first decoder step, the input b t is the vehicle driving characteristics observed at time t; h t-1 is the hidden feature of the graph extracted by the encoder; z t and r t are the update gate and reset gate, σ is the sigmoid activation function; W z and W r is the weight matrix for updating and resetting the gating calculation step; W is the generation The weight matrix in the step; (1-z t )*h t-1 represents the update gate z t Used to selectively forget the hidden features of the current input; Indicates selective storage of the current input b t The information of the information can be decoded by GRU to obtain the current real observation state; Set up a normal traffic scene, construct an observation state dataset through online sampling, and use the observation state dataset to train the reconstruction module; The AOSD module and the GTOSR module are combined as a risk handling module.

8. The robust decision-making method for interconnected autonomous vehicles in mixed traffic flow based on deep reinforcement learning as claimed in claim 7, characterized in that: The process of step 7 is as follows: Integrate the risk processing module in the decision model to build a robust decision framework. At each time step, observe the state s t It is first processed by risk and then passed to the decision model; If the observed state s t If the AOSD module determines that the state is risky, the GTOSR module will reconstruct a state To replace the state t , guide the agent back to a safe state; then, the decision model obtained in the training phase is reconstructed according to the state Make a decision. If the observed state is judged to be a risk-free state, the decision model will make a decision based on the observed state s t Make decisions directly.

Images