Graph convolutional neural network and reinforcement learning fused highland cooperative control method

Abstract

The invention provides a highland cooperative control method fusing a graph convolutional neural network and reinforcement learning, through controlling urban road traffic signal lamps, an expressway ramp signal machine and a variable speed limit unit, expressway traffic congestion is relieved, the urban traffic passing efficiency is improved, road driving vehicles are controlled by means of the signal lamps and the speed limit unit, and the traffic efficiency is improved. Traffic flow scheduling optimization in a global road network is achieved; the whole structure adopts a distributed cooperative control framework, is controlled by a decision-making agent, and performs autonomous decision-making and adjustment according to real-time traffic conditions, so that the adaptability and robustness of the system are enhanced, and the defects of large data calculation amount, difficulty in processing, control strategy lag and the like caused by traditional centralized control are effectively avoided; the graph convolutional neural network model is constructed to aggregate the feature information of the peripheral decision intelligent agent, and the control weight is dynamically allocated, so that the information sharing capability is enhanced, the suboptimal decision caused by lack of comprehensive perception of the surrounding environment is prevented, and the cooperation efficiency of the whole system is improved.

Description

Technical Field

[0001] The present invention relates to the field of traffic control and intelligent traffic technology, and more specifically to a highland collaborative control method integrating graph convolutional neural network and reinforcement learning. Background Art

[0002] At present, in the field of urban and highway traffic control, fixed or feedback timing strategies and control technologies are widely used. Among them, the fixed traffic signal control strategy is the most traditional and simple, and is mostly used in urban road intersections. Its main characteristics are that the cycle and phase time of the signal light are based on artificial presets, and are periodically cycled according to a fixed duration, and are not adjusted with the changes in traffic flow. It lacks adaptability and dynamism and cannot cope with real-time changes in traffic flow. This leads to traffic congestion and low traffic efficiency when traffic demand fluctuates greatly. The feedback traffic signal control strategy dynamically adjusts the signal timing based on traffic flow feedback and is mostly used in highway entrance ramps. Although it meets the adaptability and dynamism requirements of traffic signal control to a certain extent, it maintains a single processing strategy for the feedback information of the controlled section, and cannot adapt to areas with more complex traffic flow through self-learning. In addition, the feedback traffic signal control strategy makes decisions based on the traffic status data of a single intersection or section, and lacks information sharing between different control units, which easily aggravates the local optimal problem of traffic flow scheduling, thereby causing regional congestion.

[0003] In terms of overall structure, compared with distributed control, the centralized control in existing research has a large amount of data calculation, complex and difficult processing, and strong dependence on the central server. Once a single point failure of an intelligent agent occurs, it is possible to paralyze the entire traffic control system. In addition, the centralized control system manages all intelligent agents in the entire road network through centralized unified processing and scheduling. If the road network covers a large area, the central control system makes a unified decision and then feeds back to each intelligent agent, which can easily cause the control strategy to lag, thereby posing a threat to vehicle driving safety and failing to meet the requirements of traffic control for real-time and safety.

[0004] In addition, in the prior art, for road sections where traffic status data collection equipment is not installed, mathematical interpolation or fitting models are usually used to infer or estimate missing data. Although this method can make up for the problem of missing data information in unmonitored sections to a certain extent, the interpolation or fitting method relies on static or historical data, which not only has limited data accuracy and poor real-time performance, but also easily causes data error accumulation when used for a long time, reducing the stability of the overall traffic control system. Summary of the invention

[0005] In order to solve the congestion problem caused by the lack of coordination between expressways and urban ground roads, the present invention proposes a road network traffic control method combining graph convolutional neural networks and reinforcement learning. The method first uses a traffic information collection module to obtain ground intersection signals, ramp signal light controller states and road traffic data. For sections without fixed collection equipment, vehicle-road collaboration technology is used to collect data through automatic networked vehicles. Each ground intersection signal light, entrance ramp controller and variable speed limit control unit is equipped with a behavior decision-making agent based on reinforcement learning, and the key spatiotemporal features extracted by the graph convolutional network are used as the input of the decision-making agent to optimize the decision-making process. In the scenario of considering the collaboration of multiple decision-making agents, the structured information output by the graph convolutional neural network is used as the data input for self-learning and training to improve the strategy effect and finally achieve continuous iteration to the optimal strategy. The decision-making agent optimizes signal timing and vehicle speed, adjusts the speed of ramp release vehicles and highway main line traffic, and comprehensively considers factors such as travel time, road vehicle density and queue length to ensure the smooth flow of the highway main line and quickly clear the ramp stranded vehicles to avoid affecting the traffic of urban ground roads. This method effectively addresses traffic congestion and ramp oversaturation by collaboratively managing highway and ground traffic, thereby achieving optimized coordination between highways and urban ground transportation.

[0006] The highland collaborative control method integrating graph convolutional neural network and reinforcement learning includes the following steps:

[0007] Step 1: Collect traffic status data through the traffic information collection module, and build a traffic data communication layer to form a distributed interconnected traffic information network;

[0008] Step 2, preprocessing the data collected in step 1 through a data preprocessing module;

[0009] Step 3: Take the preprocessed data in step 2 as input, perform time-series graph convolution processing on it through the graph convolution neural network module, and output the traffic status;

[0010] Step 4: Input the traffic state extraction features in step 3 into the reinforcement learning module, generate the optimal control strategy through continuous training feedback, and pass it to the corresponding traffic control unit.

[0011] Step 5: The traffic control unit converts the output of the control strategy into a traffic instruction and sends it to the associated vehicle or related carrier through communication.

[0012] As a further improvement of the present invention, in step 1, the traffic information network includes a node set and an edge set in the network, the node set represents the behavior decision-making agent, and the edge set represents the communication relationship between the decision-making agents.

[0013] As a further improvement of the present invention, the traffic information network regards each behavior decision agent as a network node, and for each decision agent, that is, the network node v i Establish the feature vector of real-time traffic status v i ∈V, where t i is the timestamp, recording the data collection time; V is the node set, covering all decision-making agents in the road network; d i is the traffic status data, including the queue length, vehicle density and average speed of urban ground roads, expressway main lines and entrance ramp sub-sections; k i For data labeling, identify the decision-making agent that collects data input, that is, the decision-making agent with the shortest straight-line distance to the data source.

[0014] As a further improvement of the present invention, by acquiring and recording the data collected in step 1, a decision agent v in the traffic data communication layer is constructed. i The historical traffic data in the past T time interval is defined as the data vector in represents the traffic state characteristics obtained by the decision agent i at time t. In step 2, the specific steps of data preprocessing include: data detection, data smoothing and data standardization.

[0015] As a further improvement of the present invention, the time series graph convolution processing in step 3 is specifically as follows: the data is preprocessed to obtain a standardized data vector containing time and space dimension information As the module input, based on the traffic data communication layer, by stacking a layer of temporal convolution and a multi-layer graph convolution neural network, the spatiotemporal correlation characteristics between different decision-making agents are captured to obtain the correlation vector O of the traffic status data. i .

[0016] As a further improvement of the present invention, in step 4, the reinforcement learning module includes a training module and an execution module, wherein:

[0017] The execution module includes a traffic control unit and a strategy network corresponding to the decision-making agent in the traffic data communication layer after collecting data;

[0018] The training module includes a reward calculation module, a sampling buffer, and a value network.

[0019] As a further improvement of the present invention, the reward calculation module in the training module includes:

[0020] Reward calculation for the length of vehicle queues at highway entrance ramps:

[0021]

[0022] In the formula, q max is the maximum queue length of the entrance ramp, q is the actual queue length of the ramp, Δq is the rate of change of the queue length, α is the acceleration reward impact factor, and k base is the basic reward coefficient, Q rm represents the reward value corresponding to the queue length of the entrance ramp, k i is the coefficient of the reward function, where k i Indicates the sensitivity of the reward value to changes in the queue length q;

[0023] Bonus calculation for total vehicle travel time:

[0024]

[0025] In the formula, is the actual vehicle travel time, T max k is the longest travel time acceptable to drivers, and its value is determined based on the actual situation of the highway. j is the sensitivity coefficient, T tr Indicates the reward value corresponding to the vehicle travel time;

[0026] Reward calculation for vehicle density at an urban road intersection or highway main line at any time t:

[0027]

[0028] In the formula, c max is the maximum capacity of the road, d t,i is the vehicle density of road section i at time t, β is the reward discount coefficient;

[0029] Introduce weight coefficients to calculate the comprehensive reward function r:

[0030] r=T tr +μ 1 Q rm +μ 2 D ld

[0031] Where r is the weighted average of the above three rewards, μ 1 and μ 2 are the proportional coefficients of queue length and vehicle density bonus compared to travel time bonus.

[0032] As a further improvement of the present invention, in step 5, for two different traffic systems, expressways and urban roads, the weight coefficients of different reward values ​​are differentiatedly designed in the reward calculation module to achieve multi-strategy collaborative control, specifically:

[0033] In the highway traffic system, increasing the queue length Q rm The weight μ1 , prompting the decision-making agent to prioritize reducing the ramp queue length and quickly clear the stranded vehicles on the ramp;

[0034] In the urban road traffic system, increasing vehicle density D ld The weight μ 2 , ensuring that the decision-making agent can prioritize traffic flow in congested areas of intersections;

[0035] In the global traffic network system, by adjusting the weight μ 1 and μ 2 , increasing the weight of the total travel time in the comprehensive reward function r, and guiding the decision-making agent to balance local and global traffic demands.

[0036] As a further improvement of the present invention, the traffic control unit includes a signal light control unit laid at the intersection of the ground urban road, a variable speed limit control unit of the main line of the expressway, and a signal light control unit of the entrance ramp, each control unit corresponds to a decision-making intelligent agent, and the action a selected by the decision-making intelligent agent is t The traffic control module converts the action instructions into corresponding traffic control instructions, and sends the instructions to the associated decision-making intelligent agent or the vehicles on the controlled road section through the communication network of the traffic information collection module. The traffic light control unit at the intersection is used to convert the action instructions into the timing of each phase of the traffic light. The traffic light control unit at the entrance ramp is used to convert the vehicle release instructions of the ramp into the red-green signal cycle. The variable speed limit control unit is used to convert the speed limit instructions into a specific speed limit value.

[0037] As a further improvement of the present invention, in step 5, the method for converting the variable speed limit instruction into a specific speed limit value by the traffic control unit includes the following steps:

[0038] Step 5-2-1, obtain the variable speed limit instruction at the current time t from the execution module.

[0039] Step 5-2-2, obtain the vehicle density ρ of the variable speed limit section through the road camera, and the calculation formula is as follows:

[0040]

[0041] Where N is the number of vehicles in the control area section, and L is the road length of the control area section.

[0042] Step 5-2-3, set the speed limit range of the variable speed control area and determine the basic speed limit value, such as the minimum speed limit v min and the maximum speed limit v max .

[0043] Step 5-2-4, check whether the vehicle density ρ in the control area reaches the threshold ζ. If the vehicle density on the road in the current variable speed limit control area does not reach the threshold ζ, keep the original speed limit value and skip the speed limit value update step. If the vehicle density exceeds the threshold ζ, execute the following update steps 5-2-5 and 5-2-6.

[0044] Step 5-2-5, calculate the speed limit value v of the road section according to the vehicle density in the control area limit :

[0045]

[0046] In the formula, v max is the maximum speed limit set in step 5-2-3, ρ max It is the maximum vehicle density that the controlled road section can withstand.

[0047] Step 5-2-6, update the specific speed limit value, limit v limit In the set speed limit range:

[0048] v limit =max(v min ,min(v max ,v limit ))

[0049] Step 5-2-7, update the specific speed limit value v limit Applied to the variable speed limit information board, the information board VSLD transmits the speed limit information to the vehicles traveling on the controlled section through the electronic display screen.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] (1) The present invention proposes a collaborative control method that integrates urban ground roads and highways. The method aims to solve the congestion problem caused by the lack of coordination between highways and urban ground roads. It not only considers the traffic management of the highway main line and its entrance ramps, but also includes the traffic control of urban ground roads in the collaborative control scope. Traditional methods to alleviate highway congestion often rely on variable speed limit control of the highway main line and regulation and release of ramps. However, this method lacks coordination with urban ground roads and is prone to a large influx of urban road traffic, resulting in a backlog of vehicles on the ramps, causing congestion in the intersection area of ​​highways and urban roads, thereby reducing the overall traffic efficiency of the urban road network. The highland collaborative control scheme is based on real-time traffic data and multi-strategy collaborative traffic scheduling between the two traffic systems of urban roads and highways, maximizing the use of traffic resources of urban roads and highways. On the basis of alleviating highway congestion, the adverse effects of ramp queues on urban ground roads are avoided. This is an important advantage of the present invention compared with traditional control methods.

[0052] (2) The present invention proposes a distributed multi-decision agent collaborative control strategy, which regards each controller or traffic light as an independent behavioral decision agent and assigns an independent control unit. The strategy supports decision agents to make autonomous decisions based on local real-time data and current traffic scenarios, and converge to the optimal strategy through self-training and continuous optimization, thereby improving the adaptability and robustness of the system. By introducing GCN to capture the spatiotemporal correlation characteristics between different decision agents, the decision agents can not only make independent decisions based on the traffic conditions within their own visible areas, but also share traffic status data with neighboring decision agents through short-range communication technology to achieve collaborative decision-making. Traditional traffic control methods usually rely on a central control system to uniformly dispatch and manage all decision agents in the entire road network. The main drawback of this centralized control method is that the central system needs to receive and process a large amount of traffic data from different decision agents in real time, make a unified decision, and then feed it back to each decision agent. Once a single point of failure occurs, the entire traffic control system may be paralyzed. In contrast, the distributed decision agent collaborative control strategy based on graph convolutional neural network can reduce the dependence on the central server, while avoiding the problem of inaccurate decision-making caused by information lag, and significantly improve the data processing efficiency and the system's adaptability.

[0053] (3) The highland collaborative control method based on graph convolutional network (GCN) and reinforcement learning PPO proposed in this invention can dynamically adjust the control strategy by analyzing traffic status data such as vehicle density, queue length and average speed. This method adopts a cyclic update mechanism of "feature extraction-online decision-making-offline training-evaluation comparison-iterative optimization" so that the strategy can be continuously optimized according to real-time traffic data and interactive scenarios. Compared with the traditional single reinforcement learning method, the combination of GCN can effectively capture the spatiotemporal correlation between decision-making agents, so that each decision-making agent not only considers its own state when making decisions, but also comprehensively considers the situation of surrounding decision-making agents, avoiding the problem of local optimal strategy. At the same time, the proximal policy optimization algorithm (PPO) of the Actor-Critic architecture supports online decision-making based on real-time traffic status, ensuring high real-time performance and response speed of traffic control. And PPO can limit the range of policy changes through clipping technology to ensure the stability of the update process. In addition, the reinforcement learning model can efficiently use empirical data for offline training, reduce dependence on a large amount of new data, and improve the learning efficiency and convergence speed of the model.

[0054] (4) The present invention designs a complete highland collaborative control system with strong adaptability and learning capabilities. The system consists of traffic information collection, data preprocessing, graph convolutional neural network and reinforcement learning modules. The modules work closely together to achieve a complete closed loop from data collection and processing to strategy output and optimization. Compared with a control system with a single function or fixed rules, the cross-regional multi-decision-making intelligent collaborative control system proposed in the present invention is more scalable and can be adjusted according to the actual traffic scene requirements, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the application scenario and highland collaborative control network model of the present invention.

[0056] Figure 2 It is a schematic diagram of the system module structure in the technical solution of the present invention.

[0057] Figure 3 Schematic diagram of the graph convolutional neural network layer structure of the present invention.

[0058] Figure 4 This is a training flow chart of the reinforcement learning module of the Actor-Critic architecture of the present invention.

[0059] Figure 5 The figure is a conversion flow chart of the ramp release instruction and the traffic light signal cycle of the present invention.

[0060] Figure 6 It is a conversion flow chart of the variable speed limit instruction and the specific speed limit value of the present invention. DETAILED DESCRIPTION

[0061] The present invention is described in detail below in conjunction with the various embodiments shown in the accompanying drawings, but it should be noted that these embodiments are not limitations of the present invention, and any equivalent transformations or substitutions in functions, methods, or structures made by ordinary technicians in the field based on these embodiments are all within the scope of protection of the present invention.

[0062] A highland collaborative control method integrating graph convolutional neural network and reinforcement learning is provided. The method is based on a system including a traffic information collection module, a data preprocessing module, a graph convolutional neural network module, a reinforcement learning module, and several traffic control units. The method includes the following steps:

[0063] Step 1, collect traffic status data through the traffic information collection module, and build a traffic data communication layer to form a distributed interconnected traffic information network; the traffic information collection module is mainly used to collect the status information of each control unit in the road network and the traffic information of the traffic flow on the road to form a real-time traffic status network. Each control unit corresponds to an independent behavior decision-making intelligent body, including intersection lights, ramp lights and variable speed limit information boards. The data collection unit includes but is not limited to electronic equipment such as road cameras, coil sensors and infrared speed meters. The traffic status information obtained is not limited to the queue length of the entrance ramp and the city intersection, the average vehicle speed and the road traffic density. For sections that lack fixed collection equipment, vehicle-road collaborative technology is used to collect data through automatic connected vehicles (CAVs) to make up for the problem of missing data on sections without equipment and form a complete traffic information network.

[0064] In order to realize the sharing of traffic status information among different decision-making agents, a traffic data communication layer based on vehicle networking technology is constructed in the traffic information collection module according to the collected original data. This layer aims to realize the sharing of traffic data among different decision-making agents within a certain range through the Cellular Vehicle-to-Everything (C-V2X) or Dedicated Short Range Communication (DSRC) network, forming a distributed interconnected traffic information network G = (V, E). The node set V in the network represents each behavior decision-making agent, and the edge set E represents the communication relationship between decision-making agents.

[0065] The traffic information network regards each behavior decision agent as a network node, and for each decision agent, i.e., network node v i Establish the feature vector of real-time traffic status v i ∈V where t i is the timestamp, recording the data collection time; V is the node set, covering all decision-making agents in the road network; d i is the traffic status data, including the queue length, vehicle density and average speed of urban ground roads, expressway main lines and entrance ramp sub-sections; k i For data labeling, identify the decision-making agent that collects data input, that is, the decision-making agent with the shortest straight-line distance to the data source.

[0066] Step 2: preprocess the data collected in step 1 through the data preprocessing module; the data preprocessing module will obtain and record the decision-making agent v through the traffic data collection module. i The historical traffic data in the past T time interval is defined as the data vector in represents the traffic state characteristics obtained by decision agent i at time t. This module will i Perform data detection, smoothing and standardization operations. The specific steps are as follows:

[0067] Data detection: When real-time monitoring and collecting traffic status data, data sensors and other electronic devices may output extreme abnormal values ​​due to failure or aging. In response to data anomalies, the preprocessing module uses anomaly criteria based on normal distribution to perform data detection. First, according to the data vector X i Calculate the mean of each component in Then by comparing Deviations from the mean detect whether they are abnormal data:

[0068]

[0069]

[0070] Data smoothing: After the preprocessing module filters out abnormal values ​​through data detection, in order to reduce the impact of short-term data fluctuations, such as sudden braking or braking of the vehicle, the exponentially weighted moving average (EWMA) method is used to calculate the data feature vector of the decision-making agent i at each moment in the past T period. To make corrections, the specific steps are as follows:

[0071]

[0072] In the formula, δ is the smoothing coefficient, and the left side of the formula is the smoothed traffic status data, It is the smoothed traffic status data at time t-1.

[0073] Data standardization: After data detection and smoothing, the feature vector needs to be standardized. Since different features such as queue length, vehicle density and average speed have different dimensions and data ranges, the preprocessing module uses the Z-score standardization method to standardize each state feature. To process:

[0074]

[0075]

[0076] In the formula, is the feature v i The standard deviation of is the standardized traffic state feature vector of decision agent i at time t;

[0077] The standardized data vector is

[0078] Status data i Including the queue length q of urban surface roads, highway main lines and entrance ramp sub-sections t,i , vehicle density d t,i and average speed v t,i , the algorithm for obtaining is:

[0079]

[0080] Where N stop is the number of vehicles that stop and wait in the road segment, is the average vehicle length, v i is the speed of each vehicle, L is the total length of the road section, q t,i represents the queue length of section i, d t,i represents the vehicle density of road section i, v t,i represents the average speed of vehicles traveling in road segment i, and N represents the number of vehicles in the road segment.

[0081] Step 3: Take the preprocessed data in step 2 as input, perform time-series graph convolution processing on it through the graph convolution neural network module, and output the traffic status;

[0082] Graph Convolutional Neural Network module to normalize data vectors As module input. Contains time and space dimension information, so the temporal graph convolutional neural network (T-GCN) is used to process the data output by the preprocessing module Based on the traffic data communication layer, by stacking a layer of temporal convolution and a multi-layer graph convolution neural network, the spatiotemporal correlation characteristics between different decision-making agents are captured to obtain the correlation vector O of the traffic status data. i , the steps are:

[0083] According to the data vector X in step 2 i The definition of decision agent v i The characteristics in the past T period are expressed as The output from the data preprocessing module can be expressed as Right now

[0084] Will Input the temporal convolution layer and use 1D convolution to perform convolution operation on the time series features of decision agent i:

[0085]

[0086] The specific formula of the one-dimensional convolution function Conv1D in the temporal convolution layer is as follows:

[0087]

[0088]

[0089] In the formula, the output is the characteristic of decision agent i in the time dimension, σ i is the ReLU activation function, W time It is the convolution kernel of the temporal convolution layer, with a length of k.

[0090] At the output of the temporal convolutional layer On this basis, the hidden layer h of the graph convolutional network is constructed. i , associate the node traffic status characteristics in time and space:

[0091]

[0092] In the formula, A is the adjacency matrix of the graph, I is a diagonal matrix, D is the degree matrix of the graph, and W is the (k) is the weight matrix of the kth hidden layer, σ i is the ReLU activation function, represents the output of the k+1th layer in the hidden layer of decision agent i, where

[0093] After stacking multiple layers of graph convolution operations, the output decision agent traffic status information correlation vector O i :

[0094]

[0095] In the formula, the graph convolutional network uses the last layer of the hidden layer as the output layer. is the output after K layers of graph convolution operation, W o is the learnable weight matrix corresponding to decision agent i.

[0096] Step 4: Input the traffic state extraction features in step 3 into the reinforcement learning module, generate the optimal control strategy through continuous training feedback, and pass it to the corresponding traffic control unit. The reinforcement learning module adopts the Actor-Critic framework, including a training module and an execution module, where:

[0097] The execution module consists of the traffic control unit and the strategy network corresponding to the decision-making agent. The Actor network t Generate real-time traffic control strategyπ θ (a t |s t ), the decision agent selects action a according to probability samplingt And pass it to the corresponding traffic control unit, the traffic unit will need to perform action a t The data is converted into corresponding traffic instructions and sent to the associated decision-making agent or the vehicles on the controlled road section through the communication network. The feedback data is obtained by the traffic information collection module. Figure 4 As shown, the information correlation vector O finally output by the graph convolutional neural network module i , after linear transformation, the corresponding probability distribution strategy π is generated θ , the decision agent selects action a based on probability sampling t And it is executed by the traffic control unit.

[0098] The training module consists of a reward calculation module, a sample buffer, and a value network. The Critic network predicts the state value based on the interaction data in the sample buffer, and calculates the advantage function based on the accumulated reward value to guide the Actor network update. After the Actor network and Critic network parameters are updated, a new round of iterative training is performed until the strategy converges to the optimal strategy and stops training. The sample buffer stores the state s for a period of time T. t , action a t , reward r t , and the corresponding next state s t+1 After accumulating a certain amount of data, the training module is trained based on the buffer data. The reward value calculation takes into account the queue length Q of the high-speed entrance ramp rm , the total vehicle travel time T tr and vehicle density D ld The weighted average of the three:

[0099] r=T tr +μ 1 Q rm +μ 2 D ld

[0100] Where r is the weighted average of the above three rewards, μ 1 and μ 2 are the ratio coefficients of queue length and vehicle density bonus to travel time bonus respectively. Since the road network area selected in the embodiment includes expressways and urban road traffic systems, the weight μ should be adjusted from the perspective of multi-strategy coordinated control. 1 and μ 2 , increase the weight of the total travel time in the reward function r, and guide the decision-making agent to balance the local and global traffic demands in the global traffic network system, thereby improving the traffic efficiency of the entire road network.

[0101] Traffic control units include signal light control units installed at intersections of urban roads, variable speed limit control units on the main lines of expressways, and signal light control units on entrance ramps. Each control unit corresponds to a decision-making agent, which takes the action a selected by the decision-making agent. t The traffic control unit is used to convert the action instructions into corresponding traffic control instructions, and send the instructions to the associated decision-making intelligent body or the vehicles on the controlled road section through the communication network of the traffic information collection module. The signal light control unit at the intersection is used to convert the action instructions into the timing of each phase of the signal light, the signal light control unit at the entrance ramp is used to convert the vehicle release instructions of the ramp into the red and green signal cycle, and the variable speed limit control unit is used to convert the speed limit instructions into a specific speed limit value.

[0102] The execution and training process of the training module includes the following specific steps:

[0103] Step 4-1: Traffic status information correlation vector O output from the graph convolutional neural network module i Extract the decision agent node features as state s t Enter the reinforcement learning module.

[0104] Step 4-2, at time t, the Actor network has state vector s t Perform linear transformation to generate the corresponding probability distribution strategy π θ :

[0105] π θ (a t |s t )=softmax(W T s t +b)

[0106] In the formula, W T is the transposed weight matrix of the policy network, which determines the state vector s t The influence weight of each feature on action selection, b is the bias vector of linear transformation, so that the policy network can better fit the complex nonlinear relationship, and softmax is the activation function, which is defined as follows:

[0107]

[0108] In the formula, z i is the state vector s t The influence weight value corresponding to the i-th action after linear transformation, The exponential sum of all action-influencing weights is used to normalize and generate the traffic control strategy π θ (a t |s t )’s probability distribution.

[0109] Step 4-3, the decision agent generates a traffic control strategy π based on the Actor network θ (a t |s t ) performs probability sampling and selects action a t And pass it to the corresponding traffic control unit. After the traffic unit interacts with the environment, repeat steps 1-4 to obtain the next state information s t+1 , together with the immediate reward r obtained from the reward calculation module t , forming a traffic status data tuple {(s t ,a t ,r t ,s t+1 )}, temporarily stored in the sampling buffer.

[0110] Step 4-4, after collecting traffic status data for a period of time T, the Critic network evaluates the state s based on the data set in the sampling buffer t The expected return V(s t ):

[0111]

[0112] In the formula, represents the expected value of reward based on strategy π, γ k is the discount factor, which represents the weight of future rewards, r t+k Represents the decision-making agent at time t+k based on state s t Instant rewards received.

[0113] Step 4-5: For each data tuple in the sampling buffer, the Critic network uses the Bellman equation and the value function V(s t ) Estimate the state s t Next, perform specific actions a t The expected return Q(s t ,a t ):

[0114]

[0115] In the formula, r t is the immediate reward in the data tuple, γ is the discount factor, is the next state s in the data tuple t+1 expected value.

[0116] Step 4-6: The training module calculates action a based on the V and Q values ​​obtained in steps 4-4 and 4-5. t Advantage function Used to measure the state t Next, perform a specific action at Advantages compared to other possible actions, which guide the strategy update of the Actor network:

[0117]

[0118] In the formula, Q(s t ,a t ) means in state s t Next, perform action a t The expected future return, while V(s t ) does not consider the specific action, only based on the state s t The expected return that can be obtained.

[0119] After completing data sampling and advantage function calculation in steps 4-7, the training module uses the proximal policy optimization algorithm (PPO) and the Adam optimizer to update the network parameters of the Actor-Critic, specifically:

[0120] Both the policy network and the evaluation network minimize the loss function and update the network parameters θ and θ in combination with the Adam optimizer gradient optimization. V .

[0121] Evaluate the value loss function L of the network VF and parameter θ V The optimization process is as follows:

[0122]

[0123] In the formula, N is the number of training data, R t is the actual cumulative return, V(s t |θ V ) is state s t The estimated value of the value function under .

[0124]

[0125] In the formula, α critic is the adaptive learning rate in the Adam optimizer, is the value loss function L VF About evaluating network parameters θ V gradient.

[0126] The policy loss function L of the policy network PPO The optimization process of parameters θ is as follows:

[0127]

[0128]

[0129] In the formula, r t (θ) is the policy ratio, which indicates the ratio of the new and old policies; ∈ is the clipping coefficient, which is a parameter used to control the policy update amplitude to prevent the policy network from updating too much; clip is the policy clipping function, which is used to limit the policy update to ensure that the policy does not fluctuate drastically during training.

[0130]

[0131] In the formula, α actor is the adaptive learning rate in the Adam optimizer, is the strategy loss function L PPO The gradient with respect to the Actor network parameters θ.

[0132] Step 4-8, the Critic network minimizes the value loss function L VF (θ V ) to update:

[0133]

[0134] In the formula, N is the number of training data, R t is the actual cumulative return, V(s t |θ V ) is state s t The estimated value of the value function under .

[0135] Combined with Adam optimizer to update network parameters θ V :

[0136]

[0137] Among them, α critic is the adaptive learning rate in the Adam optimizer, is the value loss function L VF About Critic Network Parameter θ V gradient.

[0138] Step 4-9, the Actor network minimizes the policy loss function L PPO (θ) is updated:

[0139]

[0140]

[0141] In the formula, r t(θ) is the strategy ratio, which indicates the ratio of the new strategy to the old strategy; ∈ is the clipping coefficient, which is used to control the parameter of the strategy update amplitude to prevent the network from updating too much; clip is the strategy clipping function, which is used to limit the strategy update to ensure that the strategy does not fluctuate drastically during the training process. Combined with the Adam optimizer, update the network parameter θ:

[0142]

[0143] Among them, α actor is the adaptive learning rate in the Adam optimizer, is the strategy loss function L PPO The gradient with respect to the Actor network parameters θ.

[0144] Steps 4-10: In the updating process of the Actor and Critic networks, in order to prevent the strategy from converging to the local optimal solution too early, entropy rewards are introduced in the optimization process of the Actor network to increase the randomness of the strategy. By affecting the long-term cumulative rewards of the decision-making agent, the decision-making agent is encouraged to explore the global optimal solution of the strategy:

[0145]

[0146] In the formula, is the cumulative reward of the decision-making agent at time t, r t is the immediate reward obtained in the reward module, β is the weight coefficient of the entropy reward, is the entropy value of the strategy, which indicates the randomness of the strategy.

[0147] After completing the update of the policy network and the evaluation network, a new round of iterations will be continued until the strategy converges to the optimal strategy and then stops training. The convergence condition is that the loss function value no longer decreases significantly or the average reward reaches the set threshold. When the multi-decision agent environment changes significantly or the average reward is lower than the set minimum value, the sampling buffer is cleared, new traffic status data is obtained through the traffic information collection module, and a new training process is restarted.

[0148] Steps 4-11: The reinforcement learning module uses a combination of online decision making and offline training. In each round of training, the decision agent makes decisions based on the current strategy π θ Real-time decision making and continuous feedback data acquisition through the traffic information collection module. After a certain period of time T of data accumulation, the training module is trained based on the data set in the sampling buffer. After multiple rounds of training, the network parameters θ and θ V It will be gradually optimized until the model converges and reaches the optimal control strategy.

[0149] The reward calculation module in the training module includes:

[0150] Reward calculation for the length of vehicle queues at highway entrance ramps:

[0151]

[0152] In the formula, q max is the maximum queue length of the entrance ramp, q is the actual queue length of the ramp, Δq is the rate of change of the queue length, α is the acceleration reward impact factor, and k base is the basic reward coefficient, Q rm represents the reward value corresponding to the queue length of the entrance ramp, k i is the coefficient of the reward function, where k i Indicates the sensitivity of the reward value to changes in the queue length q.

[0153] The specific advantages of the settings are as follows:

[0154] Dynamic Adjustment: Introduce Δq as the queue length change rate, and adjust the impact of this traffic state parameter on the reward through the weighting factor α, so that the reward can be flexibly adjusted according to the change of queue length, and it is easier to capture the trend of traffic flow changes to respond to different traffic conditions. At the same time, introduce sensitivity k i To express the sensitivity of the reward to changes in q, and to increase the controllability of the reward.

[0155] Growth balance: The designed formula uses the logarithm log(1+q) so that when the queue length q is small, the reward changes faster, and when q grows to a certain extent, the change tends to be balanced. This design can prevent the reward value from growing too fast when the queue length is large, and balance the reward effect under different queue lengths.

[0156] Saturation control: In the formula To control the saturation effect of queue length, when q approaches q max The function tends to be stable when the time is up, indicating that the reward gradually reaches the upper limit. This design can avoid excessive rewards caused by too long queues.

[0157] Bonus calculation for total vehicle travel time:

[0158]

[0159] In the formula, is the actual vehicle travel time, T max k is the longest travel time acceptable to drivers, and its value is determined based on the actual situation of the highway. j is the sensitivity coefficient, T tr Represents the reward value corresponding to the vehicle travel time.

[0160] The specific advantages of the settings are as follows:

[0161] Realistic considerations: Tmax is the longest travel time acceptable to the driver. As the denominator, it can effectively control the downward trend of the reward after the travel time exceeds the threshold. This design reflects the system's hope that the reward can balance the driver's expected time limit.

[0162] Dynamic adjustment: As above, the reward calculation also introduces a sensitivity coefficient, which can be manually adjusted k in different situations j To adapt to different traffic flows and road conditions.

[0163] Non-linear calculation: The max function is used to ensure that the minimum reward value is -1 to prevent excessive negative rewards from affecting system stability.

[0164] Reward calculation for vehicle density at an urban road intersection or highway main line at any time t:

[0165]

[0166] In the formula, c max is the maximum capacity of the road, d t,i is the vehicle density of road section i at time t, and β is the reward discount coefficient.

[0167] The specific advantages of the settings are as follows:

[0168] pass The rate of density change can be captured, which enables the system to identify rising or falling density trends and dynamically adjust rewards. This time-based density change rate calculation is very suitable for short-term traffic regulation, helping to respond to congestion in a timely manner and achieve trend identification;

[0169] pass According to the current density d and the maximum road capacity c max This control mechanism is particularly effective on roads with high traffic volume, helping to avoid road overload.

[0170] The smoothing process of this formula uses the hyperbolic tangent function tanh to process the reward so that the reward value tends to be stable when it approaches the limit, avoiding the unlimited growth or decrease of the reward value.

[0171] Introduce weight coefficients to calculate the comprehensive reward function r:

[0172] r=T tr +μ 1 Q rm +μ 2 D ld

[0173] Where r is the weighted average of the above three rewards, μ1 and μ 2 are the proportional coefficients of queue length and vehicle density bonus compared to travel time bonus.

[0174] Step 5: The traffic control unit converts the output of the control strategy into a traffic instruction and sends it to the associated vehicle or related carrier through communication.

[0175] According to the setting of the reward calculation module, the weight coefficients of different reward values ​​are designed for two different traffic systems, highways and urban roads. The weight coefficients define the proportion of three different rewards in the comprehensive reward function, which is used to achieve multi-strategy collaborative control. Specifically:

[0176] In the highway traffic system, increasing the queue length Q rm The weight μ 1 , prompting the decision-making agent to prioritize reducing the length of ramp queues, quickly clearing stranded vehicles on the ramp, and preventing traffic congestion at the connecting sections of highways and urban roads.

[0177] In the urban road traffic system, increasing vehicle density D ld The weight μ 2 , ensuring that the decision-making agent can prioritize traffic flow in the congested area of ​​the intersection and prevent congestion from spreading to other urban intersections.

[0178] In the global traffic network system, local traffic control cannot reflect the overall system efficiency. 1 and μ 2 , increasing the weight of the total travel time in the comprehensive reward function r, guiding the decision-making agent to balance local and global traffic demands, thereby improving the traffic efficiency of the entire road network.

[0179] like Figure 5 As shown, according to the traffic control unit, the vehicle release instruction of the highway entrance ramp is converted into a signal cycle, and the conversion method includes the following steps:

[0180] Step 5-1-1, obtain the ramp release flow instruction at the current time t from the execution module.

[0181] Step 5-1-2, check whether the duration of the current phase of the ramp control signal light has reached the preset minimum duration. If the current phase duration has exceeded the minimum duration, execute the following update steps 5-1-3 and 5-1-4 to reallocate and start a new red-green signal light cycle. If the minimum duration is not reached, any phase switching action is deemed invalid, and the ramp control signal light needs to continue to maintain the original phase, that is, skip the update steps 5-1-3 and 5-1-4.

[0182] Step 5-1-3: The ramp release flow is updated to release N vehicles per minute.

[0183] Step 5-1-4, calculate the timing of the red and green signals, the specific formula is as follows:

[0184]

[0185] RT=T-GT

[0186] Where GT is the duration of the green light after the update, and RT is the duration of the red light after the update. is the average release time for each vehicle. If the green light duration exceeds the signal cycle T, the red light duration is set to zero, and the green light duration is equal to the signal cycle T.

[0187] Step 5-1-5, after the update is completed, until the next signal cycle starts, the ramp signal control unit determines whether to release the vehicle based on the status of the traffic light.

[0188] Step 5-1-6, repeat the above steps until the control ends.

[0189] like Figure 6 As shown, according to converting the variable speed limit instruction into a specific speed limit value in the traffic control unit, the conversion method includes the following steps:

[0190] Step 5-2-1, obtain the variable speed limit instruction at the current time t from the execution module.

[0191] Step 5-2-2, obtain the vehicle density ρ of the variable speed limit section through the road camera, and the calculation formula is as follows:

[0192]

[0193] Where N is the number of vehicles in the control area section, and L is the road length of the control area section.

[0194] Step 5-2-3, set the speed limit range of the variable speed control area and determine the basic speed limit value, such as the minimum speed limit v min and the maximum speed limit v max .

[0195] Step 5-2-4, check whether the vehicle density ρ in the control area reaches the threshold ζ. If the vehicle density on the road in the current variable speed limit control area does not reach the threshold ζ, keep the original speed limit value and skip the speed limit value update step. If the vehicle density exceeds the threshold ζ, execute the following update steps 5-2-5 and 5-2-6.

[0196] Step 5-2-5, calculate the speed limit value v of the road section according to the vehicle density in the control area limit :

[0197]

[0198] In the formula, v max is the maximum speed limit set in step 5-2-3, ρ max It is the maximum vehicle density that the controlled road section can withstand.

[0199] Step 5-2-6, update the specific speed limit value, limit v limit In the set speed limit range:

[0200] v limit =max(v min ,min(v max ,v limit ))

[0201] Step 5-2-7, update the specific speed limit value v limit Applied to the variable speed limit information board (VSLD), the information board (VSLD) transmits the speed limit information to the vehicles traveling on the controlled section through the electronic display screen.

[0202] According to the traffic control unit, the control instructions of the city intersection signal lights are converted into the timing of each phase, and the conversion method includes the following steps:

[0203] Step 5-3-1, the decision agent selects the optimal control instruction a according to the current traffic status t , and the instruction a t Sent to the signal light control unit at the intersection.

[0204] Step 5-3-2, according to instruction a t Determine the target phase p that the signal light needs to execute t And the corresponding duration l t .

[0205] Step 5-3-3, obtain the current phase p and duration l of the intersection traffic light.

[0206] Step 5-3-4, if the current phase p is equal to the target phase p t Consistent, and the duration l does not reach the duration l required by the target phase t , the current phase is maintained and the remaining duration l is executed t -l; if the duration l has exceeded the required duration l of the target phase t , it automatically switches to the next phase.

[0207] Step 5-3-5, if the current phase p is equal to the target phase p t If they are inconsistent, switch to the target phase p immediatelyt , and start executing the new phase timing until the specified phase duration l is completed t .

[0208] like Figure 1 As shown, the highland collaborative control method, the data collection area includes urban ground roads, highway main lines and highway entrance ramps, and the collection equipment includes but is not limited to road cameras, coil sensors, infrared speed meters and other electronic equipment. For road sections lacking fixed collection equipment, vehicle-road collaborative technology is used to collect data through automatic connected vehicles (CAVs). The collaborative control system includes a traffic information collection module, a data preprocessing module, a graph convolutional neural network module, a reinforcement learning module and several traffic control units. The composition and connection of the modules are shown in the figure. Figure 2 shown.

[0209] The memory bank of the graph convolutional neural network will be marked by data k i Identify the traffic data of any decision agent i in the past T period and define it as the original data vector X i , after the data preprocessing module, the standardized data is obtained In this example, the time length T is defined as 1 hour, and five decision-making agents in the road network are selected as embodiments, including three intersection signal light decision-making agents, one variable speed limit decision-making agent, and one ramp control signal decision-making agent.

[0210] Intersection signal lights core The short-range communication network composed of the center is represented by G = (V 5 ,E 4 ), that is, the DSRC network covers five decision agents and four data transmission channels. The surrounding decision agents send their own traffic status data to the decision agent v core , forming the traffic status information matrix of the regional communication decision-making agent at the current moment Among them, the traffic state characteristics of each decision agent Timestamp i Record with a timer, data label k i Corresponding to the number of the road network decision agent, the state data d i The queue length q including the segment j t,j , vehicle density d t,j and the average speed v t,j , the specific calculation formula is as follows:

[0211]

[0212] Among them, N stop is the number of vehicles that stop and wait in road segment j, is the average vehicle length, v k is the speed of each vehicle, L is the total length of the road segment, and N represents the number of vehicles in the road segment.

[0213] like Figure 3 As shown, the decision agent v core After calculating the degree matrix D and association matrix A of the communication network G, the time dimension features are obtained through temporal convolution Then, by stacking multiple layers of graph convolution, the traffic status information correlation vector O is output i . Among them, the degree matrix D and the association matrix A are:

[0214]

[0215] The traffic status information correlation vector O of the decision-making agent i The calculation formula is as follows:

[0216]

[0217] Where W o is the learnable weight matrix corresponding to decision agent i, is the output of the Kth layer in the hidden layer,

[0218] like Figure 6 As shown, the method for converting the variable speed limit instruction and the specific speed limit value includes the following steps:

[0219] Step a: Obtain the variable speed limit instruction at the current time t from the execution module;

[0220] Step b: Obtain the number of vehicles N and the road length L on the controlled section through the road camera, and calculate the vehicle density of the variable speed limit section. The calculation formula is as follows:

[0221]

[0222] Step c: Set the speed limit range of the variable speed control area and specify the minimum speed limit v min =40km / h, maximum speed limit v max =80km / h.

[0223] Step d: Set the vehicle density threshold ζ = 80 veh / km, and the maximum tolerable vehicle density of the controlled section ρ max =120veh / km, check whether the vehicle density ρ in the control area exceeds the threshold ζ. If the vehicle density ρ does not reach the threshold, skip the following update steps. If the vehicle density ρ exceeds the threshold ζ, perform the following steps to update the speed limit:

[0224] Step d-1: Calculate the speed limit value v of the road section according to the vehicle density in the control area limit:

[0225]

[0226] Step d-2: Update the specific speed limit value, limit v limit In the set speed limit range:

[0227] v limit =max(v min ,min(v max ,v limit ))

[0228] Step e: Update the specific speed limit value v limit Applied to the variable speed limit information board, the information board (VSLD) transmits speed limit information to vehicles traveling on the controlled road section through an electronic display screen.

[0229] The collaborative control method of urban ground roads and highways of the present invention not only considers traffic management from the main highway line and its entrance ramp itself, but also includes the traffic control of urban ground roads in the collaborative control range, which can solve the problem of traffic congestion on highways and increased queue length on entrance ramps affecting the connection with ground roads. The method has the characteristics of real-time, accuracy, robustness, adaptability and scalability, which is conducive to the traffic management department to manage cross-regional traffic in the city and improve the traffic efficiency of the global road network.

[0230] This application proposes a collaborative control method that integrates highways and urban intersections. By intelligently controlling urban road traffic lights, highway ramp signals, and variable speed limit units, it aims to alleviate highway traffic congestion and improve urban traffic efficiency. Compared with a single optimized urban or highway traffic system, the method proposed in this invention can balance the congestion conditions of two different systems, control road vehicles with the help of signal lights and speed limit units, and achieve traffic flow scheduling optimization in the global road network.

[0231] In terms of overall structure, this method adopts a distributed collaborative control framework. The traffic lights, ramp signals and variable speed limit units at urban intersections are all controlled by independent decision-making agents. The decision-making agents make autonomous decisions and adjustments based on local real-time traffic conditions, thereby enhancing the adaptability and robustness of the system and effectively avoiding the defects of centralized control such as large amount of data calculation, difficult processing, and lagging control strategies.

[0232] At the same time, a graph convolutional neural network model is constructed to aggregate the feature information of surrounding decision-making agents and dynamically allocate control weights, which enhances information sharing among multiple decision-making agents and prevents suboptimal decisions made by a single decision-making agent due to lack of comprehensive perception of the surrounding environment, thereby improving the collaborative efficiency of the overall system.

[0233] In addition, this method improves the safety and traffic efficiency of merging vehicles through a variable speed limit system and ramp lights, and combines the city's ground road network to coordinately control the phase delay of intersection lights and ramp controllers to reduce the length of vehicle queues on the entrance ramp and prevent congestion from spreading to ground sections. For sections that are not equipped with road detectors, vehicle-road collaboration technology is used to dynamically provide real-time road traffic status, compensate for data loss, and form a real-time traffic data network.

[0234] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A highland collaborative control method integrating graph convolutional neural network and reinforcement learning, characterized in that: The following steps are involved: Step 1: Collect traffic status data through the traffic information collection module, and build a traffic data communication layer to form a distributed interconnected traffic information network; Step 2, preprocessing the data collected in step 1 through a data preprocessing module; Step 3: Take the preprocessed data in step 2 as input, perform time-series graph convolution processing on it through the graph convolution neural network module, and output the traffic status; Step 4: Input the traffic state extraction features in step 3 into the reinforcement learning module, generate the optimal control strategy through continuous training feedback, and pass it to the corresponding traffic control unit. Step 5: The traffic control unit converts the output of the control strategy into a traffic instruction and sends it to the associated vehicle or related carrier through communication.

2. The highland collaborative control method integrating graph convolutional neural network and reinforcement learning according to claim 1 is characterized in that: In step 1, the traffic information network includes a node set and an edge set in the network, the node set represents the behavior decision-making agent, and the edge set represents the communication relationship between the decision-making agents.

3. The highland collaborative control method integrating graph convolutional neural network and reinforcement learning according to claim 2 is characterized in that: The traffic information network regards each behavior decision agent as a network node, and for each decision agent, i.e., network node v i Establish the feature vector of real-time traffic status v i ∈V, where t i is the timestamp, recording the data collection time; V is the node set, covering all decision-making agents in the road network; d i is the traffic status data, including the queue length, vehicle density and average speed of urban ground roads, expressway main lines and entrance ramp sub-sections; k i For data labeling, identify the decision-making agent that collects data input, that is, the decision-making agent with the shortest straight-line distance to the data source.

4. The highland collaborative control method integrating graph convolutional neural network and reinforcement learning according to claim 1 is characterized in that: By acquiring and recording the data collected in step 1, the decision-making intelligent agent v in the traffic data communication layer is constructed. i The historical traffic data in the past T time interval is defined as the data vector in represents the traffic state characteristics obtained by the decision agent i at time t. In step 2, the specific steps of data preprocessing include: data detection, data smoothing and data standardization.

5. The highland collaborative control method integrating graph convolutional neural network and reinforcement learning according to claim 1 is characterized in that: The time series graph convolution processing in step 3 is specifically as follows: after preprocessing the data, a standardized data vector containing time and space dimension information is obtained. As the module input, based on the traffic data communication layer, by stacking a layer of temporal convolution and a multi-layer graph convolution neural network, the spatiotemporal correlation characteristics between different decision-making agents are captured to obtain the correlation vector O of the traffic status data. i .

6. The highland collaborative control method integrating graph convolutional neural network and reinforcement learning according to claim 1 is characterized in that: In step 4, the reinforcement learning module includes a training module and an execution module, wherein: The execution module includes a traffic control unit and a strategy network corresponding to the decision-making agent in the traffic data communication layer after collecting data; The training module includes a reward calculation module, a sampling buffer, and a value network.

7. The highland collaborative control method integrating graph convolutional neural network and reinforcement learning according to claim 6 is characterized in that: The reward calculation module in the training module includes: Reward calculation for the length of vehicle queues at highway entrance ramps: In the formula, q max is the maximum queue length of the entrance ramp, q is the actual queue length of the ramp, Δq is the rate of change of the queue length, α is the acceleration reward impact factor, k base is the basic reward coefficient, Q rm represents the reward value corresponding to the queue length of the entrance ramp, k i is the coefficient of the reward function, where k i Indicates the sensitivity of the reward value to changes in the queue length q; Bonus calculation for total vehicle travel time: In the formula, is the actual vehicle travel time, T max k is the longest travel time acceptable to drivers, and its value is determined based on the actual situation of the highway. j is the sensitivity coefficient, T tr Indicates the reward value corresponding to the vehicle travel time; Reward calculation for vehicle density at an urban road intersection or highway main line at any time t: In the formula, c max is the maximum capacity of the road, d t,i is the vehicle density of road section i at time t, β is the reward discount coefficient; Introduce weight coefficients to calculate the comprehensive reward function r: r=T tr +μ1Q rm +μ2D ld Where r is the weighted average of the above three rewards, μ1 and μ2 are the proportional coefficients of queue length and vehicle density rewards to travel time rewards, respectively.

8. The highland collaborative control method integrating graph convolutional neural network and reinforcement learning according to claim 7 is characterized in that: In step 5, for two different traffic systems, namely, expressways and urban roads, the reward calculation module differentially designs weight coefficients of different reward values ​​to achieve multi-strategy coordinated control, specifically: In the highway traffic system, increasing the queue length Q rm The weight μ1 prompts the decision-making agent to prioritize reducing the ramp queue length and quickly clear the stranded vehicles on the ramp; In the urban road traffic system, increasing vehicle density D ld The weight μ2 ensures that the decision-making agent can prioritize the traffic flow in the congested area of ​​the intersection; In the global transportation network system, by adjusting the weights μ1 and μ2, the weight of the total travel time in the comprehensive reward function r is increased, guiding the decision-making agent to balance local and global transportation demands.

9. The highland collaborative control method integrating graph convolutional neural network and reinforcement learning according to claim 6 is characterized in that: The traffic control unit includes a signal light control unit installed at the intersection of the ground urban road, a variable speed limit control unit on the main line of the expressway, and a signal light control unit on the entrance ramp. Each control unit corresponds to a decision-making agent. t The traffic control module converts the action instructions into corresponding traffic control instructions, and sends the instructions to the associated decision-making intelligent agent or the vehicles on the controlled road section through the communication network of the traffic information collection module. The traffic light control unit at the intersection is used to convert the action instructions into the timing of each phase of the traffic light. The traffic light control unit at the entrance ramp is used to convert the vehicle release instructions of the ramp into the red-green signal cycle. The variable speed limit control unit is used to convert the speed limit instructions into a specific speed limit value.

10. The highland collaborative control method integrating graph convolutional neural network and reinforcement learning according to claim 9 is characterized in that: In step 5, the method for converting the variable speed limit instruction into a specific speed limit value by the traffic control unit includes the following steps: Step 5-2-1, obtaining a variable speed limit instruction at the current time t from the execution module; Step 5-2-2, obtain the vehicle density ρ of the variable speed limit section through the road camera, and the calculation formula is as follows: Where N is the number of vehicles in the control area section, and L is the road length of the control area section; Step 5-2-3, set the speed limit range of the variable speed control area and determine the basic speed limit value, such as the minimum speed limit v min and the maximum speed limit v max ; Step 5-2-4, check whether the vehicle density ρ in the control area reaches the threshold value ζ. If the vehicle density on the road in the current variable speed limit control area does not reach the threshold value ζ, keep the original speed limit value and skip the speed limit value update step; if the vehicle density exceeds the threshold value ζ, execute the following update steps 5-2-5 and 5-2-6; Step 5-2-5, calculate the speed limit value v of the road section according to the vehicle density in the control area limit : In the formula, v max is the maximum speed limit set in step 5-2-3, ρ max is the maximum vehicle density that the controlled road section can bear; Step 5-2-6, update the specific speed limit value, limit v limit In the set speed limit range: v limit =max(v min ,min(v max ,v limit )) Step 5-2-7, update the specific speed limit value v limit Applied to the variable speed limit information board VSLD, the information board VSLD transmits the speed limit information to the vehicles traveling on the controlled section through the electronic display screen.

Images