A method and device for multi-section traffic control on highways based on reinforcement learning

Abstract

This invention discloses a multi-section traffic flow control method for highways based on reinforcement learning, belonging to the field of highway traffic flow control technology. The method includes: constructing local state vectors for each section based on traffic state data; constructing a section correlation matrix based on the highway network topology, whereby the section correlation matrix is ​​used to quantify the traffic correlation strength between sections; fusing the local state vectors of each section based on the section correlation matrix using matrix multiplication to obtain a global state vector; inputting the global state vector into a trained reinforcement learning model to output an optimal combination of control actions, which is used to form control commands to control the highway network. This invention can adapt to dynamic fluctuations in traffic flow, has a global perspective, and can adaptively optimize based on real-time traffic conditions.

Description

Technical Field

[0001] This invention relates to the field of highway traffic control technology, and in particular to a method and apparatus for multi-section traffic control on highways based on reinforcement learning. Background Technology

[0002] The core task of highway traffic control is to optimize road network efficiency and alleviate congestion by adjusting traffic signals and lane usage.

[0003] Currently, highway traffic flow control mostly employs fixed control strategies or local optimization methods based on single sections, relying on preset rules or simple traffic statistics models. Existing technologies have significant limitations: first, they struggle to adapt to dynamic fluctuations in traffic flow, with fixed strategies lagging in scenarios such as peak hours and sudden accidents; second, they lack a global perspective, as control measures in a single section may trigger cascading congestion in adjacent sections; and third, control parameters largely depend on manual experience, resulting in strong subjectivity and an inability to adaptively optimize based on real-time traffic conditions.

[0004] Therefore, a method and device for multi-section traffic control on highways based on reinforcement learning was developed to solve the above problems. Summary of the Invention

[0005] This invention proposes a method and device for multi-section traffic control on highways based on reinforcement learning, in order to solve the problems that existing highway traffic control models are unable to adapt to dynamic fluctuations in traffic flow, lack a global perspective, and cannot adaptively optimize according to real-time traffic conditions.

[0006] The present invention achieves the above objectives through the following technical solutions:

[0007] This invention provides a reinforcement learning-based method for multi-section traffic control on highways, comprising:

[0008] Acquire traffic status data and highway network topology for each section of the highway, with each section controlled independently;

[0009] Construct local state vectors for each section based on traffic status data;

[0010] An interval correlation matrix is ​​constructed based on the highway network topology. The interval correlation matrix is ​​used to quantify the traffic correlation strength between each interval.

[0011] The global state vector is obtained by fusing the local state vectors of each interval based on the interval correlation matrix using matrix multiplication.

[0012] The global state vector is input into the trained reinforcement learning model, which outputs the optimal combination of control actions. The optimal combination of control actions is used to form control commands, which are used to control the highway network.

[0013] Furthermore, an interval correlation matrix M is constructed based on the highway network topology. The matrix elements in the correlation matrix M are defined as follows:

[0014] ;

[0015] in, For the correlation distance threshold, For the association weight, satisfying Where i and j represent the indices of the intervals, i being the row index of the matrix, indicating the i-th interval. j represents the column index, and j represents the j-th interval. .

[0016] Furthermore, the association weights are set according to the following rules:

[0017] like and If they are directly adjacent intervals, then ;

[0018] like and If there is a gap of 1 interval and the travel time is ≤10 minutes, then ;

[0019] like and If there are two or more intervals, but they belong to the same main traffic artery, then ;

[0020] Correlation Matrix Symmetry must be satisfied And diagonal elements .

[0021] Furthermore, based on the interval correlation matrix, the local state vectors of each interval are fused using matrix multiplication, including:

[0022] Stack the local state vectors of all intervals column-wise to form Local state matrix of order :

[0023] ;

[0024] Each column corresponds to a local state vector for an interval;

[0025] By fusing local state and interval correlation features through matrix multiplication, a global state vector is generated. :

[0026] ;

[0027] The dimension of this vector is Its elements Where q is the state index, and its value range is... m represents the number of traffic status indicators for a single interval; k is the index of the target interval, with a value range of [value missing]. ; i is the index of the traversed interval, and its value range is It retains the first The local characteristics of each indicator in each interval are further analyzed through the correlation matrix. Incorporating the interval By considering the correlation with other intervals, a unified representation of the state of multiple intervals is achieved.

[0028] Furthermore, the reinforcement learning model is a deep Q-network, specifically including:

[0029] The input layer has the same number of neurons as the global state vector, and the activation function is ReLU.

[0030] The hidden layers consist of two layers. The number of neurons in the first layer is twice the dimension of the global state vector, while the number of neurons in the second layer is the same as the dimension of the global state vector. The activation function for both layers is ReLU.

[0031] The output layer has the number of neurons equal to the size of the action set, and outputs the Q-value corresponding to each action.

[0032] Furthermore, the reward function of the reinforcement learning model consists of single-interval traffic efficiency reward, multi-interval collaborative reward, control cost, and congestion penalty.

[0033] Furthermore, the reward function of the reinforcement learning model for:

[0034] ;

[0035] in, This is a reward for single-section traffic efficiency. For multi-interval collaborative rewards, To control costs, Penalties for congestion , , , These are the weights for single-section traffic efficiency rewards, multi-section collaborative rewards, control costs, and congestion penalties, respectively. t represents time.

[0036] ;

[0037] For the first Execute actions in each interval The average speed before The average vehicle speed after execution. The maximum speed is designed for this section. where n is the total number of intervals;

[0038] ;

[0039] in, For the first Traffic density after actions are performed in each section The average traffic density across all sections. The maximum permissible average density of the road network. A larger value indicates a more balanced density across the intervals.

[0040] ;

[0041] in, For the first The current value of each control action. The adjusted value. This is used to correct overly frequent movements.

[0042] ;

[0043] in, This is the congestion density threshold.

[0044] Furthermore, the training process of a reinforcement learning model includes:

[0045] Initialize the model's core parameters;

[0046] Based on historical traffic data, different traffic scenarios are simulated to obtain a sample set, and the following operations are performed:

[0047] Randomly select the initial state ,according to Greedy strategy for selecting actions ;

[0048] Calculate the reward after performing the action and the next state ;

[0049] Sample Store in the experience replay buffer ;

[0050] Repeat the above process until the number of samples stored in the buffer reaches the capacity threshold.

[0051] Priority experience replay sampling includes:

[0052] Sampling is performed according to sample priority. The definition is as follows:

[0053]

[0054] in, , For timing difference error, , This represents the action at time t+1. This represents the state at time t+1. Indicates the state at time t. Indicates the action at time t. Indicates the target Q value. This indicates the evaluation of the Q value. This represents the reward value at time t. Indicates the discount factor. The constant is a tiny constant used in priority experience playback, and its purpose is to prevent the timing difference error from being equal to 0.

[0055] Sampling probability , These are priority weight coefficients; sample weights are updated after sampling. , The correction coefficient gradually increases to 1 with each training iteration, and h represents the sample traversal index, with a value range of [value missing]. , This represents the total number of samples.

[0056] Network parameter updates include:

[0057] Calculate the target Q value: ;

[0058] Calculate the evaluation Q value: ;

[0059] Define the loss function: , Batch size;

[0060] The Adam optimizer is used to minimize the loss function, and the evaluation network is then updated. Parameters;

[0061] Each iteration Next, synchronize the target network Parameters;

[0062] Training termination conditions include:

[0063] When the loss function values ​​of 10 consecutive batches Furthermore, the average reward value on the validation set increases by less than [a certain percentage] over five consecutive iterations. When the time is right, stop offline training and save the network parameters as a pre-trained model.

[0064] Furthermore, the global state vector is input into the trained reinforcement learning model, which outputs the optimal combination of control actions. This optimal combination of control actions is used to form control commands, including:

[0065] according to - Greedy strategy for selecting actions:

[0066]

[0067] Optimal action combination output The corresponding control strategies for each interval (such as...) The variable speed limit value for interval 1. (e.g., the flow adjustment coefficient for the entrance ramp of section 2). For exploration rate;

[0068] The control execution module converts the optimal action combination into specific control commands and sends them to the roadside control equipment to execute the control actions;

[0069] Traffic status after data collection and control Calculate real-time rewards ;

[0070] Sample The data is stored in an online experience buffer and fine-tuned online every 30 minutes to update and evaluate the network parameters.

[0071] The present invention also provides an apparatus for the aforementioned reinforcement learning-based multi-section traffic control method for highways, comprising:

[0072] The traffic status acquisition module is used to acquire traffic status data and highway network topology for each section of the highway, and each section is controlled independently.

[0073] The multi-interval state modeling module is used to construct local state vectors for each interval based on traffic state data, construct interval correlation matrices based on the highway network topology, quantify the traffic correlation strength between intervals, and fuse the local state vectors of each interval based on the interval correlation matrix using matrix multiplication to obtain a global state vector.

[0074] The reinforcement learning decision module is used to input the global state vector into the trained reinforcement learning model and output the optimal combination of control actions. The optimal combination of control actions is used to form control commands, which are used to control the highway network.

[0075] Furthermore, it also includes a control execution module for executing control commands.

[0076] The beneficial effects of this invention are as follows:

[0077] The present invention proposes a method and device for multi-section traffic control of highways based on reinforcement learning. It constructs a state space that adapts to the road network topology, designs a reward function that takes into account both global and local factors, solves the problem of multi-section decision coupling, can adapt to dynamic fluctuations in traffic flow, has a global perspective, and can adaptively optimize according to real-time traffic conditions. Attached Figure Description

[0078] Figure 1 This is a flowchart of a multi-section traffic control method for highways based on reinforcement learning, as proposed in this application. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0080] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0081] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0082] The applicant found that although reinforcement learning has been applied in single-scenario optimization, in multi-section collaborative control of highways, how to construct a state space that adapts to the road network topology, design a reward function that takes into account both global and local factors, and solve the problem of multi-section decision coupling are still technical challenges that have not been overcome.

[0083] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0084] like Figure 1As shown, the device of this invention consists of a traffic state acquisition module, a multi-section state modeling module, a reinforcement learning decision-making module, and a control execution module. The traffic state acquisition module collects data such as traffic flow, average vehicle speed, traffic density, and queue length in each section in real time through roadside sensors, vehicle terminals, and other devices, forming a standardized state dataset. The multi-section state modeling module is deployed in the cloud and, based on the road network topology, fuses and encodes the states of each section to construct a unified state space that includes the states within each section and the characteristics of the relationships between sections. The reinforcement learning decision-making module is also deployed in the cloud and adopts a deep Q-network (DQN) architecture. Through offline training and online fine-tuning, the agent learns multi-section collaborative control strategies and outputs the optimal combination of control actions for each section. The control execution module converts the control actions output by the decision module into lane control commands, speed limit sign adjustment signals, ramp barrier control signals, etc., and applies them to the actual road network.

[0085] I. Multi-interval state modeling method.

[0086] The core of multi-interval state modeling is to transform scattered local states within intervals into a unified global state representation through standardization, topological correlation modeling, and matrix fusion operations. This provides high-dimensional and comprehensive input data for reinforcement learning decision-making. The specific steps are as follows:

[0087] Step 1: Standardize the local state vector of the interval.

[0088] For each control zone ( The original data corresponding to the status index set is extracted from the traffic status acquisition module and denoted as follows: , , indicating the first The interval is the first The original value of each indicator.

[0089] To eliminate the impact of differences in the units of measurement of different indicators on the modeling accuracy, the Min-Max standardization method is used to normalize the original data, as shown in the following formula:

[0090] ;

[0091] in, , For all intervals, the first The original data set of the indicators The minimum value of this set. The maximum value of this set is the normalized data. .

[0092] Based on the standardized index values, construct the first... Local state vectors of each interval :

[0093] ;

[0094] The dimension of this vector is It fully characterizes the traffic flow features of a single section.

[0095] Step 2: Constructing the interval correlation matrix.

[0096] Based on the highway network topology (such as interval connectivity, travel direction, distance threshold, etc.), construct Interval Inference Matrix This is used to quantify the connectivity strength between intervals, and the matrix elements are defined as follows:

[0097] ;

[0098] in, The associated distance threshold is set according to the actual road network size, with a recommended value of 5-10 kilometers. For the association weight, satisfying The association weights are set according to the following rules:

[0099] (1) If and If the intervals are directly adjacent (without other intervals between them), then ;

[0100] (2) If and If there is a gap of 1 interval and the travel time is ≤10 minutes, then ;

[0101] (3) If and If there are two or more intervals, but they belong to the same main traffic artery, then .

[0102] Correlation Matrix Symmetry must be satisfied And diagonal elements .

[0103] Step 3: Global state vector fusion and generation.

[0104] Stack the local state vectors of all intervals column-wise to form Local state matrix of order ,

[0105] ;

[0106] Each column corresponds to a local state vector for an interval.

[0107] By fusing local state and interval correlation features through matrix multiplication, a global state vector is generated. ,

[0108] ;

[0109] The dimension of this vector is Its elements Where q is the state index, and its value range is... m represents the number of traffic status indicators for a single interval; k is the index of the target interval, with a value range of [value missing]. ; i is the index of the traversed interval, and its value range is It retains the first The local characteristics of each indicator in each interval are further analyzed through the correlation matrix. Incorporating the interval By considering the correlation with other intervals, a unified representation of the state of multiple intervals is achieved.

[0110] II. Strengthen learning decision-making and control methods.

[0111] This method is based on a deep Q-network (DQN) architecture and uses a two-stage mode of offline training and online fine-tuning to enable the agent to learn multi-interval cooperative control strategies. It also introduces priority experience replay and adaptive learning rate mechanisms to improve the efficiency and stability of policy optimization. The specific steps are as follows:

[0112] Step 1: Initialize the reinforcement learning model.

[0113] (1) Network architecture definition.

[0114] A 3-layer fully connected neural network is used as the evaluation network for DQN. With the target network The network structure is as follows:

[0115] Input layer: The number of neurons is equal to the dimension of the global state vector. The activation function is ReLU;

[0116] Hidden layers: 2 layers, the first layer has 2×(m×n) neurons and the second layer has m×n neurons, both with ReLU activation function;

[0117] Output layer: The number of neurons is equal to the size of the control action set. Without an activation function, output the Q-values ​​corresponding to each action.

[0118] (2) Initialize core parameters.

[0119] Learning rate The initial value is set to 0.005, and an adaptive adjustment strategy is adopted. The formula is as follows: , To determine the number of training iterations, , which is the decay coefficient of the learning rate, to ensure that the learning rate decreases as the training progresses;

[0120] Discount factor Set to 0.9 to balance immediate and long-term rewards;

[0121] Experience replay buffer The capacity is set to 10,000 to store samples of state, action, reward, and next state. ;

[0122] Target network update cycle The parameters of the target network are updated every 100 iterations, which means that all the parameters of the evaluation network are completely copied to the target network.

[0123] Exploration rate The initial value is set to 0.9, using... - Greedy strategy, formula is: .

[0124] Step 2: Design the reward function.

[0125] reward function Taking into account single-section traffic efficiency, multi-section coordination, congestion mitigation effect, and control cost, a weighted summation method is adopted, as shown in the following formula:

[0126] ;

[0127] The definitions of each item are as follows:

[0128] (1) Single-section traffic efficiency bonus .

[0129] ;

[0130] in, For the first Execute actions in each interval The average speed before The average vehicle speed after execution. Design the maximum speed for this section (e.g., 120 km / h). .

[0131] (2) Multi-interval collaborative reward .

[0132] ;

[0133] in, For the first Traffic density after actions are performed in each section The average traffic density across all sections. The maximum permissible average density of the road network. A larger value indicates a more balanced density across the intervals.

[0134] (3) Cost control .

[0135] ;

[0136] in, For the first The current value of each control action. The adjusted value. This is used to correct overly frequent movements.

[0137] (4) Congestion penalties .

[0138] ;

[0139] in, The congestion density threshold (set according to road section design standards, recommended at 20 pcu / km, where pcu represents the equivalent of a standard passenger car) and the weighting coefficients satisfy the following: .

[0140] Step 3: Offline training process.

[0141] (1) Sample collection.

[0142] Based on historical traffic data (peak hours, off-peak hours, and incident periods over the past year), simulate different traffic scenarios and perform the following operations:

[0143] Randomly select the initial state ,according to Greedy strategy for selecting actions ;

[0144] Calculate the reward after performing the action and the next state ;

[0145] Sample Store in the experience replay buffer ;

[0146] Repeat the above process until the number of samples stored in the buffer reaches the capacity threshold.

[0147] (2) Priority experience replay sampling.

[0148] To improve training efficiency, samples are sampled according to priority. The definition is as follows:

[0149] ;

[0150] in, For timing difference error, , and All of these are corresponding actions;

[0151] Sampling probability , These are priority weight coefficients; sample weights are updated after sampling. , The value was gradually increased to 1.0 with each training iteration.

[0152] (3) Network parameter update.

[0153] Calculate the target Q value: ;

[0154] Calculate the evaluation Q value: ;

[0155] Define the loss function: , , where is the batch size;

[0156] The Adam optimizer is used to minimize the loss function, and the evaluation network is then updated. Parameters;

[0157] Each iteration Next, synchronize the target network The parameters.

[0158] (4) Conditions for termination of training.

[0159] When the loss function values ​​of 10 consecutive batches Furthermore, the average reward value on the validation set increases by less than [a certain percentage] over five consecutive iterations. When the time is right, stop offline training and save the network parameters as a pre-trained model.

[0160] Step 4: Online decision-making and closed-loop control.

[0161] (1) Real-time status input.

[0162] The traffic status acquisition module collects road network data every 5 seconds, and the multi-interval status modeling module generates a real-time global status vector. Input into the pre-trained DQN evaluation network .

[0163] (2) Optimal action output.

[0164] according to - Greedy strategy (online phase) (Fixed to 0.1) Select action:

[0165] ;

[0166] Optimal action combination output The corresponding control strategies for each interval, such as The variable speed limit value for interval 1. For example, the flow adjustment coefficient of the entrance ramp in section 2.

[0167] (3) Control action execution and feedback fine-tuning.

[0168] The control execution module converts the optimal action combination into specific control commands and sends them to roadside control equipment (such as variable speed limit signs, ramp barrier controllers, etc.) to execute the control actions.

[0169] Traffic status after data collection and control Calculate real-time rewards ;

[0170] Sample Stored in an online experience buffer, fine-tuned online every 30 minutes (50 iterations, learning rate...). Update and evaluate network parameters to ensure the strategy adapts to real-time traffic changes.

[0171] The specific implementation process of this invention is as follows:

[0172] First, complete the device initialization and start the traffic status acquisition module. Through roadside radar, video surveillance, ETC system and other equipment, collect data such as traffic flow, average vehicle speed, traffic density and queue length in each section in real time to form a standardized status dataset.

[0173] Second, perform multi-interval state modeling, extract local state vectors from each interval, construct interval correlation matrices, and fuse them to obtain the global state vector.

[0174] Third, the reinforcement learning decision-making module is activated, the pre-trained deep Q-network model is loaded, and the optimal combination of control actions is output based on the real-time global state vector.

[0175] Finally, the control execution module converts the control actions into lane adjustment commands, speed limit signals, ramp flow control commands, etc., and sends them to the traffic control equipment. At the same time, the real-time traffic status after control is fed back to the decision module to achieve dynamic collaborative control.

[0176] The advantages of this invention compared to the prior art are as follows:

[0177] (1) The present invention adopts a reinforcement learning adaptive decision-making mechanism, which can respond to the dynamic changes in traffic flow in real time without the need for manual intervention to adjust control parameters, thereby improving the flexibility and timeliness of traffic control.

[0178] (2) The present invention constructs a multi-interval global state model and avoids chain congestion caused by local optimization through a collaborative control strategy, thereby achieving an overall improvement in road network traffic efficiency.

[0179] (3) The data-driven training method of this invention eliminates the subjective bias of human experience, and the reward function takes into account both local and global interests, making the control decision more scientific and reasonable.

[0180] (4) The control process of this invention is simple and efficient, requires no complex mathematical calculations, can be quickly deployed based on existing traffic sensing equipment, adapts to highway networks of different scales, and has a wide range of application scenarios.

[0181] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for multi-section traffic control on highways based on reinforcement learning, characterized in that, include: Acquire traffic status data and highway network topology for each section of the highway, with each section controlled independently; Construct local state vectors for each section based on traffic status data; An interval correlation matrix is ​​constructed based on the highway network topology. The interval correlation matrix is ​​used to quantify the traffic correlation strength between each interval. The global state vector is obtained by fusing the local state vectors of each interval based on the interval correlation matrix using matrix multiplication. The global state vector is input into the trained reinforcement learning model, which outputs the optimal combination of control actions. The optimal combination of control actions is used to form control commands, which are used to control the highway network. Construct an interval correlation matrix M based on the highway network topology; Based on the interval correlation matrix, the local state vectors of each interval are fused using matrix multiplication, including: Stack the local state vectors of all intervals column-wise to form Local state matrix of order : ； Each column corresponds to a local state vector for an interval; By fusing local state and interval correlation features through matrix multiplication, a global state vector is generated. : ； The dimension of this vector is Its elements Where q is the state index, and its value range is... m represents the number of traffic status indicators for a single interval; k is the index of the target interval, with a value range of [value missing]. ; i is the index of the traversed interval, and its value range is ; Reward function of reinforcement learning model for: ； in, This is a reward for single-section traffic efficiency. For multi-interval collaborative rewards, To control costs, Penalties for congestion , , , These are the weights for single-section traffic efficiency rewards, multi-section collaborative rewards, control costs, and congestion penalties, respectively. t represents time. ； For the first Execute actions in each interval The average speed before The average vehicle speed after execution. The maximum speed is designed for this section. where n is the total number of intervals; ； in, For the first Traffic density after actions are performed in each section The average traffic density across all sections. The maximum permissible average density of the road network. A larger value indicates a more balanced density across the intervals. ； in, For the first The current value of each control action. The adjusted value. This is used to correct overly frequent movements. ； in, This is the congestion density threshold.

2. The method for multi-section traffic control of highways based on reinforcement learning according to claim 1, characterized in that, The matrix elements in the interval correlation matrix M are defined as follows: ； in, For the correlation distance threshold, For the association weight, satisfying , where i and j represent the indices of the interval.

3. The method for multi-section traffic control of highways based on reinforcement learning according to claim 2, characterized in that, Association weights are set according to the following rules: like and If they are directly adjacent intervals, then ; like and If there is a gap of 1 interval and the travel time is ≤10 minutes, then ; like and If there are two or more intervals, but they belong to the same main traffic artery, then ; Correlation Matrix Symmetry must be satisfied And diagonal elements .

4. The method for multi-section traffic control of highways based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning model is a deep Q-network, specifically including: The input layer has the same number of neurons as the global state vector, and the activation function is ReLU. The hidden layers consist of two layers. The number of neurons in the first layer is twice the dimension of the global state vector, while the number of neurons in the second layer is the same as the dimension of the global state vector. The activation function for both layers is ReLU. The output layer has the number of neurons equal to the size of the action set, and outputs the Q-value corresponding to each action.

5. The method for multi-section traffic control of highways based on reinforcement learning according to claim 1, characterized in that, The training process of a reinforcement learning model includes: Initialize the model's core parameters; Based on historical traffic data, different traffic scenarios are simulated to obtain a sample set, and the following operations are performed: Randomly select the initial state ,according to Greedy strategy for selecting actions ; Calculate the reward after performing the action and the next state ; Sample Store in the experience replay buffer ; Repeat the above process until the number of samples stored in the buffer reaches the capacity threshold. Priority experience replay sampling includes: Sampling is performed according to sample priority. The definition is as follows: in, For timing difference error, , This represents the action at time t+1. This represents the state at time t+1. Indicates the state at time t. Indicates the action at time t. Indicates the target Q value. This indicates the evaluation of the Q value. This represents the reward value at time t. Indicates the discount factor. It is a constant, a tiny constant in priority experience playback, and its function is to avoid the timing difference error from being equal to 0; Sampling probability , These are priority weight coefficients; sample weights are updated after sampling. , As training iterations increase, h gradually increases to 1.0, where h represents the sample traversal index, and its value ranges from 1 to 1.

0. , This represents the total number of samples. Network parameter updates include: Calculate the target Q value: ; Calculate the evaluation Q value: ; Define the loss function: , Batch size; The Adam optimizer is used to minimize the loss function, and the evaluation network is then updated. Parameters; Each iteration Next, synchronize the target network Parameters; Training termination conditions include: When the loss function values ​​of 10 consecutive batches Furthermore, the average reward value on the validation set increases by less than [a certain percentage] over five consecutive iterations. When the time is right, stop offline training and save the network parameters as a pre-trained model.

6. The method for multi-section traffic control of highways based on reinforcement learning according to claim 5, characterized in that, The global state vector is input into the trained reinforcement learning model, which outputs the optimal combination of control actions. This optimal combination of control actions is used to form control commands, including: according to - Greedy strategy for selecting actions: Optimal action combination output The corresponding control strategies for each interval For exploration rate; The control execution module converts the optimal action combination into specific control commands and sends them to the roadside control equipment to execute the control actions; Traffic status after data collection and control Calculate real-time rewards ; Sample The data is stored in an online experience buffer and fine-tuned online every 30 minutes to update and evaluate the network parameters.

7. An apparatus for a reinforcement learning-based multi-section traffic control method for highways as described in any one of claims 1-6, characterized in that, include: The traffic status acquisition module is used to acquire traffic status data and highway network topology for each section of the highway, and each section is controlled independently. The multi-interval state modeling module is used to construct local state vectors for each interval based on traffic state data, construct interval correlation matrices based on the highway network topology, quantify the traffic correlation strength between intervals, and fuse the local state vectors of each interval based on the interval correlation matrix using matrix multiplication to obtain a global state vector. The reinforcement learning decision module is used to input the global state vector into the trained reinforcement learning model and output the optimal combination of control actions. The optimal combination of control actions is used to form control commands, which are used to control the highway network.

Images