Highway construction area step-by-step variable speed limit control method based on dual depth Q network

Abstract

The invention discloses an expressway construction area step-by-step variable speed limit control method based on a dual depth Q network, and the method comprises the steps: constructing an environment model interacting with a reinforcement learning agent, and the environment model comprises a state space, an action space and a reward function; constructing a reinforcement learning model based on a dual depth Q network, and generating a training data set through iterative interaction between the intelligent agent and the environment model; and training the dual depth Q network by using the training data set and taking maximization of accumulated rewards as a target, and realizing step-by-step variable speed limit control of the highway construction area based on a trained model. According to the method, data driving is taken as a main part, the overall income brought by step-by-step variable speed limit control is represented by comprehensively using driving safety, high efficiency and system stability indexes, a more intelligent and dynamic speed limit control strategy is constructed, and the traffic efficiency and the traffic safety level are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation information processing technology, specifically to a method for stepwise variable speed limit control in highway construction zones based on dual deep Q-networks. Background Technology

[0002] In recent years, with the continuous increase in motor vehicle ownership and travel demand, my country's highway construction has entered a phase of reconstruction and expansion. Currently, reconstruction and expansion projects generally adopt a "construction while maintaining traffic flow" model, leading to lane closures and traffic diversions becoming commonplace in construction areas. These projects are characterized by large spatial spans, long durations, and significant reductions in traffic capacity, resulting in frequent "speed precipitates" when vehicles enter the work area, seriously threatening driving safety. International road traffic safety research shows that the incidence of rear-end collisions and intrusions in reconstruction and expansion work areas can be 2-3 times higher than on regular road sections.

[0003] Currently, traffic control in construction zones of multi-lane highways in my country mainly relies on static, tiered speed limits or repeated speed limit reminders to reduce vehicle speeds within the controlled sections. These speed limits are largely determined based on regulations and historical experience; once implemented, they cannot be arbitrarily adjusted and are difficult to dynamically adapt to real-time factors such as traffic flow and construction progress. This results in poor adaptability and significant limitations in control effectiveness. Especially in conditions of high traffic density and dispersed speed distribution, static speed limits often fail to effectively smooth traffic flow and alleviate conflicts, hindering improvements in traffic efficiency and safety levels within construction zones.

[0004] Therefore, how to construct a more intelligent and dynamic speed limit control strategy while ensuring driving safety is a problem that urgently needs to be solved by those skilled in the art, so as to improve traffic conditions and increase traffic efficiency in highway construction areas. Summary of the Invention

[0005] In view of the above problems, this invention proposes a stepwise variable speed limit control method for highway construction zones based on dual deep Q-networks. This method treats the speed limit control in construction zones as a Markov decision process, and reinforcement learning provides an effective paradigm for solving this type of problem. By comprehensively considering traffic efficiency, safety, and system stability, a multi-objective reward function is constructed to optimize the speed limit strategy by maximizing the cumulative reward.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: This invention provides a method for stepwise variable speed limit control in highway construction zones based on dual depth Q-networks, comprising the following steps: S101: Construct an environment model for interaction with the reinforcement learning agent, the environment model including a state space, an action space, and a reward function; S102: Construct a reinforcement learning model based on a dual deep Q-network, and generate a training dataset through iterative interaction between the agent and the environment model; S103: Using the training dataset, with the goal of maximizing the cumulative reward, train the dual deep Q network, and implement stepwise variable speed limit control in the highway construction area based on the trained model.

[0007] In one embodiment, in step S101: The state space includes the traffic volume, occupancy rate, and average speed of the road segment in the construction area within a time period t. The action space includes discrete road segment speed limits, which are taken based on preset intervals; The reward function is a multi-objective function, which includes a weighted sum of safety, efficiency, and stability indicators.

[0008] In one embodiment, the reward function comprehensively considers driving safety indicators, efficiency indicators, and system stability indicators; therefore, the reward function calculation formula is as follows:

[0009] in, , and Custom weights are assigned to the three metrics, with values ​​ranging from [value range missing]. ; As a safety indicator, it is characterized by the standard deviation of vehicle speeds within the speed-limited zone. A lower standard deviation indicates that the vehicle speeds are more uniform within that section of road. For the first The vehicle's speed during the current speed limit period. The average speed of all vehicles during the current speed limit period, where N is the total number of vehicles; As an efficiency indicator, it is calculated by comparing the average vehicle speed with the road's maximum speed limit threshold. The ratio is used to represent; As a system stability indicator, it is quantitatively characterized based on vehicle queue length; L This is the actual queue length. The preset queue length value. This is the penalty coefficient.

[0010] In one embodiment, the specific process of generating the training dataset in step S102 includes: The initial data sequence is obtained through iterative mapping between the state space and the action space; The initial data sequence is reconstructed into a structure of state-action-next state tuples, which is used to calculate the predicted Q value and the target Q value.

[0011] In one embodiment, step S103, training the dual deep Q-network, includes: A loss function is constructed based on the predicted Q-value, the target Q-value, and the immediate reward. The loss function is optimized using gradient descent, and the network parameters are updated until the model converges.

[0012] In one embodiment, the loss function formula is as follows:

[0013] Where r is the reward function; This is a discount factor used to measure the impact on rewards earned in future time periods; These are the parameters of the Q-network; These are the parameters of the target Q-network.

[0014] In one embodiment, the dual deep Q-network structurally introduces two parallel branches: The state value branch is used to estimate the importance of the current state. The action advantage branch is used to estimate the relative advantage of each action in this state. The final Q value is obtained by combining the two:

[0015] Q(s, a) represents the Q value of taking action a in state s; V(s) represents the state value function of state s; A(s,a) represents the advantage function of taking action a in state s. Indicates the size of the action space; This represents traversing all actions in the action space and calculating the mean of the dominance function for all actions in the current state.

[0016] As can be seen from the above technical solution, compared with the prior art, the present invention has the following technical advantages: (1) The most prominent feature of this invention is that it takes data-driven as the core concept, models the traffic control in the construction area as a Markov decision process, and realizes intelligent control based on dual deep Q network, which breaks through the limitations of the traditional speed limit control mode based on fixed rules.

[0017] (2) This invention uses a carefully designed multi-objective reward function to synergistically optimize traffic efficiency, safety and system stability, so that the control strategy can achieve a balance in multiple performance dimensions and improve the overall performance of the system.

[0018] (3) The dual deep Q network architecture adopted can estimate the state value and action advantage independently through its unique value stream and advantage stream separation mechanism, thereby learning the value function more accurately, effectively improving the stability and convergence efficiency of policy learning, and ensuring the high reliability of the control policy in complex traffic environment. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 A flowchart of a progressively variable speed limit control method for highway construction areas provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall principle of the method flow provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of a SUMO scene in the construction area provided in an embodiment of the present invention; Figure 4 The average cumulative reward change curve during the algorithm training process is provided in the embodiments of the present invention; Figure 5 This is a diagram showing the change in safety indicators of a construction section during a variable speed limit process, provided in an embodiment of the present invention. Figure 6 This is a graph showing the change in efficiency indicators of road sections during variable speed limit construction, provided in an embodiment of the present invention. Figure 7 This is a graph showing the change in stability indicators of a construction section during a variable speed limit process, provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] This invention discloses a method for stepwise variable speed limit control in highway construction zones based on dual depth Q-networks, referring to... Figure 1 As shown, it includes the following steps: S101: Construct an environment model for interaction with the reinforcement learning agent, the environment model including a state space, an action space, and a reward function; S102: Construct a reinforcement learning model based on a dual deep Q-network, and generate a training dataset through iterative interaction between the agent and the environment model; S103: Using the training dataset, with the goal of maximizing the cumulative reward, train the dual deep Q network, and implement stepwise variable speed limit control in the highway construction area based on the trained model.

[0023] In step S101, refer to Figure 2 As shown, based on a real highway construction zone scenario, a dual deep Q-network environment model is established in the SUMO traffic simulation platform to build a reinforcement learning environment model. The environment model describes the traffic dynamics of the highway construction zone, interacts with the reinforcement learning agent, and generates data. It mainly includes three modules: state space, action space, and reward function, specifically defined as follows: (1) Definition of state space: The driving state of vehicles in the highway construction zone is affected by multiple factors. The state space includes representative variables such as traffic volume, occupancy rate, and average speed. For the current time period The state space is shown in equation (1): (1) in, , ,and , respectively representing the control time period Inner Traffic flow, occupancy rate, and average speed of the segment; (2) Definition of motion space: In the stepwise variable speed limit control in the high-speed construction zone, motion refers to the movement of a vehicle within the variable speed limit control area during the control time period. The speed limit must be observed within the designated area. Furthermore, in each zone with progressively variable speed limits, the speed limit increases progressively based on the action value. Δv (for example Δv =10km / h). For example, in each training cycle, the speed limit value 'a' of the reinforcement learning agent will be directly assigned to the downstream speed limit segments 5 and 6. For the segments 4, 3, 2, and 1 sequentially from downstream to upstream, their speed limits will be a+10, a+20, a+30, and a+40 respectively. Furthermore, the maximum speed limit cannot exceed the default speed limit of the highway segment. Current time cycle. motion space express.

[0024] (3) Definition of reward function r. The reward function is a multi-objective function, which includes a weighted sum of safety, efficiency, and stability indicators: First, define the driving safety indicators respectively. High efficiency indicators and system stability indicators The specific descriptions of each performance indicator are as follows.

[0025] The speed is characterized by the standard deviation of the speed of vehicles in the speed-limited zone. A lower speed standard deviation indicates that the vehicle speed in this section is more uniform, which reduces the risk of collision caused by the speed difference between vehicles, as shown in formula (2): (2) in, It is the first The vehicle's speed during the current speed limit period. This represents the average speed of all vehicles within the current speed limit period, where N is the total number of vehicles. The smaller the value, the lower the driving risk.

[0026] Characterized by average vehicle speed, using average speed With the maximum speed limit threshold of the road The degree of closeness is quantified, that is: ratio representation; as shown in formula (3): (3) in, The larger the value, the higher the traffic efficiency of the road segment.

[0027] The quantification is based on vehicle queue length, as shown in formula (4): (4) in, L Represents the actual queue length. This represents the preset queue length value. This represents the penalty coefficient. Vehicles queuing within a specific range are not penalized; however, once the queue length exceeds a preset threshold, the penalty increases exponentially with the queue length.

[0028] Based on the above three indicators, using , and The weighted sum is used as the reward function, as shown in formula (5): (5) in, , and Custom weights are assigned to the three metrics, with values ​​ranging from [value range missing]. .

[0029] In step S102, the reinforcement learning network is constructed as follows: Figure 2 As shown, training data can be generated.

[0030] Due to the action space For continuous variables, a reinforcement learning model is built using a dual deep Q-network. Unlike traditional single Q-value estimation networks, the dual deep Q-network introduces two parallel branches in its structure: a state value branch is used to estimate the importance of the current state. The action advantage branch is used to estimate the relative advantage of each action in this state. The final Q value is obtained by combining the two: (6) Q(s, a) represents the Q value of taking action a in state s; V(s) represents the state value function of state s; A(s,a) represents the advantage function of taking action a in state s. Indicates the size of the action space; This represents traversing all actions in the action space and calculating the mean of the dominance function for all actions in the current state.

[0031] The model comprises two sub-networks: a Q-network and a target Q-network, both with identical structures consisting of two convolutional layers. A training dataset is generated based on the iterative interaction between the agent and the environment model. Simultaneously, the predicted Q value and the target Q value are calculated.

[0032] (1) Training the online Q-network. Using the current state. and actions Train the Q-network to obtain the "predicted Q-value". , indicating the selection action The value, among which, These are the hyperparameters of the Q-network.

[0033] (2) Training the target Q-network. Given the state data for the next cycle. Using the target Q network from state Predict the maximum Q-value among all actions performed, i.e., the "target Q-value". ,in, These are the hyperparameters of the target Q-network.

[0034] Specifically, dataset The construction process is as follows: Step 21: Based on the initial state of the environment model Randomly select actions This leads to the state of the next cycle. The data sequence is obtained through iterative mapping of state space to action space. ; Step 22: According to The structure of data sequences Reconstruction involves refactoring the initial data sequence into a structure of state-action-next state tuples. .

[0035] In step S103, the speed limit control in the construction area is optimized.

[0036] like Figure 2 As shown, based on the training of the dual deep Q-network, the model can ultimately output the optimal action according to the current traffic flow state. This is described in two steps.

[0037] (1) Construct a loss function based on the predicted Q-value, the target Q-value, and the reward r calculated from the data samples. As shown in formula (7): (7) in, This is a discount factor used to measure the impact on rewards earned over future time periods.

[0038] (2) Optimize the loss function using gradient descent. And update the parameters of the Q network. and parameters in the target Q network Speed ​​control was optimized through multiple iterations.

[0039] For example, the control method for progressively variable speed limits in highway construction zones based on dual-depth Q-networks, as described in this invention, is as follows: Step 1) Instantiate the environment model. For example... Figure 3 As shown, the speed limit zone of a certain highway construction area is 2260m long and is divided into 6 variable speed limit sub-sections. Among them, sections 1-4 are progressively variable speed limit zones. The instantiated environment model based on this is as follows: (1) Determining the state space. In the scenario of a highway construction zone, the traffic flow is complex, and multiple variables should be selected to comprehensively depict the real-time operation of the traffic flow. However, too many variables will result in excessive data dimensionality and make model solving difficult. Based on this, traffic volume, occupancy rate, and average speed are selected as state variables. In addition, road segments 4, 5, and 6 are selected as state observation segments, for a total of 3×3=9 state variables.

[0040] (2) Determining the movement space. To reduce the movement space and prevent frequent and significant vehicle deceleration, the speed limit is discretized at 5 km / h intervals. Considering the basic operation of traffic flow and the maximum speed limit of the road segment, the movement space... .

[0041] (3) Determining the reward function. Let , and All values ​​being 1 indicates that the importance placed on driving safety, efficiency, and system stability is equal. .

[0042] 2) Construct a deep Q-network and iteratively interact with the environment model.

[0043] (1) Based on the initial state of the environment model Randomly select actions This leads to the state of the next cycle. By iteratively mapping state to action, a data sequence is obtained. .

[0044] (2) Because Q network is based on As input, the target Q-network uses As input, data sequence To refactor, In this way, the "predicted Q value" and the "target Q value" can be obtained.

[0045] 3) Model training and testing to verify the results.

[0046] (1) Calculate the loss function according to formula (7) and optimize it using gradient descent. Set the number of training rounds to 300, each round to 1 hour, and train 10 times. Plot the average cumulative reward change curve obtained in each training round, as shown below. Figure 4 As shown, after 200 training rounds, the average cumulative reward converged to around 3200, and the final average cumulative reward was 3250.12, indicating that the model has convergence.

[0047] (2) For the trained deep Q network (usually called "agent"), conduct a stepwise variable speed limit control experiment in the high-speed construction area scenario recreated by SUMO, and evaluate the safety indicators respectively. High efficiency indicators and stability indicators The standard deviation of speed, travel time, and number of vehicles in queues were selected for quantitative analysis, such as... Figure 5 , Figure 6 and Figure 7 As shown.

[0048] In this diagram, the green line representing DuDQN (Dual Deep Q-Network) indicates the core control method proposed in this invention. The black line corresponding to the absence of variable speed limits represents existing technology. Figure 5 , Figure 6 and Figure 7 The curves shown are the results obtained after experiments were conducted on the SUMO simulation platform using the DuDQN method, aiming to verify the effectiveness of this method.

[0049] Depend on Figure 5 It can be seen that the mean standard deviation of speed under agent control decreased from 7.94 to 4.83, indicating a significant improvement in vehicle driving stability; Figure 6 It can be seen that the average travel time decreased from 59.8s to 51.0s, indicating improved traffic efficiency; Figure 7 It can be seen that the number of vehicles in the queue was lower than the speed limit without variable speed for most periods of time, and traffic delays were effectively controlled. Combined with its performance in terms of speed stability and traffic efficiency, the overall traffic operation status was significantly optimized.

[0050] 4) Finally, assuming the model has been deployed to a highway construction area, such as... Figure 3 As shown, road sections 4, 5, and 6 are the status observation zones. The implementation process will be explained step-by-step using a control time cycle, for example, updating the speed limit every 30 minutes: Step 1: Status awareness, data collection; The system collects real-time traffic data from the construction area using sensors. For example, over time period t, it observes the traffic volume on road segments 4, 5, and 6. Market share and average speed , forming the state space Specific values ​​might be: traffic volume of 1200 vehicles / hour, occupancy rate of 15%, and average speed of 60 km / h. These data are then input into the deployed DuDQN model.

[0051] Step 2: Model reasoning and action output; The DuDQN model is based on the current state. Calculate the Q value for each possible speed-limited action using formula (6), and select the action with the largest Q value as the optimal speed limit.

[0052] For example, the model comes from the action space. In a speed range of {40, 45, 50, 55, 60, 65, 70, 75, 80} km / h, the optimal output action is a(t) = 60 km / h, which is the speed limit for this period. The decision-making process considers multiple objectives and rewards: safety (small speed standard deviation); efficiency (high average speed); and stability (short queue length).

[0053] Step 3: Perform the action and adjust the speed limit; The system sends the speed limit of 60 km / h to the variable speed limit signs in the construction area and updates the display. At the same time, according to the principle of progressive speed limit, the speed limit of the upstream sub-section increases by Δv = 10 km / h in turn, such as 70 km / h for section 4, 80 km / h for section 3, etc., to ensure a smooth transition in vehicle speed.

[0054] Step 4: Effectiveness evaluation and feedback; In the next time period t+1, the system collects data again, calculates the reward r, and uses formula (5). For example, if the speed standard deviation decreases after the current speed limit is reached ( Improved), average speed close to the maximum speed limit ( Optimization), reduction of queuing vehicles ( If the model parameters are improved, the reward r will be positive. The model parameters are fine-tuned through online learning. The loss function of formula (7) is used to continuously optimize the strategy.

[0055] In the above embodiments, firstly, a dual deep Q-network environment model is established based on a real highway construction zone scenario, and the state space and action space in the environment model are determined. Secondly, traffic efficiency, safety, and stability evaluation indicators are extracted from the state and action variables, and a multi-objective reward function is constructed through weighted fusion. Subsequently, the deep Q-network model is trained with the goal of maximizing cumulative rewards through continuous interaction between the agent and the environment. Finally, the model's control effect is verified based on the SUMO simulation platform, and the effectiveness of the strategy is evaluated by comparing and analyzing various performance indicators. The method of this invention can be integrated into existing traffic control systems, achieving low-latency decision-making through edge computing devices to ensure the real-time performance of speed limit strategies.

[0056] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for stepwise variable speed limit control in highway construction zones based on dual deep Q-networks, characterized in that, Includes the following steps: S101: Construct an environment model for interaction with the reinforcement learning agent, the environment model including a state space, an action space, and a reward function; S102: Construct a reinforcement learning model based on a dual deep Q-network, and generate a training dataset through iterative interaction between the agent and the environment model; S103: Using the training dataset, with the goal of maximizing the cumulative reward, train the dual deep Q network, and implement stepwise variable speed limit control in the highway construction area based on the trained model.

2. The method according to claim 1, characterized in that, In step S101: The state space includes the traffic volume, occupancy rate, and average speed of the road segment in the construction area within a time period t. The action space includes discrete road segment speed limits, which are taken based on preset intervals; The reward function is a multi-objective function, which includes a weighted sum of safety, efficiency, and stability indicators.

3. The method according to claim 2, characterized in that, The reward function comprehensively considers driving safety indicators, efficiency indicators, and system stability indicators. Therefore, the reward function calculation formula is: in, , and Custom weights are assigned to the three metrics, with values ​​ranging from [value range missing]. ; As a safety indicator, it is characterized by the standard deviation of vehicle speeds within the speed-limited zone. A lower standard deviation indicates that the vehicle speeds are more uniform within that section of road. For the first The vehicle's speed during the current speed limit period. The average speed of all vehicles during the current speed limit period, where N is the total number of vehicles; As an efficiency indicator, it is calculated by comparing the average vehicle speed with the road's maximum speed limit threshold. The ratio is used to represent; As a system stability indicator, it is quantitatively characterized based on vehicle queue length; L This is the actual queue length. The preset queue length value. This is the penalty coefficient.

4. The method according to claim 1, characterized in that, In step S102, the specific process of generating the training dataset includes: The initial data sequence is obtained through iterative mapping between the state space and the action space; The initial data sequence is reconstructed into a structure of state-action-next state tuples, which is used to calculate the predicted Q value and the target Q value.

5. The method according to claim 4, characterized in that, In step S103, training the dual deep Q-network includes: A loss function is constructed based on the predicted Q-value, the target Q-value, and the immediate reward. The loss function is optimized using gradient descent, and the network parameters are updated until the model converges.

6. The method according to claim 5, characterized in that, The loss function formula is as follows: Where r is the reward function; This is a discount factor used to measure the impact on rewards earned in future time periods; These are the parameters of the Q-network; These are the parameters of the target Q-network.

7. The method according to claim 5, characterized in that, The dual deep Q-network introduces two parallel branches in its structure: The state value branch is used to estimate the importance of the current state. ; The action advantage branch is used to estimate the relative advantage of each action in this state. The final Q value is obtained by combining the two: Q(s, a) represents the Q value of taking action a in state s; V(s) represents the state value function of state s; A(s,a) represents the advantage function of taking action a in state s. Indicates the size of the action space; This represents traversing all actions in the action space and calculating the mean of the dominance function for all actions in the current state.

Images