A variable speed limit control method for expressways under digital twin conditions

Abstract

The present invention relates to a method for controlling variable speed limits on expressways under digital twin conditions, including building a scenario simulation model, generating a benchmark variable speed limit collaborative control strategy; constructing traffic efficiency indicators and traffic safety indicators, evaluating the effectiveness of the strategy, and building a historical data event library; building a simulation model for each new scenario in the historical data event library, and using a continuous learning algorithm to update and optimize the benchmark variable speed limit control strategy in each scenario in turn; performing parallel testing of variable speed limit control strategies for multiple scenarios to obtain the optimal variable speed limit control strategy, which is applied to a real environment for variable speed limit control. Compared with the prior art, the present invention preserves the knowledge memory of the variable speed limit control strategy in the learning process of each scenario, improves applicability, can effectively optimize the operation of expressway traffic flow, realizes efficient optimization of the variable speed limit collaborative control strategy, and ensures the effectiveness of the variable speed limit control strategy in the empirical application process.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic control, and in particular to a variable speed limit control method for an expressway under digital twin conditions. Background Art

[0002] Variable speed limit (VSL) is a key component of active traffic management system (ATMS). Variable speed limit control includes speed limits for road sections in various scenarios such as traffic congestion, secondary accident prevention and extreme weather, and dynamically adjusts control strategies to improve traffic flow.

[0003] The document “Developing crash modification factors for variable speed limits” (FHWA, 2021. https: / / www.fhwa.dot.gov / publications / research / safety / 21080 / 21080.pdf) and the document “Assessment of dynamic speed limits on freeway A20 near Rotterdam” (Hoogendoorn, SP, Daamen, W., Hoogendoorn, RG, Goemans, JW, 2013. Transportation research record 2380(1), 61-71.) disclose that variable speed limit control can reduce the accident rate by 34% and traffic congestion time by 18%, which is of great significance to improving traffic safety and traffic efficiency on expressways.

[0004] The core of variable speed limit control lies in the formulation of control strategies. At present, the mainstream trend is to use deep reinforcement learning algorithms to conduct research on variable speed limit control strategies. However, the existing research on variable speed limit control strategies based on deep reinforcement learning mainly focuses on the training of single-scenario control strategies, and has poor applicability in real multi-scenario environments. Traffic flow scenarios are complex and diverse, and it is not feasible to traverse and learn all scenarios. Under multi-scenario training conditions, deep reinforcement learning has a serious problem of scene forgetting, which leads to the performance degradation of variable speed limit control strategies in "old scenarios" trained in the early stage. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method for generating a variable speed limit control strategy for an expressway under digital twin conditions.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A variable speed limit control method for an expressway under digital twin conditions, the method comprising the following steps:

[0008] Step 1: Build an event scenario simulation model for the current road section as an interactive environment, and use a multi-agent deep reinforcement learning algorithm to generate a benchmark variable speed limit collaborative control strategy;

[0009] Step 2: Construct traffic efficiency index and traffic safety index to form comprehensive evaluation index, apply benchmark variable speed limit control strategy in real multi-scenario environment and evaluate the traffic flow improvement effect in real time. When the improvement effect is less than the threshold, store the corresponding event information in the historical event database;

[0010] Step 3: Build a scenario simulation model for the corresponding scenario based on the event information in the historical event database, and use a continuous learning algorithm to update and optimize the benchmark variable speed limit control strategy in each scenario in turn to obtain a multi-scenario variable speed limit control strategy;

[0011] Step 4: Simulate the actual traffic flow environment of the road section under digital twin conditions, conduct parallel tests of multi-scenario variable speed limit control strategies in the simulated environment, and obtain the optimal variable speed limit control strategy based on the comprehensive improvement effect of traffic safety and traffic efficiency, and apply it to the actual environment for variable speed limit control.

[0012] Furthermore, the event information includes event type, event duration and traffic flow information.

[0013] Furthermore, step 1 specifically includes the following steps:

[0014] Step 11: Based on the existing event information of the current road section, an event scenario simulation model is built in the simulation platform;

[0015] Step 12: Use the constructed scenario simulation model as the interactive environment, use the multi-agent deep reinforcement learning algorithm, and use the variable speed limit controller as the agent to perform simulation interactive trial and error training;

[0016] Step 13: When the multi-agent deep reinforcement learning algorithm network converges, the training ends and a benchmark variable speed limit collaborative control strategy is obtained.

[0017] Furthermore, the event scenario simulation model includes a car-following model and a lane-changing model.

[0018] Furthermore, the multi-agent deep reinforcement learning algorithm includes agents, states, actions and reward functions, where the agent is a variable speed limit controller for a section, the state is a data set including the mean speed of the section, the standard deviation of the speed of the section, the mean flow rate of the section and the standard deviation of the flow rate of the section, the action is the minimum speed limit and the maximum speed limit of the section, and the reward function is the objective function of the algorithm training. The reward function formula is as follows:

[0019] r=1.2AS-2SS-0.2V

[0020] Where r is the reward function value, AS is the mean speed of the road section, SS is the standard deviation of the speed of the road section, the units of AS and SS are both km / h, V is the total number of vehicles with a speed lower than 50 km / h, and only the values ​​of the calculated quantities are calculated without calculating the units.

[0021] Furthermore, step 3 specifically includes the following steps:

[0022] Step 31: build a scene simulation model for each event in the event library formed in step 2;

[0023] Step 32: Use the multi-agent deep reinforcement learning algorithm to perform strategy update optimization in the new scenario, use the continuous learning algorithm to limit the gradient update space during the deep network update process, and store the knowledge memory of each scenario after the training of the scenario is completed;

[0024] Step 33: Repeat step 32 until all scenarios are learned, and obtain a multi-scenario variable speed limit control strategy.

[0025] Furthermore, the knowledge memory of the scene in step 32 is represented by the network parameter matrix R after the scene training is completed, and the gradient update space calculation formula is:

[0026]

[0027] In the formula, is the gradient generated during the training process, M is the union of the network parameter matrix R of the knowledge memory of all trained scenes, and M T is the transpose of M.

[0028] Furthermore, step 4 specifically includes the following steps:

[0029] Step 41: Use digital twin technology to simulate a real traffic flow environment, and perform parallel testing on the benchmark variable speed limit control strategy generated in step 1 and the multi-scenario variable speed limit control strategy optimized in step 3 in the digital twin environment;

[0030] Step 42: using the traffic efficiency index and traffic safety index proposed in step 2 as comprehensive strategy evaluation indexes, respectively calculating the improvement of the strategy evaluation indexes under the intervention of the baseline strategy and the optimized multi-scenario strategy;

[0031] Step 43: The strategy with the largest improvement in the index is used as the final variable speed limit control strategy and applied to a real expressway environment.

[0032] Furthermore, the traffic safety index is the remaining collision time E 1 The traffic efficiency index is the total travel time E 2 , the calculation method of comprehensive evaluation index is as follows:

[0033] E=W 1 *E 1 +W 2 *E 2

[0034] In the formula, E is the comprehensive evaluation index, W 1 Represents the remaining collision time E 1 The weight, W 2 Represents the total travel time E 2 The weight of .

[0035] Furthermore, the formula for calculating the improvement of the strategy evaluation index is as follows:

[0036]

[0037] Where ΔE is the improvement of the strategy evaluation index, E A is the comprehensive evaluation index value without intervention, E A ′ is the comprehensive evaluation index value under strategic intervention.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. The present invention uses a road section scenario simulation model to generate a benchmark variable speed limit collaborative control strategy, uses comprehensive evaluation indicators for real-time evaluation, re-models the scenarios with insufficient improvement effects, and updates and optimizes the basic strategy. By simulating parallel tests of variable speed limit control strategies in multiple scenarios, the optimal variable speed limit control strategy is obtained, and the results are applied to the real environment, thereby improving the reliability of the variable speed limit control strategy.

[0040] 2. The present invention uses a continuous learning algorithm to preserve the knowledge memory of the variable speed limit control strategy during the learning process of each scenario, thereby improving the applicability of the variable speed limit control strategy in a realistic multi-scenario environment.

[0041] 3. The present invention comprehensively considers both traffic safety and traffic efficiency, and carries out research on the generation and optimization of variable speed limit control strategies, which can effectively optimize the operation of expressway traffic flow.

[0042] 4. The present invention adopts a multi-agent deep reinforcement learning algorithm, uses a deep network to capture continuous high-dimensional state input features, shares global information of the system during the training process, performs multi-section linkage, and realizes efficient optimization of the variable speed limit collaborative control strategy.

[0043] 5. The present invention conducts parallel testing and optimization of the benchmark strategy and the optimization strategy under digital twin conditions, further ensuring the effectiveness of the variable speed limit control strategy in the empirical application process. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0045] Figure 2 It is a variable speed limit control strategy learning framework based on multi-agent deep reinforcement learning in an example of the present invention.

[0046] Figure 3 Schematic diagram of a traffic congestion scenario in an embodiment of the present invention.

[0047] Figure 4 Schematic diagram of a high accident risk scenario in an embodiment of the present invention.

[0048] Figure 5 Schematic diagram of a secondary accident prevention scenario in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0050] Example

[0051] The present invention discloses a variable speed limit control method for expressways under digital twin conditions, such as Figure 1 The process shown in the figure specifically includes the following steps:

[0052] Step 1: Benchmark collaborative strategy generation: Build an event scenario simulation model for the current road section as an interactive environment, and use a multi-agent deep reinforcement learning algorithm to generate a benchmark variable speed limit collaborative control strategy;

[0053] Step 2: Strategy effect evaluation and event database construction: Construct traffic efficiency indicators and traffic safety indicators to form comprehensive evaluation indicators, apply the benchmark variable speed limit control strategy in a real multi-scenario environment and evaluate the traffic flow improvement effect in real time. When the improvement effect is less than 18%, store the corresponding event information in the historical event database;

[0054] Step 3: Continuous update and optimization of strategies: Build a simulation model for each new scenario based on the historical event database, and use a continuous learning algorithm to update and optimize the benchmark variable speed limit control strategy in each scenario in turn to obtain a multi-scenario variable speed limit control strategy;

[0055] Step 4: Parallel testing and optimization of strategies: Simulate the actual traffic flow environment of the road section under digital twin conditions, conduct parallel testing of multi-scenario variable speed limit control strategies in the simulation environment, and obtain the optimal variable speed limit control strategy based on the comprehensive improvement effect on traffic safety and traffic efficiency, and apply it to variable speed limit control in the real environment.

[0056] Preferably, the event information in the historical event database includes: event type (such as accident, congestion, etc.), event duration, and traffic flow information (volume, speed).

[0057] Step 1 specifically includes the following steps:

[0058] Step 11: Based on the historical event information stored in the historical event database, an event scenario simulation model is built in the SUMO (Urban Traffic Simulation) platform;

[0059] Step 12: Use the constructed scenario simulation model as the interactive environment, use the multi-agent deep reinforcement learning algorithm, regard the variable speed limit controller as an agent, and conduct simulation interactive trial and error training;

[0060] Step 13: When the agent network converges, the training ends and a benchmark variable speed limit collaborative control strategy is obtained.

[0061] Preferably, a car-following model and a lane-changing model are used in the event scenario simulation model, the car-following model is an intelligent driver model, and the lane-changing model is an LC2013 model.

[0062] The multi-agent deep reinforcement learning algorithm is an Actor-Critic (participant-evaluator) architecture. The control units are divided by the roadside speed limit information release equipment. Each control unit is regarded as an agent. Through the global information sharing of the system during the training process, multi-road section coordination is carried out. The specific agent, state, action and reward function design are as follows:

[0063] Agent: The variable speed limit controller of the road section is regarded as an agent. The agent of each road section can set different speed limits for the road section area.

[0064] Status: the set of the mean speed of the road section, the standard deviation of the speed of the road section, the mean flow of the road section, and the standard deviation of the flow of the road section;

[0065] Action: The range is the minimum speed limit and maximum speed limit of the road section, with an interval of 5km / h;

[0066] Reward function: The reward function is the objective function of algorithm training, and the formula is as follows:

[0067] r=1.2AS-2SS-0.2V

[0068] Where r is the reward function value, AS is the mean speed of the road section, SS is the standard deviation of the speed of the road section, the units of AS and SS are both km / h, V is the total number of low-speed vehicles (vehicles with a speed lower than 50 km / h), and only the values ​​of the calculated quantities are calculated without calculating the units.

[0069] In this embodiment, the expressway section considered is a 12km one-way three-lane section, which is divided into 6 sections at intervals of 2km. The speed limit information is released through the gantry, and the traffic flow information is collected through sensing devices such as coils. The default speed limit is 100km / h, the minimum speed limit is 60km / h, the simulation time is 1h, and the flow input is 5400veh / h. Figure 2 shown.

[0070] The historical event scenario is a traffic congestion scenario. In order to generate a traffic congestion wave that propagates from downstream to upstream, the vehicle speed at 11.5km to 12km is set to 30km / h in 5 to 10 minutes. The schematic diagram is as follows: Figure 3 shown.

[0071] The comprehensive evaluation index of traffic efficiency and traffic safety in step 2 consists of two parts: traffic safety index and traffic efficiency index. The traffic safety index is the remaining collision time (Time to collision, TTC):

[0072] E 1 =-TTC total

[0073] Where, TTC total is the overall TTC of the road section, that is, the sum of the TTCs between all two adjacent vehicles on the road section. The TTC calculation formula is as follows:

[0074]

[0075] Where ΔD is the distance between the two vehicles and ΔV is the speed difference between the two vehicles.

[0076] The traffic efficiency indicator is the total travel time (TTT), which is the sum of the travel time of all vehicles on the road section and is set as:

[0077] E 2 =-TTT

[0078] Finally, considering the traffic safety and efficiency, the comprehensive evaluation indicators are as follows:

[0079] E=W 1 *E 1 +W 2 *E 2

[0080] W 1 Represents the weight of traffic safety index, W 2 It represents the weight of the traffic efficiency index, and its value is determined based on the actual situation of the expressway.

[0081] In this embodiment, in order to balance the two types of indicators, W 1 , W 2 They are determined to be 0.5 and 1.5 respectively. Added to the event library are high accident risk scenarios (hereinafter referred to as scenario 2) and secondary accident prevention scenarios (hereinafter referred to as scenario 3). The main performance characteristics of the high accident risk scenario are large fluctuations in vehicle speed on the road section, which is achieved by setting the "SpeedDev (traffic flow speed standard deviation)" parameter in SUMO to 0.9; the secondary accident prevention scenario is simulated by stopping a vehicle at 11.5km for a period of 5 to 10 minutes. Example diagrams of scenarios 2 and 3 are shown below. Figure 4 and Figure 5 shown.

[0082] Step 3 specifically includes the following steps:

[0083] Step 31: build a scene simulation model for each event in the event library formed in step 2;

[0084] Step 32: Use the multi-agent deep reinforcement learning algorithm to perform strategy update optimization in the new scenario, use the continuous learning algorithm to limit the gradient update space during the deep network update process, and store the knowledge memory of each scenario after the training of the scenario is completed;

[0085] Step 33: Repeat step 32 until all scenarios are learned, and obtain a multi-scenario variable speed limit control strategy.

[0086] The continuous learning algorithm in step 3 includes limiting the gradient update space during the scene training process and storing the scene knowledge memory after the scene training is completed. The scene knowledge memory is represented by the network parameter matrix R after the scene training is completed. The gradient update space calculation formula is:

[0087]

[0088] In the formula, is the gradient generated during the training process, M is the union of the network parameter matrix R of the knowledge memory of all previously trained scenes, and M T is the transpose of M.

[0089] In this embodiment, the benchmark strategy is continuously optimized on the high accident risk scenario (scenario 2) and the secondary accident prevention scenario (scenario 3) stored in step 2 in sequence.

[0090] Step 4 specifically includes the following steps:

[0091] Step 41: Use digital twin technology to simulate a real traffic flow environment, and perform parallel testing on the benchmark variable speed limit control strategy generated in step 1 and the multi-scenario variable speed limit control strategy optimized in step 3 in the digital twin environment;

[0092] Step 42: Using the comprehensive indicators of traffic efficiency and traffic safety proposed in step 2 as strategy evaluation indicators, respectively calculate the improvement of the strategy evaluation indicators under the intervention of the baseline strategy and the optimized multi-scenario strategy;

[0093] Step 43: The strategy with the largest improvement in the index is used as the final variable speed limit control strategy and applied to a real expressway environment.

[0094] The formula for calculating the improvement of the strategy evaluation index is as follows:

[0095]

[0096] In the formula, ΔE is the improvement of the strategy evaluation index, E is the index value without intervention, and E ′ is the indicator value under strategic intervention.

[0097] In this embodiment, the calculation results of the improvement of the indicators of the baseline strategy and the optimized multi-scenario strategy in parallel tests on three scenarios are shown in Table 1:

[0098] Table 1 Improvement of comprehensive evaluation indicators under the two strategy intervention conditions in each scenario

[0099]

[0100] As shown in Table 1, the performance of the strategy after optimization in step 3 is better than the baseline strategy in each scenario, with an average improvement of 12.08%. Therefore, the optimized strategy is applied to the actual environment.

[0101] In summary, the present invention uses a continuous learning algorithm to retain the knowledge memory of the variable speed limit control strategy in each scenario during the multi-scenario training process, overcoming the defect of poor applicability of the traditional variable speed limit control strategy generation method in a realistic multi-scenario environment. At the same time, the strategy is tested and optimized in parallel under digital twin conditions, ensuring the effectiveness of the strategy in the empirical application process, which is helpful to optimize the operation of expressway traffic flow.

[0102] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A variable speed limit control method for expressways under digital twin conditions, characterized in that: The method comprises the following steps: Step 1: Build an event scenario simulation model for the current road section as an interactive environment, and use a multi-agent deep reinforcement learning algorithm to generate a benchmark variable speed limit collaborative control strategy; Step 2: Construct traffic efficiency index and traffic safety index to form comprehensive evaluation index, apply benchmark variable speed limit control strategy in real multi-scenario environment and evaluate the traffic flow improvement effect in real time. When the improvement effect is less than the threshold, store the corresponding event information in the historical event database; Step 3: Build a scenario simulation model for the corresponding scenario based on the event information in the historical event database, wherein the event information includes event type, event duration, and traffic flow information, and use a continuous learning algorithm to update and optimize the benchmark variable speed limit control strategy in each scenario in turn to obtain a multi-scenario variable speed limit control strategy; Step 4: Under digital twin conditions, simulate the actual traffic flow environment of the road section to obtain a simulation environment, conduct parallel tests of multi-scenario variable speed limit control strategies in the simulation environment, and obtain the optimal variable speed limit control strategy based on the comprehensive improvement effect on traffic safety and traffic efficiency, and apply it to the actual environment for variable speed limit control.

2. The method for controlling variable speed limit on expressways under digital twin conditions according to claim 1 is characterized in that: The step 1 specifically comprises the following steps: Step 11: Based on the existing event information of the current road section, an event scenario simulation model is built in the simulation platform; Step 12: Use the constructed scenario simulation model as the interactive environment, use the multi-agent deep reinforcement learning algorithm, and use the variable speed limit controller as the agent to perform simulation interactive trial and error training; Step 13: When the multi-agent deep reinforcement learning algorithm network converges, the training ends and a benchmark variable speed limit collaborative control strategy is obtained.

3. The method for controlling variable speed limit on expressways under digital twin conditions according to claim 2 is characterized in that: The event scenario simulation model includes a car-following model and a lane-changing model.

4. The method for controlling variable speed limit on expressways under digital twin conditions according to claim 2 is characterized in that: The multi-agent deep reinforcement learning algorithm includes an agent, a state, an action and a reward function. The agent is a variable speed limit controller for a section. The state is a data set including the mean speed of the section, the standard deviation of the speed of the section, the mean flow rate of the section and the standard deviation of the flow rate of the section. The action is the minimum speed limit and the maximum speed limit of the section. The reward function is the objective function of the algorithm training. The reward function formula is as follows: r=1.2AS-2SS-0.2V Where r is the reward function value, AS is the mean speed of the road section, SS is the standard deviation of the speed of the road section, the units of AS and SS are both km / h, and V is the total number of vehicles with a speed lower than 50 km / h.

5. The method for controlling variable speed limit on expressways under digital twin conditions according to claim 1 is characterized in that: The step 3 specifically includes the following steps: Step 31: build a scene simulation model for each event in the event library formed in step 2; Step 32: Use the multi-agent deep reinforcement learning algorithm to perform strategy update optimization in the new scenario, use the continuous learning algorithm to limit the gradient update space during the deep network update process, and store the knowledge memory of each scenario after the training of the scenario is completed; Step 33: Repeat step 32 until all scenarios are learned, and obtain a multi-scenario variable speed limit control strategy.

6. The method for controlling variable speed limit on expressways under digital twin conditions according to claim 5 is characterized in that: The knowledge memory of the scene in step 32 is represented by the network parameter matrix R after the scene training is completed, and the gradient update space calculation formula is: In the formula, is the gradient generated during the training process, M is the union of the network parameter matrix R of the knowledge memory of all trained scenes, and M T is the transpose of M.

7. The method for controlling variable speed limit on expressways under digital twin conditions according to claim 1 is characterized in that: The step 4 specifically comprises the following steps: Step 41: Use digital twin technology to simulate a real traffic flow environment, and perform parallel testing on the benchmark variable speed limit control strategy generated in step 1 and the multi-scenario variable speed limit control strategy optimized in step 3 in the digital twin environment; Step 42: using the traffic efficiency index and traffic safety index proposed in step 2 as comprehensive strategy evaluation indexes, respectively calculating the improvement of the strategy evaluation indexes under the intervention of the baseline strategy and the optimized multi-scenario strategy; Step 43: compare the improvement of the index of the baseline strategy and the improvement of the index of the strategy under the intervention of the optimized multi-scenario strategy, and use the one with the larger improvement as the final variable speed limit control strategy, and apply it to the real expressway environment.

8. The method for controlling variable speed limit on expressways under digital twin conditions according to claim 7 is characterized in that: The traffic safety index is the remaining collision time E1, the traffic efficiency index is the total travel time E2, and the calculation method of the comprehensive evaluation index is as follows: E=W1*E1+W2*E2 Where E is a comprehensive evaluation index, W1 represents the weight of the remaining collision time E1, and W2 represents the weight of the total travel time E2.

9. The method for controlling variable speed limit on expressways under digital twin conditions according to claim 8 is characterized in that: The formula for calculating the improvement of the strategy evaluation index is as follows: Where ΔE is the improvement of the strategy evaluation index, E A is the comprehensive evaluation index value without intervention, E A ′ It is the comprehensive evaluation index value under strategic intervention.

Images