Deep reinforcement learning highway variable speed limit control method based on multi-target dynamic weight distribution

Abstract

The invention discloses a deep reinforcement learning highway variable speed limit control method based on multi-target dynamic weight distribution, and the method comprises the steps: 1, collecting the traffic flow information of a road section, and obtaining the traffic state data; 2, defining an action parameter and a reward function of the deep reinforcement learning method; 3, constructing a network model of deep reinforcement learning, and establishing a playback memory bank storage sample; and 4, reward function weight updating is carried out based on an entropy weight method, an optimized control strategy is obtained based on a sampling training deep reinforcement learning network model, and speed limit control is carried out on each control unit. According to the method, dynamic weight distribution of the traffic efficiency and the safety reward function is achieved, the method can adapt to a complex traffic environment, meanwhile, the operation efficiency of the expressway is effectively improved, and the safety risk is reduced.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic control technology, specifically to a deep reinforcement learning method for variable speed limit control on highways using multi-objective dynamic weight allocation. This method is applicable to variable speed limit control on highways in a connected environment and can achieve coordinated optimization of highway operating efficiency and traffic safety by dynamically adjusting the priority of traffic control objectives. Background Technology

[0002] With the continuous growth of global motor vehicle ownership, highway traffic congestion has become increasingly prominent, especially during morning and evening rush hours. Insufficient road capacity easily leads to a series of problems such as traffic delays and increased traffic safety risks. The construction of new lanes or expansion of highway networks is constrained by land resources, construction costs, and environmental limitations, making large-scale implementation difficult. Therefore, traffic control strategies that directly manage highway merging zones, such as variable speed limits and ramp flow control, have become key approaches to improving the operational efficiency of highway merging zones.

[0003] Machine learning, especially reinforcement learning, constructs mappings directly without a model, freeing it from the dependence of traditional models on explicit traffic equations. However, most reinforcement learning-based variable speed limit control focuses only on a single objective of traffic efficiency or traffic safety, lacking collaborative optimization of multiple objectives. Furthermore, the weights of these objectives cannot be dynamically adjusted according to real-time traffic conditions, leading to unstable control performance in complex traffic scenarios. Therefore, there is an urgent need for a variable speed limit control method that can achieve dynamic weight optimization of multiple objectives and adapt to intelligent connected environments to solve the traffic management challenges of highways. Summary of the Invention

[0004] Purpose of the invention: To address the shortcomings of existing technologies, this invention provides a deep reinforcement learning-based variable speed limit control method for highways that employs multi-objective dynamic weight allocation. The aim is to dynamically adjust speed limits by allocating weights based on traffic efficiency and safety benefits, thereby ensuring that traffic control strategies can adapt to constantly changing traffic conditions and further improve traffic efficiency while enhancing traffic safety.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: The present invention discloses a variable speed limit control method for highways based on deep reinforcement learning with multi-objective dynamic weight allocation. The method is characterized by applying equal distances along the main line of the highway according to the vehicle's direction of travel. L Deployed I Each roadside unit (RSU) is used to divide the mainline of the highway into sections. I Each sub-segment is numbered as follows: i , The one that merges with the entrance ramp is numbered as follows: Sub-segments and numbered I The sub-segment is defined as the bottleneck area, numbered as The sub-segment is defined as the acceleration zone, before Each sub-segment is defined as a variable speed limit control area; the variable speed limit control method for highways includes the following steps: Step 1: In the t Traffic information for each sub-road segment is collected during each control cycle, and the traffic information for the first sub-road segment is obtained. t Under each control cycle I Traffic status information for individual road segments ; Using equation (1) to obtain the first i Each sub-section is in the first t Density under each control cycle Thus, the first t Traffic status information under each control cycle ; (1) In equation (1), For the first i Each road segment is under the current control cycle t The number of vehicles below; Step 2: Define the first t Speed ​​limiting action under each control cycle and mixed rewards ; Step 3: Construct deep reinforcement learning network models, including: immediate decision network model, immediate evaluation network model, long-term decision network model, and long-term evaluation network model; Step 4: Establish the playback memory sample set ; Step 5: When replaying the memory bank The number of samples stored in it exceeds Then, it indicates that the weight update cycle has been reached, and the weights of the mixed reward are updated using the entropy weight method; Step 6: Train the deep reinforcement learning network model to obtain the trained deep reinforcement learning network model, and then use the trained instant decision network model to output the corresponding speed limiting action based on the real-time input state information.

[0006] The present invention provides a method for variable speed limit control on highways based on deep reinforcement learning with multi-objective dynamic weight allocation, characterized in that step 2 includes: Step 2.1: Define the first t Speed ​​limiting action under each control cycle ,in, Indicates the first tThe first control cycle Speed ​​limit values ​​for each variable speed limit control zone; Step 2.2: Use equation (2) to obtain the first... t Bottleneck section under each control cycle Efficiency bonus for internal vehicles : (2) In equation (2), Represented as the first t Each control cycle is numbered as follows I Vehicles in the bottleneck section d speed, Indicates the first t Each control cycle is numbered as follows I The number of all vehicles within the bottleneck section. It is the maximum speed of the vehicle in free-flow conditions; Indicates the first I Each roadside unit (RSU) obtained its number as follows: I Within the bottleneck section t The first control cycle d The speed of the vehicle; Step 2.3: Calculate the first step using equation (3). t Each control cycle is numbered as follows I The bottleneck section d Time of vehicle collision ; (3) In equation (3), and They represent the first t Each control cycle is numbered as follows I The bottleneck section d Vehicles and their first The longitudinal position of the vehicle in front. Indicates the length of the vehicle in front. and They represent the first t Each control cycle is numbered as follows I The bottleneck section d Vehicles and their first The speed of the car in front; Indicates the first I The roadside unit RSU obtains the first I Within each sub-section of the road t The first control cycle d The location of the vehicle; Indicates the first I The roadside unit RSU obtains the first I Within each sub-section of the roadt The first control cycle d- The position of the vehicle in front; Indicates the first I The roadside unit RSU obtains the first I Within each sub-section of the road t The first control cycle d- The speed of the vehicle in front; Step 2.4: Calculate the first step using equation (4). t Each control cycle is numbered as follows I Cumulative collision time within bottleneck sections : (4) In equation (4), It is the collision time threshold. Indicates the first t Each control cycle is numbered as follows I The time between vehicle collisions that meet the conditions within the bottleneck section of the road; Step 2.5: Obtain the first step using equation (5). t Each control cycle is numbered as follows I Safety rewards within bottleneck sections ; (5) Step 2.6: Obtain the first step using equation (6). t Each control cycle is numbered as follows I Mixed rewards within bottleneck sections : (6) In equation (6), and They represent and The weights assigned to each number are all positive, and .

[0007] Furthermore, step 3 includes: Step 3.1: Construct an instantaneous decision-making network model, including: an input layer containing one neuron, each containing... one neuron The hidden layer consists of one output layer with two neurons; the parameters for initializing the instantaneous decision network model are: ; Step 3.2: Construct an instant evaluation network model, including: two input layers, each containing one neuron, each containing... one neuron Hidden layers, where the first layer is... The hidden layer is a fully connected layer, and the output layer contains one neuron; the parameters for initializing the instant evaluation network model are: ; Step 3.3: Construct a long-term decision network model with the same structure as the instantaneous decision network model, and set the parameters of the long-term decision network model... Construct a long-term evaluation network model with the same structure as the instant evaluation network model, and set the parameters of the long-term evaluation network model to be... .

[0008] Furthermore, step 4 includes: Step 4.1: The instantaneous decision-making network model receives the first... t Traffic status information under each control cycle Then, an exploratory speed-limited action is generated using equation (7). ; (7) In equation (7), This represents the policy function of an instantaneous decision-making network model. For the first t Random noise under each control cycle; Step 4.2: The t Traffic status under each control cycle Execute speed limiting action After that, the first State under each control cycle and with the t Efficiency bonus under each control cycle and security rewards Together as a sample Store to playback memory middle.

[0009] Furthermore, step 5 includes: Step 5.1: From Extract the nearest M The efficiency reward and security reward of the sample constitute the first sample. t Reward matrix under each control period ,in, Indicates the first The first control cycle Efficiency reward per sample Indicates the first The first control cycle Safety reward for each sample; Step 5.2: Calculate the first step using equations (8) and (9). The first control cycle Efficiency reward probability of a sample and the The probability of a safe reward for a sample : (8) (9) Step 5.3: Calculate the first step using equations (10) and (11). Information entropy of efficiency reward under each control cycle Information entropy of security rewards : (10) (11) Step 5.4: Calculate the first step using equations (12) and (13). The target weight of efficiency reward under each control cycle and the target weight of the security reward function : (12) (13) Step 5.5: Use equations (14) and (15) respectively to... and Perform a smooth update to obtain the weight of the updated efficiency reward. and the weight of the security function and will Assign to , Assign to , (14) (15) In equations (14) and (15), This indicates the proportion of the reward function weights; and They represent the first The weights of efficiency rewards and safety functions under each control cycle, when During initialization ; ; It is a random value between 0 and 1.

[0010] Furthermore, step 6 includes: Step 6.1: When replaying the memory bank The number of samples stored in the batch exceeds the training batch size. Then, from the playback memory bank Extract Sample ,in, Indicates the first The first control cycle k The state of each sample Indicates the first The first control cycle k Speed-limiting actions for each sample, Indicates the first The first control cycle k Efficiency reward per sample Indicates the first The first control cycle k Safety reward for each sample Indicates the first The first control cycle k The state of each sample; Step 6.2: Use equation (16) to obtain the first... The first control cycle k Mixed reward for each sample : (16) Step 6.3: Calculate the first step using equation (17). The first control cycle k Target score for each sample : (17) In equation (17), Indicates the discount factor. The output of the long-term decision network model represents the first... The first control cycle k The strategy for each sample This indicates the output of the long-term evaluation network model on the first... The first control cycle k The rating value of the state-action pair for each sample; Step 6.4: Calculate the instant evaluation network model in the first step using equation (18). The output of the first control cycle k The rating of a state-action pair for each sample : (18) In equation (18), The output of the instantaneous decision network model represents the first... The first control cycle k The strategy for each sample This indicates the immediate evaluation of the network model output for the first... The first control cyclek The rating value of the state-action pair for each sample; Step 6.5: Construct an instant evaluation network model using equation (19) in the first step. Loss function under each control cycle : (19) Step 6.6: Construct the parameter update formula for the instant evaluation network model using equation (20): (20) In equation (20), This indicates the learning rate for real-time evaluation of the network model; This indicates calculating the gradient of the parameters of the instant evaluation network model. This indicates assignment; Step 6.7: Construct an instant decision-making network model using equation (21) in the first step. Objective function under each control cycle : (twenty one) In equation (21), The instantaneous decision network model is represented in the first... The output of the first control cycle k The rating value of the state-action pair for each sample; Step 6.8: Construct the parameter update formula for the instant decision-making network model using equation (22): (twenty two) In equation (22), This represents the learning rate of the instantaneous decision-making network model; This indicates the gradient of the parameters of the real-time evaluation decision model; Step 6.9: Construct the parameter update formulas for the long-term scoring network model and the long-term decision network model using equations (23) and (24), respectively; (twenty three) (twenty four) In equations (23) and (24), Indicates the weight of soft updates; Step 6.10: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would t +1 is assigned to t Then, return to step 6.1 and execute sequentially until the cycle ends. t Reaching the set maximum cycle T This process yields the trained deep reinforcement learning network model.

[0011] The present invention provides an electronic device, including a memory and a processor, characterized in that the memory is used to store a program supporting the processor in executing the highway variable speed limit control method, and the processor is configured to execute the program stored in the memory.

[0012] The present invention discloses a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, performs the steps of the highway variable speed limit control method.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention addresses highway traffic control scenarios and proposes an innovative variable speed limit decision-making method. This method combines entropy weighting with deep reinforcement learning, constructing a comprehensive reward model that dynamically evaluates traffic efficiency and operational safety, thereby achieving intelligent decision-making for speed limit strategies. The control system of this invention can automatically determine the optimal speed limit value for different road segments and at different times based on real-time traffic conditions, thus effectively improving the overall quality of traffic operation.

[0014] 2. Traditional variable speed limit control models typically rely on static rules or preset strategies, which often struggle to achieve optimal control performance in response to rapid changes in traffic flow, conflicting objectives (efficiency versus safety), and vehicle behavior disturbances. This invention, from the perspective of intelligent transportation systems, constructs a dynamic entropy weight evaluation method. This method integrates traffic efficiency and safety indicators into the reward function and automatically adjusts their weights using the entropy weight method, enabling the control system to adaptively change its optimization focus based on actual traffic conditions. This dynamic weighting mechanism significantly improves the system's flexibility and control accuracy, allowing the variable speed limit strategy to adapt to different traffic phases, traffic flow compositions, and external environmental disturbances.

[0015] 3. This invention integrates an entropy weight adjustment mechanism with a deep reinforcement learning training model, enabling the control system to continuously learn the optimal speed limit strategy using a large number of samples in the playback memory. The deep reinforcement learning-based training model performs real-time feedback learning in each control cycle, continuously optimizing the policy network based on comprehensive rewards and Bellman evaluation values. This invention significantly improves the applicability and robustness of variable speed limit control on highways in dynamic environments, not only increasing road traffic efficiency and reducing speed fluctuations and congestion risks, but also greatly enhancing the intelligence level of automated traffic management systems. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a highway scene according to the present invention; Figure 2 This is a diagram showing the relationships and interactions between the various modules within the control system of the present invention. Figure 3 This is a flowchart of the deep reinforcement learning-based variable speed limit control method based on multi-objective dynamic weight allocation, as described in this invention. Detailed Implementation

[0017] In this embodiment, a highway variable speed limit control method based on deep reinforcement learning with multi-objective dynamic weight allocation is proposed, which involves equal-distance speed limits along the main line of the highway according to the direction of vehicle travel. L Deployed I Each roadside unit (RSU) is used to divide the mainline of the highway into sections. I Each sub-segment is numbered as follows: i , The one that merges with the entrance ramp is numbered as follows: Sub-segments and numbered I The sub-segment is defined as the bottleneck area, numbered as The sub-segment is defined as the acceleration zone, before Each sub-segment is defined as a variable speed limit control area; in this embodiment, such as Figure 1 As shown, L The value is 200m. I The value is 6. The first three sub-segments are variable speed limit control areas, sub-segment number 4 is an acceleration area, and sub-segments numbered 5 and 6 are bottleneck areas; for example... Figure 2 and Figure 3 As shown, the variable speed limit control method for highways includes the following steps: Step 1: In the t Traffic information for each sub-road segment is collected during each control cycle, and the traffic information for the first sub-road segment is obtained. t Under each control cycle I Traffic status information for individual road segments In this embodiment, t The duration is 60 seconds.

[0018] Using equation (1) to obtain the first i Each sub-section is in the first t Density under each control cycle Thus, the first t Traffic status information under each control cycle ; (1) In equation (1), For the first i Each road segment is under the current control cycle t The number of vehicles below.

[0019] Step 2: Define the first t Speed ​​limiting action under each control cycle and mixed rewards ; Step 2.1: Define the first t Speed ​​limiting action under each control cycle ,in, Indicates the first t The first control cycle The speed limit value of each variable speed limit control zone; in this embodiment, the speed limit value The value range is [60 km / h, 120 km / h].

[0020] Step 2.2: Use equation (2) to obtain the first... t Bottleneck section under each control cycle Efficiency bonus for internal vehicles : (2) In equation (2), Represented as the first t Each control cycle is numbered as follows I Vehicles in the bottleneck section d speed, Indicates the first t Each control cycle is numbered as follows I The number of all vehicles within the bottleneck section. It is the maximum speed of the vehicle in free-flow conditions; Indicates the first I Each roadside unit (RSU) obtained its number as follows: I Within the bottleneck section t The first control cycle d The vehicle's speed; in this embodiment, the free-flow velocity. Take 120 km / h.

[0021] Step 2.3: Calculate the first step using equation (3). t Each control cycle is numbered as follows I The bottleneck section d Time of vehicle collision ; (3) In equation (3), and They represent the first t Each control cycle is numbered as follows I The bottleneck section d Vehicles and their first The longitudinal position of the vehicle in front. Indicates the length of the vehicle in front. and They represent the first tEach control cycle is numbered as follows I The bottleneck section d Vehicles and their first The speed of the car in front; Indicates the first I The roadside unit RSU obtains the first I Within each sub-section of the road t The first control cycle d The location of the vehicle; Indicates the first I The roadside unit RSU obtains the first I Within each sub-section of the road t The first control cycle d- The position of the vehicle in front; Indicates the first I The roadside unit RSU obtains the first I Within each sub-section of the road t The first control cycle d- The speed of the vehicle in front; in this embodiment, Take 5 m.

[0022] Step 2.4: Calculate the first step using equation (4). t Each control cycle is numbered as follows I Cumulative collision time within bottleneck sections : (4) In equation (4), It is the collision time threshold. Indicates the first t Each control cycle is numbered as follows I The vehicle collision time within the bottleneck section that meets the conditions; in this embodiment, Take 3 seconds.

[0023] Step 2.5: Obtain the first step using equation (5). t Each control cycle is numbered as follows I Safety rewards within bottleneck sections ; (5) Step 2.6: Obtain the first step using equation (6). t Each control cycle is numbered as follows I Mixed rewards within bottleneck sections : (6) In equation (6), and They represent and The weights assigned to each number are all positive, and .

[0024] Step 3: Construct deep reinforcement learning network models, including: immediate decision network model, immediate evaluation network model, long-term decision network model, and long-term evaluation network model; Step 3.1: Construct an instantaneous decision-making network model, including: an input layer containing one neuron, each containing... one neuron The hidden layer consists of one output layer with two neurons; the parameters for initializing the instantaneous decision network model are: In this embodiment, The value is 128. The value is 2.

[0025] Step 3.2: Construct an instant evaluation network model, including: two input layers, each containing one neuron, each containing... one neuron Hidden layers, where the first layer is... The hidden layer is a fully connected layer, and the output layer contains one neuron; the parameters for initializing the instant evaluation network model are: In this embodiment, The value is 128. The value is 2.

[0026] Step 3.3: Construct a long-term decision network model with the same structure as the instantaneous decision network model, and set the parameters of the long-term decision network model... Construct a long-term evaluation network model with the same structure as the instant evaluation network model, and set the parameters of the long-term evaluation network model to be... .

[0027] Step 4: Establish the playback memory sample set ; Step 4.1: The instantaneous decision-making network model receives the first... t Traffic status information under each control cycle Then, an exploratory speed-limited action is generated using equation (7). ; (7) In equation (7), This represents the policy function of an instantaneous decision-making network model. For the first t Random noise under each control cycle.

[0028] Step 4.2: The t Traffic status under each control cycle Execute speed limiting action After that, the first State under each control cycle and with the t Efficiency bonus under each control cycle and security rewards Together as a sample Store to playback memory middle.

[0029] Step 5: Update the weights of the mixed reward using the entropy weight method; Step 5.1: When replaying the memory bank The number of samples stored in it exceeds After that, it indicates that the weight update cycle has been reached, and from... Extract the nearest M The efficiency reward and security reward of the sample constitute the first sample. t Reward matrix under each control period ,in, Indicates the first The first control cycle Efficiency reward per sample Indicates the first The first control cycle The safety reward for each sample; in this embodiment... M The value is 50.

[0030] Step 5.2: Calculate the first step using equations (8) and (9). The first control cycle Efficiency reward probability of a sample and the The probability of a safe reward for a sample : (8) (9) Step 5.3: Calculate the first step using equations (10) and (11). Information entropy of efficiency reward under each control cycle Information entropy of security rewards : (10) (11) Step 5.4: Calculate the first step using equations (12) and (13). The target weight of efficiency reward under each control cycle and the target weight of the security reward function : (12) (13) Step 5.5: Use equations (14) and (15) respectively to... and Perform a smooth update to obtain the weight of the updated efficiency reward. and the weight of the security function and will Assign to , Assign to , (14) (15) In equations (14) and (15), This indicates the proportion of the reward function weights; and They represent the first The weights of efficiency rewards and safety functions under each control cycle, when During initialization ; ; It is a random value between 0 and 1; in this embodiment, Take 0.9.

[0031] Step 6: Train the deep reinforcement learning network model to obtain the trained deep reinforcement learning network model and acquire the current state parameters. This allows the trained real-time decision-making network model to output speed-limiting actions. And assign it to vehicles in each sub-segment within the variable speed limit control area; Step 6.1: When replaying the memory bank The number of samples stored in the batch exceeds the training batch size. Then, from the playback memory bank Extract Sample ,in, Indicates the first The first control cycle k The state of each sample Indicates the first The first control cycle k Speed-limiting actions for each sample, Indicates the first The first control cycle k Efficiency reward per sample Indicates the first The first control cycle k Safety reward for each sample Indicates the first The first control cycle k The state of each sample; in this embodiment, The value is 32.

[0032] Step 6.2: Use equation (16) to obtain the first... The first control cycle k Mixed reward for each sample : (16) Step 6.3: Calculate the first step using equation (17). The first control cycle k Target score for each sample : (17) In equation (17), Indicates the discount factor. The output of the long-term decision network model represents the first... The first control cycle k The strategy for each sample This indicates the output of the long-term evaluation network model on the first... The first control cycle k The rating value of the state-action pair for each sample; in this embodiment, Take 0.9.

[0033] Step 6.4: Calculate the instant evaluation network model in the first step using equation (18). The output of the first control cycle k The rating of a state-action pair for each sample : (18) In equation (18), The output of the instantaneous decision network model represents the first... The first control cycle k The strategy for each sample This indicates the immediate evaluation of the network model output for the first... The first control cycle k The rating of a state-action pair for each sample.

[0034] Step 6.5: Construct an instant evaluation network model using equation (19) in the first step. Loss function under each control cycle : (19) Step 6.6: Construct the parameter update formula for the instant evaluation network model using equation (20): (20) In equation (20), This indicates the learning rate for real-time evaluation of the network model; This indicates calculating the gradient of the parameters of the instant evaluation network model. This indicates assignment; in this embodiment, Take 0.0002.

[0035] Step 6.7: Construct an instant decision-making network model using equation (21) in the first step. Objective function under each control cycle : (twenty one) In equation (21), The instantaneous decision network model is represented in the first... The output of the first control cycle k The rating of a state-action pair for each sample.

[0036] Step 6.8: Construct the parameter update formula for the instant decision-making network model using equation (22): (twenty two) In equation (22), This represents the learning rate of the instantaneous decision-making network model; This represents calculating the gradient of the parameters of the instant evaluation decision model; in this embodiment, Take 0.0005.

[0037] Step 6.9: Construct the parameter update formulas for the long-term scoring network model and the long-term decision network model using equations (23) and (24), respectively; (twenty three) (twenty four) In equations (23) and (24), This represents the weight of the soft update; in this embodiment, Take 0.01.

[0038] Step 6.10: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would t +1 is assigned to t Then, return to step 6.1 and execute sequentially until the cycle ends. t Reaching the set maximum cycle T Thus, a trained deep reinforcement learning network model is obtained; in this embodiment, T The duration is set to 18000 seconds.

[0039] Step 6.11: Utilize the trained real-time decision network to output corresponding speed limiting actions based on the real-time input state information, and assign them to each variable speed limiting control unit for execution.

[0040] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0041] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. A method for variable speed limit control on highways based on deep reinforcement learning with multi-objective dynamic weight allocation, characterized in that, On the main line of the highway, at equal intervals according to the direction of vehicle travel. L Deployed I Each roadside unit (RSU) is used to divide the mainline of the highway into sections. I Each sub-segment is numbered as follows: i , The one that merges with the entrance ramp is numbered as follows: Sub-segments and numbered I The sub-segment is defined as the bottleneck area, numbered as The sub-segment is defined as the acceleration zone, before Each sub-segment is defined as a variable speed limit control area; the variable speed limit control method for highways includes the following steps: Step 1: In the t Traffic information for each sub-road segment is collected during each control cycle, and the traffic information for the first sub-road segment is obtained. t Under each control cycle I Traffic status information for individual road segments ; Using equation (1) to obtain the first i Each sub-section is in the first t Density under each control cycle Thus, the first t Traffic status information under each control cycle ; (1) In equation (1), For the first i Each road segment is under the current control cycle t The number of vehicles below; Step 2: Define the first t Speed ​​limiting action under each control cycle and mixed rewards ; Step 3: Construct deep reinforcement learning network models, including: immediate decision network model, immediate evaluation network model, long-term decision network model, and long-term evaluation network model; Step 4: Establish the playback memory sample set ; Step 5: When replaying the memory bank The number of samples stored in it exceeds Then, it indicates that the weight update cycle has been reached, and the weights of the mixed reward are updated using the entropy weight method; Step 6: Train the deep reinforcement learning network model to obtain the trained deep reinforcement learning network model, and then use the trained instant decision network model to output the corresponding speed limiting action based on the real-time input state information.

2. The method for variable speed limit control on highways based on deep reinforcement learning with multi-objective dynamic weight allocation according to claim 1, characterized in that, Step 2 includes: Step 2.1: Define the first t Speed ​​limiting action under each control cycle ,in, Indicates the first t The first control cycle Speed ​​limit values ​​for each variable speed limit control zone; Step 2.2: Use equation (2) to obtain the first... t Bottleneck section under each control cycle Efficiency bonus for internal vehicles : (2) In equation (2), Represented as the first t Each control cycle is numbered as follows I Vehicles in the bottleneck section d speed, Indicates the first t Each control cycle is numbered as follows I The number of all vehicles within the bottleneck section. It is the maximum speed of the vehicle in free-flow conditions; Indicates the first I Each roadside unit (RSU) was assigned the number as follows: I Within the bottleneck section t The first control cycle d The speed of the vehicle; Step 2.3: Calculate the first step using equation (3). t Each control cycle is numbered as follows I The bottleneck section d Time of vehicle collision ; (3) In equation (3), and They represent the first t Each control cycle is numbered as follows I The bottleneck section d Vehicles and their first The longitudinal position of the vehicle in front. Indicates the length of the vehicle in front. and They represent the first t Each control cycle is numbered as follows I The bottleneck section d Vehicles and their first The speed of the car in front; Indicates the first I The roadside unit RSU obtains the first I Within each sub-section of the road t The first control cycle d The location of the vehicle; Indicates the first I The roadside unit RSU obtains the first I Within each sub-section of the road t The first control cycle d- The position of the vehicle in front; Indicates the first I The roadside unit RSU obtains the first I Within each sub-section of the road t The first control cycle d- The speed of the car in front; Step 2.4: Calculate the first step using equation (4). t Each control cycle is numbered as follows I Cumulative collision time within bottleneck sections : (4) In equation (4), It is the collision time threshold. Indicates the first t Each control cycle is numbered as follows I The time between vehicle collisions that meet the conditions within the bottleneck section of the road; Step 2.5: Obtain the first step using equation (5). t Each control cycle is numbered as follows I Safety rewards within bottleneck sections ; (5) Step 2.6: Obtain the first step using equation (6). t Each control cycle is numbered as follows I Mixed rewards within bottleneck sections : (6) In equation (6), and They represent and The weights assigned to each number are all positive, and .

3. The method for variable speed limit control on highways based on deep reinforcement learning with multi-objective dynamic weight allocation according to claim 2, characterized in that, Step 3 includes: Step 3.1: Construct an instantaneous decision-making network model, including: an input layer containing one neuron, each containing... one neuron The hidden layer consists of one output layer with two neurons; the parameters for initializing the instantaneous decision network model are: ; Step 3.2: Construct an instant evaluation network model, including: two input layers, each containing one neuron, each containing... one neuron Hidden layers, where the first layer is... The hidden layer is a fully connected layer, and the output layer contains one neuron; the parameters for initializing the instant evaluation network model are: ; Step 3.3: Construct a long-term decision network model with the same structure as the instantaneous decision network model, and set the parameters of the long-term decision network model... Construct a long-term evaluation network model with the same structure as the instant evaluation network model, and set the parameters of the long-term evaluation network model to be... .

4. The method for variable speed limit control on highways based on deep reinforcement learning with multi-objective dynamic weight allocation according to claim 3, characterized in that, Step 4 includes: Step 4.1: The instantaneous decision-making network model receives the first... t Traffic status information under each control cycle Then, an exploratory speed-limited action is generated using equation (7). ; (7) In equation (7), This represents the policy function of an instantaneous decision-making network model. For the first t Random noise under each control cycle; Step 4.2: The t Traffic status under each control cycle Execute speed limiting action After that, the first State under each control cycle and with the t Efficiency bonus under each control cycle and security rewards Together as a sample Store to playback memory middle.

5. The method for variable speed limit control on highways based on deep reinforcement learning with multi-objective dynamic weight allocation according to claim 4, characterized in that, Step 5 includes: Step 5.1: From Extract the nearest M The efficiency reward and security reward of the sample constitute the first sample. t Reward matrix under each control period ,in, Indicates the first The first control cycle Efficiency reward per sample Indicates the first The first control cycle Safety reward for each sample; Step 5.2: Calculate the first step using equations (8) and (9). The first control cycle Efficiency reward probability of a sample and the The probability of a safe reward for a sample : (8) (9) Step 5.3: Calculate the first step using equations (10) and (11). Information entropy of efficiency reward under each control cycle Information entropy of security rewards : (10) (11) Step 5.4: Calculate the first step using equations (12) and (13). The target weight of efficiency reward under each control cycle and the target weight of the security reward function : (12) (13) Step 5.5: Use equations (14) and (15) respectively to... and Perform a smooth update to obtain the weight of the updated efficiency reward. and the weight of the security function and will Assign to , Assign to , (14) (15) In equations (14) and (15), This indicates the proportion of the reward function weights; and They represent the first The weights of efficiency rewards and safety functions under each control cycle, when During initialization ; ; It is a random value between 0 and 1.

6. The method for variable speed limit control on highways based on deep reinforcement learning with multi-objective dynamic weight allocation according to claim 5, characterized in that, Step 6 includes: Step 6.1: When replaying the memory bank The number of samples stored in the batch exceeds the training batch size. Then, from the playback memory bank Extract Sample ,in, Indicates the first The first control cycle k The state of each sample Indicates the first The first control cycle k Speed-limiting actions for each sample, Indicates the first The first control cycle k Efficiency reward per sample Indicates the first The first control cycle k Safety reward for each sample Indicates the first The first control cycle k The state of each sample; Step 6.2: Use equation (16) to obtain the first... The first control cycle k Mixed reward for each sample : (16) Step 6.3: Calculate the first step using equation (17). The first control cycle k Target score for each sample : (17) In equation (17), Indicates the discount factor. The output of the long-term decision network model represents the first... The first control cycle k The strategy for each sample This indicates the output of the long-term evaluation network model on the first... The first control cycle k The rating of the state-action pair for each sample; Step 6.4: Calculate the instant evaluation network model in the first step using equation (18). The output of the first control cycle k The rating of a state-action pair for each sample : (18) In equation (18), The output of the instantaneous decision network model represents the first... The first control cycle k The strategy for each sample This indicates the immediate evaluation of the network model output for the first... The first control cycle k The rating of the state-action pair for each sample; Step 6.5: Construct an instant evaluation network model using equation (19) in the first step. Loss function under each control cycle : (19) Step 6.6: Construct the parameter update formula for the instant evaluation network model using equation (20): (20) In equation (20), This indicates the learning rate for real-time evaluation of the network model; This indicates calculating the gradient of the parameters of the instant evaluation network model. This indicates assignment; Step 6.7: Construct an instant decision-making network model using equation (21) in the first step. Objective function under each control cycle : (21) In equation (21), The instantaneous decision network model is represented in the first... The output of the first control cycle k The rating of the state-action pair for each sample; Step 6.8: Construct the parameter update formula for the instant decision-making network model using equation (22): (22) In equation (22), This represents the learning rate of the instantaneous decision-making network model; This indicates the gradient of the parameters of the real-time evaluation decision model; Step 6.9: Construct the parameter update formulas for the long-term scoring network model and the long-term decision network model using equations (23) and (24), respectively; (23) (24) In equations (23) and (24), Indicates the weight of soft updates; Step 6.10: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would t +1 is assigned to t Then, return to step 6.1 and execute sequentially until the cycle ends. t Reaching the set maximum cycle T This process yields the trained deep reinforcement learning network model.

7. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the highway variable speed limit control method according to any one of claims 1-6, and the processor is configured to execute the program stored in the memory.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the highway variable speed limit control method according to any one of claims 1-6.

Images