A single ramp control method for highways considering hard shoulder opening based on reinforcement learning

Abstract

The present application relates to a method for controlling a single ramp on a highway based on reinforcement learning and considering the opening of a hard shoulder. The method includes: constructing a state space of a decision-making agent according to the changes in the traffic flow motion state of the highway and considering the opening of the hard shoulder, defining the current phase of the ramp signal light as the action space of the decision-making agent, defining the change value after squaring the sum of the ramp and the main lane queue as the reward function of the decision-making agent, using a reinforcement learning algorithm enhanced by rule knowledge to train the decision-making agent, training until the reward value obtained by calculating the reward function fluctuates within a preset range, obtaining a trained decision-making agent, obtaining the real-time traffic flow motion state of the highway and considering the opening of the hard shoulder as state data, inputting it into the trained decision-making agent for analysis and outputting a ramp control strategy to control the ramp of the highway. The overall operating efficiency of the transportation system is effectively enhanced.

Description

Technical Field

[0001] The present application relates to the field of intelligent connected transportation technology, and in particular to a single-ramp control method for highways based on reinforcement learning and considering the opening of hard shoulders. Background Art

[0002] Expressways are roads designed for high-speed driving of cars. As expressways develop towards intelligence, the scientific and technological concept of "smart expressways" has gradually taken shape. Smart expressways propose to introduce Internet thinking and technology to reconstruct and rebuild the electromechanical systems and management services of traditional expressways. The initial construction task is to improve the level of expressway operation management and travel service quality through information exchange and sharing, data fusion and mining, and realize the informatization and intelligence of provincial expressway monitoring and management, emergency command, decision-making assistance, business handling, travel guidance and other services.

[0003] In current highway traffic scenarios, the congested bottleneck section formed by ramp merging is one of the main reasons affecting the traffic efficiency and traffic safety of highways. Traditional ramp control strategies are mostly static control and cannot be dynamically adjusted according to the real-time traffic flow operation status, resulting in low overall operation efficiency of the highway traffic system. Summary of the invention

[0004] Based on this, it is necessary to provide a highway single ramp control method based on reinforcement learning and considering the opening of hard shoulders to address the above technical problems, which can improve the overall operating efficiency of the highway traffic system.

[0005] A single ramp control method for a highway considering hard shoulder opening based on reinforcement learning, the method comprising:

[0006] According to the changes in the traffic flow movement state of the highway and considering the opening of the hard shoulder, the state space of the decision-making intelligent agent is constructed;

[0007] The current phase of the ramp signal light is defined as the action space of the decision-making agent, and the effective control domain of the ramp signal light is dynamically adjusted;

[0008] The change value after taking the square of the sum of the ramp and main lane queues is defined as the reward function of the decision-making agent;

[0009] The decision-making agent is trained by using a reinforcement learning algorithm enhanced by rule knowledge, until the reward value obtained by calculating the reward function fluctuates within a preset range, thereby obtaining a trained decision-making agent;

[0010] Obtain the traffic flow movement state of the real-time expressway and consider the hard shoulder opening situation as state data, input it into the trained decision-making agent for analysis and output the ramp control strategy, so as to control the ramp of the expressway according to the ramp control strategy.

[0011] In one embodiment, the state space s includes 6 state data, and the expression of the state space s is:

[0012] s = [s1, s2, s3, s4, s5, s6] T

[0013] Among them, s1 represents the direction with greater traffic pressure in the ramp merging area, s1 ∈ {1, 2}. Assuming that the total travel delays of vehicles in the main line direction and the ramp direction in the ramp merging area are d1 and d2 respectively, when d1 > d2, s1 = 1, indicating that the traffic pressure in the main line lane of the ramp merging area is greater; when d1 < d2, s1 = 2, indicating that the traffic pressure in the ramp lane of the ramp merging area is greater; s2 represents the current phase of the ramp signal light, s2 ∈ {1, 2}, s2 = 1 represents the green light phase, allowing ramp vehicles to merge into the main line, and s2 = 2 represents the red light phase, not allowing ramp vehicles to merge into the main line; s3 represents the queue length level upstream of the main line lane, s3 ∈ {1, 2, 3, 4, 5}, Among them, Q main represents the ratio of the upstream queue length on the main line to the length of the signal light control area; s4 represents the queue length level of the ramp lane, s4 ∈ {1, 2, 3, 4, 5}, Among them, Q ramp represents the ratio of the ramp queue length to the defined effective control length of the ramp; s5 represents whether the hard shoulder downstream of the ramp merging area is temporarily open, s5 ∈ {1, 2}. When s5 = 1, it means that the hard shoulder downstream of the ramp merging area is temporarily open to relieve the congestion of the traffic bottleneck in the merging area; when s5 = 2, it means that the hard shoulder downstream of the ramp merging area is not temporarily open. s6 represents the opening range of the hard shoulder downstream of the ramp merging area, s6 ∈ {1, 2, 3, 4, 5, 6, 7, 8, 9}, Among them, x refers to the distance between the hard shoulder opening starting point and the merging point, and y refers to the opening length of the hard shoulder.

[0014] In one embodiment, the ratio Q main of the upstream queue length on the main line to the length of the signal light control area is expressed as:

[0015]

[0016] D main = V main ·T control

[0017] Among them, q main represents the upstream queue length of the main line, D main Indicates the length of the control domain in the main line direction, V main represents the average speed of vehicles on the main line, T control Indicates the control time interval.

[0018] In one embodiment, the ratio Q of the ramp queue length to the defined ramp effective control length is ramp The expression is:

[0019]

[0020]

[0021] Among them, q ramp represents the ramp queue length, D ramp represents the length of the control domain in the ramp direction, V ramp represents the average speed of vehicles on the ramp, T control RAMP LENGTH Indicates the length of the ramp lane.

[0022] In one embodiment, the action space A is expressed as:

[0023] A={RAMP,MAIN}

[0024] Among them, RAMP means that vehicles from the ramp direction can pass; MAIN means that vehicles from the ramp direction cannot pass.

[0025] In one embodiment, the reward function r(s,a) is expressed as:

[0026]

[0027] in, represents the queue length of the main lane at time t, represents the queue length of the ramp at t, represents the queue length of the main lane at time t-1, Represents the queue length of the ramp at t-1.

[0028] In one embodiment, the reinforcement learning algorithm enhanced by rule knowledge is used to train the decision agent to a fluctuation range of the reward value of the reward function to obtain a trained decision agent, including:

[0029] At the beginning of training, the rule-based ALINEA control algorithm is first used to control the traffic volume merging from the ramp to the main line, and the state data, ramp control strategy, reward value and state data after control are collected and stored in the experience pool as sample data;

[0030] After the sample data in the experience pool reaches a preset number, the sample data in the experience pool is used to train the decision-making agent based on reinforcement learning, and the state data, ramp control strategy, reward value and state data after regulation generated during the training of the decision-making agent are also stored in the experience pool as sample data;

[0031] The sample data in the experience pool is recycled to train the decision-making agent based on reinforcement learning until the reward value calculated by the reward function fluctuates within a preset range, thereby obtaining a trained decision-making agent.

[0032] The above-mentioned single ramp control method for highways based on reinforcement learning and considering the opening of hard shoulders, constructs the state space of the decision-making agent according to the changes in the traffic flow movement state of the highway and the opening of the hard shoulders, defines the change of the current phase of the ramp signal light as the action space of the decision-making agent, dynamically adjusts the effective control domain of the ramp signal light, and defines the change value after the square of the sum of the ramp and the main lane queue as the reward function of the decision-making agent, and uses the reinforcement learning algorithm enhanced by rule knowledge to train the decision-making agent until the reward value calculated by the reward function fluctuates within a preset range, obtains the trained decision-making agent, obtains the real-time traffic flow movement state of the highway and considers the opening of the hard shoulders, and inputs it into the trained decision-making agent for analysis and output of the ramp control strategy, so as to control the ramp of the highway according to the ramp control strategy. The synergy of ramp control and hard shoulder opening is achieved, and dynamic adjustments are made in real time according to the traffic flow operation status, which effectively enhances the overall operation efficiency of the transportation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a flow chart of a single ramp control method for a highway considering hard shoulder opening based on reinforcement learning in one embodiment;

[0034] Figure 2 A schematic diagram of various areas of a highway traffic scene in one embodiment;

[0035] Figure 3 A schematic diagram of action switching in an embodiment;

[0036] Figure 4 A schematic diagram of an action RAMP in one embodiment;

[0037] Figure 5is a schematic diagram of action MAIN in one embodiment;

[0038] Figure 6 A schematic diagram of dual experience pool training logic in one embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0040] In view of the congestion bottleneck caused by ramp merging in highway traffic scenarios, its impact on traffic efficiency and traffic safety is minimized as much as possible. This application proposes a highway single ramp control method based on reinforcement learning and considering the opening of hard shoulders, which combines ramp flow control with the opening of hard shoulders of roads, sets a ramp control strategy based on reinforcement learning algorithms, and dynamically adjusts the traffic flow from the ramp to the main road according to the real-time operating status of the merging area.

[0041] In one embodiment, Figure 1 As shown, a single ramp control method for highways considering the opening of hard shoulders based on reinforcement learning is provided. The method is applied to a terminal as an example for explanation, and includes the following steps:

[0042] Step S220, constructing the state space of the decision-making agent according to the change of the traffic flow movement state of the expressway and considering the opening of the hard shoulder.

[0043] Among them, Figure 2 The schematic diagram of each area of ​​the highway traffic scene shown includes the main line upstream environment state perception area, the main line downstream environment state perception area and the ramp environment state perception area.

[0044] In one embodiment, the state space s includes 6 state data, and the expression of the state space s is:

[0045] s=[s1,s2,s3,s4,s5,s6] T

[0046] Among them, s1 represents the direction with relatively high traffic pressure in the ramp merging area, s1 ∈ {1, 2}. Assume that the total travel delays of vehicles in the main line direction and the ramp direction in the ramp merging area are d1 and d2 respectively. When d1 > d2, s1 = 1, indicating that the traffic pressure in the main line lane of the ramp merging area is relatively high; when d1 < d2, s1 = 2, indicating that the traffic pressure in the ramp lane of the ramp merging area is relatively high; s2 represents the current phase of the ramp signal, s2 ∈ {1, 2}, s2 = 1 represents the green light phase, allowing ramp vehicles to merge into the main line, and s2 = 2 represents the red light phase, not allowing ramp vehicles to merge into the main line; s3 represents the queue length level upstream of the main line lane, s3 ∈ {1, 2, 3, 4, 5}, Among them, Q main represents the ratio of the upstream queue length on the main line to the signal control domain length; s4 represents the queue length level of the ramp lane, s4 ∈ {1, 2, 3, 4, 5}, Among them, Q ramp represents the ratio of the ramp queue length to the defined effective control length of the ramp; s5 represents whether the hard shoulder downstream of the ramp merging area is temporarily open, s5 ∈ {1, 2}. When s5 = 1, it indicates that the hard shoulder downstream of the ramp merging area is temporarily open to relieve the congestion of the traffic bottleneck in the merging area; when s5 = 2, it indicates that the hard shoulder downstream of the ramp merging area is not temporarily open. s6 represents the opening range of the hard shoulder downstream of the ramp merging area, s6 ∈ {1, 2, 3, 4, 5, 6, 7, 8, 9}, Among them, x refers to the distance between the starting point of the hard shoulder opening and the merging point, and y refers to the opening length of the hard shoulder.

[0047] Among them, the ramp merging area operation imbalance model is used to characterize the unbalanced operation trend of local traffic flow, so as to distinguish the difference degree of the traffic states of the main line and the ramp at the ramp intersection. Therefore, the direction with relatively high traffic pressure in the ramp merging area is used as one of the state components in the state space.

[0048] In an example, the values of x and y can be (0, 1000, 2000).

[0049] In an embodiment, the ratio Q main of the upstream queue length on the main line to the signal control domain length

[0050]

[0051] D main = V main ·T control

[0052] Among them, q main represents the upstream queue length on the main line, D mainIndicates the length of the control domain in the main line direction, V main represents the average speed of vehicles on the main line, T control Indicates the control time interval.

[0053] In one embodiment, the ratio Q of the ramp queue length to the defined ramp effective control length is ramP The expression is:

[0054]

[0055]

[0056] Among them, q ramp represents the ramp queue length, D ramp represents the length of the control domain in the ramp direction, V ramp represents the average speed of vehicles on the ramp, T control RAMP LENGTH Indicates the length of the ramp lane.

[0057] Step S240, changing the current phase of the ramp signal light is defined as the action space of the decision-making agent, and dynamically adjusting the effective control domain of the ramp signal light.

[0058] In one embodiment, the expression of the action space A is:

[0059] A={RAMP,MAIN}

[0060] Among them, RAMP means that vehicles from the ramp direction can pass; MAIN means that vehicles from the ramp direction cannot pass.

[0061] Here, the action is defined as changing the current phase of the ramp signal. Figure 3 As shown in the figure, changing the current phase of the ramp signal can be that the same action is selected in the two decisions, that is, the current phase is 1, and the next phase is still phase 1, or the current phase is 2, and the next phase is still phase 2; the actions selected in the two decisions are different, that is, the current phase is 1, and the next phase is switched to phase 2, or the current phase is 2, and the next phase is switched to phase 1. If the decision-making agent selects the same action in two adjacent decisions, the phase will continue to maintain a cycle. If the two adjacent actions are inconsistent, a yellow signal stage (duration 3 seconds) will be added between the two actions.

[0062] In one example, when the action of action space A is RAMP, the schematic diagram of action RAMP is as follows: Figure 4 As shown, when the action in action space A is MAIN, the schematic diagram of action MAIN is as follows Figure 5 shown.

[0063] Step S260, defining the change in the square of the sum of the ramp and main lane queues as the reward function of the decision-making agent.

[0064] It should be understood that under the same traffic capacity, the lower the traffic density, the higher the average speed of vehicles will be, thereby effectively reducing travel delays. Therefore, the agent's reward function can be defined as the change in the square of the sum of the ramp and main lane queues.

[0065] In one embodiment, the reward function r(s,a) is expressed as:

[0066]

[0067] in, represents the queue length of the main lane at time t, represents the queue length of the ramp at t, represents the queue length of the main lane at time t-1, Represents the queue length of the ramp at t-1.

[0068] Among them, vehicles with a speed lower than 10km / h can be set as congested queue vehicles to balance the queue lengths at different stages.

[0069] Step S280, using a reinforcement learning algorithm enhanced by rule knowledge to train the decision-making agent, until the reward value obtained by calculating the reward function fluctuates within a preset range, and a trained decision-making agent is obtained.

[0070] Among them, the reinforcement learning algorithm enhanced by rule knowledge can be to use the rule knowledge of the traditional ramp control strategy, priority experience playback and joint training mechanism to enhance the robustness and convergence speed of the control strategy based on reinforcement learning training. On the one hand, the rule knowledge of the traditional ramp control strategy can guide the decision-making agent to a relatively reasonable training direction in the early stage of training, reduce the ineffective exploration of the decision-making agent as much as possible, and improve the convergence speed of the algorithm; on the other hand, the traditional control strategy based on rule knowledge can verify the effectiveness of the decision of the reinforcement learning training control strategy, play a "correction" role, ensure the basic lower limit effect of the ramp control strategy, and enable it to output relatively stable control decisions. Therefore, it can be understood that rule knowledge strengthens the constraints for reinforcement learning and avoids unreasonable actions. It enables the network to not only learn the dynamic rules of the environment, but also use these rules to guide the decision-making process. The advantage of this method is that it can make full use of human knowledge and experience to improve the accuracy and efficiency of decision-making.

[0071] In one embodiment, a decision agent is trained using a reinforcement learning algorithm enhanced by rule knowledge, and the training is performed to a fluctuation range of a reward value of a reward function to obtain a trained decision agent, including:

[0072] In the initial stage of training, the rule-based ALINEA control algorithm is first used to control the traffic volume merging from the ramp into the main line, and the state data, ramp control strategy, reward value and state data after control are collected and stored in the experience pool as sample data; after the sample data in the experience pool reaches the preset number, the sample data in the experience pool is used to train the decision-making agent based on reinforcement learning, and the state data, ramp control strategy, reward value and state data after control generated during the training of the decision-making agent are also stored in the experience pool as sample data; the sample data in the experience pool is recycled to train the decision-making agent based on reinforcement learning until the reward value obtained by calculating the reward function fluctuates within the preset range, thereby obtaining a trained decision-making agent.

[0073] It should be understood that the experience pool and training mechanism based on rule knowledge and trial and error mechanism are introduced to improve the robustness and practicality of the algorithm.

[0074] Among them, after the environment is initialized, at the beginning of training, the control strategy of the rule-based ALINEA control algorithm is first used to regulate the traffic volume merging from the ramp into the main line, and the corresponding state and control decision after processing are input into the experience pool, and the sample data in this experience pool is used to train the decision-making agent. At the same time, the experience data generated by the control strategy based on reinforcement learning training is also input into the experience pool, gradually reducing the impact of the experience data generated by the control strategy of the rule-based ALINEA control algorithm on the decision-making agent, so that the control strategy proposed in this application can show more flexibility to demand fluctuations in traffic scenarios while ensuring the basic control effect, and finally improve the robustness and convergence speed of the reinforcement learning control algorithm.

[0075] In one embodiment, dual experience pools can also be used to store experience data obtained by the control strategy of the rule-based ALINEA control algorithm and experience data obtained by the control strategy based on reinforcement learning. Figure 6As shown in the figure, in training, the agent that can collect the experience data collected by the strategy in each thread is called a learning agent, and the agent that actually interacts with the environment to collect experience in each thread is called an executive agent. In order to realize the rule knowledge enhancement mechanism, two executive agents are instantiated, namely rule-driven agents and data-driven agents. In the early stage of training, the rule-driven agent first interacts with the environment to obtain experience data and stores it in the rule knowledge experience pool; then the learning agent uses the data in the rule experience knowledge experience pool for preliminary training to avoid inefficient exploration in the early stage of training; thereafter, the data-driven agent copies the neural network parameters of the learning agent, starts to interact with the environment to obtain experience data, and stores it in the trial and error mechanism experience pool, after which the learning agent starts to use the data in the experience pool for learning and training, and at the same time, the data-driven agent iteratively copies the neural network parameters of the learning agent.

[0076] Step S300, obtain the real-time traffic flow movement status of the highway and consider the hard shoulder opening status as status data, input it into the trained decision-making intelligent agent for analysis and output of the ramp control strategy, so as to control the ramp of the highway according to the ramp control strategy.

[0077] In one example, when a trained decision-making agent is used in ramp control on a highway, the state data, ramp control strategy, reward value and regulated state data generated can also be stored in the experience pool as sample data. By continuously enriching the data in the experience pool, the decision-making agent can be further trained and the network parameters of the decision-making agent can be continuously optimized to enable the decision-making agent to have better performance.

[0078] It should be understood that the decision-making intelligent agent trained by the reinforcement learning algorithm enhanced by rule knowledge can dynamically adjust the traffic flow from the ramp to the main road according to the real-time operating status of the highway merging area. Further combining the ramp flow control with the opening of the hard shoulder of the road, the opening of the hard shoulder is included in the environmental state information, so that the reinforcement learning algorithm can combine the current shoulder opening status to more effectively regulate the merging of ramp vehicles and maximize the use of the increased traffic capacity of the hard shoulder.

[0079] The above-mentioned single ramp control method for highways based on reinforcement learning and considering the opening of hard shoulders constructs the state space of the decision-making agent according to the changes in the traffic flow movement state of the highway and the opening of the hard shoulders, defines the change of the current phase of the ramp signal as the action space of the decision-making agent, dynamically adjusts the effective control domain of the ramp signal, and defines the change value after squared sum of the ramp and main lane queues as the reward function of the decision-making agent. The decision-making agent is trained by the reinforcement learning algorithm enhanced by rule knowledge until the reward value obtained by the reward function calculation fluctuates within a preset range, and the trained decision-making agent is obtained. The real-time traffic flow movement state of the highway and the opening of the hard shoulders are obtained as state data, which are input into the trained decision-making agent for analysis and output of the ramp control strategy, so as to control the ramp of the highway according to the ramp control strategy. The synergy of ramp control and hard shoulder opening is achieved, and dynamic adjustments are made in real time according to the traffic flow operation status, which effectively enhances the overall operation efficiency of the transportation system.

[0080] This application combines ramp control with the opening of the hard shoulder. On the one hand, it can make use of the improvement in operating capacity brought about by the opening of the hard shoulder to release more ramp vehicles to merge into the main line while ensuring the traffic efficiency of the merging area, so as to ensure the basic right of way of ramp vehicles; on the other hand, the opening of the hard shoulder is used as part of the environmental state information of the decision-making intelligent body, so that the ramp control strategy based on reinforcement learning can fully combine its different opening conditions to specifically regulate the traffic volume merging from the ramp into the main line, so as to achieve a control effect that is neither "conservative" nor "aggressive".

[0081] This application proposes a control strategy based on reinforcement learning training for the single ramp control problem, so that it can output control decisions that can cope with fluctuating traffic demand scenarios without traffic flow modeling; at the same time, in response to the limitations of the reinforcement learning algorithm, this application constructs a dual experience pool of rule knowledge and trial-and-error mechanism to guide it in a relatively reasonable exploration direction in the early stage of strategic intelligent agent training, thereby improving the convergence speed and stability of the algorithm; in addition, the training and priority experience replay mechanism is used to improve the sample sampling efficiency of the algorithm.

[0082] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0083] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A highway single ramp control method considering hard shoulder opening based on reinforcement learning, characterized in that: The method comprises: According to the changes in the traffic flow movement state of the highway and considering the opening of the hard shoulder, the state space of the decision-making intelligent agent is constructed; The current phase of the ramp signal light is defined as the action space of the decision-making agent, and the effective control domain of the ramp signal light is dynamically adjusted; The change value after taking the square of the sum of the ramp and main lane queues is defined as the reward function of the decision-making agent; The decision-making agent is trained by using a reinforcement learning algorithm enhanced by rule knowledge, until the reward value obtained by calculating the reward function fluctuates within a preset range, thereby obtaining a trained decision-making agent; The real-time traffic flow movement status of the highway and the opening status of the hard shoulder are obtained as status data, which are input into the trained decision-making intelligent agent for analysis and output of the ramp control strategy, so as to control the ramp of the highway according to the ramp control strategy.

2. The method according to claim 1, characterized in that The state space s Includes 6 state data, the state space s The expression is: ， in, Indicates the direction of greater traffic pressure in the ramp merging area. , assuming that the total travel delays of vehicles in the mainline direction and ramp direction of the ramp merging area are , ,when hour, , indicating that the traffic pressure on the main lane of the ramp merging area is relatively high. hour, , indicating that the traffic pressure on the ramp lanes in the ramp merging area is relatively high; Indicates the current phase of the ramp signal. , Represents the green light phase, allowing ramp vehicles to merge into the main line. Represents the red light phase, which does not allow ramp vehicles to merge into the main line; represents the queue length level upstream of the main lane, , ,in, Represents the ratio of the upstream queue length of the main line to the length of the signal control domain; represents the queue length level of the ramp lane, , ,in, It represents the ratio of the ramp queue length to the defined ramp effective control length; Indicates whether the hard shoulder downstream of the ramp merge area is temporarily open. ,when When , it means that the hard shoulder downstream of the ramp merging area is temporarily opened to relieve the congestion of the traffic bottleneck in the merging area; when When , it means that the hard shoulder downstream of the ramp merge area is not temporarily opened. Indicates the open area of ​​the hard shoulder downstream of the ramp merging area. , ,in, x Refers to the distance between the opening start of the hard shoulder and the merging point. y Refers to the open length of the hard shoulder.

3. The method according to claim 2, characterized in that The ratio of the upstream queue length of the main line to the length of the signal light control area The expression is: , ; in, represents the upstream queue length of the main line, Indicates the length of the control field in the main line direction. represents the average speed of vehicles on the main line, Indicates the control time interval.

4. The method according to claim 2, characterized in that: The ratio of the ramp queue length to the defined ramp effective control length The expression is: , ; in, represents the ramp queue length, Indicates the length of the control domain in the ramp direction, represents the average speed of vehicles on the ramp, Indicates the control time interval, Indicates the length of the ramp lane.

5. The method according to claim 1, characterized in that The action space A The expression is: ; in, Indicates that vehicles coming from the ramp direction can pass; It means that vehicles coming from the ramp direction cannot pass.

6. The method according to claim 1, characterized in that The reward function The expression is: ; in, Represents the main lane t The queue length at the time, Represents the ramp t The queue length at the time, Represents the main lane t -1 queue length at time, Represents the ramp t -1 The queue length at time.

7. The method according to claim 1, characterized in that The method of using a reinforcement learning algorithm enhanced by rule knowledge to train the decision agent to a fluctuation range of the reward value of the reward function to obtain a trained decision agent includes: At the beginning of training, the rule-based ALINEA control algorithm is first used to control the traffic volume merging from the ramp to the main line, and the state data, ramp control strategy, reward value and state data after control are collected and stored in the experience pool as sample data; After the sample data in the experience pool reaches a preset number, the sample data in the experience pool is used to train the decision-making agent based on reinforcement learning, and the state data, ramp control strategy, reward value and state data after regulation generated during the training of the decision-making agent are also stored in the experience pool as sample data; The sample data in the experience pool is recycled to train the decision-making agent based on reinforcement learning until the reward value calculated by the reward function fluctuates within a preset range, thereby obtaining a trained decision-making agent.

Images