Intersection reinforcement learning control method, device and medium in mixed traffic environment

Abstract

The application discloses a mixed traffic environment intersection reinforcement learning control method, equipment and medium, the application introduces CAV special phase and CAV special lane of free lane direction, through the first intelligent agent adjusts CAV special lane mode according to CAV penetration, the second intelligent agent executes corresponding signal light phase scheme according to CAV special lane mode, and dynamically adjusts signal light state according to real-time traffic flow, when signal light is in CAV special phase, the third intelligent agent dynamically determines the right of way of lane according to the current CAV traffic state, and releases vehicles based on the right of way;Through this control method, the lane mode and signal light state can be dynamically optimized under different CAV penetration, and the vehicle passage is flexibly coordinated under CAV special phase, compared with dynamic signal light and traditional fixed signal light, the average delay time and average queue length of vehicles are reduced, and the passage efficiency is increased.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, specifically to a method, device, and medium for reinforcement learning control of intersections in mixed traffic environments. Background Technology

[0002] Road intersections serve as core hubs and bottlenecks in urban transportation. When vehicles turn at these intersections, traffic demand is higher than on adjacent roads, potentially leading to conflicts between vehicles traveling in different directions, reducing capacity, and further exacerbating congestion. Therefore, optimizing intersection traffic management and improving capacity are crucial for alleviating traffic congestion.

[0003] Currently, intersections mainly use traffic lights with fixed cycles for control. However, dynamic changes in traffic flow can lead to discrepancies between actual and expected traffic volume. To address this issue, intelligent traffic light control strategies have emerged, which dynamically adjust the signal light status based on real-time traffic flow information to improve traffic efficiency.

[0004] Furthermore, in a fully connected autonomous vehicle (CAV) environment, the control strategy for unsignalized intersections leverages vehicle-to-everything (V2X) technology to achieve vehicle coordination, significantly improving traffic efficiency. In recent years, autonomous driving technology has developed rapidly. For example, Baidu's "Radish Express" has been tested in multiple locations, and several intelligent connected vehicle testing demonstration zones and pilot cities have been established nationwide, providing support for the implementation of unsignalized intersection control strategies.

[0005] However, for a considerable period to come, human-driven vehicles (HVs) and driverless vehicles (CAVs) will coexist in mixed traffic environments. While unsignalized intersection control strategies can improve efficiency, the lack of traffic light guidance makes HV behavior unpredictable and compromises safety.

[0006] Therefore, most studies still rely on traffic lights to manage CAVs and HVs. The conventional approach is to allow CAVs and HVs to share lanes, improving efficiency by adjusting CAV trajectories and traffic light states. However, this method cannot fully utilize the potential of CAVs and struggles to overcome the uncertainties posed by HVs. Control strategies based on dedicated CAV lanes physically isolate CAVs and HVs, reducing interaction conflicts, enabling CAVs to operate more efficiently, and optimizing intersection traffic flow.

[0007] Existing research on dedicated CAV lanes largely focuses on static lane allocation. However, under varying traffic flow and CAV penetration rates, this can lead to wasted lane resources or exacerbated congestion. Some scholars have begun research on dynamic dedicated lane control. For example, some studies propose that left-turning and straight-going CAVs use independent dedicated lanes for their respective phases. However, this separated design requires more lane infrastructure, and when the proportion of oncoming traffic is uneven, some lanes are underutilized, resulting in resource waste. Other studies propose a dynamic allocation method where left-turning and straight-going vehicles share dedicated CAV lanes. However, this method involves frequent lane function switching when traffic flow fluctuates significantly, potentially reducing overall traffic efficiency.

[0008] Furthermore, with the development of artificial intelligence, machine learning has begun to be applied to intelligent transportation systems. Deep reinforcement learning (DRL), as the learning system closest to the human brain, can efficiently solve complex decision-making problems. However, its application in mixed traffic environments is still limited, and most research relies on optimization-based algorithms. These traditional methods have theoretical advantages, but they require a large amount of computational resources to solve for the global optimum, which may lead to computational delays and an inability to respond promptly to changes in traffic conditions. In practical applications, with increasing traffic flow and environmental complexity, it is difficult to guarantee real-time performance. Summary of the Invention

[0009] Based on the problems raised in the background technology above, the purpose of this invention is to provide a reinforcement learning control method, device, and medium for intersections in mixed traffic environments. By introducing dedicated phases for traffic vehicular activity (CAV) and dedicated lanes for CAV in the free lane direction, the conflict between CAV and traffic vehicles (HV) is reduced. The mode of the dedicated CAV lanes is dynamically adjusted according to the current CAV penetration rate. Then, the state of the traffic lights is dynamically adjusted according to the current traffic flow. The control method is generated by updating the training based on the interaction between the intersection in the mixed traffic environment and the deep reinforcement learning framework, thereby solving the intersection coordination problem in mixed traffic environments.

[0010] This invention is achieved through the following technical solution:

[0011] The first aspect of this invention provides a reinforcement learning control method for intersections in a mixed traffic environment, comprising the following steps:

[0012] The intersection in a mixed traffic environment is defined as an environment; wherein, the environment includes lanes, and the lanes are divided into ordinary lanes and CAV-dedicated lanes;

[0013] Construct a first intelligent agent, a second intelligent agent, and a third intelligent agent;

[0014] The first intelligent agent perceives the state of the environment and executes... - Greedy strategy selects CAV dedicated lane mode;

[0015] The second intelligent agent senses the state of the environment under the influence of the CAV dedicated lane mode and executes... - Greedy strategy for phase selection;

[0016] The third agent senses the state of the environment under the given phase and executes... - A greedy strategy allocates the right-of-way for vehicles in the CAV-dedicated lane.

[0017] In the above technical solution, the intersection in a mixed traffic environment is defined as the environment. In a mixed traffic environment oriented towards CAVs and HVs, this method divides the lanes into ordinary lanes and dedicated CAV lanes. The introduction of dedicated CAV lanes isolates the mutual influence between CAVs and HVs. The dedicated CAV lanes are set as free-direction lanes, allowing right turns, straight-through traffic, and left turns. Ordinary lanes are set as fixed lanes, meaning that left-turning HVs can only pass through the intersection from the fixed left-turn ordinary lanes. Based on this, dedicated CAV phases are provided for the dedicated CAV lanes. Within these dedicated phases, this method allows for free control of CAV passage through the intersection. In other phases, traffic lights separate conflicting vehicles, allocating conflicting HV traffic flows to different phases. HVs, on the other hand, follow the traditional method, passing through the intersection according to traffic light instructions.

[0018] The deep reinforcement learning framework includes three agents: the first agent (Lane-Pattern Agent), the second agent (Traffic-Signal Agent), and the third agent (CAV-Coordination Agent). The first agent, the second agent, and the third agent are dynamically coupled with the environment in sequence.

[0019] First, the first agent adjusts the CAV-dedicated lane mode based on the CAV penetration rate. Specifically, the first agent senses the ratio of CAVs to HVs in the environment and executes... - Greedy strategy determines the action - selects the CAV dedicated lane mode and executes the action to apply the selected CAV dedicated lane mode to the environment.

[0020] Secondly, the second intelligent agent dynamically adjusts and determines the traffic light phase based on the CAV dedicated lane mode and real-time traffic flow. Specifically, the second intelligent agent perceives the state of the environment after applying the selected CAV dedicated lane mode, and executes... - A greedy strategy determines the action—selects a phase, and executes the action by applying the selected phase to the environment.

[0021] Finally, the third agent is used to allocate right-of-way to vehicles in the CAV-dedicated lane under the selected phase. The third agent perceives the state of the environment after applying the selected phase, and executes... - Greedy strategy determines actions - allocate right-of-way for vehicles in dedicated CAV lanes, and regulate CAV passage through intersections based on the allocated right-of-way to control mixed traffic flow.

[0022] In one alternative embodiment, the first agent performs the following steps:

[0023] Obtain the number of CAV vehicles and HV vehicles in the environment at time t, and use the ratio of the number of CAV vehicles to the number of HV vehicles as the state at time t. ;

[0024] The CAV lane mode will be selected as the action, and the state will be determined using an ε-greedy strategy. Action at time t ;

[0025] Perform the action and obtain the execution of the action. The reward at time t is calculated based on the number of vehicles passing through after that. .

[0026] In an optional embodiment, after the first agent has finished executing, the method further includes: setting the state... The aforementioned actions ,award and the state of the environment at time t+1 The experience is integrated and placed into an experience replay pool, and the first agent is trained using the experience in the experience replay pool.

[0027] In one alternative embodiment, the second intelligent agent performs the following steps:

[0028] Obtain the queue length of the lane corresponding to each traffic light phase in the environment at time t, and use the queue length of the lane corresponding to each traffic light phase as the state at time t. ;

[0029] The phase of the selected CAV dedicated lane mode is taken as the action, and the state is determined using an ε-greedy strategy. Action at time t ;

[0030] Perform the action and obtain the execution of the action. The reward at time t is calculated based on the number of vehicles passing through after that. .

[0031] In an optional embodiment, after the second agent has finished executing, the method further includes: setting the state... The aforementioned actions ,award and the state of the environment at time t+1 The experience is integrated and placed into an experience replay pool, and the second agent is trained using the experience in the experience replay pool.

[0032] In one alternative embodiment, the third agent performs the following steps:

[0033] Obtain the traffic state in the environment at time t, and use the traffic state as the state at time t. The traffic state is encoded using discrete traffic state coding, which includes a position matrix, a speed matrix, a path matrix, and a release priority matrix.

[0034] Assigning right-of-way to vehicles in the CAV-dedicated lane is taken as an action, and the state is determined using an ε-greedy strategy. Action at time t ;

[0035] Perform the action and obtain the execution of the action. The reward at time t is calculated based on the subsequent vehicle passage data, queue length, and release priority. .

[0036] In one optional embodiment, the reward at time t is calculated based on the vehicle passage data, the queue length, and the release priority. The calculation process is as follows:

[0037]

[0038] In the above formula, Let be the number of vehicles passing through at time t. The number of arms, For the number of lanes, Let be the queue length at time t. Let t be the priority for allowing passage. , , As weight.

[0039] In an optional embodiment, after the third agent has finished executing, the method further includes: setting the state... The aforementioned actions ,award and the state of the environment at time t+1 The experience is integrated and placed into an experience replay pool, and the third agent is trained using the experience in the experience replay pool.

[0040] A second aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a reinforcement learning control method for intersections in a mixed traffic environment.

[0041] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a reinforcement learning control method for intersections in a mixed traffic environment.

[0042] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0043] This invention introduces dedicated CAV (Carrying Availability) phases and dedicated CAV lanes for free-lane traffic. A first intelligent agent adjusts the dedicated CAV lane mode based on CAV penetration rate. A second intelligent agent executes the corresponding traffic light phase scheme based on the dedicated CAV lane mode and dynamically adjusts the traffic light status based on real-time traffic flow. When the traffic light is in a dedicated CAV phase, a third intelligent agent dynamically determines the lane's right-of-way based on the current CAV traffic state and releases vehicles based on this right-of-way. This control method dynamically optimizes lane modes and traffic light status under different CAV penetration rates, while flexibly coordinating vehicle passage under dedicated CAV phases. This maximizes overall traffic efficiency and reduces intersection delays. Compared to dynamic traffic lights and traditional fixed traffic lights, it reduces average vehicle delay time and average queue length, increasing traffic efficiency. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0045] Figure 1 This is a diagram illustrating the overall architecture of the intersection reinforcement learning control method in a mixed traffic environment provided in Embodiment 1 of the present invention.

[0046] Figure 2 This is a schematic diagram of the structure of an intersection in a mixed traffic environment provided in Embodiment 1 of the present invention;

[0047] Figure 3This is a schematic diagram of the lane direction provided in Embodiment 1 of the present invention;

[0048] Figure 4 This is a schematic diagram of the architecture for training an intelligent agent according to Embodiment 1 of the present invention;

[0049] Figure 5 This is a schematic diagram of the CAV dedicated lane mode provided in Embodiment 1 of the present invention;

[0050] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 2 of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0052] Example 1

[0053] Figure 1 This is a diagram illustrating the overall architecture of the intersection reinforcement learning control method in a mixed traffic environment provided in Embodiment 1 of the present invention. Figure 1 As shown, the reinforcement learning control method for intersections in mixed traffic environments includes the following steps:

[0054] The intersection in a mixed traffic environment is defined as an environment; wherein, the environment includes lanes, and the lanes are divided into ordinary lanes and CAV-dedicated lanes;

[0055] Construct a first intelligent agent, a second intelligent agent, and a third intelligent agent;

[0056] The first intelligent agent perceives the state of the environment and executes... - Greedy strategy selects CAV dedicated lane mode;

[0057] The second intelligent agent senses the state of the environment under the influence of the CAV dedicated lane mode and executes... - Greedy strategy for phase selection;

[0058] The third agent senses the state of the environment under the given phase and executes... - A greedy strategy allocates the right-of-way for vehicles in the CAV-dedicated lane.

[0059] It should be noted that this invention comprises two parts: an environment and a deep reinforcement learning framework. By generating a control method through the interaction between the environment and the deep reinforcement learning framework, the problem of intersection coordination in mixed traffic environments can be solved.

[0060] Among them, intersections in mixed traffic environments are set as environments, such as Figure 2 As shown, in a mixed traffic environment oriented towards CAV and HV, this method divides lanes into ordinary lanes and dedicated CAV lanes, using dedicated CAV lanes to isolate the mutual influence between CAV and HV; for example... Figure 3 As shown, the CAV-dedicated lane is set as a free-direction lane, allowing right turns, straight-through traffic, and left turns; the regular lane is set as a fixed lane, meaning that left-turning HVs can only pass through the intersection from the fixed left-turn regular lane. Based on this, a dedicated CAV phase is provided for the CAV-dedicated lane. Within the dedicated CAV phase, this method is used to freely control the passage of CAVs through the intersection; in other phases, traffic lights are used to separate conflicting vehicles, allocating conflicting HV traffic flows to different phases, while HVs proceed through the intersection entirely in the traditional manner, following the traffic light instructions.

[0061] The deep reinforcement learning framework includes three agents: the first agent (Lane-Pattern Agent), the second agent (Traffic-Signal Agent), and the third agent (CAV-Coordination Agent). The first agent, the second agent, and the third agent are dynamically coupled with the environment in sequence.

[0062] First, the first agent adjusts the CAV-dedicated lane mode based on the CAV penetration rate. Specifically, the first agent senses the ratio of CAVs to HVs in the environment and executes... - Greedy strategy determines the action - selects the CAV dedicated lane mode and executes the action to apply the selected CAV dedicated lane mode to the environment.

[0063] Secondly, the second intelligent agent dynamically adjusts the traffic light phase based on the CAV dedicated lane mode and real-time traffic flow. Specifically, the second intelligent agent perceives the state of the environment after applying the selected CAV dedicated lane mode, and executes... - A greedy strategy determines the action—selects a phase, and executes the action by applying the selected phase to the environment.

[0064] Finally, the third agent is used to allocate right-of-way to vehicles in the CAV-dedicated lane under the selected phase. The third agent perceives the state of the environment after applying the selected phase, and executes... - Greedy strategy determines actions - allocate right-of-way for vehicles in dedicated CAV lanes, and regulate CAV passage through intersections based on the allocated right-of-way to control mixed traffic flow.

[0065] By introducing dedicated phases and lanes for CAVs, the dedicated lane mode and traffic light status can be dynamically adjusted according to real-time traffic conditions and CAV penetration rate. This maximizes CAV efficiency while ensuring the efficiency of traffic vehicles (HVs), thereby significantly reducing the average delay time and average queue length of vehicles at intersections in mixed traffic environments.

[0066] Furthermore, the first, second, and third intelligent agents are constructed and trained based on the DDQN network structure. The training process of the first, second, and third intelligent agents based on the DDQN network structure is as follows: Figure 4 As shown, the agent obtains the state from the environment and executes... A greedy strategy determines actions, rewards are given for executing actions, and the environment transitions to the next state. The state, action, reward, and next state are stored in an experience replay pool for training the agent. In this embodiment, the states obtained by the first, second, and third agents from the environment, the actions determined based on those states, the rewards obtained after executing the actions, and the next state obtained after the environment transition are all determined based on the functions they implement.

[0067] In one alternative embodiment, the first intelligent agent performs the function of adjusting the CAV-dedicated lane mode according to the CAV penetration rate.

[0068] In this embodiment, the penetration rate of CAV is determined by the ratio of the number of CAV vehicles to the number of HV vehicles. Therefore, the first agent obtains the number of CAV vehicles and HV vehicles in the environment at time t, calculates the ratio of CAV vehicles to HV vehicles, and uses this ratio as the state at time t. Wherein, the state at time t It is expressed as follows:

[0069]

[0070] In the above formula, The proportions of CAV It is composed of the proportion of HV.

[0071] The CAV lane mode will be selected as the action, and the state will be determined using an ε-greedy strategy. Action at time t ,action It is expressed as follows:

[0072]

[0073] It should be noted that, This is a set of CAV-dedicated lane modes, including three modes, denoted by 1, 2, and 3. This embodiment uses a three-lane configuration as an example. The CAV-dedicated lane modes are as follows: Figure 5 As shown, the first CAV dedicated lane mode does not include a CAV dedicated lane; the second CAV dedicated lane mode includes one CAV dedicated lane, and the CAV dedicated lane is located in the middle; the third CAV dedicated lane mode includes two CAV dedicated lanes, and the two CAV dedicated lanes are adjacent to each other.

[0074] The first agent uses an ε-greedy strategy. Determined status Action at time t That is, the selection action from the three CAV dedicated lane modes is based on the ratio of CAV vehicles to HV vehicles at time t.

[0075] Execute the action at time t That is, the environment is configured according to the CAV dedicated lane mode, the number of vehicles passing through in the selected CAV dedicated lane mode is obtained, and the reward at time t is calculated based on the number of vehicles passing through in the selected CAV dedicated lane mode. ,award The calculation is as follows:

[0076]

[0077] In the above formula, Let be the number of vehicles passing through at time t+T. Let t represent the number of vehicles passing through at time t, and T represent the duration of the lane mode.

[0078] At this point, the environment transitions to the next state. ; will state ,action ,award The state of the environment at time t+1 Integrate into experience The experience is placed into the experience replay pool, and the first agent is trained using the experience in the experience replay pool.

[0079] In one optional embodiment, the second intelligent agent monitors the environment after the first intelligent agent selects the CAV dedicated lane mode and applies it to the environment. Its function is to dynamically adjust the traffic light phase and determine the phase based on the CAV dedicated lane mode and real-time traffic flow.

[0080] In this embodiment, the second agent obtains the queue length of the lane corresponding to each traffic light phase in the environment at time t, and uses the queue length of the lane corresponding to each traffic light phase as the state at time t. ,state It is expressed as follows:

[0081]

[0082] In the above formula, Let be the queue length of the lane corresponding to the i-th traffic light phase at time t. This represents the total number of traffic light phases.

[0083] The phase of the selected CAV dedicated lane mode is taken as the action, and the state is determined using an ε-greedy strategy. Action at time t ,action It is expressed as follows:

[0084]

[0085] The second agent employs an ε-greedy strategy. Determined status Action at time t That is, the queue length based on the traffic light phase corresponding to the lane is from Select one phase from the signal light phases.

[0086] Execute the action at time t That is, applying the selected phase to the environment.

[0087] Obtain the number of vehicles passing through the selected CAV dedicated lane mode, and calculate the reward at time t based on the number of vehicles passing through the selected CAV dedicated lane mode. ,award The calculation is as follows:

[0088]

[0089] In the above formula, Let t be the number of vehicles passing through at time t+τ, and τ be the duration of a phase.

[0090] At this point, the environment transitions to the next state. ; will state ,action ,award The state of the environment at time t+1 Integrate into experience The second agent is trained using the experience in the experience replay pool.

[0091] The second intelligent agent executes the corresponding traffic light phase scheme according to the CAV dedicated lane mode, and dynamically adjusts the traffic light status according to real-time traffic flow.

[0092] In one alternative embodiment, the third agent allocates right-of-way to vehicles in the CAV dedicated lane under the CAV dedicated phase after the second agent selects the CAV dedicated lane mode.

[0093] In this embodiment, the third agent acquires the traffic state in the environment at time t and uses the traffic state as the state at time t. ,state It is expressed as follows:

[0094]

[0095] In the above formula, Let be the position matrix at time t. Let be the velocity matrix at time t. Let be the path matrix at time t. Let be the release priority matrix at time t.

[0096] It should be noted that the traffic state in this embodiment adopts Discrete Traffic State Coding (DTSE), which consists of four matrices: a position matrix, a speed matrix, a path matrix, and a release priority matrix. Each matrix is... matrix, The matrix represents the conflict zone at an intersection in a mixed traffic environment. Therefore, the traffic state obtained in this embodiment is the traffic state of the conflict zone at an intersection in a mixed traffic environment. Each position in the matrix corresponds to a gridded position in the intersection. If there is a vehicle at the corresponding position, the matrix inputs the corresponding information; otherwise, it is 0. The release priority corresponds to the vehicle's delay time; the greater the delay time, the higher the priority.

[0097] The action of allocating the right-of-way for vehicles in the CAV-dedicated lane is adopted in this embodiment. The binary vector represents the allocation of right-of-way for vehicles in the CAV dedicated lane, and the state is determined by an ε-greedy strategy. Action at time t ,action It is expressed as follows:

[0098]

[0099] In the above formula, Indicates the number of arms. Indicates the number of lanes. Indicates time t at On the boom Right of way in the lane.

[0100] Execute action and obtain the action to be performed. The reward at time t is calculated by combining the vehicle passage data, queue length, and release priority, taking into account the three factors: the number of vehicles passing through, queue length, and release priority. ,award It is expressed as follows:

[0101]

[0102] In the above formula, Let be the number of vehicles passing through at time t. The number of arms, For the number of lanes, Let be the queue length at time t. Let t be the priority for allowing passage. , , For weights.

[0103] At this point, the environment transitions to the next state. ; will state ,action ,award The state of the environment at time t+1 Integrate into experience The experience is placed into the experience replay pool, and the third agent is trained using the experience in the experience replay pool.

[0104] In summary, by having the first intelligent agent adjust the CAV-dedicated lane mode based on the CAV penetration rate, the second intelligent agent executes the corresponding traffic light phase scheme based on the CAV-dedicated lane mode, and dynamically adjusts the traffic light status according to real-time traffic flow, and the third intelligent agent dynamically determines the lane's right-of-way based on the current CAV traffic status when the traffic light is in the CAV-dedicated phase, and releases vehicles based on the right-of-way, this control method can dynamically optimize lane modes and traffic light status under different CAV penetration rates, while flexibly coordinating vehicle passage under the CAV-dedicated phase, thereby maximizing overall traffic efficiency and reducing intersection delays.

[0105] Example 2

[0106] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 2 of the present invention, as shown below. Figure 6 As shown, the electronic device includes a processor 21, a memory 22, an input device 23, and an output device 24; the number of processors 21 in the computer device can be one or more. Figure 6Taking a processor 21 as an example; the processor 21, memory 22, input device 23, and output device 24 in an electronic device can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0107] The memory 22, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules. The processor 21 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 22, thereby implementing the intersection reinforcement learning control method in the mixed traffic environment of Embodiment 1.

[0108] The memory 22 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 22 may further include memory remotely located relative to the processor 21, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0109] Input device 23 can be used to receive user input such as ID and password. Output device 24 is used to output the network configuration page.

[0110] Example 3

[0111] Embodiment 3 of the present invention also provides a computer-readable storage medium, wherein the computer-executable instructions, when executed by a computer processor, are used to implement the intersection reinforcement learning control method in a mixed traffic environment as provided in Embodiment 1.

[0112] The storage medium containing computer-executable instructions provided in the embodiments of the present invention is not limited to the method operation provided in Embodiment 1, but can also execute related operations in the intersection reinforcement learning control method in a mixed traffic environment provided in any embodiment of the present invention.

[0113] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A reinforcement learning control method for intersections in mixed traffic environments, characterized in that, The steps include the following: The intersection in a mixed traffic environment is defined as an environment; wherein, the environment includes lanes, and the lanes are divided into ordinary lanes and CAV-dedicated lanes; Construct a first intelligent agent, a second intelligent agent, and a third intelligent agent; The first intelligent agent perceives the state of the environment and executes... - A greedy strategy selects the CAV dedicated lane mode; the first agent executes the following steps: Obtain the number of CAV vehicles and HV vehicles in the environment at time t1, and use the ratio of the number of CAV vehicles to the number of HV vehicles as the state at time t1. ; The CAV dedicated lane mode will be selected as the action, and the state will be determined using an ε-greedy strategy. Actions at time t1 ; Perform the action and obtain the execution of the action. The reward at time t1 is calculated based on the number of vehicles passing through after that. ; The second intelligent agent senses the state of the environment under the influence of the CAV dedicated lane mode and executes... - A greedy strategy is used to select the phase; the second agent performs the following steps: Obtain the queue length of the lane corresponding to each traffic light phase in the environment at time t2, and use the queue length of the lane corresponding to each traffic light phase as the state at time t2. ; The selected phase will be used as the action, and the state will be determined using an ε-greedy strategy. Actions at time t2 ; Perform the action and obtain the execution of the action. The reward at time t2 is calculated based on the number of vehicles passing through after that. ; The third agent senses the state of the environment under the given phase and executes... - A greedy strategy allocates the right-of-way for vehicles in the CAV-dedicated lane; the third agent performs the following steps: Obtain the traffic state in the environment at time t3, and use the traffic state as the state at time t3. The traffic state is encoded using discrete traffic state coding, which includes a position matrix, a speed matrix, a path matrix, and a release priority matrix. Assigning right-of-way to vehicles in the CAV-dedicated lane is taken as an action, and the state is determined using an ε-greedy strategy. Actions at time t3 ; Perform the action and obtain the execution of the action. The reward at time t3 is calculated based on the number of vehicles passing through, the queue length, and the release priority. .

2. The intersection reinforcement learning control method in a mixed traffic environment according to claim 1, characterized in that, After the first agent completes its execution, the method further includes: setting the state. The aforementioned actions The aforementioned reward and the state of the environment at time t1+1 The experience is integrated and placed into an experience replay pool, and the first agent is trained using the experience in the experience replay pool.

3. The intersection reinforcement learning control method in a mixed traffic environment according to claim 1, characterized in that, After the second agent completes its execution, the process further includes: setting the state. The aforementioned actions The aforementioned reward and the state of the environment at time t2+1 The experience is integrated and placed into an experience replay pool, and the second agent is trained using the experience in the experience replay pool.

4. The intersection reinforcement learning control method in a mixed traffic environment according to claim 1, characterized in that, After the third agent completes its execution, the method further includes: setting the state. The aforementioned actions The aforementioned reward and the state of the environment at time t3+1 The experience is integrated and placed into an experience replay pool, and the third agent is trained using the experience in the experience replay pool.

5. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the intersection reinforcement learning control method in any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the intersection reinforcement learning control method in a mixed traffic environment as described in any one of claims 1 to 4.

Images