Environmental perception traffic signal control method based on reinforcement learning

Abstract

The invention relates to an environmental perception traffic signal control method based on reinforcement learning, and belongs to the technical field of computers. According to the method, traffic signals are intelligently regulated and controlled by sensing environment information and traffic states and combining road states. Firstly, a traffic signal regulation and control strategy is learned in a simulation environment by using a reinforcement learning algorithm, and then the obtained regulation and control strategy is applied to a real scene. The specific implementation comprises the steps of collecting environmental perception data, constructing a simulated traffic environment, configuring a control strategy, setting a typical traffic scene, collecting environmental information, fusing features, setting performance indexes, reinforcing learning optimization, applying a real scene, establishing a feedback mechanism and the like. Through combination of reinforcement learning and environment perception technologies, the method can dynamically adapt to traffic demands in different time and environments, effectively alleviates traffic congestion, improves traffic operation efficiency, and provides an effective scheme for improving urban traffic conditions.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology and relates to an environment-aware traffic signal control method based on reinforcement learning. Background Art

[0002] Traditional traffic signal control relies on manual investigation and field settings to formulate control plans. Traffic signals under this configuration cannot dynamically respond to traffic needs according to actual conditions. The introduction of artificial intelligence technology can effectively integrate historical experience and real-time information to achieve the purpose of making traffic operations smoother and improving the travel experience of traffic participants. Although some non-traditional traffic signal control methods can be adjusted dynamically, environmental factors such as slippery road surfaces and visibility are not taken into account, and the effect is poor under severe weather conditions. Therefore, the solution of using artificial intelligence technology to perceive environmental factors and optimize traffic signal control has attracted widespread attention in academia and industry, and has produced practical solutions.

[0003] Due to the dynamics and complexity of the traffic system and the limited nature of traffic signal control actions, the traffic signal control problem can be modeled as a Markov decision process (MDP), in which the traffic state and signal control actions jointly determine the future traffic conditions. Through this modeling, the benefits of different control strategies can be quantified, and the signal configuration strategy can be optimized with the goal of improving the overall benefit, ultimately increasing traffic flow and reducing congestion. Reinforcement learning (RL) methods are constantly developing, and they have mature applications in complex decision-making and dynamic environment adaptation. In recent years, with the advancement of deep learning technology, RL has made significant breakthroughs in solving high-dimensional state space problems, promoting the rapid development of fields such as games, robot control, and financial decision-making. Reinforcement learning methods can continuously learn and optimize strategies based on reward signals through interaction with the environment, thereby realizing real-time and dynamic traffic signal control. Therefore, reinforcement learning is naturally and effectively applicable to solving traffic signal control problems and can improve the operating efficiency of the overall traffic system.

[0004] By combining the traffic signal control strategy obtained by reinforcement learning with the real-time environmental information perceived by the neural network, the intelligent adjustment and optimization of traffic signals can be achieved, thereby dynamically adapting to traffic needs at different times and environments. By simulating different traffic conditions in a simulated environment, various traffic conditions in reality can be effectively simulated and responded to, so as to train iterative control strategies and optimize control schemes in a virtual environment, which provides valuable experience and data support for traffic signal control in reality. This method can not only effectively learn and simulate changes in traffic flow, but also improve the ability to optimize traffic signal control, thereby improving the efficiency and safety of urban traffic. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide an environment-aware traffic signal control method based on reinforcement learning. By perceiving environmental information and traffic status, combined with road status, traffic signals can be intelligently controlled. The combined use of multimodal information can take various factors into consideration, so as to control traffic with reasonable and appropriate strategies. By learning traffic control schemes under various circumstances through reinforcement learning algorithms, traffic congestion can be effectively alleviated and traffic operation efficiency can be improved.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] The present invention provides a method for sensing real-time environmental information and traffic conditions and adjusting signal control strategies. The overall process is roughly as follows: first, a reinforcement learning algorithm is used in a simulated environment to learn and control traffic signal strategies, and then the obtained control strategies are applied to real-world scenarios.

[0008] The implementation method includes the following steps:

[0009] Step 1) Collect and integrate environmental perception data, including road conditions, weather information, and traffic conditions. Clean and pre-process the data to ensure its accuracy and usability;

[0010] Step 2) Use the SUMO (Simulation of Urban Mobility) platform as the simulation environment. The data used includes multiple intersections. Each intersection is a crossroad or a three-way intersection. The exit of each intersection has three lanes (left turn, straight and right turn), and the entrance is a double lane. Configure the data file and build the simulated traffic environment information in the simulation software. Set the traffic data in the simulation environment, including the road network structure, traffic flow settings, etc., and configure the initial traffic signal strategy;

[0011] Step 3) Configure the initial control strategy in the simulation environment, configure basic information such as road network information and traffic flow, and add it as environmental information;

[0012] Step 4) Set up a variety of typical traffic scenarios with reference to reality, including different vehicle types, climate conditions and road conditions. In each scenario, preset different driving behavior parameters, such as acceleration, deceleration, environmental deceleration and start delay, to simulate real driving behavior;

[0013] Step 5) Collect weather conditions at a certain moment through network information, and use sensors to collect road conditions and traffic conditions. Convert this information into a multimodal vector and form a state matrix;

[0014] Step 6), using the feature fusion module to extract and fuse features of the state matrix, so as to comprehensively consider various environmental factors and generate a comprehensive state representation;

[0015] Step 7) Set performance indicators for measuring strategies, including average vehicle passing time, intersection throughput, queue length, and average vehicle waiting time;

[0016] Step 8), apply the original strategy and the obtained fusion feature information to the simulation environment, obtain the performance score, and optimize the strategy through the reinforcement learning model;

[0017] Step 9) Apply the trained and optimized traffic signal control strategy to real scenarios, monitor its effect and collect feedback data to further adjust and optimize the strategy;

[0018] Step 10) Establish a feedback mechanism to regularly evaluate and update traffic signal control strategies to adapt to changing traffic conditions and environmental conditions and ensure the efficient operation of the transportation system.

[0019] The beneficial effects of the present invention are:

[0020] (1) Dynamic adaptability

[0021] Through sensors and network information, data such as weather, road conditions and traffic flow are obtained in real time, and environmental factors are incorporated into the control strategy, so that traffic signals can be dynamically adjusted according to actual road conditions to adapt to traffic needs at different times and environments. Reinforcement learning algorithms are used to continuously learn and optimize control strategies, adjust signal light timing based on historical experience and real-time feedback, and achieve optimal control of traffic signals.

[0022] (2) Improved efficiency

[0023] By dynamically adjusting the timing of traffic lights, traffic congestion can be effectively alleviated, the capacity of intersections can be improved, the queuing time of vehicles can be reduced, and the efficiency of traffic operation can be improved. By dynamically adjusting the green light time according to the traffic flow, the capacity of intersections can be maximized and the utilization efficiency of roads can be improved.

[0024] (3) Improved security

[0025] By optimizing traffic signal control, traffic conflicts can be effectively reduced, the probability of traffic accidents can be lowered, and traffic safety can be improved. Vehicle waiting time can be shortened, driver anxiety can be reduced, and the traffic experience of traffic participants can be improved.

[0026] (4) Energy conservation and emission reduction

[0027] By optimizing traffic signal control, reducing vehicle waiting time at intersections and reducing vehicle idling time, exhaust emissions and energy consumption can be reduced, achieving energy conservation and emission reduction.

[0028] (5) Flexibility and scalability

[0029] This method is based on a mature simulation platform and reinforcement learning algorithm, is easy to deploy and apply, and can be applied to traffic intersections of different sizes and types. This method can be expanded according to actual needs. For example, more environmental perception data can be added, such as vehicle flow prediction and pedestrian flow, to further improve the accuracy and effectiveness of the control strategy.

[0030] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0032] Figure 1 The basic idea of ​​the environment-aware traffic signal control method based on reinforcement learning described in the present invention;

[0033] Figure 2 It is a schematic diagram of the convolution-based traffic condition extraction process described in the present invention. DETAILED DESCRIPTION

[0034] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0035] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0036] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0037] See also Figure 1 and Figure 2 , the present invention describes an environment-aware traffic signal control method based on reinforcement learning, and the detailed steps are as follows:

[0038] Step 1) Experimental environment configuration, including algorithm and simulation operating environment;

[0039] Step 2) Set up the simulation environment. The data used include road network information, traffic flow data and traffic signal templates, which correspond to the three files flow.json, roadnet.json and signal_plan_template.txt respectively. According to the data files, the simulation traffic environment E is constructed in the simulation software. real and E sim , where E real With E sim The constructs are different and serve as real-world tests of strategy performance;

[0040] Step 3), use M=<S,A,P,r,γ> To describe the Markov decision process MDP in the traffic signal control problem, where S represents the system state space, A represents the set of action spaces, and P describes the next state s t+1 ∈S is the probability distribution of the transfer, r represents the reward, π θ is the policy generated by the network with θ as parameter and γ is the discount factor.

[0041] Step 4) The training process of the reinforcement learning algorithm is to learn an optimal strategy that can maximize the long-term expectation of the cumulative reward. The role of γ is to make the current decision have a greater impact on the cumulative reward. The discounted cumulative reward is:

[0042]

[0043] Using the value Q function, its role is to evaluate the expected return of taking an action in a specific state and find the action a that may lead to the maximum long-term reward r, which enables the agent to choose the optimal action to maximize the long-term benefits. The update of the Q function follows the Bellman equation:

[0044] Q(s t ,a,t)=r t+1 +γ·maxQ(s t+1 ,a t+1 ,t+1) (2)

[0045] The action-value function Q at time t is the sum of the reward at the next timestamp t+1 and the maximum potential future reward.

[0046] Step 5) Use the E real Trajectory data to improve the simulation environment E sim , to enhance algorithm usability. sim The iterative behavior in is a Markov decision process MDP, using P φ Describe the cause action a in the simulation environment t The occurrence of state s t To s t+1 The probability of transfer. real ={τ 1 ,τ 2 ,…,τ I} is based on real-world trajectory data, where Refers to the real Execution strategy π θ The collected state transition trajectory of the i-th lane, P * Describes the state Next, action occurs The probability of * Describe the minimization difference in dynamic transfer, P * Describes the probability of state transition in the real world, d(·) represents the absolute value of the difference:

[0047]

[0048] Forward Model Use from E real The purpose is to train with data from t and operation a t Predict the next possible state

[0049]

[0050] Inverse Model Use from E sim The purpose is to train from the current state s t Predict the action that leads to a given next state The inverse model transforms the predictions of the forward model and the current state s t As input:

[0051]

[0052] According to the strategy θ Predicted action a t The resulting state-action pair t ,a t >, in E sim Medium Action Will correct E sim The s t+1 , making it close to the predicted E real The next state in This allows the dynamics in the simulation to Close to real-world dynamics Thus, due to P φ Close to P * , in E sim Strategies learned from θ , and in the transfer to E real There will be a smaller performance gap.

[0053] Step 5), use the feature extraction module to extract traffic condition information X t :

[0054] X t =w 1 *Conv 1 (W(t))+w 1 *Conv 2 (R * )+w 3 *Embed(T(t)) (6)

[0055] X t =ReLU(Linear(X t )) (7)

[0056] Among them, w i is the corresponding feature weight, Conv i It is a convolution module, and Embed is a time embedding module. After the initial fusion of features, the feature input is obtained after a linear transformation and an activation function transformation.

[0057] ​Step 6) Improve the forward model. After adding traffic condition related information, the representation of the forward model changes to:

[0058]

[0059] Forward Model Approximated by a neural network, using Indicates that from the real environment E real The state and action sampled from the trajectory of , then its parameter φ + The expression for optimization using minimization of mean squared error (MSE) loss is:

[0060]

[0061] Step 7) Improve the inverse model. After adding traffic condition related information, the representation of the inverse model changes to:

[0062]

[0063] Inverse Model is also approximated by the neural network, because the target action belongs to the discrete value of the signal control problem defined in the existing work, so φ - It is optimized by minimizing the categorical cross entropy (MCE) loss:

[0064]

[0065] Example: Traffic signal control at intersections based on deep reinforcement learning

[0066] 1. Data collection and preprocessing

[0067] Environmental perception data:

[0068] Weather conditions: Sunny;

[0069] Road slipperiness: dry;

[0070] Visibility: Good

[0071] Road condition data:

[0072] Number of exit lanes: 4

[0073] Right turn lane: Yes

[0074] Left turn lane: Yes

[0075] Sidewalk: Yes

[0076] Traffic status data:

[0077] Import channel 1:

[0078] Lane 1: Number of vehicles: 3, queue length: 16m, medium and large vehicles: none, throughput efficiency: 25.4 vehicles / min;

[0079] Lane 2: Number of vehicles: 6, queue length: 35m, medium and large vehicles: none, throughput efficiency: 16.2 vehicles / min;

[0080] Lane 3: Number of vehicles: 6, queue length: 34m, medium and large vehicles: yes, throughput efficiency: 16.2 vehicles / min;

[0081] Lane 4: Number of vehicles: 5, queue length: 28m, medium and large vehicles: none, throughput efficiency: 15.0 vehicles / min;

[0082] Import channel 2:

[0083] Lane 1: Number of vehicles: 7, queue length: 45m, medium and large vehicles: yes, throughput efficiency: 33.2 vehicles / min;

[0084] Lane 2: Number of vehicles: 7, queue length: 44m, medium and large vehicles: none, throughput efficiency: 15.6 vehicles / min;

[0085] Lane 3: Number of vehicles: 6, queue length: 35m, medium and large vehicles: yes, throughput efficiency: 16.2 vehicles / min;

[0086] Lane 4: Number of vehicles: 6, queue length: 34m, medium and large vehicles: none, throughput efficiency: 13.8 vehicles / min;

[0087] Import channel 3:

[0088] Lane 1: Number of vehicles: 4, queue length: 23, medium and large vehicles: yes, throughput rate: 18.6 vehicles / min;

[0089] Lane 2: Number of vehicles: 5, queue length: 29m, medium and large vehicles: yes, throughput efficiency: 16.6 vehicles / min;

[0090] Lane 3: Number of vehicles: 5, queue length: 27m, medium and large vehicles: yes, throughput efficiency: 16.2 vehicles / min;

[0091] Lane 4: Number of vehicles: 3, queue length: 17m, medium and large vehicles: none, throughput rate: 15 vehicles / min;

[0092] Import channel 4:

[0093] Lane 1: Number of vehicles: 1, queue length: 5m, medium and large vehicles: none, throughput efficiency: 7.2 vehicles / min;

[0094] Lane 2: Number of vehicles: 3, queue length: 17m, medium and large vehicles: none, throughput efficiency: 13.2 vehicles / min;

[0095] Lane 3: Number of vehicles: 2, queue length: 13m, medium and large vehicles: none, throughput efficiency: 15.6 vehicles / min;

[0096] Lane 4: Number of vehicles: 3, queue length: 17m, medium and large vehicles: none, throughput efficiency: 15.6 vehicles / min;

[0097] 2. Simulation environment construction

[0098] SUMO platform: Use the SUMO platform to build a traffic simulation environment including intersections to simulate traffic conditions under sunny days, dry roads, and good visibility.

[0099] Road network structure: Four entrance roads and four exit roads are set up, and each intersection is a double lane.

[0100] Traffic flow setting: Same as the data collection part.

[0101] Initial traffic signal strategy:

[0102] Green light time: 30 seconds in each direction

[0103] Red light time: 30 seconds in each direction

[0104] 3. Reinforcement learning model training:

[0105] State space: Environmental perception data and traffic status data are input into the model as the state space. For example, weather conditions, road wetness, visibility, road conditions, traffic flow at each entrance, medium and large vehicles, passing efficiency and other information are used as state features.

[0106] Action space: The action space is defined as adjusting the green light time in each direction, for example, the adjustment range is 20 seconds to 40 seconds.

[0107] Reward function: The reward value is calculated based on indicators such as average vehicle passing time, intersection throughput, queue length, and average vehicle waiting time. For example, the reward value can be set to the difference between throughput and queue length to encourage the model to improve throughput and reduce queue length.

[0108] Deep reinforcement learning algorithm: Use the DQN algorithm to train the model and learn the optimal traffic signal control strategy under different environmental conditions.

[0109] 4. Real-world application:

[0110] Data collection: Collect environmental perception data and traffic status data in real time and input them into the trained DQN model.

[0111] Decision-making: The DQN model outputs the optimal action based on the current state, for example, extending the green light time in the congested direction and shortening the green light time in the unobstructed direction. Assume that the model outputs the following actions:

[0112] Entrance lanes 1 and 3: 30 seconds for straight ahead green light, 25 seconds for left turn green light

[0113] Entrance Lanes 2 and 4: 35 seconds for straight ahead green light, 30 seconds for left turn green light

[0114] Traffic signal control: Adjust traffic signals based on the actions output by the model, for example, extend the green light time in the congested direction and shorten the green light time in the unobstructed direction.

[0115] 5. Feedback and optimization:

[0116] Data collection: Collect traffic data in real scenarios, such as average vehicle passing time, intersection throughput, queue length, average vehicle waiting time, etc.

[0117] Evaluation: Evaluate the effectiveness of the DQN model control strategy, for example, by comparing it with the initial strategy and analyzing changes in metrics such as throughput and queue length.

[0118] Feedback: Feedback the evaluation results to the DQN model to further optimize the model parameters and strategies.

[0119] Update: The DQN model is updated regularly to adapt to changing traffic and environmental conditions.

[0120] Expected results:

[0121] Improve traffic operation efficiency: By dynamically adjusting traffic signals, traffic congestion can be effectively alleviated and the traffic capacity of intersections can be improved.

[0122] Reduce traffic delays: shorten the average waiting time of vehicles and improve the travel experience of traffic participants.

[0123] Improve traffic safety: Reduce the probability of traffic accidents by optimizing traffic signal control.

[0124] Energy saving and emission reduction: reduce vehicle idling time, reduce exhaust emissions and energy consumption.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for controlling environmentally aware traffic signals based on reinforcement learning, characterized in that: The following steps are involved: Step 1): Collect and integrate environmental perception data, including road conditions, weather information, and traffic conditions, and clean and pre-process the data; Step 2): Use the SUMO (Simulation of Urban Mobility) platform to build a simulated traffic environment, set the traffic data in the simulation environment, including the road network structure and traffic flow settings, and configure the initial traffic signal strategy; Step 3): Configure the initial control strategy in the simulation environment, configure the basic information of the road network and traffic flow, and add it as environmental information; Step 4): Set a variety of typical traffic scenarios based on reality. In each scenario, preset different driving behavior parameters to simulate real driving behavior. Step 5): Collect weather conditions at a certain moment through network information, use sensors to collect road conditions and traffic conditions, convert this information into multimodal vectors, and form a state matrix; Step 6): Use the feature fusion module to extract and fuse features of the state matrix, comprehensively consider various environmental factors, and generate a comprehensive state representation; Step 7): Set performance indicators for measuring strategies, including average vehicle passing time, intersection throughput, queue length, and average vehicle waiting time; Step 8): Apply the original strategy and the obtained fusion feature information to the simulation environment, obtain the performance score, and optimize the strategy through the reinforcement learning model; Step 9): Apply the trained and optimized traffic signal control strategy to real scenarios, monitor its effect and collect feedback data to further adjust and optimize the strategy; Step 10): Establish a feedback mechanism to regularly evaluate and update traffic signal control strategies to adapt to changing traffic conditions and environmental conditions to ensure the efficient operation of the transportation system.

2. The method for controlling environment-aware traffic signals based on reinforcement learning according to claim 1, characterized in that: In step 5), the road conditions collected by sensors include road surface slipperiness and visibility.

3. The method for controlling environment-aware traffic signals based on reinforcement learning according to claim 1, characterized in that: In step 6), the feature fusion module includes a convolution module, a time embedding module and a linear transformation module.

4. The method for controlling environment-aware traffic signals based on reinforcement learning according to claim 1, characterized in that: In step 8), the reinforcement learning model adopts a deep reinforcement learning method.

5. The method for controlling environment-aware traffic signals based on reinforcement learning according to claim 1, characterized in that: In step 9), the feedback data includes traffic flow, average vehicle passing time, intersection throughput, queue length and average vehicle waiting time.

Images