Single-point traffic signal control method and device, storage medium and electronic equipment

Abstract

The application discloses a single-point traffic signal control method, comprising the following steps: determining predicted phase state information of each phase in the next signal cycle according to historical lane data of each phase of the single-point intersection; performing merging processing on the predicted phase state information of each phase to obtain predicted intersection state information; inputting the predicted intersection state information into a pre-trained reinforcement learning model, and outputting a cycle length of the next signal cycle corresponding to the predicted intersection state information after processing of the reinforcement learning model; and allocating phase lengths of each phase in the next signal cycle according to the cycle length of the next signal cycle. The application can be applied to a saturated road condition and has good generality.

Description

Technical Field

[0001] This application relates to traffic signal control technology, and in particular to a single-point traffic signal control method, device, storage medium, and electronic device. Background Technology

[0002] With the continuous development of urbanization, urban traffic flow is increasing, and traffic signal control technology has been widely applied.

[0003] In traffic signal control, single-point traffic signal control mainly involves determining the signal timing at a single intersection. Currently, this is primarily achieved through analytical algorithms or data-driven periodic single-point signal timing algorithms. However, both methods have their own limitations.

[0004] One drawback of analytical algorithms is that mathematical modeling methods can only model unsaturated situations. In saturated situations, the nonlinearity and complex time-varying nature of traffic make modeling difficult, rendering analytical methods generally unusable. While data-driven periodic single-point signal timing algorithms can achieve good results through multiple rounds of training in simulations, they suffer from drawbacks such as difficulty in generalization and deployment. That is, results trained in one scenario are unlikely to be effective in all scenarios, and online training is difficult to support in real-world signal control environments. Summary of the Invention

[0005] This application provides a single-point traffic signal control method, device, storage medium, and electronic device, which can be applied to road saturation conditions and has good generalization ability.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] A single-point traffic signal control method, comprising:

[0008] Based on the historical lane data of each phase of the single-point intersection, the predicted phase state information of each phase in the next signal cycle of the traffic signal is determined.

[0009] The predicted phase state information of each phase is merged to obtain the predicted intersection state information; wherein, the predicted intersection state information is used to measure the overall congestion level of the intersection, including each phase.

[0010] The predicted intersection state information is input into a pre-trained reinforcement learning model, and after processing by the reinforcement learning model, the cycle duration of the next signal cycle corresponding to the predicted intersection state information is output.

[0011] Based on the duration of the next signal cycle, a phase duration is assigned to each phase of the next signal cycle.

[0012] Preferably, the step of allocating phase durations to each phase of the next signal cycle based on the cycle duration of the next signal cycle includes:

[0013] Based on the predicted phase state information of each phase, the duration ratio of each phase is determined;

[0014] The duration of the next signal cycle is allocated to each phase according to the duration ratio of each phase, thus obtaining the phase duration of each phase.

[0015] Preferably, the duration ratio of each phase is positively correlated with the congestion level of the corresponding phase.

[0016] Preferably, the predicted intersection state information is used as the state of the reinforcement learning model, and the duration of the next signal cycle is selected as the action of the reinforcement learning model.

[0017] The optional period duration of the next signal period increases as the predicted intersection state information increases, serving as a training constraint for the reinforcement learning model.

[0018] Preferably, when training the reinforcement learning model, the duration range of the next signal cycle is set to [C]. base -d,C base +d], selecting the period duration of the next signal period within the period duration range; wherein, the C base The reference period duration is positively correlated with the predicted intersection state information, and d is a preset constant.

[0019] Preferably, C is calculated using the Webster single-point algorithm based on the sum of the set critical lane flow ratios. w , will the C w As the C base ;

[0020] If the sum of the critical lane flow rate ratios is less than a set high saturation threshold, then the period duration range of the next signal cycle is set to [C]. w -d,C w +d];

[0021] If the sum of the critical lane flow rate ratios is greater than or equal to the high saturation threshold, the duration range of the next signal cycle is set to [C′]. w C max ]; wherein, the C′ wThe reference cycle duration is calculated when the sum of the critical lane flow rate ratios equals the high saturation threshold, and the C... max The maximum allowed cycle duration.

[0022] Preferably, the reward of the reinforcement learning model is a weighted sum of at least one of the following: queue length at the green light start time, number of vehicles in the queue at the green light start time, queue length at the red light start time, number of vehicles in the queue at the red light start time, average vehicle delay, average number of stops per vehicle, and traffic flow within a set time period; wherein, when calculating the weighted sum, each weight is greater than or equal to 0.

[0023] Preferably, the merging process of the predicted phase state information of each phase includes: calculating a weighted sum of the predicted phase state information of each phase.

[0024] Preferably, determining the predicted phase state information for each phase of the next signal cycle based on the acquired historical lane data includes:

[0025] Based on lane data in each phase of a specified historical signal cycle, lane status information for the corresponding phase in each specified historical signal cycle is determined using the signal cycle as the unit.

[0026] Using signal cycles as units, the actual phase state information of the corresponding phase in each specified historical signal cycle is determined based on the lane state information in each phase of each specified historical signal cycle.

[0027] Based on the actual phase state information of each phase in all the specified historical periods, the predicted phase state information of the corresponding phase in the next signal period is determined.

[0028] Preferably, determining the actual phase state information of the corresponding phase in each of the specified historical signal periods includes:

[0029] Based on the lane data of the corresponding phase within a specified historical signal period, determine the queue length or number of vehicles in the queue at the green light start time for each lane and the corresponding phase, which serves as the lane status information for the corresponding phase within the corresponding historical signal period.

[0030] The actual phase state information of the corresponding phase in the specified historical signal period is determined based on the maximum value of the lane state information of all lanes contained in each phase within the specified historical signal period.

[0031] A single-point traffic signal control device includes: an intersection state prediction unit, a model processing unit, and a cycle allocation unit;

[0032] The intersection state prediction unit is used to determine the predicted phase state information of each phase in the next signal cycle of the traffic signal based on the historical lane data of each phase of the acquired single-point intersection; it is also used to merge the predicted phase state information of each phase to obtain the predicted intersection state information; wherein, the predicted intersection state information is used to measure the overall congestion level of the intersection, including each phase.

[0033] The model processing unit is used to input the predicted intersection state information into a pre-trained reinforcement learning model, and after processing by the reinforcement learning model, output the cycle duration of the next signal cycle corresponding to the predicted intersection state information.

[0034] The period allocation unit is used to allocate phase duration to each phase of the next signal period according to the period duration of the next signal period.

[0035] Preferably, in the period allocation unit, the step of allocating phase durations for each phase of the next signal period according to the period duration of the next signal period includes:

[0036] Based on the predicted phase state information of each phase, the duration ratio of each phase is determined;

[0037] The duration of the next signal cycle is allocated to each phase according to the duration ratio of each phase, thus obtaining the phase duration of each phase.

[0038] Ideally, the duration ratio of each phase is positively correlated with the congestion level of the corresponding phase.

[0039] Preferably, in the model processing unit, the predicted intersection state information is used as the state of the reinforcement learning model, and the duration of the next signal cycle is selected as the action of the reinforcement learning model.

[0040] When training the reinforcement learning model, the optional period duration of the next signal period increases as the predicted intersection state information increases, which serves as a training constraint for the reinforcement learning model.

[0041] Preferably, when training the reinforcement learning model, the duration range of the next signal cycle is set to [C]. base -d,C base +d], selecting the period duration of the next signal period within the period duration range; wherein, the C base The reference period duration is positively correlated with the predicted intersection state information, and d is a preset constant.

[0042] Preferably, C is calculated using the Webster single-point algorithm based on the sum of the set critical lane flow ratios. w , will the C w As the C base ;

[0043] If the sum of the critical lane flow rate ratios is less than a set high saturation threshold, then the period duration range of the next signal cycle is set to [C]. w -d,C w +d];

[0044] If the sum of the critical lane flow rate ratios is greater than or equal to the high saturation threshold, the duration range of the next signal cycle is set to [C′]. w C max ]; wherein, the C′ w The reference cycle duration is calculated when the sum of the critical lane flow rate ratios equals the high saturation threshold, and the C... max The maximum allowed cycle duration.

[0045] Preferably, the reward of the reinforcement learning model is a weighted sum of at least one of the following: queue length at the green light start time, number of vehicles in the queue at the green light start time, queue length at the red light start time, number of vehicles in the queue at the red light start time, average vehicle delay, average number of stops per vehicle, and traffic flow within a set time period; wherein, when calculating the weighted sum, each weight is greater than or equal to 0.

[0046] Preferably, in the intersection state prediction unit, the merging process of the predicted phase state information of each phase includes: calculating a weighted sum of the predicted phase state information of each phase.

[0047] Preferably, in the intersection state prediction unit, determining the predicted phase state information of each phase in the next signal cycle based on the acquired historical lane data includes:

[0048] Based on lane data in each phase of a specified historical signal cycle, lane status information for the corresponding phase in each specified historical signal cycle is determined using the signal cycle as the unit.

[0049] Using signal cycles as units, the actual phase state information of the corresponding phase in each specified historical signal cycle is determined based on the lane state information of each phase in each specified historical signal cycle.

[0050] Based on the actual phase state information of each phase in all the specified historical periods, the predicted phase state information of the corresponding phase in the next signal period is determined.

[0051] Preferably, in the intersection state prediction unit, determining the actual phase state information of the corresponding phase in each specified historical signal period includes:

[0052] Based on the lane data of each phase within a specified historical signal period, determine the queue length or number of vehicles in the queue at the green light start time for each lane and corresponding phase, which serves as the lane status information for the corresponding phase within the corresponding historical signal period.

[0053] The actual phase state information of the corresponding phase in the specified historical signal period is determined based on the maximum value of the lane state information of all lanes contained in each phase within the specified historical signal period.

[0054] A computer-readable storage medium storing computer instructions thereon, characterized in that, when the instructions are executed by a processor, they can implement the single-point traffic signal control method described above.

[0055] An electronic device, comprising at least a computer-readable storage medium and a processor;

[0056] The processor is configured to read the executable instructions from the computer-readable storage medium and execute the instructions to implement the single-point traffic signal control method described in any of the preceding claims.

[0057] As can be seen from the above technical solution, this application predicts the predicted phase state information of each phase in the next signal cycle of the traffic signal based on the historical lane data of each phase of the single-point intersection; uses the predicted phase state information of each phase to determine the predicted intersection state information; inputs the predicted intersection state information into a pre-trained reinforcement learning model, and outputs the cycle duration of the next signal cycle corresponding to the predicted intersection state information after processing by the reinforcement learning model; and allocates the duration of each phase to the current signal cycle according to the cycle duration of the current signal cycle. Through the above processing, the phase state information of each phase in the predicted next signal cycle is merged to obtain the predicted intersection state information, which is then used as the input to the reinforcement learning model. Since the input to the reinforcement learning model is no longer an ordered sequence of several phase states, but rather the predicted intersection state information reflecting the overall state of the intersection, the output of the reinforcement learning model is no longer the phase durations corresponding to each phase in sequence, but rather the cycle duration of the entire signal cycle. Therefore, the entire reinforcement learning model reflects the mapping relationship between the overall state of the intersection and the entire cycle duration of the signal cycle, and is no longer related to the order of the phases. This avoids the constraints of the release mode when the ordered phase sequence is used as the input to the reinforcement learning model, making the reinforcement learning model more adaptable to different release modes; at the same time, it also makes the scheme more adaptable to road congestion. Attached Figure Description

[0058] Figure 1 A schematic diagram of the basic structure of a reinforcement learning model;

[0059] Figure 2 This is a schematic diagram of the basic process of the single-point traffic signal control method in this application;

[0060] Figure 3 This is a schematic diagram of the specific flow of the single-point traffic signal control method in a specific embodiment of this application;

[0061] Figure 4 A diagram illustrating queuing and traffic data;

[0062] Figure 5 This is a schematic diagram of the single-port release mode;

[0063] Figure 6 This is a schematic diagram of a common double-ring overlapping pattern;

[0064] Figure 7 This is a schematic diagram showing the relationship between the action space range and the intersection saturation.

[0065] Figure 8 This is a schematic diagram of the basic structure of the single-point traffic signal control device in this application;

[0066] Figure 9 A schematic diagram of the basic structure of the electronic device provided in this application. Detailed Implementation

[0067] To make the objectives, technical means, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings.

[0068] First, let me introduce reinforcement learning models. Reinforcement learning is a major category within the machine learning family. It enables machines to learn how to achieve high scores in an environment through continuous trial and error, accumulating experience and learning from it. Figure 1 A schematic diagram of the basic structure of a reinforcement learning model.

[0069] exist Figure 1 In the reinforcement learning model shown, when an agent performs a task, it first interacts with its environment through action A. Under the influence of action A and the environment, the agent generates a new state, and the environment provides an immediate reward. This process continues, with the agent constantly interacting with the environment to generate a large amount of data. The reinforcement learning algorithm uses this data to modify its action policy, interacts with the environment again to generate new data, and uses this new data to further improve its behavior. After several iterations of learning, the agent eventually learns the optimal action (optimal policy) to complete the task.

[0070] The biggest difference between reinforcement learning and supervised learning is that reinforcement learning lacks the labels used in supervised learning. Reinforcement learning only provides rewards, but these rewards are different from the labels used in supervised learning; they are not given in advance but rather deferred. Furthermore, each step in reinforcement learning is closely related to the sequence of events. In contrast, the training data in supervised learning is generally independent, without this sequential dependency.

[0071] Next, we will introduce the use of reinforcement learning models for single-point traffic signal control. Before proceeding with the details, let's first explain the concept of phase in traffic signal control technology.

[0072] In traffic signal control technology, a "phase" is defined as the display state of a signal group corresponding to one or more traffic flows that simultaneously have the right-of-way. Based on this, the phase of a single-point intersection refers to a certain traffic state at the intersection, such as east-west straight, east-west left turn, north-south straight, and north-south left turn. A signal cycle consists of the sum of all pre-set phase time periods.

[0073] When using reinforcement learning models for single-point traffic signal control, the phase state information of each phase at the intersection can be arranged in a certain order to form a phase state sequence, which is then input into the reinforcement learning model to obtain the phase duration of each phase corresponding to this phase state sequence. However, since the phase state sequence input to the reinforcement learning model is related to the arrangement order of each phase, and the arrangement order of each phase is different under different traffic release modes, the reinforcement learning model trained according to one phase arrangement order performs poorly for other phase arrangement orders. This results in poor generalization of the reinforcement learning model, meaning that the model is not very versatile under different traffic release modes.

[0074] In view of this, the basic idea of ​​this application is to transform the phase states of each phase into non-sequential states for training reinforcement learning models, thereby improving the versatility of reinforcement learning models.

[0075] Figure 2 This is a schematic diagram illustrating the basic flow of the single-point traffic signal control method in this application. Figure 2 As shown, the method includes:

[0076] Step 201: Based on the historical lane data of each phase of the single-point intersection, determine the predicted phase state information of each phase in the next signal cycle of the traffic signal.

[0077] Each phase occupies a different time period within a signal cycle. Within each phase's time period, lane data is acquired corresponding to that time period; this is called the lane data for that phase. Based on the historical lane data for each phase, the predicted phase state information for that phase is determined.

[0078] Step 202: Merge the phase state information of each predicted phase to obtain the predicted intersection state information.

[0079] To remove the sequentiality between phases, the phase state information of each phase is merged to obtain the predicted intersection state information. This predicted intersection state information is used to measure the overall congestion level of a single-point intersection, including all phases. Specifically, the merging process for the phase state information can be achieved through methods such as weighted summation. This weighted sum reflects the overall condition of each phase, i.e., the state of the single-point intersection. Based on this, the weighted sum of the phase state information of each phase is called the intersection state information. The predicted intersection state information is obtained by merging the predicted phase state information of each phase. Of course, this example only uses weighted summation; in practical applications, appropriate merging methods can be selected as needed, such as weighted average, exponential weighted average, or other merging methods.

[0080] Step 203: Input the predicted intersection state information into the pre-trained reinforcement learning model. After processing by the reinforcement learning model, output the duration of the next signal cycle corresponding to the predicted intersection state information.

[0081] In this application, the predicted intersection state information is input into the reinforcement learning model. The model output is no longer the specific duration of each phase, but the sum of the durations of each phase, which is the period duration of the signal cycle. Specifically, this signal cycle duration is the predicted period duration of the next signal cycle.

[0082] Step 204: Assign phase duration to each phase of the next signal cycle according to the cycle duration of the next signal cycle.

[0083] When timing a signal for a single point, the duration of each phase needs to be determined. Step 203 determines the sum of the durations of all phases, which is the period duration. This step allocates a specific duration to each phase based on this total sum of phase durations. The allocation of phase durations can be based on the different phase states of each phase.

[0084] This concludes the single-point traffic signal control method process in this application. Through the above process, the intersection state information, reflecting the overall state of each phase, is used as input to the reinforcement learning model, thereby enhancing the model's adaptability to different release patterns. Furthermore, leveraging the advantages of reinforcement learning, the solution can also effectively adapt to road saturation conditions.

[0085] The specific implementation of this application is illustrated below through a specific embodiment.

[0086] Figure 3 This is a schematic diagram illustrating the specific process of single-point traffic signal control in a particular embodiment of this application. For example... Figure 3 As shown, the method includes:

[0087] Step 301: The detection equipment collects and reports lane data of a single intersection in real time.

[0088] The device for detecting lane data can be installed on the signal poles at the exit lanes of an intersection, thereby covering every lane of all approach lanes at the intersection. This allows for the real-time acquisition of lane data for each lane, such as queue length, number of vehicles in the queue, and departure flow.

[0089] Real-time detection data will be uploaded to traffic signals and the regional central control server, and reported opportunistically based on the configuration of the detection devices. Radar-type detection devices typically upload data at a high frequency, uploading data to edge nodes such as traffic signals at a frequency of 1Hz, thus allowing data from very small time units to be reported. Video-type detection devices upload data at a lower frequency compared to radar-type devices, and data can be uploaded based on vehicle passage events.

[0090] Step 302: Preprocess the reported lane data.

[0091] Based on the lane data reported by the detection equipment and combined with the signal control scheme for the current signal cycle, the lane data reported by the detection equipment is preprocessed to obtain feature data at key time points. Furthermore, the preprocessing and storage of lane data should be continuous to constantly generate data for determining the signal control scheme for the next signal cycle.

[0092] Specifically, the lane data acquired in the current cycle is data obtained by the detection equipment at certain intervals, such as 1 second. However, when generating traffic signal control schemes, data from specific moments or durations is required, such as lane data at the moment the green light turns on or lane flow within the current signal cycle. Therefore, this step requires preprocessing the reported lane data to extract the necessary lane data or accumulate the corresponding lane data within a specific duration.

[0093] The lane data mainly used in this embodiment is:

[0094] (1) Queuing data: the queue length and number of vehicles in the queue when the green light turns on in each phase, and the queue length and number of vehicles in the queue when the red light turns on in each phase. These queuing data are obviously data at characteristic time points. It is necessary to combine the signal control scheme of the current signal cycle to determine the red or green light turning on time of each phase, and then obtain the corresponding data at the corresponding time point from the reported data.

[0095] (2) Traffic flow data: Departure traffic during the green light period usually refers to statistical data within a set time period;

[0096] (3) Flow velocity data: the average flow velocity of the departing flow during the phase green light period.

[0097] like Figure 4 As shown, taking a single-loop 4-phase release mode as an example, a schematic diagram of queuing data and flow data is given. Figure 4 As can be seen, in the time dimension, each phase occupies a different time period in a signal cycle. Within the time period occupied by each phase, there is lane data acquired for that time period, which is called lane data in that phase. The relevant data of each lane in each phase needs to be processed and stored according to the signal cycle to obtain the corresponding queuing data and traffic flow data.

[0098] Step 303: Based on the historical lane data of each phase of the single-point intersection, determine the predicted phase state of each phase in the next signal cycle.

[0099] In this application, the timing scheme for the next signal cycle is predicted based on historical lane data. Therefore, it is necessary to predict the phase state information of each phase in the next cycle to obtain the predicted phase state information.

[0100] The predicted phase state information for the next signal cycle can be predicted based on the actual phase state information of several historical signal cycles. Specific processing may include:

[0101] Based on lane data in each phase of a specified historical signal period, the lane status information of the corresponding phase in that specified historical signal period is determined, using signal period as the unit. Based on lane status information in each phase of each specified historical signal period, the actual phase status information in that specified historical signal period is determined, using signal period as the unit. Finally, based on the actual phase status information of each phase in all specified historical periods, the predicted phase status information of the corresponding phase in the next signal period is determined.

[0102] As can be seen from the above processing, when predicting the predicted phase state information for the next signal cycle, it is first necessary to determine the actual phase state information of each phase in several specified historical signal cycles, on a signal cycle basis. Then, based on the actual phase state information of each phase in all specified historical signal cycles, the actual phase state information of each phase can be calculated from the lane state information contained in the phase for each historical signal cycle. Generally, lane state information can be used to reflect the arrival, departure, and queuing status of vehicles in that lane. For example, lane state information can be determined based on queuing data and traffic flow data from the aforementioned lane data.

[0103] Here's a concrete example: the number of vehicles queuing in a lane at the moment the green light turns on can be used as the lane state. For each phase, the maximum value of the lane state across all lanes within a historical signal cycle is taken as the actual phase state g_n for the corresponding phase within that signal cycle. Then, based on the phase states g_n(tn)…g_n(t-2), g_n(t-1), g_n(t) within several specified historical signal cycles, the predicted phase state for the same phase in the next signal cycle is determined. Specifically, various existing prediction methods can be used to determine the predicted phase state for the next cycle based on the actual phase states of historical signal cycles.

[0104] Step 304: Determine the predicted intersection status information based on the predicted phase status information of each phase.

[0105] When using reinforcement learning models for traffic signal timing, the state information and duration (or green ratio) of each phase are typically used as the state and action of the reinforcement learning model, thus obtaining a corresponding list of states and actions. However, since phase is an ordered parameter, the expression of the state and action list implicitly contains the concept of phase order, which is related to the intersection's release pattern. For example, a common single-way release pattern (i.e., the release order) is "East-West-South-North," as shown below. Figure 5 As shown.

[0106] Specifically, in reinforcement learning training, the state is represented as predicted phase state information S = (s1, s2, s3, s4), and the corresponding action is designed as predicted phase duration A = (a1, a2, a3, a4). However, the phase order is the same as the intersection's traffic release pattern. Therefore, if an intersection or a traffic release pattern is changed, for example, "East-South-West-North", the model trained according to the phase states corresponding to the original traffic release pattern will not be applicable to single-point intersections with changed traffic release sequences.

[0107] Furthermore, in cases where the release pattern is relatively simple, the originally trained model can be used by changing the phase sequence. However, in cases where the release pattern is complex, such as common ones... Figure 6 As shown in the double-ring overlapping pattern, the trained model will be relatively specific and difficult to apply to other release patterns.

[0108] In view of this, in order to improve generalization and reduce the state space, the state of the input reinforcement learning model is adjusted in this application. Instead of using a predicted phase sequence S = (s1, s2, s3, s4) as the state, the overall state of the intersection is determined by merging the predicted phase state information of each phase, which is called the predicted intersection state information. The predicted intersection state information is used as the input of the reinforcement learning model. This step is used to merge the predicted phase state information. In this embodiment, the merging process is implemented by weighted summation as an example to illustrate the determination of the predicted intersection state information, as shown in Equation (1).

[0109]

[0110] Where, k i For phase state information s i The weights, specifically k i The value of k can be set according to the needs of the actual scenario. For example, all k can be set. i Set to 1. In this way, this application no longer uses an ordered sequence of several predicted phase state information as the state, but uses the predicted intersection state information of the entire intersection determined by each predicted phase state information as the input state of the reinforcement learning model, thereby avoiding the input order of the phase sequence and being applicable to different release modes.

[0111] In practical engineering implementation, in order to reduce the state space, the predicted intersection state information S can be discretized. For example, the intersection state information can be mapped to a state space with an interval of 5 [5,10,15,..] based on the principle of rounding up.

[0112] Step 305: Input the predicted intersection state information into the pre-trained reinforcement learning model. After processing by the reinforcement learning model, output the duration of the next signal cycle corresponding to the predicted intersection state information.

[0113] In this step, the predicted intersection state information determined in step 304 is input into the reinforcement learning model to obtain the cycle duration of the next signal cycle corresponding to the predicted intersection state information.

[0114] Specifically, in reinforcement learning models, actions and states are corresponding. In this application, the input to the reinforcement learning model is the predicted intersection state information, which in this embodiment is the weighted sum of the predicted phase state information. Therefore, the corresponding action should also be the sum of the actions of each phase, that is, the sum of the phase durations of each phase, i.e., the period duration of the signal cycle. The theoretical support for this is that, in the field of signal control, under reasonable signal allocation, the overall average delay of an intersection is related to the period size, so the period duration can be directly used as the action.

[0115] Furthermore, as mentioned earlier, the predicted intersection state information can also be processed using other merging methods. Since merging processing is performed, regardless of the method used, it reflects the merged information of each predicted phase state, i.e., the overall state of the intersection, and correspondingly, it can be correlated with the signal cycle duration. For example, when the merging processing is a weighted average, the state can correspond to the average duration of each phase. Then, multiplying the average duration by the number of phases can also yield the signal cycle duration, which can still be used as the action of the reinforcement learning model.

[0116] In practical engineering implementation, in order to reduce the motion space, the motion can be discretized. Only a portion of the motion is retained in the motion space, and the maximum and minimum values ​​of the motion space are given according to the actual upper and lower limits.

[0117] In other words, the reinforcement learning model in this application provides a mapping relationship between the predicted intersection state information and the duration of the next signal cycle. Therefore, based on the input predicted intersection state information, the duration of the next signal cycle can be determined.

[0118] Step 306: Allocate the duration of each phase to the next signal cycle according to the cycle duration of the next signal cycle.

[0119] The duration of the next signal cycle has been determined by the aforementioned reinforcement learning model. Considering that signal timing requires the duration of each phase, it is necessary to further allocate a duration to each phase based on the duration of the next signal cycle.

[0120] Regarding the allocation of periods, this application provides an analytical knowledge model, which calculates the predicted phase state s of each phase based on the aforementioned step 203. i First, determine the duration ratio of each phase; then, allocate the duration of the next signal cycle to each phase according to the duration ratio of each phase, thus obtaining the duration g of each phase. i ,Right now

[0121]

[0122] Where C is the duration of the next signal cycle.

[0123] The duration ratio of each phase can be positively correlated with the congestion level of the corresponding phase. For example, when the actual phase state of the historical signal period used to calculate the predicted phase state is determined by the queue length or the number of vehicles in the queue, the phase duration ratio can be allocated according to the corresponding queue length or the number of vehicles in the queue. Generally, the larger the queue length or the number of vehicles in the queue, the longer the phase duration.

[0124] At this point, Figure 3The traffic signal control method flow in the illustrated embodiment has ended.

[0125] In addition, in the single-point traffic signal control method of this application, the acquisition of lane state, phase state information and predicted intersection state information are all done using analytical methods. Then, a reinforcement learning model is used to determine the duration of the next signal cycle, and finally, an analytical method is used to determine the duration of each phase.

[0126] The above processing method directly uses a pre-trained reinforcement learning model. The following explains the relevant processing during the training of the reinforcement learning model.

[0127] The reinforcement learning model in this application can be trained online using a large amount of real-world data. However, considering the low efficiency of online training and the low rate of real-world case exploration, a simulation-based training method is preferred. Regarding the specific training method, any data-driven machine learning method is acceptable; Q-Learning reinforcement learning will be used as an example below.

[0128] A basic simulation model can be built based on traffic simulation software, establishing fundamental road network and traffic flow information. This simulation model then generates various data required for reinforcement learning training. Model training can be performed in multiple rounds for the same scenario, with each round representing a complete scenario run. In traffic signal control, if the simulation scenario duration is 3600 seconds, each simulation round will run for 3600 seconds. In practical engineering training, a multi-process framework is often used to accelerate training, which will not be elaborated upon here.

[0129] During specific training, intersection state information is calculated based on simulation data, and the cycle duration is used as the action. For example, the weighted sum of predicted phase state information is used to predict intersection state information. Figure 1 As shown, during training, the reward corresponding to the selected action is obtained based on the feedback from the environment. Then, the reward for that action is learned in a certain way. When using a Q-Learning reinforcement learning model, this manifests as updating the Q-table. After all rounds of training are completed, the reinforcement learning model file can be obtained by processing the final updated Q-table through steps such as pruning and format conversion.

[0130] The reward is used to represent the quality of the allocated time period. Different objectives correspond to different rewards. In single-point traffic control scenarios, commonly used metrics include the queue at the start of red / green lights, average vehicle delay, average number of stops per vehicle, and traffic volume. Therefore, the reward design can be expressed as:

[0131] R=k1Queue+k2Delay+k3Stop+k0Volume

[0132] Specific scenarios can be tailored with appropriate weights based on different control objectives. Each weight can be greater than or equal to 0; a weight of 0 indicates that the reward does not involve the corresponding parameter. The parameters mentioned above (e.g., queue at traffic light start time, average vehicle delay, average number of stops per vehicle, and traffic volume) can be real-time, used to calculate immediate rewards, or historical averages, used to calculate the model's historical rewards, such as rewards from the beginning of training to the current training phase.

[0133] Furthermore, the optimal actions obtained by training directly using reinforcement learning may be randomly distributed and poorly correlated with the state. While a set of results may be usable in its training scenario, it may not be ideal for other scenarios. In other words, actions obtained through completely random exploration are more suitable for the training scenario, and their generalization ability needs improvement. Considering that in real-world applications, as intersection congestion increases, the given optimal cycle time generally also increases, meaning there is a positive correlation between predicted congestion states and optimal actions, this existing knowledge can be used to impose a positive correlation constraint on the action space for the current state. This aims to improve the generalization usability of the training results. Additionally, appropriately reducing the length of the corresponding action space based on the current state can reduce meaningless exploration and avoid unreasonable solutions, thereby improving the algorithm's training efficiency and effectiveness.

[0134] Specifically, when training a reinforcement learning model, the duration range of the next signal cycle (i.e., the action space) can be set to [C]. base -d,C base +d], and select the period duration of the next signal period within this period duration range. C w C is a baseline cycle length determined based on the current intersection conditions before each action selection. To achieve a positive correlation between the state and the corresponding action space, C... base The value is positively correlated with the predicted intersection state information. 'd' is a preset constant that can be set according to the actual scenario, for example, 'd = 20 seconds'. The above processing ensures that as the predicted intersection state information increases, the selectable cycle length within the action space increases, and the length of the action space (i.e., 2*d) can be set according to the actual scenario.

[0135] Below is a specific example of a state-action space constraint method, which may include: during the training of a reinforcement learning model, before each action selection (i.e., before determining the duration of the next signal cycle), the Webster single-point algorithm, a classic algorithm in traffic control, is used to calculate the baseline cycle duration C based on the sum of the flow rate ratios of preset key lanes. w , the C w As mentioned above, C base When constraining the action space corresponding to a certain state, the currently available action space can be restricted to [C]. w -20°C w +20], that is, the actual range of selectable actions is defined by a 20-second fluctuation around the baseline period (the action space still needs to be discretized; this step is just to constrain its upper and lower bounds again). The baseline period is calculated using the sum of the critical lane flow ratios. Since the sum of the critical lane flow ratios reflects the saturation of the intersection and is related to the predicted intersection state, and the Webster single-point algorithm itself ensures that the calculated baseline period increases with the overall saturation, the action space based on the state-Webster single-point period also generally exhibits this trend. Figure 7 This diagram illustrates the relationship between the action space range and intersection saturation. When calculating the baseline period using the Webster algorithm, some additional considerations are needed. In practical applications, the Webster single-point algorithm is typically not suitable for high saturation conditions. Therefore, the action space calculation based on the baseline period is only applicable when the saturation is below a certain level. For high saturation conditions, the action space setting can be further restricted. A specific example is given below, where the action space is set to [C] when the sum of the overall critical lane flow ratios at the intersection is less than a set high saturation threshold (e.g., 0.8). w -20°C w +20], when the sum of the overall critical lane flow ratios at the intersection is greater than or equal to a high saturation threshold (e.g., 0.8), the action space can be set to [C′]. w C max ], where C′ w C is the baseline cycle length calculated when the sum of the critical lane flow ratios equals a high saturation threshold (e.g., 0.8). max The maximum allowed cycle duration.

[0136] Based on this, optionally in this application, when selecting actions during reinforcement learning model training, a trend constraint is applied to the action space that increases with the increase of predicted intersection state information, to ensure the generalization ability of the trained model. That is, the selectable period length of the signal cycle increases with the increase of intersection state information, serving as a training constraint for the reinforcement learning model. Through training with the aforementioned constraints, the trained reinforcement learning model can possess better generalization ability and be further applied to more scenarios.

[0137] It is important to note that the selection of actions (i.e., cycle duration) has a unique approach in reinforcement learning training. Generally, random actions are used extensively at the beginning of the training phase to increase the scope of exploration. As the training rounds progress, the probability of random exploration is gradually reduced, while the probability of actions that have performed well under the same conditions in the past is gradually increased.

[0138] The above describes the specific implementation of the single-point traffic signal control method in this application.

[0139] In the control method of this application, the cycle calculation problem of a single-point intersection is decomposed into sub-problems such as lane state recognition, actual phase state recognition, phase state prediction, intersection state information recognition, cycle calculation corresponding to intersection state information, and time allocation of each phase in the cycle. The divide-and-conquer approach is used to comprehensively adopt different solutions based on the different characteristics of these different problems.

[0140] Specifically, for simple, modelable sub-problems (such as lane state recognition, actual phase state recognition, phase state prediction, intersection state recognition, and time allocation of each phase in a cycle), an analytical model is established, and a knowledge-based approach is used to solve them, avoiding the instability and difficulty in interpretation inherent in data-driven methods. For the highly complex nonlinear optimal cycle problem, a data-driven approach is used, i.e., a reinforcement learning model is trained. This training process only requires constructing a scenario in simulation to obtain a usable agent, avoiding the difficulty of modeling complex nonlinear problems that analytical algorithms require. In other words, this application comprehensively adopts a data-plus-knowledge-driven approach for the generation of periodic single-point signal control schemes.

[0141] The divide-and-conquer approach makes each problem independent and meaningful, and each step interpretable. This approach also facilitates continuous iterative improvement of the algorithm; as long as the accuracy of solving each individual problem is continuously improved, the overall effectiveness of the method will be enhanced.

[0142] As described above, the single-point traffic control method of this application connects the training process and the actual project through a model file. Therefore, this solution can continuously improve the on-site control solution by continuously training and optimizing the reinforcement learning model, that is, by continuously updating the on-site model file.

[0143] More specifically, in reinforcement learning models, the design of states and actions adopts the merging of states (e.g., the merging state) and actions. Instead of using an ordered sequence of several phase states as the state, the state is the merging information of all phases of the entire intersection (i.e., the merging state). Instead of using an ordered sequence of several phase durations as the action, the action is the sum of all phase durations of the entire intersection (i.e., the sum action).

[0144] The above-mentioned design of merging states and actions can be integrated with multiple release modes, avoiding the problem that data-driven algorithms are limited to specific phase sequence release modes due to the direct use of lanes or phases as the dimensions of states and actions during training. This improves the generalization ability of the algorithm to different release modes and the possibility of engineering implementation.

[0145] Furthermore, knowledge-based relevance constraints can be incorporated into the state-action generation process. This requires that the action (i.e., the duration of the next signal cycle) increases as the predicted state information (i.e., the predicted intersection state information) increases, rather than directly training data-driven algorithms to obtain patterns in states and actions. This approach, based on on-site control experience and incorporating knowledge-based relevance design, effectively increases the algorithm's generalization ability and achieves better overall results across various scenarios.

[0146] This application also provides a single-point traffic signal control device, which can be used to implement the above-described traffic signal control method. Figure 8 This is a schematic diagram of the basic structure of the single-point traffic signal control device in this application. Figure 8 As shown, the device includes: an intersection state prediction unit, a model processing unit, and a cycle allocation unit.

[0147] The intersection state prediction unit is used to determine the predicted phase state information of each phase in the next signal cycle of the traffic signal based on the historical lane data of each phase of the acquired single-point intersection; it is also used to merge the predicted phase state information of each phase to obtain the predicted intersection state information; wherein, the predicted intersection state information is used to measure the overall congestion level of the intersection, including each phase.

[0148] The model processing unit is used to input the predicted intersection state information into the pre-trained reinforcement learning model, and after processing by the reinforcement learning model, output the cycle duration of the next signal cycle corresponding to the predicted intersection state information.

[0149] The period allocation unit is used to allocate phase duration to each phase of the next signal cycle according to the period duration of the next signal cycle.

[0150] Optionally, in the period allocation unit, the phase duration of each phase of the next signal period is allocated according to the period duration of the next signal period, which may specifically include:

[0151] Based on the predicted phase state information of each phase, determine the duration ratio of each phase;

[0152] The duration of the next signal cycle is allocated to each phase according to the duration ratio of each phase, thus obtaining the phase duration of each phase.

[0153] Optionally, the duration ratio of each phase is positively correlated with the congestion level of the corresponding phase.

[0154] Optionally, in the model processing unit, the predicted intersection state information is used as the state of the reinforcement learning model, and the duration of the next signal cycle is selected as the action of the reinforcement learning model.

[0155] When training a reinforcement learning model, the optional period length of the next signal period increases as the predicted intersection state information increases, which serves as a training constraint for the reinforcement learning model.

[0156] Optionally, when training the reinforcement learning model, the duration range of the next signal cycle is set to [C]. base -d,C base +d], select the period duration of the next signal period within the period duration range; where, C base The baseline period is the duration that is positively correlated with the predicted intersection status information, and d is a preset constant.

[0157] Optionally, C can be calculated using the Webster single-point algorithm based on the sum of the set critical lane flow ratios. w , will the C w As the C base ;

[0158] If the sum of the critical lane flow rates is less than the set high saturation threshold, then the cycle duration range of the next signal cycle is set to [C]. w -d,C w +d];

[0159] If the sum of the key lane flow rates is greater than or equal to the high saturation threshold, the cycle duration range for the next signal cycle is set to [C′]. w C max ]; where C′ wC is the baseline cycle length calculated when the sum of the critical lane flow rate ratios equals the high saturation threshold. max The maximum allowed cycle duration.

[0160] Optionally, the reward of the reinforcement learning model is a weighted sum of at least one of the following: queue length when the green light turns on, number of vehicles in the queue when the green light turns on, queue length when the red light turns on, number of vehicles in the queue when the red light turns on, average vehicle delay, average number of stops per vehicle, and traffic flow within a set time period; wherein, each weight is greater than or equal to 0 when calculating the weighted sum.

[0161] Optionally, in the intersection state prediction unit, merging the predicted phase state information of each phase includes: calculating a weighted sum of the predicted phase state information of each phase.

[0162] Optionally, in the intersection state prediction unit, the predicted phase state information for each phase in the next signal cycle is determined based on the acquired historical lane data, which may specifically include:

[0163] Based on lane data in each phase of a specified historical signal cycle, lane status information for the corresponding phase in each specified historical signal cycle is determined using the signal cycle as the unit.

[0164] Using signal cycles as units, the actual phase state information of the corresponding phase in each specified historical signal cycle is determined based on the lane state information of each phase in each specified historical signal cycle.

[0165] Based on the actual phase state information of each phase in all specified historical periods, determine the predicted phase state information of the corresponding phase in the next signal period.

[0166] Optionally, in the intersection state prediction unit, the process of determining the actual phase state information of the corresponding phase in each specified historical signal period may specifically include:

[0167] Based on the lane data of the corresponding phase within a specified historical signal period, determine the queue length or number of vehicles in the queue at the green light start time for each lane and the corresponding phase, which serves as the lane status information for the corresponding phase within the corresponding historical signal period.

[0168] The actual phase state information for the corresponding phase in the specified historical signal period is determined based on the maximum value of lane state information in all lanes contained in each phase within the specified historical signal period.

[0169] This application also provides a computer-readable storage medium that stores instructions, which, when executed by a processor, can perform the steps in the single-point traffic signal control method described above. In practical applications, the computer-readable medium may be included in the devices / apparatus / systems of the above embodiments, or it may exist independently and not assembled into the device / apparatus / system. The instructions stored in the computer-readable storage medium, when executed by a processor, can perform the steps in the single-point traffic signal control method described above.

[0170] According to the embodiments disclosed in this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof, but not intended to limit the scope of protection of this application. In the embodiments disclosed in this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0171] Figure 9 An electronic device is also provided for this application. For example... Figure 9 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0172] The electronic device may include a processor 901 with one or more processing cores, a memory 902 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 902 is executed, a single-point traffic signal control method can be implemented.

[0173] Specifically, in practical applications, this electronic device may also include components such as a power supply 903 and an input / output unit 904. Those skilled in the art will understand that... Figure 9 The structure of the electronic device shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0174] The processor 901 is the control center of the electronic device. It connects various parts of the electronic device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 902, and calling data stored in the memory 902, it performs various functions of the server and processes data, thereby monitoring the electronic device as a whole.

[0175] Memory 902 can be used to store software programs and modules, i.e., the aforementioned computer-readable storage medium. Processor 901 executes various functional applications and data processing by running the software programs and modules stored in memory 902. Memory 902 may primarily include a program storage area and a data storage area, wherein the program storage area may store the operating system, application programs required for at least one function, etc.; the data storage area may store data created according to the use of the server, etc. In addition, memory 902 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory 902 may also include a memory controller to provide processor 901 with access to memory 902.

[0176] The electronic device also includes a power supply 903 that supplies power to the various components. This power supply can be logically connected to the processor 901 via a power management system, enabling functions such as charging, discharging, and power consumption management. The power supply 903 may also include one or more DC or AC power supplies, a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator, or any other components.

[0177] The electronic device may also include an input / output unit 904, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. The input unit output 904 can also be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof.

[0178] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A single-point traffic signal control method, characterized in that, Including: Determine the predicted phase state information of each phase in the next signal cycle of the traffic signal according to the historical lane data in each phase of the obtained single-point intersection; Perform a merging process on the predicted phase state information of each phase to obtain the predicted intersection state information; wherein, the predicted intersection state information is used to measure the overall congestion degree of the intersection including each phase; Input the predicted intersection state information into a pre-trained reinforcement learning model, and after being processed by the reinforcement learning model, output the cycle duration of the next signal cycle corresponding to the predicted intersection state information; According to the cycle duration of the next signal cycle, allocate the phase duration for each phase of the next signal cycle.

2. The method according to claim 1, wherein The allocating the phase duration for each phase of the next signal cycle according to the cycle duration of the next signal cycle includes: Determine the duration ratio of each phase according to the predicted phase state information of each phase; Allocate the cycle duration of the next signal cycle to each phase according to the duration ratio of each phase to obtain the phase duration of each phase.

3. The method according to claim 2, wherein The duration ratio of each phase is positively correlated with the congestion degree of the corresponding phase.

4. The method according to claim 1, wherein The predicted intersection state information is used as the state of the reinforcement learning model, and the cycle duration of the next signal cycle is selected as the action of the reinforcement learning model; Taking that the optional cycle duration of the next signal cycle becomes longer as the predicted intersection state information increases as the training constraint of the reinforcement learning model.

5. The method according to claim 4, wherein When training the reinforcement learning model, set the cycle duration range of the next signal cycle to , and select the cycle duration of the next signal cycle within the cycle duration range; where the is a reference cycle duration positively correlated with the predicted intersection state information, and d is a preset constant.

6. The method according to claim 5, wherein Calculated using Webster's single-point algorithm based on the sum of the set critical lane flow ratios , and taking the said as the said ; If the sum of the key lane flow rate ratios is less than the set high saturation threshold, the cycle length range of the next signal cycle is set to ; If the sum of the critical lane flow rate ratios is greater than or equal to the high saturation threshold, set the cycle length range of the next signal cycle to ; where, the is the reference cycle length calculated when the sum of the critical lane flow rate ratios is equal to the high saturation threshold, and the is the maximum allowable cycle length.

7. The method according to claim 1, wherein The reward of the reinforcement learning model is: the weighted sum of at least two of the queue length at the moment of green light start, the number of queuing vehicles at the moment of green light start, the queue length at the moment of red light start, the number of queuing vehicles at the moment of red light start, the average vehicle delay, the average vehicle stop times, and the traffic flow within a set time; wherein, each weight value is greater than or equal to 0 when calculating the weighted sum.

8. The method according to claim 1, wherein The performing a merging process on the predicted phase state information of each phase includes: calculating the weighted sum of the predicted phase state information of each phase.

9. The method according to claim 1, wherein The determining the predicted phase state information of each phase in the next signal cycle according to the obtained historical lane data includes: Taking the signal cycle as a unit, determine the lane state information of the corresponding phase in each specified historical signal cycle according to the lane data in each phase of the specified historical signal cycle; Taking the signal cycle as a unit, determine the actual phase state information of the corresponding phase in each specified historical signal cycle according to the lane state information in each phase of each specified historical signal cycle; Determine the predicted phase state information of the corresponding phase in the next signal cycle according to the actual phase state information of each phase in all the specified historical cycles.

10. The method according to claim 9, wherein The determining the actual phase state information of the corresponding phase in each specified historical signal cycle includes: According to the lane data of the corresponding phase in the specified historical signal cycle, determine the queue length or the number of queuing vehicles at the moment of green light start of each lane corresponding to the phase as the lane state information of the corresponding phase in the corresponding historical signal cycle; Determine the actual phase state information of the specified historical signal cycle for the corresponding phase according to the maximum value of the lane state information in all lanes included in each phase within the specified historical signal cycle.

11. A single-point traffic signal control device, characterized in that, Including: An intersection state prediction unit, a model processing unit, and a cycle allocation unit; The intersection state prediction unit is configured to determine the predicted phase state information of each phase in the next signal cycle of the traffic signal according to the obtained historical lane data of each phase of the single-point intersection; and is further configured to perform a merging process on the predicted phase state information of each phase to obtain predicted intersection state information; wherein, the predicted intersection state information is used to measure the overall congestion degree of the intersection including each phase. The model processing unit is configured to input the predicted intersection state information into a pre-trained reinforcement learning model, and output the cycle duration of the next signal cycle corresponding to the predicted intersection state information after being processed by the reinforcement learning model. The cycle allocation unit is configured to allocate phase durations to each phase of the next signal cycle according to the cycle duration of the next signal cycle.

12. A computer-readable storage medium having computer instructions stored thereon, characterized in that, When the instruction is executed by a processor, it can implement the single-point traffic signal control method according to any one of claims 1 to 10.

13. An electronic device, characterized in that, The electronic device at least includes a computer-readable storage medium and further includes a processor; The processor is configured to read an executable instruction from the computer-readable storage medium and execute the instruction to implement the single-point traffic signal control method according to any one of claims 1 to 10 above.

Images