A universal model system for signal light control and signal light control method

Abstract

In order to reduce the time of model training and increase the possibility of large-scale deployment of traffic light control, the present invention provides a general model system for traffic light control, including a feature extraction module and an action prediction module. The feature extraction module is used to convert the traffic flow information of the intersection into an intersection information feature vector, wherein the traffic flow information includes the average traffic flow under each traffic flow signal, the maximum lane occupancy within K seconds, the average occupancy, the direction of movement, the number of lanes included, the duration of the movement, whether the duration is the minimum green light time, and whether the current signal is green. The action prediction module is used to predict the duration of the current traffic light state of the intersection based on the intersection information feature vector. Using "setting the current phase duration" as an action not only retains the concept of the traffic cycle, but also ensures the safety of traffic participants such as drivers and pedestrians.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic control, and in particular to a universal model system for signal light control and a signal light control method. Background Art

[0002] Proper control of traffic lights can effectively reduce congestion in road networks. However, traditional fixed-mode traffic light control is no longer sufficient to cope with increasingly complex road network conditions. With the success of deep reinforcement learning (DRL) in various fields, research has begun to use DRL methods to control traffic lights, enabling timely adjustment of traffic light states based on real-time road network conditions. This approach can significantly reduce average vehicle waiting times. Some studies have used value-based approaches, while others have used policy-based approaches. The action designs used in these studies vary, including "selecting the next phase" and "maintaining or switching to the next phase."

[0003] While existing reinforcement learning-based methods have made significant progress in traffic light control, they still require training new models for new intersections with different structures. Specifically, when faced with intersections of varying structures, such as intersections of varying shapes (crossroads, T-junctions), varying numbers of lanes, and varying signal phase structures, the model must be redesigned and trained from scratch. This necessitates training and obtaining optimal policies for every signal light in a large-scale urban road network, which consumes considerable time and computational resources. To address this issue, several studies have attempted to train universal models for traffic light control. For example, Chen et al. employed a parameter sharing method based on FRAP to adapt to new scenarios, demonstrating promising performance at scales involving thousands of traffic lights. However, this approach still employs the "next phase selection" as the agent's action design. Once the agent selects an action, the signal phase can be selected from all non-conflicting combinations, thus ignoring pedestrian perception and violating driver driving habits. Consequently, it is difficult to maintain the original traffic light phase structure. For example, some studies have used the AttendLight algorithm and combined it with an attention mechanism to train a universal model for intersections with different structures and traffic flow distributions. While this universal model can maintain the phase structure of the intersection, its action design also involves "selecting the next phase," which can cause traffic signals to change in a random order, potentially putting drivers and pedestrians in unsafe situations. Summary of the Invention

[0004] In order to solve some or all of the problems in the prior art, the present invention provides a general model system for signal light control, which includes:

[0005] a feature extraction module for converting traffic flow information at the intersection into an intersection information feature vector, wherein the traffic flow information includes average traffic flow under each traffic flow signal, maximum lane occupancy within K seconds, average occupancy, lane function, number of lanes, duration of movement, whether the duration is the minimum green time, and whether the current signal is green; and

[0006] An action prediction module is used to predict the duration of the current traffic light state of the intersection based on the intersection information feature vector.

[0007] Furthermore, the general model system also includes a state value prediction module, which is used to predict the total reward obtained from each state to the end of the simulation based on the intersection information feature vector, wherein the reward includes the negative queue length.

[0008] Furthermore, the traffic flow signals include: northbound, northbound left turn, eastbound, eastbound left turn, westbound, westbound left turn, southbound, and southbound left turn.

[0009] Furthermore, if there are less than eight actual traffic flow signals at the intersection, the traffic flow information corresponding to the missing traffic flow signals is recorded as 0.

[0010] Furthermore, the feature extraction module includes a multi-layer structure:

[0011] A traffic flow information extraction layer, which is used to extract information about each traffic flow; and

[0012] The intersection information extraction layer is used to extract intersection information based on the traffic flow information.

[0013] Furthermore, the traffic flow information extraction layer and the intersection information extraction layer both use one-dimensional convolutional layers, and

[0014] The feature extraction module further includes a flattening layer and at least one fully connected layer.

[0015] Furthermore, the action prediction module includes at least one fully connected layer.

[0016] Furthermore, the state value prediction module includes at least one fully connected layer.

[0017] Furthermore, the duration of the current traffic light state includes the duration of the current green light, which is selected from a preset duration set according to the intersection information feature vector.

[0018] Furthermore, each duration in the preset duration set ranges from 5 to 70 seconds.

[0019] Furthermore, the difference between any two durations in the preset duration set is no more than 5 seconds.

[0020] Based on the general model system described above, another aspect of the present invention provides a signal light control method, comprising:

[0021] Extracting intersection information feature vectors by the feature extraction module;

[0022] Calculating the duration of the green light at the current intersection according to the intersection information feature vector by the action prediction module; and

[0023] The green light of the control signal light switches to yellow or red after the stated duration.

[0024] Furthermore, the signal light control method further includes:

[0025] The general model is trained, wherein the training set used for training is enhanced by shuffling the traffic turning flow.

[0026] Furthermore, the traffic diversion flow disorder includes:

[0027] After the traffic flow information of the intersection is obtained, the traffic flow information of the intersection is disrupted to obtain new traffic flow information of the intersection.

[0028] Furthermore, the training uses negative queue lengths as rewards.

[0029] The present invention provides a universal model system for signal light control and a signal light control method, which designs a universal model for intersections of different structures, thereby effectively reducing the time for model training, making large-scale deployment of signal light control possible. At the same time, the universal model and signal light control method adopt "setting the current phase duration" as an action, which not only retains the concept of traffic cycle, but also ensures the safety of drivers, pedestrians and other traffic participants. Through testing using an urban traffic simulator, it was found that the performance of the universal model is close to that of the model trained directly in a single environment, and its model training time can be reduced by more than 80% compared to the model trained directly in a single environment. Therefore, a large amount of computing resources can be saved when there are many intersections, making it possible to apply it on a large scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] To further illustrate the above and other advantages and features of various embodiments of the present invention, a more detailed description of various embodiments of the present invention will be presented with reference to the accompanying drawings. It will be understood that these drawings depict only typical embodiments of the present invention and are not to be considered as limiting the scope thereof. In the drawings, for clarity, identical or corresponding components will be represented by the same or similar reference numerals.

[0031] Figure 1A schematic structural diagram showing a general model system for signal light control according to an embodiment of the present invention;

[0032] Figure 2 A schematic flow chart showing a signal light control method according to an embodiment of the present invention; and

[0033] Figures 3a-3c Schematic diagrams showing the changes in cumulative rewards for different models at three test intersections. DETAILED DESCRIPTION

[0034] In the following description, the present invention is described with reference to various embodiments. However, those skilled in the art will recognize that the various embodiments may be implemented without one or more of the specific details or with other alternative and / or additional methods or components. In other cases, well-known structures or operations are not shown or described in detail to avoid obscuring the inventive aspects of the present invention. Similarly, specific numbers and configurations are set forth for illustrative purposes in order to provide a comprehensive understanding of the embodiments of the present invention. However, the present invention is not limited to these specific details.

[0035] In this specification, reference to "one embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. The appearances of the phrase "in one embodiment" in various places in this specification are not necessarily all referring to the same embodiment.

[0036] It should be noted that the embodiments of the present invention describe the method steps in a specific order, but this is only for the purpose of illustrating the specific embodiment and does not limit the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0037] In the present invention, a module may be implemented by software, hardware, firmware or a combination thereof.

[0038] While reinforcement learning methods have made significant progress in traffic light control, a major challenge is that models must be redesigned and trained from scratch for intersections with varying structures, such as intersections of varying shapes (crossroads, T-junctions), varying numbers of lanes, and varying signal phase structures. Training and optimizing policies for every signal light in a large-scale urban road network would be time-consuming and computationally intensive. Existing work has designed general models to handle intersections of varying structures; however, all of these approaches utilize the "next phase selection" action, which can lead to safety issues because the random selection of phases disrupts the traditional traffic cycle.

[0039] Based on this, the present invention proposes a universal signal light control method based on reinforcement learning and data enhancement. This method can be used to train a universal model to control intersections of different structures, thereby reducing the time for model training and making large-scale deployment of signal light control possible. To achieve the above objectives, the present invention first extracts features through traffic movement rather than extracting features according to lanes, thereby ensuring that different traffic intersections have features of the same size. Secondly, the present invention uses "setting the current phase duration" as the action design, which not only retains the concept of traffic cycle, but also ensures the safety of traffic participants such as drivers and pedestrians, to ensure that different phase structures are safe and generalizable. Through this design, the model has the same structure at different intersections. In addition, in the present invention, a new data enhancement method for traffic signal control is adopted, called "traffic movement shuffle". By incorporating data enhancement into reinforcement learning training, the performance of the generalized model can be further improved.

[0040] The solution of the present invention will be further described below in conjunction with the accompanying drawings of the embodiments.

[0041] Figure 1 FIG. 1 is a schematic diagram showing a general model system for signal light control according to an embodiment of the present invention. Figure 1 As shown, a general model system for traffic light control includes a feature extraction module 101 and an action prediction module 102 .

[0042] The feature extraction module 101 is used to convert the traffic flow information of the intersection into an intersection information feature vector. Since the number of lanes at different intersections may vary, if the traditional method is used to extract features according to the lanes, the size of the state space of different intersections will be different. As is known to all, no matter how many lanes there are at an intersection, there are only eight traffic turning flow signals (movement signals). Specifically, the eight traffic flow signals include northbound, northbound left turn, eastbound, eastbound left turn, westbound, westbound left turn, southbound, and southbound left turn. Based on this, in an embodiment of the present invention, the observation state of the agent is defined as in In one embodiment of the present invention, the information of the first traffic diversion flow is It includes three parts: traffic characteristics, traffic turning flow characteristics and traffic signal characteristics. The traffic characteristics indicate the degree of congestion at the intersection, including the average traffic flow, the maximum lane occupancy within K seconds, and the traffic signal characteristics. max and average occupancy meanThe traffic turning flow characteristics mainly refer to the information of the traffic flow itself, which includes the lane function. The lane function refers to the driving direction allowed by the lane, including one or more of straight, left turn, and right turn. Based on this, the traffic turning flow characteristics specifically include whether the direction of movement is straight. straight , and the number of lanes Lanes, and the traffic signal characteristics include the duration of the movement Duration, whether the duration is the minimum green light time Is minGreen , and whether the current traffic signal flow is green light Is Green In summary, the information of the first traffic diversion flow as follows:

[0043]

[0044] The eight features mentioned above can be easily obtained, which makes the feature extraction method mentioned above very practical in actual deployment.

[0045] Because some intersections, such as three-way intersections, have fewer than eight traffic turn flows, to ensure the same size of state space at different intersections, in one embodiment of the present invention, when the number of traffic turn flows at an intersection is less than eight, the traffic flow information corresponding to the missing traffic flow signals can be supplemented with zeros. For example, a common three-way intersection only uses four traffic turn flow signals: eastbound, eastbound left turn, westbound, and southbound left turn. In practice, zero padding can be used on the remaining four traffic turn flow features to maintain the same size of the observed state at the three-way intersection as at the intersection.

[0046] In order to achieve the feature extraction and conversion as described above, in one embodiment of the present invention, the feature extraction module includes a multi-layer structure, wherein the multi-layer structure includes at least a traffic flow information extraction layer and an intersection information extraction layer. The traffic flow information extraction layer is used to extract information about each traffic flow, and the intersection information extraction layer is used to extract intersection information based on the traffic flow information. Table 1 shows a structural schematic diagram of a general model in one embodiment of the present invention. As shown in Table 1, in one embodiment of the present invention, in order to enhance the generalization ability, one-dimensional convolution is used to extract information about traffic flows and intersections. Specifically, the feature extraction module first uses a one-dimensional convolution layer with an input channel size of 1, a kernel size of 1×8, and an output channel size of 128, which is used to extract information about each traffic flow. After passing through this convolution layer, the shape of the data is converted from 1×8×8 to 128×8×1, and then passes through a one-dimensional convolution with an input channel size of 128, a convolution kernel size of 8×1, and an output channel size of 256 to extract the intersection information. Finally, through a flattening layer and at least one fully connected layer, the information of the entire intersection is converted into a vector with an output length of 64.

[0047]

[0048] Table 1

[0049] The action prediction module 102 is used to perform action prediction based on the intersection information feature vector. In practical applications, the action design of the intelligent agent, namely, the control of traffic lights, must consider the safety of all traffic participants, including vehicles, drivers, and pedestrians. Therefore, although using the "select next phase" action can greatly improve traffic efficiency at an intersection, this action design changes the original traffic light phase sequence, thereby disrupting drivers' driving habits, potentially reducing driving safety and posing a safety hazard to traffic participants. Based on this, in an embodiment of the present invention, the "set current phase duration" action is selected as the intelligent agent action design. This maintains the concept of the traffic light cycle and only changes the duration of each phase. Therefore, it can be used even if the phase structure of different intersections is different. In this embodiment of the present invention, the term "phase" refers to the combination of different states of each traffic light at an intersection. Therefore, in this embodiment of the present invention, the action prediction module 102 predicts the duration of the current traffic light state at an intersection based on the intersection information feature vector. In one embodiment of the present invention, at the beginning of each phase, the motion prediction module 102 selects a time from a preset duration set based on the current intersection conditions as the green light duration for the current phase. In one embodiment of the present invention, the durations in the preset duration set range from 5 to 70 seconds. In another embodiment of the present invention, the difference between any two durations in the preset duration set is no greater than 5 seconds. For example, the duration set may be [5s, 10s, 15s, 20s, 30s, 35s, 40s, 45s, 50s, 55s, 60s, 65s, 70s].

[0050] In one embodiment of the present invention, the action prediction module 102, also known as the policy network (ActorNet), uses the intersection information feature vector as input to predict the agent's action, specifically, the green light duration for each phase. As shown in Table 1, in one embodiment of the present invention, the action prediction module includes at least one fully connected layer.

[0051] In order to improve the accuracy of the general model, in one embodiment of the present invention, data enhancement measures are also taken to further improve the generalization ability of reinforcement learning, so that the trained general model can be transferred to new tasks. Based on this, in one embodiment of the present invention, a new data enhancement method is designed, called "movement shuffle". Specifically, after obtaining the intersection information feature vector from the intersection, we shuffle the rows of the intersection information feature vector to obtain a new intersection information feature vector. This data enhancement method conforms to the characteristics of the intersection. For example, rotating a three-way intersection by 90, 180 and 270 degrees respectively will change the intersection information feature vector. The order of the intersection information feature vectors at different intersections is different. For example, at an intersection without rotation, vehicles cannot travel from north to south, so ( On the contrary, for a 90-degree rotated intersection, vehicles can go straight from south to north, so No longer 0. The original intersection At the intersection rotated 90 degrees, it will become At this time, if the reinforcement learning is only trained at one intersection, it will be difficult to handle the other three situations. However, in actual applications, the control of traffic lights should not change with the rotation of the intersection. Based on this, the data enhancement method of the disordered traffic flow features can allow the model to ignore The order of the two axes is such that no matter how the intersection rotates, the same action can be predicted.

[0052] In one embodiment of the present invention, a proximal policy optimization algorithm (PPO) is used to perform reinforcement learning training. Specifically, in addition to the action prediction module 102, the general model also includes a state value prediction module 103. The state value prediction module 103 can be called a value network (Critic Net), which also uses the intersection information feature vector as input, and predicts the value of each state based on the intersection information feature vector. The value of each state refers to the total reward obtained from the state to the end of the simulation. As shown in Table 1, the state value prediction module includes at least one fully connected layer. Since indicators such as waiting time, driving time and delay are difficult to obtain from real-world traffic detection equipment, in one embodiment of the present invention, a negative queue length is used as a reward, and the reward is normalized to limit its range and speed up the training process. The final reward function is as follows:

[0053]

[0054] in, r jIt represents the negative queue length at time j, which can be measured directly.

[0055] The value of each state is calculated as follows:

[0056]

[0057] Figure 2 A schematic flow chart illustrating a signal light control method according to an embodiment of the present invention is shown.

[0058] like Figure 2 As shown, a signal light control method includes:

[0059] First, in step 201, the model is trained. The data augmentation method described above is used to form a training set to train the designed general model. It should be understood that in some embodiments of the present invention, if the general model has been trained, this step can be omitted in actual application.

[0060] Next, in step 202, feature extraction is performed. The feature extraction module extracts the intersection information feature vector;

[0061] Next, in step 203, action prediction is performed. The action prediction module calculates the duration of the green light at the current intersection based on the intersection information feature vector; and

[0062] Finally, in step 204, the signal light is controlled to switch the green light of the signal light to yellow or red after the duration.

[0063] To validate the traffic light control method described in this embodiment of the present invention, the general model described above was trained on eight intersections with different structures and tested on three intersections with different structures. The experimental results were compared with traditional traffic control algorithms and models trained directly on a single road network. The experiments were conducted using SUMO simulation software, an open-source, microscopic, continuous, multimodal traffic simulation platform. The Traffic Control Interface (TraCI) is used to control traffic lights and obtain information about intersection traffic conditions. Inductive loop detectors are used to monitor traffic flow, and lane area detectors are used to measure lane occupancy and queue length. To match real-world scenarios, the detector length was set to 100 meters in the tests. Furthermore, a green light was followed by a three-second yellow light before turning red to ensure driver safety. To verify the effectiveness of the method, 11 intersections with different structures were constructed, each with a different number of lanes, phase structure, and / or shape. Eight intersections were used for training and three for testing. Specifically, for example, an intersection may consist of four bidirectional lanes: five lanes in the north-south direction and four lanes in each direction. The signal phases at this intersection then consist of four phases, each consisting of two traffic movement signals. Based on this, the training dataset contains three different intersection topologies, representing a large intersection with four or more lanes per road, a small intersection with three lanes per road, and a three-way intersection. For intersections with the same topology, new intersections are generated by varying the phase sequence or number of phases. For example, two intersections may have identical physical structures but different signal phase sequences or different numbers of phases. To validate the proposed model's performance on unseen intersections, three test intersections were constructed. These test intersections, for example, may have increased the number of lanes, modified the number of signal phases, or rotated the number of phases compared to the training set intersections. Furthermore, 100 unique traffic flows were generated for each intersection in the experiment, with three-quarters used for training and the remainder for evaluation. Each traffic flow lasted 30,000 seconds. In the experiment, the performance of different methods was evaluated using the waiting time of each vehicle. The waiting time of each vehicle is defined as the duration during which the vehicle speed drops below 0.1 m / s. Shorter waiting times mean that vehicles spend less time passing through the intersection, which translates to higher traffic efficiency.

[0064] By extracting features based on traffic turn flow and using "setting the current phase duration" as an action design for the agent, the method in the embodiments of the present invention can use the same model structure to control intersections with different structures. Based on this, the general model obtained by the present invention was trained on eight different intersections and then tested on these eight intersections using different traffic flows. The model was compared with the Webster model in terms of selecting the next phase (single environment), whether to switch to the next phase (single environment), setting the current phase duration (single environment), and setting the current phase duration (multiple environments).

[0065] Table 2 shows the overall results of the proposed method and other common models at training intersections. The values ​​in the table represent the average waiting times for the different methods. Clearly, the proposed method outperforms the Webster method. This is because the Webster model assumes uniform traffic flow. However, this assumption may not hold true in reality, resulting in inaccurate calculated cycle and phase durations, leading to performance degradation.

[0066] Comparing two common methods, "Set Current Phase Duration (Multiple Environments)," and the method of the present invention, both of which were trained in multiple environments, the common model trained with the present invention showed an average 10% improvement in performance across eight intersections compared to the "Set Current Phase Duration (Multiple Environments)" method. This confirms that the new data augmentation method proposed in the present invention can improve the performance of the common model.

[0067] Model Name INT1-1 INT1-2 INT1-3 INT2-1 INT2-2 INT2-3 INT3-1 INT3-2 Webster Model 55.1955 53.8775 59.9905 47.246 47.3615 10.391 9.7525 9.825 Set the duration of the current phase (multiple environments) 10.8135 9.443 12.9 12.895 12.9885 5.274 3.831 3.8415 Method of the present invention 9.9505 8.8435 11.562 11.738 11.7275 3.9385 3.8855 3.888

[0068] Table 2

[0069] Table 3 shows the performance of models trained on specific intersections using three different action designs. The model name INTm-n, where m = 1, 2, 3 and n = 1, 2, 3, represents the nth intersection in the mth intersection topology. The intersection topology can be, for example, a large intersection with four or more lanes per road, a small intersection with three lanes per road, or a three-way intersection. For the same topology m, varying the phase sequence or number of phases can generate different intersections n. It can be seen that in most cases, the "Select Next Phase" action minimizes the waiting time per vehicle because it can select a phase arbitrarily every N seconds. However, this disordered phase combination can pose driver safety concerns. When processing intersections with only two phases, the "Select Next Phase" action degenerates into other actions. For example, at intersection INT2-3, "Select Next Phase" performs worse than the other actions. Furthermore, in most cases, the "Set Current Phase Duration" action outperforms "Whether to Switch to Next Phase."

[0070]

[0071]

[0072] Table 3

[0073] Finally, the performance of the model trained in a single environment is compared with the model trained in multiple environments. According to the results in the table above, the multi-environment model will have a waiting time of approximately 2.3 seconds longer than the single-environment model, but it only needs to be trained once and can be directly migrated to other intersections, so a lot of training time can be saved. This performance loss is acceptable because the general model takes all intersections into account. It should be emphasized that the multi-environment model (that is, the general model) only needs to be trained once and then tested at 8 different intersections without further training. For the single-environment model, separate training must be performed at each intersection. Therefore, in this experiment, the training time of the single-environment model is eight times that of the multi-environment model. When more intersections are taken into account, the training time of the single-environment model will become a bigger problem.

[0074] In addition, the effectiveness of the general model of the present invention at intersections in the test set was also tested. The three test intersections differ from the intersections in the training set in terms of the number of lanes or phases. Currently, there is no reinforcement learning model that uses "set current phase duration" as an action design that can work directly in multiple intersection scenarios without any retraining or transfer learning. Therefore, for comparison purposes, the method proposed in the present invention is directly compared with the model trained on the three test intersections. At the same time, by using the intelligent agent design proposed in the present invention, intersections with different structures can have the same model structure. Therefore, the model trained in a single scene in the training set can also be compared with the model trained using multiple scenes to explore whether including different intersections in training can help improve the performance of the general model.

[0075] Table 4 shows the effects of different models on the test intersections, where "performance degradation" indicates how much the waiting time increases compared to the model trained directly on the test road network. Similar to the results on the training set, the effect of the algorithm based on reinforcement learning is still far better than the traditional traffic control algorithm, even on the new road network. For directly training the model in multiple environments, the model performance degrades to 63%, which means that the waiting time increases by 63%. After using the data enhancement proposed by the present invention, the performance of the generalized model has been greatly improved, and the average waiting time loss has been reduced to 24%. No general training is required for new intersections.

[0076]

[0077]

[0078] Table 4

[0079] As shown in Table 4, simply training in multiple environments leads to performance loss, which can be effectively alleviated by adding data augmentation. At the same time, models trained in a single environment can also be transferred to new intersections. In particular, if there is a large similarity between the new scene and the training scene, the performance degradation will be relatively small. For example, the model trained on INT1-1 performs better on INT4 than INT5 and INT6, because INT1-1 and INT4 are both 4-way intersections with the same phase structure. The only difference between INT1-1 and INT4 is the number of lanes. In contrast, changing the number of phases or the number of roads leads to a significant drop in performance.

[0080] In practice, some critical intersections require additional attention to their performance. To this end, the proposed method is used to train a general model, which is then trained on specific environments. Experimental results show that with only a small number of training steps, significant performance improvements can be achieved at new intersections.

[0081] Figures 3a-3c The figure shows the changes in the cumulative rewards of different models in the test intersections INT4, 5, and 6. The solid line shows the result of directly using the general model on the new road network without transfer training. It can be seen that the model has shown good results without any transfer learning. The long dotted line represents the model trained from scratch, and the dotted line represents the model trained based on the general model. At about 6×10 6 The model trained directly in a single environment converges in about 10 training steps, while the general model only needs about 10 6 The model converges in just 10 training steps, which is equivalent to saving about 80% of the training time while maintaining performance.

[0082] Table 5 shows the performance of the model after transfer learning. 6 After 10 training steps, the performance degradation dropped from 24% to 5%. Training the model from scratch requires at least 6×10 6 This is very attractive for practical applications, as it can greatly reduce the number of interactions with the environment, thereby greatly improving training efficiency.

[0083] Model Name INT4 INT5 INT6 Performance degradation Set the duration of the current phase (single environment) 13.1445 1.8475 3.8045 Set the duration of the current phase (multiple environments) 19.3945(↑47.548％) 3.2125(↑73.884％) 6.427(↑68.932％) ↑63.455％ Method of the present invention 16.917(↑28.7％) 1.944(↑5.223％) 5.299(↑39.282％) ↑24.402％ The method of the present invention + transfer learning 13.5725(↑3.256％) 1.939(↑4.953％) 4.1055(↑7.912％) ↑5.373％

[0084] Table 5

[0085] Although various embodiments of the present invention have been described above, it should be understood that they are presented by way of example only and not limitation. It will be apparent to those skilled in the relevant art that various combinations, modifications, and variations may be made thereto without departing from the spirit and scope of the present invention. Therefore, the breadth and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely in accordance with the appended claims and their equivalents.

Claims

1. A general model system for signal light control, characterized in that: include: a feature extraction module configured to convert traffic flow information at the intersection into an intersection information feature vector, wherein the traffic flow information includes the average traffic flow under each traffic flow signal, the maximum lane occupancy within K seconds, the average lane occupancy, the lane function, the number of lanes included, the duration of the movement, whether the duration is the minimum green light time, and whether the current signal is green, wherein there are a total of eight traffic flow signals. If the actual traffic flow signals at the intersection are less than eight, the traffic flow information corresponding to the missing traffic flow signals is recorded as 0; and An action prediction module is configured to predict the duration of the current traffic light state of the intersection based on the intersection information feature vector.

2. The universal model system according to claim 1, wherein: The system also includes a state value prediction module, which is configured to predict the total reward obtained from each state to the end of the simulation based on the intersection information feature vector, wherein the reward includes the negative queue length.

3. The universal model system according to claim 1, wherein: The traffic flow signals include: northbound, northbound left turn, eastbound, eastbound left turn, westbound, westbound left turn, southbound, and southbound left turn.

4. The universal model system according to claim 1, wherein: The feature extraction module includes a multi-layer structure: a traffic flow information extraction layer configured to extract information of each traffic flow; as well as The intersection information extraction layer is configured to extract intersection information based on the traffic flow information.

5. The universal model system according to claim 4, characterized in that The traffic flow information extraction layer and the intersection information extraction layer both use one-dimensional convolutional layers, and The feature extraction module further includes a flattening layer and at least one fully connected layer.

6. The universal model system according to claim 1, wherein: The action prediction module includes at least one fully connected layer.

7. The universal model system according to claim 2, wherein: The state value prediction module includes at least one fully connected layer.

8. The universal model system according to claim 1, wherein: The duration of the current traffic light state includes the duration of the current green light, which is selected from a preset duration set according to the intersection information feature vector.

9. The universal model system according to claim 8, wherein: The range of each duration in the preset duration set is 5 to 70 seconds.

10. The universal model system according to claim 8, wherein: The difference between any two durations in the preset duration set is no greater than 5 seconds.

11. A signal light control method, characterized in that: The signal light control signal is outputted through the universal model system according to any one of claims 1 to 10, and the signal light control method comprises the steps of: The general model is trained, wherein a training set used for the training is augmented by shuffling the traffic turning flow; Extracting intersection information feature vectors by the feature extraction module; Calculate the duration of the green light at the current intersection based on the intersection information feature vector by the action prediction module; as well as The green light of the control signal light switches to yellow or red after the stated duration.

12. The signal light control method according to claim 11, wherein: The traffic diversion flow disorder includes: After the traffic flow information of the intersection is obtained, the traffic flow information of the intersection is disrupted to obtain new traffic flow information of the intersection.

13. The signal light control method according to claim 11, wherein: The training uses negative queue lengths as rewards.

Images