Traffic signal control method for heterogeneous intersection and related device

Abstract

The application provides a traffic signal control method for heterogeneous intersections and related equipment. The method is applied to a meta-learning model and multiple sub-learning models. Model parameters of the meta-learning model are obtained, and a first update operation is performed on the model parameters of each sub-learning model according to the model parameters of the meta-learning model. A local historical training set corresponding to each intersection is used to train the corresponding sub-learning model. In response to reaching a preset timing moment, the current lane vehicle state information of each intersection is obtained, and a timing scheme is output based on the lane vehicle state information through the corresponding sub-learning model. Each intersection executes the corresponding timing scheme and feeds back response information. New response information of the corresponding intersection is obtained through each sub-learning model after the second update operation, and the meta-learning model is trained based on all the new response information.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a traffic signal control method and related equipment for heterogeneous intersections. Background Art

[0002] With the continued growth of China's automobile ownership, traffic congestion has become a difficult and hot issue in major cities. Traffic congestion increases time and space costs, wastes resources, pollutes the environment, impacts the normal conduct of official and business activities, and seriously hinders urban economic and social development. Therefore, developing smart transportation and realizing intelligent control of traffic signals are important for ensuring the health and sustainable development of cities. Traditional fixed timing schemes are the most common traffic signal control method in real life. However, due to their lack of adaptability, signal control strategies cannot be adjusted in real time according to road traffic conditions, resulting in insufficient utilization of intersection capacity. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose a traffic signal control method and related equipment for heterogeneous intersections.

[0004] Based on the above objectives, a first aspect of the present application provides a model training method for controlling traffic signals, which is applied to a meta-learning model and multiple sub-learning models, each of which corresponds to an intersection, and the intersections corresponding to the multiple sub-learning models include different types of intersections. The method includes:

[0005] Iterate the following operations until the training cutoff condition is reached,

[0006] Obtaining model parameters of the meta-learning model, and performing a first update operation on the model parameters of each sub-learning model according to the model parameters of the meta-learning model;

[0007] Use the local historical training set corresponding to each intersection to train the corresponding sub-learning model;

[0008] In response to reaching a preset timing moment, respectively obtaining the current lane vehicle status information of each intersection, and outputting a timing plan based on the lane vehicle status information through the corresponding sub-learning model;

[0009] Instruct each intersection to execute the corresponding timing plan and feedback the response information;

[0010] Perform a second update operation on the model parameters of the corresponding sub-learning model according to each response information;

[0011] Obtaining new response information of the corresponding intersection through each sub-learning model that has undergone the second updating operation;

[0012] The meta-learning model is trained based on all new response information.

[0013] Optionally, the different types of intersections have different numbers of roads and lanes.

[0014] Optionally, the input data in the local historical training set includes lane vehicle status information, and the output data includes a timing plan, the lane vehicle status information includes vehicle position, vehicle waiting time and vehicle speed, and the timing plan includes a timing cycle and intersection phase green signal ratio.

[0015] Optionally, the sub-learning model includes a feature extraction layer and a decision output layer, and the feature extraction layer includes a lane convergence layer, a phase convergence layer, and an intersection convergence layer in sequence;

[0016] Outputting a timing plan based on the lane vehicle state information through a corresponding sub-learning model includes:

[0017] Inputting the lane vehicle state information into the lane convergence layer, and outputting lane feature representation through the lane convergence layer;

[0018] Inputting the lane feature representation into the phase convergence layer, and outputting the phase state representation through the phase convergence layer;

[0019] Inputting the phase state representation into the intersection convergence layer, and outputting the intersection feature representation through the intersection convergence layer;

[0020] The intersection feature representation is input to the decision output layer, and the timing plan is output through the decision output layer.

[0021] Optionally, the decision output layer includes an Actor network and a Critic network.

[0022] Optionally, inputting the lane vehicle state information to the lane convergence layer, and outputting lane feature representation through the lane convergence layer, includes:

[0023] Embedding the lane vehicle state information based on the first parameter of the lane convergence layer to obtain a first embedding representation of each lane;

[0024] After concatenating the mean of all first embedding representations to each first embedding representation, the second embedding representation of each lane is obtained;

[0025] Multiplying the second embedding representation by the second parameter of the lane pooling layer to obtain a first attention score for each lane for each vehicle in the lane;

[0026] The first attention score is processed by a softmax function to obtain a first attention coefficient, and the transpose of the first attention coefficient is multiplied by the first embedding representation to obtain a lane feature representation of each lane.

[0027] Optionally, inputting the phase state representation to the intersection convergence layer, and outputting the intersection feature representation through the intersection convergence layer, includes:

[0028] Embedding the phase state representation based on a third parameter of the intersection convergence layer to obtain a third embedded representation of each phase;

[0029] After concatenating the mean of all third embedding representations to each third embedding representation, the fourth embedding representation of each phase is obtained;

[0030] Multiplying the fourth embedding representation by the fourth parameter of the intersection pooling layer to obtain a second attention score for each intersection for each phase in the intersection;

[0031] The second attention score is processed by a softmax function to obtain a second attention coefficient, and the transpose of the second attention coefficient is multiplied by the third embedding representation to obtain the intersection feature representation of each intersection.

[0032] A second aspect of the present application further provides a model training device for controlling traffic signals, which is applied to a meta-learning model and multiple sub-learning models, each of which corresponds to an intersection. The intersections corresponding to the multiple sub-learning models include different types of intersections, including:

[0033] A model training module, which iteratively performs the following operations until a training cutoff condition is reached, and includes:

[0034] a first updating unit, configured to obtain model parameters of the meta-learning model, and perform a first updating operation on the model parameters of each sub-learning model according to the model parameters of the meta-learning model;

[0035] The first training unit is configured to train the corresponding sub-learning model using the local historical training set corresponding to each intersection;

[0036] an output unit configured to obtain, in response to reaching a preset timing moment, current lane vehicle state information of each intersection, and output a timing plan based on the lane vehicle state information through a corresponding sub-learning model;

[0037] A feedback unit is configured to enable each intersection to execute a corresponding timing plan and feedback response information;

[0038] A second updating unit is configured to perform a second updating operation on the model parameters of the corresponding sub-learning model according to each piece of response information;

[0039] an acquiring unit configured to acquire new response information of the corresponding intersection through each sub-learning model that has undergone the second updating operation;

[0040] The second training unit is configured to train the meta-learning model based on all new response information.

[0041] The third aspect of the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method described above when executing the computer program.

[0042] The fourth aspect of the present application further provides a non-transitory computer-readable storage medium, which stores computer instructions for causing a computer to execute the method described above.

[0043] As can be seen from the above, the traffic signal control method and related equipment for heterogeneous intersections provided by the present application are applied to a meta-learning model and multiple sub-learning models, each of which corresponds to an intersection, and the intersections corresponding to the multiple sub-learning models include different types of intersections. The sub-learning model and the meta-learning model are trained using a heterogeneous traffic environment including different types of intersections to obtain a traffic signal control model with strong generalization, so that it can quickly adapt to new traffic environments. The model parameters of the meta-learning model are obtained, and the model parameters of each sub-learning model are respectively updated for the first time according to the model parameters of the meta-learning model, and the model parameters of each sub-learning model are assigned using the model parameters of the meta-learning model. The corresponding sub-learning model is trained using the local historical training set corresponding to each intersection, and the sub-learning model after the first update is trained to optimize the model parameters of the sub-learning model. In response to the arrival of a preset timing, the current lane vehicle state information for each intersection is obtained. Based on the lane vehicle state information, a timing plan is output through the corresponding sub-learning model. Each sub-learning model then outputs a timing plan for the next signal control cycle. The timing plan represents the traffic signal control method for the next signal control cycle. Each intersection executes the corresponding timing plan and provides feedback. The response information reflects the results of the timing plan execution at the intersection. Based on each response information, a second update operation is performed on the model parameters of the corresponding sub-learning model to better match the model parameters of the sub-learning model with the conditions at the corresponding intersection. Each sub-learning model that has undergone the second update operation obtains new response information for the corresponding intersection. The meta-learning model is trained based on all the new response information so that the meta-learning model learns the model parameters of each sub-learning model, thereby improving the generalization ability of the meta-learning model. When a new sub-learning model is deployed at a new intersection, it can directly adopt the model parameters of the meta-learning model. This ensures the accuracy of the model output timing plan while reducing sub-learning model training time, improving model utilization, and enabling rapid deployment of the sub-learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 A flow chart of a traffic signal control method for a heterogeneous intersection according to an embodiment of the present application;

[0046] Figure 2A schematic diagram of a heterogeneous intersection according to an embodiment of the present application;

[0047] Figure 3 A schematic diagram of a phase control method according to an embodiment of the present application;

[0048] Figure 4 A flowchart of a method for outputting a timing plan for a sub-learning model according to an embodiment of the present application;

[0049] Figure 5 This is a schematic diagram of the network structure of the feature extraction layer of an embodiment of the present application;

[0050] Figure 6 This is a schematic diagram of the network structure of the decision output layer of an embodiment of the present application;

[0051] Figure 7 This is a schematic structural diagram of a traffic signal control device for a heterogeneous intersection according to an embodiment of the present application;

[0052] Figure 8 Schematic diagram of the hardware structure of the electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0054] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0055] In recent years, with the rise of artificial intelligence, control methods based on genetic algorithms, swarm intelligence, fuzzy control, and reinforcement learning have been gradually applied to traffic signal control. Reinforcement learning, as a model-free approach, does not require precise modeling of the environment. Instead, it optimizes the model through continuous interaction with the environment, resulting in stronger learning capabilities and adaptability, and has therefore attracted increasing attention. However, the application of existing reinforcement learning algorithms in traffic signal control mainly targets homogeneous intersections, that is, training and testing the model in the same intersection scenario. The resulting model is often only applicable to the current environment and lacks the ability to generalize to other heterogeneous intersections. Furthermore, training reinforcement learning models requires a large amount of data and is time-consuming. Retraining in a new environment will inevitably hinder the rapid deployment of the model and reduce its utilization efficiency.

[0056] In view of this, the present application proposes a model training method for controlling traffic signals, which trains an intelligent signal control model with strong generalization so that it can quickly adapt to new traffic environments.

[0057] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0058] This application proposes a model training method for controlling traffic signals, which is applied to a meta-learning model and multiple sub-learning models, each of which corresponds to an intersection, and the intersections corresponding to the multiple sub-learning models include different types of intersections. Figure 1 , the method comprises the following steps:

[0059] Iterate the following operations until the training cutoff condition is reached,

[0060] Step 102: Obtain model parameters of the meta-learning model, and perform a first update operation on the model parameters of each sub-learning model according to the model parameters of the meta-learning model.

[0061] Specifically, before training the model, N intersections are randomly sampled, where N is an integer greater than or equal to 1. These N intersections include heterogeneous intersections, and a sub-learning model is deployed for each intersection. The heterogeneity of intersections is reflected in the difference in the number of roads and lanes. Figure 2Figure 1 shows a schematic diagram of a heterogeneous intersection, where A and B are intersections with four roads, and E and H are three-way intersections with three roads. The number of lanes on each road can also vary. For example, each road may have two, three, or four lanes. In example A, each road has three lanes, while in example B, each road has two lanes. The heterogeneity of intersections is also reflected in different phase control methods. The signal display state corresponding to a group of non-conflicting traffic flows simultaneously obtaining the right of way is called a signal phase, or simply phase. Traffic light signal phases are divided according to the rotation of right-of-way at the intersection within a cycle. The number of signal phases a traffic light signal control scheme has within a cycle is called multi-phase signal control. The number of signal control phases used at an intersection should be determined by the actual traffic flow conditions at the intersection. Intersections typically use two to four signal phases. If the number of phases is too small, the right of way at the intersection cannot be effectively allocated, resulting in traffic congestion and reduced traffic safety. If the number of phases is designed to be too many, although the traffic order and safety at the intersection will be improved, some travel time will be lost when switching between phases. Too many phases will lead to a decrease in the traffic capacity of the intersection and extend the waiting time of drivers at the intersection. Figure 3 Schematic diagrams of phase control methods are shown. A is a 4-phase control method at an intersection, and B is a 3-phase control method at a three-way intersection.

[0062] In this embodiment, the model is trained by using a meta-learning method. Before the first training of the meta-learning model and multiple sub-learning models, the model parameters w of the meta-learning model are randomly initialized. 0 . The model parameters w of each sub-learning model k Assign the model parameter w of the meta-learning model 0 , w k ←w 0 , Complete the initialization operation of the sub-learning model itself.

[0063] Step 104: Use the local historical training set corresponding to each intersection to train the corresponding sub-learning model.

[0064] Furthermore, the input data in the local historical training set includes lane vehicle status information, including vehicle position, waiting time, and speed, and the output data includes a timing plan. The lane vehicle status information includes vehicle position, waiting time, and speed, and the timing plan includes a timing period and intersection phase green signal ratio. Local historical data is collected to form a local historical training set, which is then used to perform local training on the sub-learning model.

[0065] Step 106: In response to reaching the preset timing, the current lane vehicle status information for each intersection is obtained. Based on this lane vehicle status information, a timing plan is output through the corresponding sub-learning model. The timing plan includes the timing cycle and the green-to-signal ratio of each intersection phase. The green-to-signal ratio refers to the proportion of time available for vehicle traffic within a traffic light cycle, that is, the ratio of the effective green light time for a phase to the cycle length. The lane vehicle status information for the current intersection is collected and output to each sub-learning model, which then outputs its own timing plan.

[0066] Step 108: Instruct each intersection to execute the corresponding timing plan and provide feedback. After obtaining the timing plan through the above steps, the traffic lights at each intersection are instructed to execute the corresponding timing plan and feedback is collected. This feedback includes traffic indicators such as intersection queue length and vehicle waiting time. Traffic indicators can reflect changes in the traffic environment and determine the effectiveness of the timing plan.

[0067] Step 110: Perform a second update operation on the model parameters of the corresponding sub-learning model according to each response information. Perform a second update on the parameters of the corresponding sub-learning model according to each response information obtained in step 108 to further optimize the model parameters of the sub-learning model.

[0068] Step 112: Obtain new response information for the corresponding intersection through each sub-learning model that has undergone the second update operation. After the model parameters of the sub-learning model are updated for the second time and the timing plan is output again, the new response information for the intersection is collected and stored in the database.

[0069] Step 114: Train the meta-learning model based on all new response information. The meta-learning model aggregates the learning experience of each sub-learning model and uses the response information collected by each sub-learning model to update its own model parameters. By considering information from heterogeneous intersections, the meta-learning model has stronger generalization and can handle signal timing tasks at heterogeneous intersections.

[0070] The above process is iterated cyclically, and the model parameters of each sub-learning model and the meta-learning model are updated alternately until the meta-learning model converges.

[0071] Based on steps 102 to 114 above, the traffic signal control method and related equipment for heterogeneous intersections provided in this embodiment are applied to a meta-learning model and multiple sub-learning models. Each sub-learning model corresponds to an intersection, and the intersections corresponding to the multiple sub-learning models include different types of intersections. The sub-learning models and the meta-learning model are trained using a heterogeneous traffic environment including different types of intersections to obtain a traffic signal control model with strong generalization capabilities, enabling it to quickly adapt to new traffic environments. The model parameters of the meta-learning model are obtained, and a first update operation is performed on the model parameters of each sub-learning model based on the model parameters of the meta-learning model. The model parameters of each sub-learning model are assigned values ​​using the model parameters of the meta-learning model. The corresponding sub-learning model is trained using the local historical training set corresponding to each intersection. The sub-learning model after the first update is trained to optimize the model parameters of the sub-learning model. In response to the arrival of a preset timing, the current lane vehicle state information for each intersection is obtained. Based on the lane vehicle state information, a timing plan is output through the corresponding sub-learning model. Each sub-learning model then outputs a timing plan for the next signal control cycle. The timing plan represents the traffic signal control method for the next signal control cycle. Each intersection executes the corresponding timing plan and provides feedback. The response information reflects the results of the timing plan execution at the intersection. Based on each response information, a second update operation is performed on the model parameters of the corresponding sub-learning model to better match the model parameters of the sub-learning model with the conditions at the corresponding intersection. Each sub-learning model that has undergone the second update operation obtains new response information for the corresponding intersection. The meta-learning model is trained based on all the new response information so that the meta-learning model learns the model parameters of each sub-learning model, thereby improving the generalization ability of the meta-learning model. When a new sub-learning model is deployed at a new intersection, it can directly adopt the model parameters of the meta-learning model. This ensures the accuracy of the model output timing plan while reducing sub-learning model training time, improving model utilization, and enabling rapid deployment of the sub-learning model.

[0072] In some embodiments, the sub-learning model includes a feature extraction layer and a decision output layer, and the feature extraction layer includes a lane convergence layer, a phase convergence layer, and an intersection convergence layer in sequence;

[0073] Based on the lane vehicle status information, the corresponding sub-learning model outputs the timing plan, referring to Figure 4 , including the following steps:

[0074] Step 402: Input the lane vehicle state information into the lane convergence layer, and output the lane feature representation through the lane convergence layer.

[0075] Furthermore, the lane vehicle state information includes vehicle position l, vehicle waiting time c, and vehicle speed v. When there are m1 vehicles in lane i, the lane vehicle state set is expressed as When there is no vehicle in the lane, The set of vehicle states in all lanes constitutes the original state s of the current intersection k k . The lane vehicle state set corresponding to each lane at each intersection Input into the lane aggregation layer.

[0076] Furthermore, the step 402 specifically includes:

[0077] The lane vehicle state information is embedded based on the first parameter of the lane convergence layer to obtain a first embedded representation for each lane. The mean of all first embedded representations is concatenated to each first embedded representation to obtain a second embedded representation for each lane. The second embedded representation is multiplied by the second parameter of the lane convergence layer to obtain a first attention score for each lane for each vehicle in that lane. The first attention score is processed using a softmax function to obtain a first attention coefficient. The transpose of the first attention coefficient is multiplied by the first embedded representation to obtain a lane feature representation for each lane.

[0078] Specifically, traverse each lane i and extract the lane state features according to the lane convergence layer. The input of the lane convergence layer is the lane vehicle state set of the lane The lane convergence layer parameters are set to First, pass the first parameter Embed the lane vehicle state set to obtain the first embedding representation n1 represents the lane embedding dimension. The mean of all first embedding representations is Splice to each Then we get the second embedding representation Then multiply the second embedding representation by the second parameter to get the first attention score Finally, the first attention score is normalized by the softmax function to obtain the first attention coefficient Multiply the transpose of the first attention coefficient with the first embedding representation to obtain the lane feature representation of lane i j represents the phase number corresponding to lane i.

[0079] Step 404: Input the lane feature representation to the phase convergence layer, and output the phase state representation through the phase convergence layer.

[0080] Specifically, traverse each phase j and extract the features of the phase state according to the phase convergence layer. The input of the phase convergence layer is the lane feature set under the phase control. m2 is the number of lanes controlled by the phase. The parameters of the phase convergence layer are The lane state set is embedded by the parameters of the phase convergence layer to obtain n2 is the phase embedding dimension. Then Perform the averaging operation to get the phase state representation

[0081] Step 406: Input the phase state representation to the intersection convergence layer, and output the intersection feature representation through the intersection convergence layer.

[0082] Furthermore, step 406 specifically includes:

[0083] The phase state representation is embedded based on the third parameter of the intersection convergence layer to obtain a third embedding representation of each phase; the mean of all third embedding representations is spliced ​​to each third embedding representation to obtain a fourth embedding representation of each phase; the fourth embedding representation is multiplied by the fourth parameter of the intersection convergence layer to obtain a second attention score of each intersection for each phase in the intersection; the second attention score is processed by the softmax function to obtain a second attention coefficient, and the transpose of the second attention coefficient is multiplied by the third embedding representation to obtain the intersection feature representation of each intersection.

[0084] Specifically, the intersection state is feature extracted according to the intersection convergence layer, and the input data of the intersection convergence layer is the phase state set m3 is the number of control phases at the intersection. The parameters of the intersection convergence layer are set to First, pass the third parameter Embed the phase state set to obtain the third embedding representation n3 represents the intersection embedding dimension. The mean of all third embedding representations is Splice to each Then we get the fourth embedding representation Then multiply the fourth embedding representation by the fourth parameter to get the second attention score Finally, the second attention score is normalized by the softmax function to obtain the second attention coefficient Multiply the transpose of the second attention coefficient by the third embedding representation to obtain the intersection feature representation of each intersection As can be seen, the intersection feature representation dimension is unaffected by the heterogeneous characteristics of the intersection and is only relevant to the definition of n3. This completes the unified representation of the traffic state at a heterogeneous intersection using the feature extraction layer. The intersection feature representation after the feature extraction layer encompasses the traffic characteristics of the entire intersection and eliminates the impact of inconsistent state dimensions caused by heterogeneous information such as the number of vehicles, lanes, and phases.

[0085] Figure 5 The network structure diagram of the feature extraction layer is shown, which includes lane aggregation layer, phase aggregation layer and intersection aggregation layer. Figure 5 As shown, the lane aggregation layer can process lanes with different numbers of vehicles and obtain a lane feature representation with unified dimensions. Further, the lane feature set under phase j control is obtained from the correspondence between phase and lane The phase convergence layer can process the phases that control different numbers of lanes and obtain a phase feature representation with unified dimensions. The phase feature set is obtained by the phase control method at the current intersection Finally, the intersection feature representation is obtained through the intersection aggregation layer (i.e., intersection feature representation) to achieve unified processing of different phase numbers.

[0086] Step 408: Input the intersection feature representation into the decision output layer, and output the timing plan through the decision output layer.

[0087] Specifically, the unified dimension representation of the heterogeneous intersection features is then fed into the decision output layer. This layer employs a reinforcement learning algorithm to maximize cumulative rewards through continuous interaction with the environment, optimizing traffic metrics such as queue length and wait time, and adaptively adjusting the timing cycle and green signal ratio. This decision output layer comprises an actor network and a critic network. Figure 6 Figure 2 shows a schematic diagram of the network structure of the decision output layer. Figure 6As shown in the figure, the actor network includes the current actor network and the target actor network, and the critic network includes the current critic network and the target critic network. The current actor network is used to output the timing plan in real time, complete the mapping of environmental state to response action, and complete parameter updates through policy gradients during network updates. The current critic network completes the real-time evaluation of the current timing plan. The target actor network is used to output the estimated timing plan, and the target critic network is used to output the evaluation of the estimated timing plan. During network updates, the temporal difference (TD) error is calculated from the outputs of the target actor network and the target critic network to complete the update of the current critic network, thereby guiding the current actor to make a better timing plan at the next moment.

[0088] In some embodiments, in response to reaching a preset timing moment, the current lane vehicle state information of each intersection is obtained respectively, and a timing plan is outputted based on the lane vehicle state information through the corresponding sub-learning model, specifically including:

[0089] Specifically, the preset timing moment is the time when the next cycle timing plan is determined in advance. When the time arrives, the current lane vehicle status information of each intersection is obtained respectively, and the feature extraction layer is used to output the intersection feature representation by the method in the above embodiment. Then the timing plan is output through the decision output layer.

[0090] Further, such as Figure 6 As shown, the current Actor network is represented according to the intersection characteristics at the current time t Output Action Where μ is the deterministic strategy corresponding to the Actor network, The network parameters of the current Actor network. Including the cycle length of the next timing cycle and the green-to-signal ratio of each phase, that is, the timing plan, with the maximum action dimension of each sub-learning model as the unified action dimension. Figure 3 For example, the phase control method A in the action That is, it is a 5-dimensional vector, the first dimension corresponds to the timing cycle length, the second to fifth dimensions correspond to the green-to-signal ratios of the four phases, and the corresponding mask matrix is ​​[1,1,1,1,1]; for Figure 3 Phase control method B in action It is also a 5-dimensional vector. The first dimension corresponds to the timing cycle length, and the second to fourth dimensions correspond to the green-to-signal ratios of the three phases. Since the three-way intersection phase control only includes three phases, the fifth dimension value is set to 0, and the corresponding mask matrix is ​​[1,1,1,1,0].

[0091] In some embodiments, each intersection is instructed to execute a corresponding timing plan and feedback response information, specifically including:

[0092] The current intersection signal light executes the issued timing plan The intersection status is determined by Transfer to And feedback intersection reward information To the decision output layer, reward information Respond to current action The control effect is defined as the intersection health index, specifically r k =ω1P+ω2D+ω3F+ω4C, where P represents the sum of the vehicle queue lengths in the entrance lane, D represents the sum of the delays of all vehicles in the entrance lane, F represents the sum of the waiting times of all vehicles in the entrance lane, C represents the throughput of the intersection within the cycle, and ω1, ω2, ω3, and ω4 are weight coefficients. By weighted summation of the above indicators, the health status of the intersection can be fully reflected and the model update of the sub-learning model can be guided. The response information includes In addition, Stored in the experience replay pool M of the sub-learning model k As the training sample of the meta-learning model, here Information that has not passed the feature extraction layer, including the lane vehicle status set of each lane

[0093] In some embodiments, performing a second update operation on the model parameters of the corresponding sub-learning model according to each piece of response information specifically includes:

[0094] The Critic network replays the pool M based on the experience k The model parameters w of the learning model such as state transfer and reward information k To update, w k Including feature extraction layer network parameters And the decision output layer network parameters are the network parameters of the current Actor network, the current Critic network, the target Actor network, and the target Critic network respectively. The current Critic network completes the real-time evaluation of the current timing scheme by fitting the Q function, which is recorded as The current Critic network updates the network parameters involved through TD error. Due to the end-to-end training method, Related to the feature extraction layer, so the update process can achieve the network parameters of the feature extraction layer as well as Minimizing the TD error can ensure the evaluation accuracy of the current Critic network. Therefore, the specific loss function is defined as Where B is the batch size, is the TD target, Q′ and μ′ correspond to the target Critic network and the target Actor network respectively. The current Actor network measures the performance of the strategy μ through the target J. The training goal is to find the optimal strategy so that By using the chain rule, the update of the current Actor network can be expressed as Similarly, Related to the feature extraction layer, so the update process can achieve the network parameters of the feature extraction layer as well as The target actor network and target critic network are updated using soft update. τ is the soft update coefficient. At this point, a round of sub-learning model training is completed, that is, the second update operation of the model parameters of the sub-learning model is completed. Each sub-learning model that has completed training uses the new model parameters to sample new state transitions, reward information and other training samples in the local environment and stores them in the new memory bank M ′ k middle.

[0095] In some embodiments, training the meta-learning model based on all new response information specifically includes:

[0096] The meta-learning model aggregates the information of each heterogeneous sub-learning model and uses the newly sampled training samples in the memory bank to update the model parameters of the meta-learning model. Since the model structure of the meta-learning model is the same as that of the sub-learning model, the update method of the model parameters of the meta-learning model can refer to the parameter update method of the above sub-learning model. The difference is that the loss function L of each sub-learning model is accumulated. k , the loss function Loss of the meta-learning model is the loss function L of each sub-learning model k The model parameter update method of the meta-learning model is β is the meta-learner learning rate. This completes one round of meta-learning model training. Repeat the above iterative process to minimize the meta-learning model's loss function until the meta-learning model converges.

[0097] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the method.

[0098] It should be noted that the above description is limited to some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0099] The present application also provides a model training device for controlling traffic signals.

[0100] refer to Figure 7 , the model training device for controlling traffic signals comprises:

[0101] The model training module 702 iteratively performs the following operations until a training cutoff condition is reached. The model training module includes:

[0102] A first updating unit 7021 is configured to obtain model parameters of the meta-learning model and perform a first updating operation on the model parameters of each sub-learning model according to the model parameters of the meta-learning model;

[0103] The first training unit 7022 is configured to train the corresponding sub-learning model using the local historical training set corresponding to each intersection;

[0104] Output unit 7023 is configured to obtain the current lane vehicle state information of each intersection in response to reaching the preset timing time, and output the timing plan based on the lane vehicle state information through the corresponding sub-learning model;

[0105] Feedback unit 7024 is configured to enable each intersection to execute the corresponding timing plan and feedback response information;

[0106] The second updating unit 7025 is configured to perform a second updating operation on the model parameters of the corresponding sub-learning model according to each response information;

[0107] An acquiring unit 7026 is configured to acquire new response information of a corresponding intersection through each sub-learning model that has undergone the second updating operation;

[0108] The second training unit 7027 is configured to train the meta-learning model based on all new response information.

[0109] In some embodiments, the different types of intersections have different numbers of roads and lanes.

[0110] In some embodiments, the input data in the local historical training set includes lane vehicle status information, and the output data includes a timing plan. The lane vehicle status information includes vehicle position, vehicle waiting time and vehicle speed. The timing plan includes a timing cycle and an intersection phase green-to-signal ratio.

[0111] In some embodiments, the sub-learning model includes a feature extraction layer and a decision output layer, wherein the feature extraction layer includes a lane convergence layer, a phase convergence layer, and an intersection convergence layer in sequence; the output unit 7023 is further configured to input the lane vehicle state information into the lane convergence layer, and output a lane feature representation through the lane convergence layer;

[0112] Inputting the lane feature representation into the phase convergence layer, and outputting the phase state representation through the phase convergence layer;

[0113] Inputting the phase state representation into the intersection convergence layer, and outputting the intersection feature representation through the intersection convergence layer;

[0114] The intersection feature representation is input to the decision output layer, and the timing plan is output through the decision output layer.

[0115] In some embodiments, the decision output layer includes an Actor network and a Critic network.

[0116] In some embodiments, the output unit 7023 is further configured to embed the lane vehicle state information based on the first parameter of the lane convergence layer to obtain a first embedded representation of each lane;

[0117] After concatenating the mean of all first embedding representations to each first embedding representation, the second embedding representation of each lane is obtained;

[0118] Multiplying the second embedding representation by the second parameter of the lane pooling layer to obtain a first attention score for each lane for each vehicle in the lane;

[0119] The first attention score is processed by a softmax function to obtain a first attention coefficient, and the transpose of the first attention coefficient is multiplied by the first embedding representation to obtain a lane feature representation of each lane.

[0120] In some embodiments, the output unit 7023 is further configured to embed the phase state representation based on a third parameter of the intersection convergence layer to obtain a third embedded representation of each phase;

[0121] After concatenating the mean of all third embedding representations to each third embedding representation, the fourth embedding representation of each phase is obtained;

[0122] Multiplying the fourth embedding representation by the fourth parameter of the intersection pooling layer to obtain a second attention score for each intersection for each phase in the intersection;

[0123] The second attention score is processed by a softmax function to obtain a second attention coefficient, and the transpose of the second attention coefficient is multiplied by the third embedding representation to obtain the intersection feature representation of each intersection.

[0124] For the convenience of description, the above devices are described as being divided into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0125] The device of the above embodiment is used to implement the corresponding model training method for controlling traffic signals in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0126] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the model training method for controlling traffic signals described in any of the above embodiments is implemented.

[0127] Figure 8 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0128] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0129] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0130] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0131] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0132] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0133] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0134] The electronic device of the above embodiment is used to implement the corresponding model training method for controlling traffic signals in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0135] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the model training method for controlling traffic signals as described in any of the above embodiments.

[0136] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0137] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the model training method for controlling traffic signals as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0138] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the scope of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0139] In addition, for simplicity of description and discussion, and in order not to make the embodiment of the application difficult to understand, the known power supply / ground connection with integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the application difficult to understand, and this also takes into account the following fact, that is, the details of the embodiment of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuit) are set forth to describe exemplary embodiments of the application, it will be apparent to those skilled in the art that the embodiment of the application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0140] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0141] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A model training method for controlling traffic signals, characterized in that: Applied to a meta-learning model and multiple sub-learning models, each of the sub-learning models corresponds to an intersection, the intersections corresponding to the multiple sub-learning models include different types of intersections, and the different types of intersections have different numbers of roads and lanes. The sub-learning models include a feature extraction layer and a decision output layer, and the feature extraction layer sequentially includes a lane convergence layer, a phase convergence layer, and an intersection convergence layer; The method comprises: Iterate the following operations until the training cutoff condition is reached, Obtaining model parameters of the meta-learning model, and performing a first update operation on the model parameters of each sub-learning model according to the model parameters of the meta-learning model; Use the local historical training set corresponding to each intersection to train the corresponding sub-learning model; In response to reaching the preset timing moment, the current lane vehicle status information of each intersection is obtained respectively, and the timing plan is output through the corresponding sub-learning model based on the lane vehicle status information: including Inputting the lane vehicle state information into the lane convergence layer, and outputting lane feature representation through the lane convergence layer; Inputting the lane feature representation into the phase convergence layer, and outputting the phase state representation through the phase convergence layer; Inputting the phase state representation into the intersection convergence layer, and outputting the intersection feature representation through the intersection convergence layer; Inputting the intersection feature representation into the decision output layer, and outputting the timing plan through the decision output layer; Instruct each intersection to execute the corresponding timing plan and feedback the response information; Perform a second update operation on the model parameters of the corresponding sub-learning model according to each response information; Obtaining new response information of the corresponding intersection through each sub-learning model that has undergone the second updating operation; The meta-learning model is trained based on all new response information.

2. The method according to claim 1, characterized in that The input data in the local historical training set includes lane vehicle status information, and the output data includes a timing plan. The lane vehicle status information includes vehicle position, vehicle waiting time and vehicle speed. The timing plan includes a timing cycle and an intersection phase green signal ratio.

3. The method according to claim 1, characterized in that The decision output layer includes an Actor network and a Critic network.

4. The method according to claim 1, wherein Inputting the lane vehicle state information to the lane convergence layer, and outputting lane feature representation through the lane convergence layer, includes: Embedding the lane vehicle state information based on the first parameter of the lane convergence layer to obtain a first embedding representation of each lane; After concatenating the mean of all first embedding representations to each first embedding representation, the second embedding representation of each lane is obtained; Multiplying the second embedding representation by the second parameter of the lane pooling layer to obtain a first attention score for each lane for each vehicle in the lane; The first attention score is processed by a softmax function to obtain a first attention coefficient, and the transpose of the first attention coefficient is multiplied by the first embedding representation to obtain a lane feature representation of each lane.

5. The method according to claim 1, characterized in that The step of inputting the phase state representation to the intersection convergence layer and outputting the intersection feature representation through the intersection convergence layer includes: Embedding the phase state representation based on a third parameter of the intersection convergence layer to obtain a third embedded representation of each phase; After concatenating the mean of all third embedding representations to each third embedding representation, the fourth embedding representation of each phase is obtained; Multiplying the fourth embedding representation by the fourth parameter of the intersection pooling layer to obtain a second attention score for each intersection for each phase in the intersection; The second attention score is processed by a softmax function to obtain a second attention coefficient, and the transpose of the second attention coefficient is multiplied by the third embedding representation to obtain the intersection feature representation of each intersection.

6. A model training device for controlling traffic signals, characterized in that: Applied to a meta-learning model and multiple sub-learning models, each of the sub-learning models corresponds to an intersection, the intersections corresponding to the multiple sub-learning models include different types of intersections, and the different types of intersections have different numbers of roads and lanes. The sub-learning model includes a feature extraction layer and a decision output layer. The feature extraction layer sequentially includes a lane convergence layer, a phase convergence layer, and an intersection convergence layer, including: A model training module, which iteratively performs the following operations until a training cutoff condition is reached, and includes: a first updating unit, configured to obtain model parameters of the meta-learning model, and perform a first updating operation on the model parameters of each sub-learning model according to the model parameters of the meta-learning model; The first training unit is configured to train the corresponding sub-learning model using the local historical training set corresponding to each intersection; The output unit is configured to obtain the current lane vehicle status information of each intersection in response to reaching the preset timing moment, and output the timing plan based on the lane vehicle status information through the corresponding sub-learning model: including Inputting the lane vehicle state information into the lane convergence layer, and outputting lane feature representation through the lane convergence layer; Inputting the lane feature representation into the phase convergence layer, and outputting the phase state representation through the phase convergence layer; Inputting the phase state representation into the intersection convergence layer, and outputting the intersection feature representation through the intersection convergence layer; Inputting the intersection feature representation into the decision output layer, and outputting the timing plan through the decision output layer; A feedback unit is configured to enable each intersection to execute a corresponding timing plan and feedback response information; A second updating unit is configured to perform a second updating operation on the model parameters of the corresponding sub-learning model according to each piece of response information; an acquiring unit configured to acquire new response information of the corresponding intersection through each sub-learning model that has undergone the second updating operation; The second training unit is configured to train the meta-learning model based on all new response information.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

Images