Traffic light joint control method, device, computer equipment and storage medium

Abstract

The present application relates to a method, apparatus, computer equipment, storage medium, and computer program product for joint control of traffic lights. The method obtains the current phase type currently being used by the traffic light at the target intersection and the current road condition data of the target intersection; generates the current traffic state of the target intersection based on the current phase type and the current road condition data; inputs the current traffic state into a trained phase prediction model and a trained time prediction model, respectively, and outputs a target phase type and a target control duration. The phase prediction model and the time prediction model are obtained by simultaneous training based on the same training sample; controls the traffic light to switch to the target phase corresponding to the target phase type, and controls the duration of the target phase according to the target control duration. By simultaneously training the phase and duration of the traffic light, a correlation is established between the phase control and the time control of the traffic light, ensuring that the matching optimal strategy is learned, thereby improving the communication efficiency of the intersection.

Description

Technical Field

[0001] The present application relates to the field of road traffic technology, and in particular to a traffic light joint control method, device, computer equipment, storage medium and computer program product. Background Art

[0002] With the progress of social development, cars have become an indispensable part of human life, such as private cars and buses that meet travel needs, and commercial vehicles that can be used for transportation. As consumption levels rise, the number of cars continues to increase, and the resulting urban traffic congestion has become one of the inevitable troubles in daily life. At road intersections, corresponding release times are given for traffic flows in different directions. Each control state of the traffic light, that is, the combination of different light colors displayed in different directions at the entrances of various intersections, is called a traffic light phase. The phase conversion order and corresponding duration of existing urban traffic lights are preset in advance, without taking into account the actual traffic conditions. When encountering special circumstances, such as rainy weather, the existing control method cannot meet the traffic needs of vehicles with different traffic flows in actual scenarios, which can easily cause traffic congestion and low traffic efficiency. Summary of the Invention

[0003] Based on this, it is necessary to provide a traffic light joint control method, device, computer equipment, computer-readable storage medium and computer program product that can efficiently and accurately adjust traffic lights to address the above technical problems.

[0004] In a first aspect, the present application provides a method for joint control of traffic lights. The method comprises:

[0005] For the traffic light at the target intersection, obtain the current phase type of the traffic light and the current traffic condition data of the target intersection;

[0006] Generate the current traffic status of the target intersection according to the current phase type and current road condition data;

[0007] The current traffic state is input into the trained phase prediction model and the trained time prediction model respectively, and the target phase type and target control duration are output. The phase prediction model and the time prediction model are obtained by simultaneous training based on the same training sample.

[0008] The control signal light is switched to the target phase corresponding to the target phase type, and the duration of the target phase is controlled according to the target control duration.

[0009] In one embodiment, the current traffic condition data includes the current number of vehicles in each lane corresponding to the target intersection and the average distance traveled by vehicles in each lane after the current phase type is put into use. Based on the current phase and current traffic condition data, the current traffic state of the target intersection is generated, including:

[0010] generating a first eigenvector for characterizing the current phase type according to the current phase type and the number of all phase types;

[0011] Generate a second eigenvector based on the current number of vehicles in each lane and the total number of lanes, and generate a third eigenvector based on the average travel distance in each lane and the total number of lanes;

[0012] A current traffic state of the target intersection is generated based on the first eigenvector, the second eigenvector, and the third eigenvector.

[0013] In one embodiment, the training process of the time prediction model includes:

[0014] Obtaining a first time training target model having the same structure as the time prediction model, and obtaining a time evaluation model and a second time training target model having the same structure as the time evaluation model;

[0015] Acquire multiple training samples, and train a second time training target model based on the training samples, the time prediction model, and the time evaluation model;

[0016] Training the first-time training target model based on the multiple training samples and the second-time training target model;

[0017] updating the parameters in the time evaluation model according to the parameters in the second time training target model and the parameters in the time evaluation model after training;

[0018] The parameters in the time prediction model are updated according to the parameters in the first time training target model and the parameters in the time prediction model after training.

[0019] In one embodiment, the phase prediction model training process includes:

[0020] Acquire multiple training samples and a phase training target model having the same structure as the phase prediction model;

[0021] Based on multiple training samples and a phase prediction model, a phase training target model is trained;

[0022] The parameters in the phase prediction model are updated according to the parameters in the trained phase training target model.

[0023] In one embodiment, the process of obtaining training samples includes:

[0024] Obtaining sample traffic states, inputting the sample traffic states into a phase prediction model and a time prediction model respectively, and outputting corresponding sample phase types and sample control durations;

[0025] Perform traffic light simulation control according to the sample traffic state, the corresponding sample phase type, and the sample control duration, and obtain the next sample traffic state and the corresponding sample reward value after the traffic light simulation control. The sample reward value is used to represent the degree of improvement in the traffic condition of the next sample traffic state;

[0026] The sample traffic state is taken as the previous sample traffic state, and the next sample traffic state is taken as the next sample traffic state. The previous traffic state, the corresponding sample phase type, the corresponding sample control duration, the corresponding sample reward value and the next traffic state constitute a training sample.

[0027] In one embodiment, the process of traffic light simulation control is implemented through a simulation model; the process of constructing the simulation model includes:

[0028] For the target intersection, real-time data is collected on the new traffic states formed by controlling the traffic lights according to different phase types and control durations under different old traffic states, as well as the reward values ​​brought by the new traffic states;

[0029] A simulation model is constructed based on the collected data. The simulation model is used to simulate the control of traffic lights according to the old traffic state, phase type and control duration, and obtain the new traffic state and corresponding reward value after the traffic light simulation control.

[0030] In one embodiment, the training sample includes a preceding traffic state and a corresponding sample control duration; and training a first time training target model based on the training sample and the time evaluation model includes:

[0031] Based on the previous traffic state, the sample control duration corresponding to the previous traffic state is evaluated by the time evaluation model to obtain the corresponding evaluation value;

[0032] A first loss function is constructed according to the evaluation value, and the parameters in the first-time training target model are trained using the first loss function.

[0033] In one embodiment, the training samples include a preceding traffic state, a corresponding sample control duration, a corresponding sample reward value, and a following traffic state; and training a second time training target model based on the training samples, the time prediction model, and the time evaluation model includes:

[0034] Input the subsequent traffic state into the time prediction model and output the corresponding sample control time of the subsequent traffic state;

[0035] Based on the following traffic state, evaluating the sample control time corresponding to the following traffic state through a time evaluation model to obtain a first score value;

[0036] Determining a training label of the training sample for a second time training target model based on the first score value and the sample reward value corresponding to the previous traffic state;

[0037] Based on the previous traffic state, the control duration of the sample corresponding to the previous traffic state is evaluated by the time evaluation model to obtain a second scoring value;

[0038] According to the difference between the second score value and the training label, a second loss function is constructed, and the parameters in the second time training target model are trained by the second loss function.

[0039] In one embodiment, the training samples include a preceding traffic state, a corresponding sample phase type, a corresponding sample reward value, and a following traffic state; and training a phase training target model based on the plurality of training samples and the phase prediction model includes:

[0040] For a current training sample among the plurality of training samples, inputting a subsequent traffic state in the current training sample into a phase prediction model, and outputting a sample phase type and a strategy value prediction value corresponding to the subsequent traffic state;

[0041] Determine the training label of the current training sample for the phase training target model based on the strategy value prediction value corresponding to the subsequent traffic state and the sample reward value corresponding to the previous traffic state;

[0042] Input the previous traffic state into the phase prediction model and output the predicted value of the strategy value corresponding to the previous traffic state;

[0043] Constructing a third loss function based on the difference between the strategy value prediction value corresponding to the previous traffic state in each training sample and the corresponding training label;

[0044] The parameters in the phase training target model are trained through the third loss function.

[0045] In a second aspect, the present application also provides a traffic light joint control device. The device includes:

[0046] A data acquisition module is used to obtain the current phase type of the traffic light and the current road condition data of the target intersection for the traffic light at the target intersection;

[0047] A state generation module is used to generate the current traffic state of the target intersection according to the current phase type and current road condition data;

[0048] The action prediction module is used to input the current traffic state into the trained phase prediction model and the trained time prediction model respectively, and output the target phase type and target control duration. The phase prediction model and the time prediction model are obtained by simultaneous training based on the same training sample;

[0049] The action execution module is used to control the traffic light to switch to the target phase corresponding to the target phase type, and control the duration of the target phase according to the target control duration.

[0050] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0051] For the traffic light at the target intersection, obtain the current phase type of the traffic light and the current traffic condition data of the target intersection;

[0052] Generate the current traffic status of the target intersection according to the current phase type and current road condition data;

[0053] The current traffic state is input into the trained phase prediction model and the trained time prediction model respectively, and the target phase type and target control duration are output. The phase prediction model and the time prediction model are obtained by simultaneous training based on the same training sample.

[0054] The control signal light is switched to the target phase corresponding to the target phase type, and the duration of the target phase is controlled according to the target control duration.

[0055] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0056] For the traffic light at the target intersection, obtain the current phase type of the traffic light and the current traffic condition data of the target intersection;

[0057] Generate the current traffic status of the target intersection according to the current phase type and current road condition data;

[0058] The current traffic state is input into the trained phase prediction model and the trained time prediction model respectively, and the target phase type and target control duration are output. The phase prediction model and the time prediction model are obtained by simultaneous training based on the same training sample.

[0059] The control signal light is switched to the target phase corresponding to the target phase type, and the duration of the target phase is controlled according to the target control duration.

[0060] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0061] For the traffic light at the target intersection, obtain the current phase type of the traffic light and the current traffic condition data of the target intersection;

[0062] Generate the current traffic status of the target intersection according to the current phase type and current road condition data;

[0063] The current traffic state is input into the trained phase prediction model and the trained time prediction model respectively, and the target phase type and target control duration are output. The phase prediction model and the time prediction model are obtained by simultaneous training based on the same training sample.

[0064] The control signal light is switched to the target phase corresponding to the target phase type, and the duration of the target phase is controlled according to the target control duration.

[0065] The aforementioned traffic light joint control method, apparatus, computer device, storage medium, and computer program product obtain, for a traffic light at a target intersection, the current phase type of the traffic light being used and the current traffic condition data for the target intersection; generate the current traffic state of the target intersection based on the current phase type and current traffic condition data; input the current traffic state into a trained phase prediction model and a trained time prediction model, respectively, to output a target phase type and a target control duration. The phase prediction model and the time prediction model are simultaneously trained based on the same training sample; and control the traffic light to switch to a target phase corresponding to the target phase type and control the duration of the target phase according to the target control duration. By simultaneously training the phase and duration of the traffic light, a correlation is established between the training data of the traffic light phase control agent and the time control agent, enabling the decision-making areas of the two agents to learn matching optimal strategies, achieving joint optimization and improving the optimal control of the traffic light phase and duration. Furthermore, the method integrates actual traffic condition data at the intersection to intelligently set the phase of the traffic light at the intersection and optimize the signal timing, thereby stimulating the potential of the intersection and improving communication efficiency at the intersection. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A diagram showing an application environment of a traffic light joint control method according to an embodiment;

[0067] Figure 2 1 is a flow chart of a traffic light joint control method according to an embodiment;

[0068] Figure 3 A schematic diagram of a phase reference of a signal light at an intersection in one embodiment;

[0069] Figure 4 Schematic diagram of a model training process in a traffic light joint control method according to an embodiment;

[0070] Figure 5 is a flow chart of a traffic light joint control method according to another embodiment;

[0071] Figure 6 1 is a flow chart of a traffic light joint control method according to another embodiment;

[0072] Figure 7 1 is a flow chart of the training process of an intelligent agent in one embodiment;

[0073] Figure 8 is a structural block diagram of a traffic signal light joint control device in one embodiment;

[0074] Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0076] The traffic light joint control method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Specifically, terminal 102 obtains real-time traffic data at the intersection and sends it to server 104. Server 104 processes the traffic data to determine the phase switching and time settings for the traffic lights at the current intersection. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated with server 104 or placed in the cloud or other network servers.

[0077] Terminal 102 may include, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0078] In one embodiment, Figure 2 As shown, a traffic signal light joint control method is provided, which is applied to Figure 1Taking the server 104 in the example as an example, the following steps are included:

[0079] Step 202: For a traffic light at a target intersection, obtain the current phase type of the traffic light and the current traffic condition data of the target intersection;

[0080] The phase type refers to the phase type of the traffic light at the target intersection. A traffic light phase specifies the time it takes for traffic to proceed in different directions. At a signalized intersection, each state of the traffic light, or the combination of different light colors displayed for different directions on each approach lane, constitutes a signal phase. For example, at an intersection, the phase types could include "straight ahead" at an east-west intersection, "left turn" at an east-west intersection, "straight ahead" at a north-south intersection, and "left turn" at a north-south intersection.

[0081] Traffic data describes the distribution of vehicles at an intersection. This data is collected by roadside equipment such as millimeter-wave radar, lidar, and cameras. It can include the number of vehicles, lane entry and exit numbers, arrival and departure times, and the length of vehicle queues in each lane.

[0082] It should be noted that the phase type of the target intersection can be obtained through roadside equipment, such as cameras at each entrance to the target intersection, or directly from control data from a traffic signal control system. This application does not impose specific limitations on this. Specifically, in one embodiment, multiple frames of images captured by multiple cameras at the target intersection at the current moment are obtained and image processing is performed on the multiple frames to determine the current phase type of the traffic light at the target intersection, the number of vehicles, the lane numbers of vehicles entering and exiting the lanes, and the time information of vehicles arriving at and leaving the intersection.

[0083] Step 204, generating the current traffic state of the target intersection according to the current phase type and the current road condition data;

[0084] The current traffic state refers to a quantity that can be identified by the agent and describes the current traffic state at the intersection. It is understood that the acquisition of the current phase type and current road condition data by the roadside equipment in step 202 is directly acquired by the physical device. When the agent processes the data and controls the traffic lights, the agent cannot directly identify this data and needs to process the acquired data, for example, by encoding the phase type through one-hot encoding to determine the one-hot vector corresponding to the current phase type. For current road condition data, for example, if the current traffic state includes lane line queue length, then when m lane lines are identified, an m-dimensional feature vector is used to represent the lane line queue length in the current traffic state, and the value of each dimension is the length of the identified vehicle queue. It should be noted that if the current traffic state has n parameter types, an m*n feature vector can be used to represent it. For example, if the current traffic state also includes the distance traveled by vehicles at the intersection, the value of each dimension is the average travel distance.

[0085] Specifically, the corresponding eigenvector of the current phase type is determined through coding; for the current road condition data, the physical value data is defined and represented to obtain the eigenvector of the vehicle distribution at the intersection; the two together constitute the current traffic state s.

[0086] Step 206: Input the current traffic state into the trained phase prediction model and the trained time prediction model, and output the target phase type and target control duration. The phase prediction model and the time prediction model are obtained by simultaneous training based on the same training sample.

[0087] The phase prediction model is a discrete phase selection agent, and the time prediction model is a continuous time allocation agent. Both agents are trained using reinforcement learning. After inputting a vector representing the current traffic state into each trained agent, the discrete phase selection agent determines the target phase type corresponding to the next moment, while the continuous time allocation agent determines the target control duration corresponding to the target phase type.

[0088] It should be noted that there is a certain correlation between the phase prediction model and the time prediction model. When performing reinforcement learning training on the two, the same batch of data from the same training data pool is used. That is, during training, for a training sample, it needs to be input into the discrete phase selection agent and the continuous time allocation agent to be trained at the same time, and the phase prediction model and the time prediction model are trained at the same time to ensure the consistency of the learning data distribution of the two, ensure joint optimization, and improve traffic efficiency.

[0089] Step 208 : Control the signal light to switch to the target phase corresponding to the target phase type, and control the duration of the target phase according to the target control duration.

[0090] After obtaining the target phase type and target control duration, the traffic light phase at the target intersection is adjusted to the target phase type and the target control duration is maintained. For example, at an intersection, the current phase type of the target intersection is east-west straight for 30 seconds. After the algorithm of this embodiment processes the traffic condition data and phase type of the current intersection, it is determined that the target phase type is east-west left turn and the target control duration is 45 seconds. In this case, the traffic light at the target intersection is adjusted to east-west left turn for 45 seconds.

[0091] It should be noted that the target phase type can be any of the phase types at the target intersection. This means the target phase type can be consistent with or inconsistent with the current phase type. A consistent target phase type indicates a large number of vehicles waiting to pass in the desired direction, so the signal phase at the target intersection remains unchanged. Furthermore, it should be noted that while the continuous time allocation agent used to predict the target duration allocates continuous time to a single duration, based on intersection control, signals are typically controlled in seconds. Therefore, the target control duration is typically calculated as an integer number of seconds to facilitate signal control.

[0092] In the method provided in the above embodiment, for a traffic light at a target intersection, the current phase type of the traffic light and the current traffic condition data of the target intersection are obtained; based on the current phase type and current traffic condition data, the current traffic state of the target intersection is generated; the current traffic state is input into a trained phase prediction model and a trained time prediction model, respectively, to output a target phase type and target control duration; the traffic light is controlled to switch to the target phase corresponding to the target phase type, and the duration of the target phase is controlled according to the target control duration. By simultaneously training the phase and duration of the traffic light, a correlation is established between the training data of the traffic light phase control agent and the time control agent, enabling the decision-making areas of the two agents to learn matching optimal strategies, achieving joint optimization, and improving the optimal control of the traffic light phase and duration. Furthermore, the method combines actual intersection traffic condition data to intelligently set the phase of the intersection traffic light and optimize the signal timing, thereby stimulating the potential of the intersection and improving intersection communication efficiency.

[0093] In one embodiment, the current traffic condition data includes the current number of vehicles in each lane corresponding to the target intersection and the average distance traveled by vehicles in each lane after the current phase type is put into use. Based on the current phase and current traffic condition data, the current traffic state of the target intersection is generated, including:

[0094] generating a first eigenvector for characterizing the current phase type according to the current phase type and the number of all phase types;

[0095] Generate a second eigenvector based on the current number of vehicles in each lane and the total number of lanes, and generate a third eigenvector based on the average travel distance in each lane and the total number of lanes;

[0096] A current traffic state of the target intersection is generated based on the first eigenvector, the second eigenvector, and the third eigenvector.

[0097] Among them, all phase types refer to the phase types of all achievable signal lights at the target intersection. For different intersections, the phase types that exist will also be different due to the different actual conditions of the intersections where the signal lights are located. For example, at different types of intersections, the total number of signal lights at the intersection and the T-junction is different, so the phase types of the signal lights at the two intersections are also different; for another example, at the same type of intersection, although there are 4 signal lights at intersection 1 and intersection 2, there are different traffic rules at the intersection. For example, the east-west lanes of intersection 1 do not allow left turns. In addition, there will be different phase type combinations for all signal lights at the same intersection. For example, at an intersection, vehicles in the same lane can be allowed to go straight and turn left at the same time. In summary, the phase type mentioned in the embodiment of the present application should be one of all phase types determined under the same rule of the target intersection, that is, the traffic signal light joint control method mentioned in the embodiment of the present application is carried out under the condition of a fixed number of phase types. If the rules for determining all phase types need to be changed, the signal light phase prediction model and time prediction model of the target intersection need to be re-learned and trained.

[0098] The total number of lanes is the set data for the target intersection. The number of vehicles in each lane can be obtained in real time by roadside equipment. By determining the number of vehicles in each lane at the current moment, a second eigenvector is generated, representing the vehicle distribution in each lane at the target intersection at the current moment. The dimension of the second eigenvector is determined by the total number of lanes at the target intersection, with the value of each dimension representing the number of vehicles in the corresponding lane. Similarly, the dimension of the third eigenvector is also determined by the total number of lanes at the target intersection, with the value of each dimension representing the average travel distance in the corresponding lane.

[0099] The average travel distance in a lane is related to the total number of vehicles entering and exiting the lane. For the current moment, the first, second, and third eigenvectors are integrated to create a vector matrix representing the current traffic state at the target intersection.

[0100] It is understandable that for an intersection, only one of all phase types can be realized at a certain moment, otherwise traffic chaos will occur. Therefore, the current phase type can be encoded by the number of all phase types to determine the first eigenvector. Figure 3 For the target intersection shown, the encoding results corresponding to the phase type are shown in the following table (blank indicates red light status):

[0101] Table 1 Intersection phase type table

[0102]

[0103] In the method provided in the above embodiment, the current phase type and current vehicle condition data of the target intersection are converted into data through unified rules to generate data that can be recognized by the intelligent agent, thereby realizing the control of the traffic light and improving the accuracy and robustness of the traffic light control system.

[0104] In one embodiment, the training process of the time prediction model includes:

[0105] Obtaining a first time training target model having the same structure as the time prediction model, and obtaining a time evaluation model and a second time training target model having the same structure as the time evaluation model;

[0106] Acquire multiple training samples, and train a second time training target model based on the training samples, the time prediction model, and the time evaluation model;

[0107] Training the first-time training target model based on the multiple training samples and the second-time training target model;

[0108] updating the parameters in the time evaluation model according to the parameters in the second time training target model and the parameters in the time evaluation model after training;

[0109] The parameters in the time prediction model are updated according to the parameters in the first time training target model and the parameters in the time prediction model after training.

[0110] The training samples are data obtained from simulated vehicle operation at the target intersection. They can also be historical vehicle operation data and corresponding phase data obtained from real-world scenarios. It should be noted that the data included in the training samples describes the interaction experience between the target intersection and the intelligent agent controlling the target intersection, including the intelligent agent's input to the target intersection and the status of the input, i.e., the feedback. The acquired data can be used directly as sample data, or it can be processed through filtering, reorganization, duplication checking, and verification before being encoded to define each sample.

[0111] The temporal prediction model includes a sample duration strategy that selects the corresponding output control action based on the input state in the sample. The purpose of training the temporal prediction model is to determine the optimal sample duration strategy. However, if the prediction model is trained directly, changes in the temporal prediction model's parameters will affect changes in the output, resulting in large errors in the training process. Therefore, a first temporal training target model with the same structure as the temporal prediction model is constructed. The first temporal training target model is used to update parameters and select actions, while the temporal prediction model calculates the corresponding score values ​​according to a fixed method. Therefore, the first temporal training target model can be trained and its parameters directly copied to the temporal prediction model.

[0112] It can be seen that the function of the first time training target model is to update the parameters of the time prediction model. Therefore, the structure of the first time training target model must be consistent with the time prediction model, otherwise its parameters cannot be used to update the parameters of the time prediction model. Similarly, the structure of the second time training target model should be consistent with the structure of the time evaluation model. The time prediction model is used to select the time corresponding to the phase action at the next moment based on the current traffic state. Therefore, the time prediction model should be a neural network model with a continuous action space, such as the Target Critic network in the Deep Deterministic Policy Gradient (DDPG) algorithm. The model outputs a data value representing a specific action. In this embodiment, the output action is a time allocation action.

[0113] In one embodiment, see Figure 4 , Target Actor network is the time prediction model, Eval Actor network is the first time training target model, Target Critic network is the time evaluation model, and Eval Critic network is the second time training target model. Input the training sample into the Target Actor network, for example, sample (s,,,′), where s represents the traffic state, a=(a phase,time ) is a joint action, including phase distribution action a phase and time allocation action a time , r is the reward fed back to the time prediction model by the simulator at the target intersection after executing the joint action a. Input (s,,,′) into the Eval Actor network neural network and output the new time allocation action a new =(s;μ), where π(s;μ) is the time allocation strategy for training the target model Eval Actor network at the first time, and μ is the weight of the EvalActor network neural network. newInput to the Eval Critic network and output action a to the Eval Actornetwork new Perform an evaluation to obtain the evaluation value π(s; μ) of the Eval Actor network, and adjust the parameters of the first-time training target model based on the evaluation value.

[0114] In one embodiment, in order to deepen the exploration of the environment and obtain the global optimal solution, a time Add random interference and control it between the minimum control time and the maximum control time, for example, the minimum control time is 10s and the maximum control time is 60s.

[0115] For the second time training target model, by inputting the sample (s,,,′) into the time prediction model, the action in the next state s′ is predicted, and then the difference between the target value and the expected value of the action is optimized, thereby realizing the parameter update of the Eval Critic network of the second time training target model.

[0116] It should be noted that each training sample corresponds to a time step. When training the time prediction model, the parameters of the time prediction model must be adjusted after each sample is input into the model. That is, at each time step, the parameter amplitude of the trained second time training target model is transferred to the time prediction model to complete the update of the time prediction model. In addition, when updating the parameters of the time evaluation model and the time prediction model based on the first time training target model and the second time training target model, the weight parameters can be directly copied or converted through a specific method, which is not specifically limited here.

[0117] In the method provided in the above embodiment, the time prediction model is trained through the four-network model structure to complete the training of the prediction model of continuous time action, thereby obtaining more accurate prediction results and making traffic light control more accurate.

[0118] In one embodiment, the phase prediction model training process includes:

[0119] Acquire multiple training samples and a phase training target model having the same structure as the phase prediction model;

[0120] Based on multiple training samples and a phase prediction model, a phase training target model is trained;

[0121] The parameters in the phase prediction model are updated according to the parameters in the trained phase training target model.

[0122] As can be seen from the above description, if the phase prediction model is trained directly, its output process will be affected by the training process, resulting in large errors in the output. Therefore, a phase training target model with the same structure as the phase prediction model is obtained, and the parameters of the phase prediction model are updated through the phase training target model. Specifically, the role of the phase training target model is to update the parameter selection action. During the training process, the phase prediction model performs a self-benefit reward evaluation as a learning target. After the training sample is input into the phase training target model, the phase training target model will estimate the action value based on the current sample state in the training sample, and update the parameters based on the evaluation value and the value obtained by the phase training target model.

[0123] It should be noted that the method provided in the embodiments of this application uses multi-sample data to soft-update the phase prediction model. This means that the training of the phase prediction model requires sample accumulation. In terms of sample acquisition, each sample acquired at a given moment trains the phase training target model, and the parameters of the phase prediction model are adjusted using the parameters of the phase training target model during each preset cycle. For example, the time corresponding to the current sample is time step t, every n time steps constitutes a training round, and the parameters of the phase prediction model are updated every M training rounds.

[0124] For the current training sample among the N training samples, the traffic state of the next sample in the current training sample is input into the phase prediction model, and the corresponding sample phase strategy and strategy value prediction value are output (the phase prediction model selects the sample phase strategy with the largest strategy value prediction value as the output sample phase strategy, which is used here to assist in constructing the training label); based on the strategy value prediction value and the reward value in the current training sample, the training label of the phase training target model corresponding to the current training sample for the phase prediction model is determined;

[0125] The sample traffic state in each training sample is input into the phase prediction model, and the corresponding strategy value prediction value is output. According to the difference between the strategy value prediction value corresponding to the sample traffic state in each training sample in these N training samples and the training label corresponding to each training sample, a third loss function is constructed; the phase training target model is trained according to the third loss function; and the parameters in the phase prediction model are updated according to the parameters in the trained phase training target model.

[0126] In the method provided in the above embodiment, the phase prediction model is updated and trained by soft updating to obtain a more accurate phase prediction model, thereby improving the accuracy of phase type selection in signal control and the robustness of the traffic light control system.

[0127] In one embodiment, see Figure 5 , the process of obtaining training samples includes:

[0128] Step 502: Obtain sample traffic status, input the sample traffic status into the phase prediction model and the time prediction model respectively, and output the corresponding sample phase type and sample control duration;

[0129] Step 504: Perform traffic light simulation control based on the sample traffic state, the corresponding sample phase type, and the sample control duration, and obtain the next sample traffic state and the corresponding sample reward value after the traffic light simulation control. The sample reward value is used to represent the degree of improvement in the traffic condition of the next sample traffic state.

[0130] In step 506, the sample traffic state is used as the previous sample traffic state, and the next sample traffic state is used as the next sample traffic state. The previous traffic state, the corresponding sample phase type, the corresponding sample control duration, the corresponding sample reward value and the next traffic state constitute a training sample.

[0131] Among them, the sample traffic state refers to the traffic state data of the target intersection obtained randomly. For example, the historical vehicle distribution data and the corresponding phase data at each moment collected by the roadside equipment (such as millimeter wave radar, lidar, camera, etc.) at the target intersection can be imported into the traffic light simulator to prepare for the interactive training between the time prediction model and the phase prediction model and the environment. Among them, the traffic light simulator is established based on the parameters of the target intersection, and the embodiment of the present application does not specifically limit the construction process of the simulator. When training is required, at time step t, the road condition data and phase data of the intersection at the current time step t are obtained from the traffic light simulator to generate the current traffic state as a sample traffic state.

[0132] For the current sample traffic state s (the previous sample traffic state), after inputting it into the phase prediction model, the phase allocation action a is obtained. phase ; After inputting into the time prediction model, the time allocation action a is obtained time , by phase distribution action a phase and time allocation action a time Constitute a joint action a=(a phase,time ) is sent to the traffic light simulator for simulation control, which then generates the new traffic state of the target intersection, i.e., the next sample traffic state s′ (the next sample traffic state). Simultaneously, the simulator determines the degree to which the joint action a improves the traffic conditions at the target intersection based on the new traffic state and provides feedback as a reward r, i.e., the reward r corresponding to the current sample traffic state s.

[0133] The above process obtains an interaction experience (s, a, r, s′), which can be stored in the training data buffer pool as a training sample. When the phase prediction model and the time prediction model need to be trained, a batch of training samples are obtained from the training data buffer pool, and the phase prediction model and the time prediction model are trained at the same time.

[0134] In the method provided in the above embodiment, a sample traffic state is simultaneously input into a time prediction model and a phase prediction model to obtain a joint action, establishing a correlation between phase allocation and action allocation. The phase allocation action and the time allocation action are used as a single interaction experience to form a training sample. Training is conducted using the same data source. During each training session, the two agents in the distributed decision-making region of this method use the same training data, thereby learning matching optimal strategies and achieving a joint optimal strategy.

[0135] In one embodiment, see Figure 6 The training samples include the previous traffic state and the corresponding sample control duration; based on the training samples and the second time training target model, the first time training target model is trained, including:

[0136] Step 602 , based on the previous traffic state, evaluate the sample control duration corresponding to the previous traffic state using the second time training target model to obtain a corresponding score value;

[0137] Step 604: construct a first loss function according to the score value, and train the parameters in the first-time training target model using the first loss function.

[0138] For the current training sample (s,,,′), see Figure 4 , input the previous sample traffic state s into the first time training target model Eval Actor network, and obtain the new time allocation action (sample control duration) a under the current sample traffic state s new ,Then the target model Eval Critic network is trained in the second time to calculate the current sample traffic state action pair (,a new )'s evaluation value in, is the weight of the Eval Critic network, where π(s;μ) is the time allocation strategy for training the target model Eval Actor network at the first time, and μ is the weight of the target model Eval Actor network at the first time.

[0139] It should be noted that the evaluation value It represents the cumulative expected return. In one embodiment, a gradient ascent algorithm can be used to maximize the cumulative expected return to update the parameters of the first-time training target model Eval Actornetwork.

[0140] In one embodiment, the first loss function may be:

[0141]

[0142] The parameter μ in the first-time training target model is updated by the first loss function.

[0143] In the method provided in the above embodiment, the parameters of the target model trained at the first time are updated by establishing a first loss function, thereby improving the effect of model training and thereby improving the control effect of the phase-to-duration of the traffic light.

[0144] In one embodiment, the training samples include a preceding traffic state, a corresponding sample control duration, a corresponding sample reward value, and a following traffic state; and training a second time training target model based on the training samples, the time prediction model, and the time evaluation model includes:

[0145] Input the subsequent traffic state into the time prediction model and output the corresponding sample control time of the subsequent traffic state;

[0146] Based on the following traffic state, evaluating the sample control time corresponding to the following traffic state through a time evaluation model to obtain a first score value;

[0147] Determining a training label of the training sample for a second time training target model based on the first score value and the sample reward value corresponding to the previous traffic state;

[0148] Based on the previous traffic state, the control time of the sample corresponding to the previous traffic state is evaluated by the time evaluation model to obtain a second scoring value;

[0149] According to the difference between the second score value and the training label, a second loss function is constructed, and the parameters in the second time training target model are trained by the second loss function.

[0150] The following traffic state in the training sample is input into the time prediction model, and the sample duration strategy corresponding to the following traffic state in the training sample is output; based on the following traffic state in the training sample, the sample duration strategy corresponding to the following traffic state in the training sample is scored through the time evaluation model to obtain a first scoring value; based on the first scoring value and the reward value of the training sample, the training label of the training sample corresponding to the second time training target model of the time evaluation model is determined.

[0151] Based on the previous traffic state in the training sample, the sample duration strategy corresponding to the previous traffic state in the training sample is scored through the time evaluation model to obtain a second scoring value; according to the difference between the second scoring value and the training label, a second loss function is constructed, and the parameters in the second time training target model are trained according to the second loss function; according to the parameters in the trained second time training target model and the parameters in the time evaluation model, the parameters in the time evaluation model are updated.

[0152] In one embodiment, see Figure 4 , the second loss function is constructed as:

[0153]

[0154] Among them, a′t ′time =(s′; ′) is the sample control time corresponding to the subsequent traffic state; a time =π(s;μ) is the sample control duration corresponding to the previous traffic state; is the first rating value, is the second rating value, It is the network weight of the Target Critic network of the time evaluation model.

[0155] In one embodiment, the training samples include a preceding traffic state, a corresponding sample phase type, a corresponding sample reward value, and a following traffic state; and training a phase training target model based on the plurality of training samples and the phase prediction model includes:

[0156] For a current training sample among the plurality of training samples, inputting a subsequent traffic state in the current training sample into a phase prediction model, and outputting a sample phase type and a strategy value prediction value corresponding to the subsequent traffic state;

[0157] Determine the training label of the current training sample for the phase training target model based on the strategy value prediction value corresponding to the subsequent traffic state and the sample reward value corresponding to the previous traffic state;

[0158] Input the previous traffic state into the phase prediction model and output the predicted value of the strategy value corresponding to the previous traffic state;

[0159] Constructing a third loss function based on the difference between the strategy value prediction value corresponding to the previous traffic state in each training sample and the corresponding training label;

[0160] The parameters in the phase training target model are trained through the third loss function.

[0161] In one embodiment, see Figure 4 , the third loss function can be:

[0162] L(θ)=E[(r+γQ t ' arget (s′,′ phase ;′)-Q eval (s,a phase ;)) 2 ]

[0163] Among them, θ is the network weight of the phase training target model Eval network; θ′ is the network weight of the phase prediction model Targetnetwork; a′ phase Indicates the maximum Q t ' arget The corresponding phase selection action is updated by the third loss function mentioned above to update the network weight θ of the Eval network.

[0164] In the method provided in the above embodiment, the parameters of the phase training target model are updated by establishing a third loss function, thereby improving the effect of model training and thereby improving the control effect of the phase corresponding duration of the traffic light.

[0165] In one embodiment, a traffic signal light joint control method based on an agent is provided. Figure 7 , the training process of the agent includes:

[0166] Get real-time information on intersection conditions;

[0167] The information is processed to obtain the current phase information and vehicle distribution. The phase information is one-hot encoded. The vehicle distribution includes the queue length of each lane line and the distance traveled by the vehicle at the intersection. Then the state s is constructed to input into the neural network.

[0168] The traffic light timing task is decomposed into two subtasks: the traffic light phase decision subtask and the traffic light time allocation subtask. Each subtask is determined by a different decision-making region. A corresponding discrete phase decision agent and continuous time allocation agent are then constructed. This method deploys the Deep QNetwork (DQN) algorithm and the Deep Deterministic Policy Gradient (DDPG) algorithm to learn the optimal discrete phase selection strategy and the optimal continuous time allocation strategy, respectively. These two algorithms simultaneously complete the traffic light timing task at a given moment.

[0169] During the training data collection phase, the two decision-making agents interact with the intersection continuously. This method uses a distributed execution method to input the current time step state s into the phase decision area and the time allocation area at the same time, and the corresponding agents simultaneously decide on the action a. phase and a timeThe result of the decision is applied to the intersection, and the intersection will give the agent a reward r. At the same time, the intersection status will change from s to the next state s′. Then (s,,,′)(where a=(a phase,time )) is stored in the training data cache pool H, and the interaction of the current time step ends.

[0170] During the agent learning phase, the agent corresponding to each decision region draws batches of interaction experiences from a cache pool for training to improve its decision-making capabilities. The training data collection phase and the agent learning phase alternate until the proposed algorithm converges and the optimal joint strategy is obtained. DQN updates the target network using a soft update every M training rounds; in DDPG, all target network weights are updated at every time step.

[0171] This distributed execution method distributes the state of implicit coupling information of the current time step to different decision-making areas at the same time. The intelligent agent can extract the coupling relationship from the state through the neural network to implement a matching joint strategy, thereby achieving joint optimization and improving the traffic efficiency brought by the intelligent agent controlling the traffic lights.

[0172] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0173] Based on the same inventive concept, embodiments of the present application also provide a traffic light joint control device for implementing the aforementioned traffic light joint control method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more traffic light joint control device embodiments provided below can be found in the aforementioned limitations of the traffic light joint control method and will not be further elaborated here.

[0174] In one embodiment, Figure 8 As shown, a traffic signal light joint control device is provided, comprising: a data acquisition module 801, a state generation module 802, an action prediction module 803 and an action execution module 804, wherein:

[0175] The data acquisition module 801 is used to acquire the current phase type of the traffic light and the current traffic condition data of the target intersection for the traffic light at the target intersection;

[0176] A state generation module 802 is used to generate the current traffic state of the target intersection according to the current phase type and the current road condition data;

[0177] The action prediction module 803 is used to input the current traffic state into the trained phase prediction model and the trained time prediction model, and output the target phase type and target control duration. The phase prediction model and the time prediction model are obtained by simultaneous training based on the same training sample;

[0178] The action execution module 804 is used to control the signal light to switch to the target phase corresponding to the target phase type, and control the duration of the target phase according to the target control duration.

[0179] In one embodiment, the state generation module 802 is further configured to:

[0180] generating a first eigenvector for characterizing the current phase type according to the current phase type and the number of all phase types;

[0181] Generate a second eigenvector based on the current number of vehicles in each lane and the total number of lanes, and generate a third eigenvector based on the average travel distance in each lane and the total number of lanes;

[0182] A current traffic state of the target intersection is generated based on the first eigenvector, the second eigenvector, and the third eigenvector.

[0183] In one embodiment, the traffic signal light joint control device further includes a first training module for:

[0184] Obtaining a first time training target model having the same structure as the time prediction model, and obtaining a time evaluation model and a second time training target model having the same structure as the time evaluation model;

[0185] Acquire multiple training samples, and train a second time training target model based on the training samples, the time prediction model, and the time evaluation model;

[0186] Training the first-time training target model based on the multiple training samples and the second-time training target model;

[0187] updating the parameters in the time evaluation model according to the parameters in the second time training target model and the parameters in the time evaluation model after training;

[0188] The parameters in the time prediction model are updated according to the parameters in the first time training target model and the parameters in the time prediction model after training.

[0189] In one embodiment, the traffic signal light joint control device further includes a second training module for:

[0190] Acquire multiple training samples and a phase training target model having the same structure as the phase prediction model;

[0191] Based on multiple training samples and a phase prediction model, a phase training target model is trained;

[0192] The parameters in the phase prediction model are updated according to the parameters in the trained phase training target model.

[0193] In one embodiment, the traffic signal light joint control device further includes a training sample acquisition module for:

[0194] Obtaining sample traffic states, inputting the sample traffic states into a phase prediction model and a time prediction model respectively, and outputting corresponding sample phase types and sample control durations;

[0195] Perform traffic light simulation control according to the sample traffic state, the corresponding sample phase type, and the sample control duration, and obtain the next sample traffic state and the corresponding sample reward value after the traffic light simulation control. The sample reward value is used to represent the degree of improvement in the traffic condition of the next sample traffic state;

[0196] The sample traffic state is taken as the previous sample traffic state, and the next sample traffic state is taken as the next sample traffic state. The previous traffic state, the corresponding sample phase type, the corresponding sample control duration, the corresponding sample reward value and the next traffic state constitute a training sample.

[0197] In one embodiment, the first training module is further configured to:

[0198] Based on the previous traffic state, the control duration of the sample corresponding to the previous traffic state is evaluated by the second time training target model to obtain a corresponding evaluation value;

[0199] A first loss function is constructed according to the evaluation value, and the parameters in the first-time training target model are trained using the first loss function.

[0200] In one embodiment, the first training module is further configured to:

[0201] Input the subsequent traffic state into the time prediction model and output the corresponding sample control time of the subsequent traffic state;

[0202] Based on the following traffic state, evaluating the sample control time corresponding to the following traffic state through a time evaluation model to obtain a first score value;

[0203] Determining a training label of the training sample for a second time training target model based on the first score value and the sample reward value corresponding to the previous traffic state;

[0204] Based on the previous traffic state, the control duration of the sample corresponding to the previous traffic state is evaluated by the time evaluation model to obtain a second scoring value;

[0205] According to the difference between the second score value and the training label, a second loss function is constructed, and the parameters in the second time training target model are trained by the second loss function.

[0206] In one embodiment, the second training module is further configured to:

[0207] For a current training sample among the plurality of training samples, inputting a subsequent traffic state in the current training sample into a phase prediction model, and outputting a sample phase type and a strategy value prediction value corresponding to the subsequent traffic state;

[0208] Determine the training label of the current training sample for the phase training target model based on the strategy value prediction value corresponding to the subsequent traffic state and the sample reward value corresponding to the previous traffic state;

[0209] Input the previous traffic state into the phase prediction model and output the predicted value of the strategy value corresponding to the previous traffic state;

[0210] Constructing a third loss function based on the difference between the strategy value prediction value corresponding to the previous traffic state in each training sample and the corresponding training label;

[0211] The parameters in the phase training target model are trained through the third loss function.

[0212] Each module in the aforementioned traffic signal joint control device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor within a computer device in the form of hardware, or may be stored in a computer device memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0213] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 9As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store traffic status data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for joint control of traffic lights is implemented.

[0214] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0215] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements all the steps provided in the above embodiments when executing the computer program.

[0216] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, all the steps provided in the above embodiments are implemented.

[0217] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, all the steps provided in the above embodiments are implemented.

[0218] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0219] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0220] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A traffic light joint control method, characterized in that: The method comprises: For a traffic light at a target intersection, obtaining a current phase type currently being used by the traffic light and current traffic condition data of the target intersection; generating a current traffic state of the target intersection according to the current phase type and the current road condition data; The current traffic state is input into a trained phase prediction model and a trained time prediction model respectively, and a target phase type and a target control duration are output. The phase prediction model and the time prediction model are obtained by simultaneous training based on the same training sample. The phase prediction model is a discrete phase selection agent, and the time prediction model is a continuous time allocation agent. The discrete phase selection agent is used to determine the target phase type corresponding to the next moment from the current moment, and the continuous time allocation agent is used to determine the target control duration corresponding to the target phase type. The signal light is controlled to switch to a target phase corresponding to the target phase type, and a duration of the target phase is controlled according to the target control duration.

2. The method according to claim 1, characterized in that The training process of the time prediction model includes: Acquire a first time training target model having the same structure as the time prediction model, and acquire a time evaluation model and a second time training target model having the same structure as the time evaluation model; Acquire multiple training samples, and train the second time training target model based on the multiple training samples, the time prediction model, and the time evaluation model; Training the first time training target model based on the multiple training samples and the second time training target model; updating the parameters in the time evaluation model according to the parameters in the trained second time training target model and the parameters in the time evaluation model; The parameters in the time prediction model are updated according to the parameters in the first time training target model after training and the parameters in the time prediction model.

3. The method according to claim 2, characterized in that The training process of the phase prediction model includes: Acquire the plurality of training samples and a phase training target model having the same structure as the phase prediction model; Training the phase training target model based on the multiple training samples and the phase prediction model; The parameters in the phase prediction model are updated according to the parameters in the trained phase training target model.

4. The method according to claim 2 or 3, characterized in that The process of obtaining the training samples includes: Acquire a sample traffic state, input the sample traffic state into the phase prediction model and the time prediction model respectively, and output a corresponding sample phase type and sample control duration; Performing traffic light simulation control according to the sample traffic state, the corresponding sample phase type, and the sample control duration, and obtaining a next sample traffic state and a corresponding sample reward value after the traffic light simulation control, wherein the sample reward value is used to represent a degree of improvement in the traffic condition of the next sample traffic state; The sample traffic state is taken as the previous sample traffic state, the next sample traffic state is taken as the subsequent sample traffic state, and the previous sample traffic state, the corresponding sample phase type, the corresponding sample control duration, the corresponding sample reward value and the subsequent sample traffic state constitute a training sample.

5. The method according to claim 2, characterized in that The training sample includes the previous traffic state and the corresponding sample control duration; and the training of the first time training target model based on the training sample and the second time training target model includes: Based on the previous traffic state, evaluating the sample control duration corresponding to the previous traffic state using the second time training target model to obtain a corresponding evaluation value; A first loss function is constructed according to the evaluation value, and parameters in the first time training target model are trained using the first loss function.

6. The method according to claim 2, characterized in that The training samples include a preceding traffic state, a corresponding sample control duration, a corresponding sample reward value, and a following traffic state; and training the second time training target model based on the training samples, the time prediction model, and the time evaluation model, including: Inputting the following traffic state into the time prediction model, and outputting the sample control duration corresponding to the following traffic state; Based on the following traffic state, evaluating the sample control duration corresponding to the following traffic state using the time evaluation model to obtain a first scoring value; determining, based on the first score value and the sample reward value corresponding to the previous traffic state, a training label of the training sample for the second time training target model; Based on the previous traffic state, evaluating the sample control duration corresponding to the previous traffic state using the time evaluation model to obtain a second scoring value; A second loss function is constructed based on the difference between the second rating value and the training label, and the parameters in the second time training target model are trained using the second loss function.

7. The method according to claim 3, characterized in that The training samples include a preceding traffic state, a corresponding sample phase type, a corresponding sample reward value, and a following traffic state; and the training of the phase training target model based on the plurality of training samples and the phase prediction model includes: For a current training sample among a plurality of training samples, inputting a subsequent traffic state in the current training sample into the phase prediction model, and outputting a sample phase type and a strategy value prediction value corresponding to the subsequent traffic state; Determining a training label of a current training sample for the phase training target model according to the strategy value prediction value corresponding to the subsequent traffic state and the sample reward value corresponding to the preceding traffic state; Inputting the previous traffic state into the phase prediction model, and outputting a strategy value prediction value corresponding to the previous traffic state; Constructing a third loss function based on the difference between the strategy value prediction value corresponding to the previous traffic state in each training sample and the corresponding training label; The parameters in the phase training target model are trained using the third loss function.

8. A traffic light joint control device, characterized in that: The device comprises: A data acquisition module, configured to acquire, for a traffic light at a target intersection, a current phase type currently being used by the traffic light and current traffic condition data of the target intersection; a state generating module, configured to generate a current traffic state of the target intersection according to the current phase type and the current road condition data; an action prediction module, configured to input the current traffic state into a trained phase prediction model and a trained time prediction model, respectively, and output a target phase type and a target control duration, wherein the phase prediction model and the time prediction model are obtained by simultaneous training based on the same training sample; the phase prediction model is a discrete phase selection agent, and the time prediction model is a continuous time allocation agent; the discrete phase selection agent is configured to determine the target phase type corresponding to the next moment after the current moment, and the continuous time allocation agent is configured to determine the target control duration corresponding to the target phase type; An action execution module is used to control the signal light to switch to the target phase corresponding to the target phase type, and control the duration of the target phase according to the target control duration.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images