Traffic signal control method and related equipment

Abstract

The embodiment of the invention provides a traffic signal control method and related equipment, and belongs to the technical field of traffic control. The method comprises the following steps: constructing a first traffic signal control model according to a preset deep reinforcement learning model; performing model pre-training on the first traffic signal control model through preset demonstration data obtained by performing simulation demonstration through a preset traffic control algorithm to obtain a second traffic signal control model; acquiring traffic environment state data, and inputting a second traffic signal control model to generate model experience data; performing hybrid interaction training on the second traffic signal control model according to preset demonstration data and model experience data to generate a target traffic signal control model; and determining a target signal timing strategy through the target traffic signal control model so as to perform traffic signal control. According to the embodiment of the invention, the control performance problem in the early stage of model training can be relieved, the model convergence speed and the priori knowledge utilization rate are improved, and the deployment reliability in an actual traffic scene is improved.

Description

Technical Field

[0001] This application relates to the field of traffic control technology, and in particular to a traffic signal control method and related equipment. Background Technology

[0002] Traffic signal control can be viewed as a typical Markov decision process. Currently, traffic signal control models exhibit extremely poor control performance in the early stages of training (e.g., causing significant queue lengths and vehicle delays), and the time cost required for model parameters to converge to a steady state is extremely high. Furthermore, the utilization rate of prior knowledge is low, severely hindering their deployment in real-world traffic scenarios.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a traffic signal control method and related equipment, which can alleviate the control performance problem in the early stage of model training, effectively improve the model convergence speed and the utilization rate of prior knowledge, and thus improve the reliability of deployment in actual traffic scenarios.

[0005] To achieve the above objectives, one aspect of this application provides a traffic signal control method, the method comprising: The first traffic signal control model is constructed based on a pre-set deep reinforcement learning model; The first traffic signal control model is pre-trained using preset demonstration data to obtain a second traffic signal control model; wherein, the preset demonstration data is obtained through traffic simulation demonstration using a preset traffic control algorithm; Dynamically acquire traffic environment status data and input it into the second traffic signal control model to generate model experience data; The second traffic signal control model is trained using the preset demonstration data and the model experience data to generate the target traffic signal control model. The target traffic signal timing strategy is determined by the target traffic signal control model, and then traffic signal control is performed using the target traffic signal timing strategy.

[0006] In some embodiments, constructing a first traffic signal control model based on a preset deep reinforcement learning model includes: A traffic signal control architecture is constructed based on a Markov decision model; wherein, the traffic signal control architecture includes state space parameters, action space parameters, state transition probability parameters, reward parameters, and reward discount factors; The first traffic signal control model is constructed based on the traffic signal control architecture combined with a preset deep Q-learning network model; wherein, the preset deep Q-learning network model is obtained by optimizing the dual deep Q-network model using a Bayesian linear regression algorithm.

[0007] In some embodiments, before performing model pre-training on the first traffic signal control model using the preset demonstration data to obtain the second traffic signal control model, the method further includes: Traffic signal control simulation is performed based on the preset Webster algorithm to generate the preset demonstration data; wherein, the preset demonstration data includes current state data, action data, reward data and next state data.

[0008] In some embodiments, the step of pre-training the first traffic signal control model using preset demonstration data to obtain a second traffic signal control model includes: Extract several demonstration sample data from the preset demonstration data; The demonstration sample data is input into the first traffic signal control model for model training, and then the first loss data is calculated through a preset composite loss function; wherein, the preset composite loss function is constructed by the TD loss function, the supervised marginal classification loss function, and the L2 regularization loss function; The network parameters are updated based on the first loss data to obtain the second traffic signal control model.

[0009] In some embodiments, the step of performing hybrid interactive training on the second traffic signal control model based on the preset demonstration data and the model experience data to generate a target traffic signal control model includes: Based on the preset demonstration data and the model experience data, target sample data is obtained by preferential sampling using a preset sampling ratio parameter; The target sample data is input into the second traffic signal control model for model training, and then the second loss data is calculated. The network parameters of the second traffic signal control model are updated using the second loss data to generate the target traffic signal control model.

[0010] In some embodiments, the step of obtaining target sample data by preferential sampling based on the preset demonstration data and the model experience data using a preset sampling ratio parameter includes: Construct an expert buffer; wherein the preset demonstration data is stored in the expert buffer; Construct a self-generated buffer; wherein the model experience data is dynamically stored in the self-generated buffer; The target sample data is obtained by performing mixed priority sampling from the expert buffer and the self-generated buffer according to the preset sampling ratio parameter.

[0011] To achieve the above objectives, another aspect of this application provides a traffic signal control device, the device comprising: The first module is used to construct the first traffic signal control model based on a preset deep reinforcement learning model. The second module is used to pre-train the first traffic signal control model using preset demonstration data to obtain a second traffic signal control model; wherein the preset demonstration data is obtained through traffic simulation demonstration using a preset traffic control algorithm; The third module is used to dynamically acquire traffic environment status data and input it into the second traffic signal control model to generate model experience data. The fourth module is used to perform hybrid interactive training on the second traffic signal control model based on the preset demonstration data and the model experience data, and generate a target traffic signal control model. The fifth module is used to determine the target signal timing strategy through the target traffic signal control model, and then to perform traffic signal control through the target signal timing strategy.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, the electronic device comprising: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0014] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method. The embodiments of this application include at least the following beneficial effects: This application provides a traffic signal control method, device, electronic device, storage medium, and program product. This solution constructs a first traffic signal control model based on a preset deep reinforcement learning model, and pre-trains the first traffic signal control model using preset demonstration data obtained from traffic simulation demonstrations of a preset traffic control algorithm, resulting in a second traffic signal control model. Next, the embodiments of this invention dynamically acquire traffic environment state data and input it into the second traffic signal control model to obtain model experience data. Then, the second traffic signal control model is subjected to hybrid interactive training by combining the preset demonstration data and the model experience data to generate a target traffic signal control model. The target traffic signal control model is then used to determine a target signal timing strategy, and traffic signal control is performed using the target signal timing strategy to achieve reliable traffic signal control. It is easily understood that the embodiments of this invention, by obtaining preset demonstration data through simulation demonstrations of a preset traffic control algorithm for model pre-training, can use the preset demonstration data as prior knowledge for guidance, effectively alleviating the control performance problems in the early stages of model training and effectively improving the model convergence speed. Meanwhile, by combining model experience data generated by the model with prior preset demonstration data to train the second traffic signal control model, the embodiments of the present invention can achieve stable transmission of prior knowledge, effectively alleviate the problem of model performance degradation, and improve the utilization rate of prior knowledge, thereby improving the reliability of deployment in actual traffic scenarios. Attached Figure Description

[0015] Figure 1 This is a flowchart of the traffic signal control method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the traffic signal control framework of BADQN-FD provided in an embodiment of the present invention; Figure 3 This is a network structure diagram of the traffic signal control method that integrates Bayesian reinforcement learning and imitation learning provided in this embodiment of the invention; Figure 4 This is a schematic diagram of traffic signal control steps provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the traffic signal control device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0017] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0018] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0021] Markov Decision Process (MDP) is a core theoretical framework for reinforcement learning, used to describe how agents achieve long-term goals through decision-making in uncertain environments. It combines Markov properties with decision optimization.

[0022] The Webster algorithm is a widely used method for allocating cycle length and green light time in traffic signal control. Its core is to determine the cycle and green light time by minimizing vehicle delays.

[0023] Bayesian Linear Regression (BLR) algorithm: It is a machine learning algorithm that integrates Bayesian statistical theory into traditional linear regression. By introducing prior and posterior distributions, it probabilistically models the model parameters, thereby providing more robust predictions and uncertainty estimates.

[0024] Deep Q-Network (DQN) model: It is a model that combines deep learning and reinforcement learning. It approximates the Q function through neural networks, which can alleviate the curse of dimensionality problem of traditional Q-learning when dealing with high-dimensional state spaces.

[0025] Traffic signal control can be viewed as a typical Markov decision process. Currently, traffic signal control models exhibit extremely poor control performance in the early stages of training (e.g., causing significant queue lengths and vehicle delays), and the time cost required for model parameters to converge to a steady state is extremely high. Furthermore, the utilization rate of prior knowledge is low, severely hindering their deployment in real-world traffic scenarios.

[0026] For example, in related technologies, when using Bayesian deep Q-networks for traffic signal control, a "start from scratch" learning paradigm is usually adopted. For instance, the weights of the Bayesian linear regression layer are initialized with a simple Gaussian distribution without information. Although this mathematical assumption is convenient for computation, it forces the agent to generate a large number of low-quality random samples through interaction with the environment in order to gradually converge this simple Gaussian distribution to a reasonable posterior distribution. When facing the high-dimensional and complex state space of traffic signal control, the agent must go through a long period of random trial and error to accumulate experience. This mechanism leads to two serious consequences: (1) extremely poor initial performance: In the early stages of training, due to the use of a simple Gaussian distribution as prior knowledge, the agent is almost unable to make reasonable signal timing decisions, resulting in extremely high queue lengths and vehicle delays, which is unacceptable in actual traffic systems; (2) slow convergence speed: Due to the lack of prior knowledge guidance, the convergence process of the Bayesian posterior distribution is extremely slow, and the computational resources are consumed in huge quantities. In addition, simply introducing expert data into reinforcement learning training (e.g., simply putting it into the experience pool) will face two deep-seated algorithmic challenges: (1) Risk of overestimating the value of unobserved actions: Expert demonstration data is not data observed by the RL agent itself. It usually only covers a small part of the state space (i.e., the path preferred by the expert), and it may not select all possible actions (unobserved actions). Although the agent selects the action corresponding to the maximum Q value, it cannot guarantee that the pre-training results are all required. If the network is pre-trained only through Q-learning loss, the agent may select the maximum Q value of the next state (resulting in an unreasonable overestimation of Q value). However, the network is likely to be updated to unobserved state-action pairs, which will propagate these values ​​throughout the Q function, causing the Bellman equation to be unsatisfied; (2) Interference and forgetting of new and old knowledge: As the agent interacts with the environment and generates a large amount of self-generated data, the early expert data is easily covered by the replay buffer, causing the agent to gradually "forget" the effective expert rules in the later training, resulting in performance degradation.

[0027] In view of this, embodiments of this application provide a traffic signal control method, device, electronic device, storage medium, and program product. This solution constructs a first traffic signal control model based on a preset deep reinforcement learning model, and pre-trains the first traffic signal control model using preset demonstration data obtained from traffic simulation demonstrations based on a preset traffic control algorithm, resulting in a second traffic signal control model. Next, embodiments of this invention dynamically acquire traffic environment state data and input it into the second traffic signal control model to obtain model experience data. Then, the second traffic signal control model is subjected to hybrid interactive training by combining the preset demonstration data and the model experience data to generate a target traffic signal control model. The target traffic signal control model is then used to determine a target signal timing strategy, and traffic signal control is performed using the target signal timing strategy, achieving reliable traffic signal control. This alleviates control performance problems in the early stages of model training, effectively improves model convergence speed and prior knowledge utilization, and thus improves the reliability of deployment in real traffic scenarios.

[0028] The traffic signal control method provided in this application relates to the field of traffic control technology. The traffic signal control method provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the traffic signal control method, but is not limited to the above forms.

[0029] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0030] Figure 1 This is an optional flowchart of the traffic signal control method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S110 to S150.

[0031] Step S110: Construct the first traffic signal control model based on the preset deep reinforcement learning model.

[0032] Step S120: The first traffic signal control model is pre-trained using preset demonstration data to obtain the second traffic signal control model. The preset demonstration data is obtained through traffic simulation demonstration using a preset traffic control algorithm.

[0033] Step S130: Dynamically acquire traffic environment status data and input it into the second traffic signal control model to generate model experience data.

[0034] Step S140: Perform hybrid interactive training on the second traffic signal control model based on preset demonstration data and model experience data to generate the target traffic signal control model.

[0035] Step S150: Determine the target signal timing strategy through the target traffic signal control model, and then perform traffic signal control through the target signal timing strategy.

[0036] In this specific embodiment, the present invention first constructs a first traffic signal control model based on a preset deep reinforcement learning model, and pre-trains the first traffic signal control model using preset demonstration data to obtain a second traffic signal control model. Specifically, the preset deep reinforcement learning model in this embodiment includes a deep Q-learning network model. The present invention first constructs an initial traffic signal control model based on the deep reinforcement learning model, such as defining model states, actions, and reward functions. Simultaneously, since there is no relevant expert demonstration data as prior knowledge in the field of traffic signal control, the present invention pre-simulates traffic using a preset traffic control algorithm to generate preset demonstration data. The preset traffic control algorithm refers to classic traffic signal control algorithms, such as the preset Webster algorithm, TRRL algorithm, ARRB algorithm, etc. Accordingly, the present invention collects relevant preset demonstration data by running the preset traffic control algorithm in a simulation environment, and then pre-trains the first traffic signal control model using the preset demonstration data. This allows the model to be quickly initialized and have good initial performance before interacting with the environment, alleviating control performance problems in the early stages of model training and effectively improving the model convergence speed. Next, this embodiment of the invention dynamically acquires traffic environment state data and inputs it into the pre-trained second traffic signal control model to generate model experience data. Then, it combines the preset demonstration data and the model experience data to perform hybrid interactive training on the second traffic signal control model, generating the target traffic signal control model. Specifically, in this embodiment, traffic environment state data refers to the current traffic state information. Correspondingly, after completing the pre-training of the first traffic signal control model, this embodiment interacts with the environment using the obtained second traffic signal control model, outputting corresponding execution action data based on the acquired traffic environment state data, thereby obtaining model experience data. Then, to ensure that expert knowledge, i.e., the preset demonstration data, persists and effectively guides the entire model training and network parameter update cycle, this embodiment combines the preset demonstration data and the model experience data to perform hybrid interactive training on the second traffic signal control model, thereby obtaining the target traffic signal control model. This achieves robust transfer of expert knowledge and alleviates the "catastrophic forgetting" problem, improves the utilization rate of prior knowledge, reduces performance degradation, and effectively improves the reliability and efficiency of model deployment in real traffic scenarios. Finally, this embodiment of the invention determines a target signal timing strategy through a target traffic signal control model, thereby controlling traffic signals using the target signal timing strategy. Specifically, in this embodiment, the target signal timing strategy refers to the traffic signal control strategy. Accordingly, this embodiment interacts with the environment through the target traffic signal control model, such as by inputting relevant environmental state information, to generate the target signal timing strategy, and then controls the traffic signals using the target signal timing strategy, thus achieving reliable traffic signal control.

[0037] In some embodiments of the present invention, a first traffic signal control model is constructed based on a preset deep reinforcement learning model, including but not limited to the following steps: A traffic signal control architecture is constructed based on a Markov decision model. This architecture includes state-space parameters, action-space parameters, state transition probability parameters, reward parameters, and reward discount factors.

[0038] The first traffic signal control model is constructed based on the traffic signal control architecture and a pre-defined deep Q-learning network model. The pre-defined deep Q-learning network model is obtained by optimizing the dual deep Q-network model using a Bayesian linear regression algorithm.

[0039] In this specific embodiment, the present invention first constructs a traffic signal control architecture based on a Markov decision model. Specifically, in this embodiment, the traffic signal control problem is treated as a typical Markov Decision Process (MDP), and a traffic signal control architecture is constructed as follows: Figure 2 As shown in the diagram, in this traffic signal control architecture, the agent (i.e., the signal control system) observes the current state of the intersection (such as vehicle information), then selects the corresponding action (i.e., traffic light phase), and evaluates the effectiveness of its action based on the rewards from environmental feedback (such as queue length). Accordingly, through continuous interaction and maximizing rewards, the agent can progressively optimize the control strategy. The traffic signal control architecture constructed in this embodiment includes state space parameters, action space parameters, state transition probability parameters, reward parameters, and reward discount factors, i.e., a five-tuple... Composition. Specifically, in the embodiments of the present invention... Representing the state space, in this embodiment of the invention, the state of the agent at time t is defined as follows: Lane density vector Combined with current signal phase information To represent state In this embodiment of the invention, using the intersection stop line as a reference, a detection zone is divided within a preset distance, such as 100m, upstream of each lane (straight, left turn, right turn) of each approach lane (east, south, west, north), and the vehicle density is calculated to construct a lane density vector. Accordingly, the lane density vector This information can be obtained through an induction loop detector (ILD) near the intersection. To simplify implementation, this embodiment of the invention uses a lane area detector in a SUMO system to collect state information, calculated using the following formula:

[0040] Where, in the formula for Lane density of the approach lane, for The current number of vehicles in the entrance lane. The sensing range of the intersection, It is the sum of the vehicle's length and the minimum distance between vehicles.

[0041] In addition, in the embodiments of the present invention This represents the action space, controlled by a non-fixed phase switching sequence method. The discrete phase set is represented as North-South Straight (NSG), North-South Left Turn (NSLG), East-West Straight (EWG), and East-West Left Turn (EWLG), i.e. Each phase has a fixed duration, such as 20 seconds, and a preset duration (such as 2 seconds) of yellow light is inserted between different phases. Meanwhile, in this embodiment of the invention... This represents the state transition probability, that is, the probability of transitioning from the current environmental state. Take a certain action Then transition to the next state The probability of this is not described in model-free methods. Furthermore, in the embodiments of this invention... The reward represents the feedback received by agent i from the environment after performing phase action A in the current state S. In this embodiment of the invention, the reward is related to the cumulative waiting time of the vehicle and is defined as follows:

[0042] Where, in the formula This represents the predicted cumulative waiting time for all vehicles after action j is taken. Given the current actual cumulative waiting time, optimizing the reward function means minimizing the cumulative waiting time.

[0043] Accordingly, in the embodiments of the present invention This represents the reward discount factor, which is between... A constant is used to scale the reward reward obtained at future time steps. For example, in this embodiment of the invention, the reward discount factor is set to 0.98.

[0044] Furthermore, in this embodiment of the invention, a first traffic signal control model is constructed by combining a traffic signal control architecture and a preset deep Q-learning network model. Specifically, this embodiment optimizes the double deep Q-network model using a Bayesian linear regression algorithm to obtain a preset deep Q-learning network model. In this embodiment, the preset deep Q-learning network model is built upon a double deep Q-network (DQN) model architecture, and a Bayesian linear regression (BLR) algorithm is introduced at the output layer to construct the preset deep Q-learning network model, which is then combined with the traffic signal architecture to construct the initial traffic signal control model. In this embodiment, the Q-function is approximated as a linear transformation of the neural network output features using the BLR algorithm. Then, during network training, the posterior distribution of the action is updated, which in turn updates the posterior distribution of the Q function. Thompson sampling is used to select the optimal signal phase in the posterior distribution of the Q function. This architecture provides the agent with the ability to measure the uncertainty of the Q value.

[0045] In some embodiments of the present invention, before performing model pre-training on the first traffic signal control model using preset demonstration data to obtain the second traffic signal control model, the traffic signal control method provided by the embodiments of the present invention further includes, but is not limited to, the following steps: Traffic signal control simulation is performed based on a pre-defined Webster algorithm to generate pre-defined demonstration data. This pre-defined demonstration data includes current state data, action data, reward data, and state data for the next time step.

[0046] In this specific embodiment, the present invention performs a traffic signal control simulation demonstration based on a preset Webster algorithm to obtain preset demonstration data. Specifically, since there are no relevant experts in the field of traffic signal control to provide demonstration data, the present invention obtains demonstration data from classic traffic signal control algorithms. Specifically, the present invention selects the preset Webster algorithm as the expert, and collects the preset demonstration data generated by it by running the expert strategy in a simulation environment. This data includes current state data, action data, reward data, and next-time state data. And use it as demonstration data for pre-training. This guides the gradient updates of the BADQN-FD model (a pre-defined deep Q-learning network model), enabling the traffic signal control model to be quickly initialized and have good initial performance before interacting with the environment.

[0047] It should be noted that ablation experiments on the model revealed that simple timing control, longest queue priority control, and Max Pressure as the expert strategy all failed to achieve optimal model convergence. Therefore, this embodiment of the invention selects a hybrid Webster algorithm and Max Pressure algorithm as the expert strategy (hereinafter referred to as the Webster method, i.e., the preset Webster algorithm). First, the saturation flow at the intersection is obtained through the SUMO detector. Arrival flow rate compared to the current time period (60s) The pressure of each signal phase at the intersection is calculated by substituting the difference in the number of vehicles upstream and downstream in the standard Max Pressure with the difference in upstream and downstream traffic flow, and the phase with the highest pressure is selected as the next phase to be activated. This method is consistent with the reward function related to waiting time set in the embodiments of the present invention. Therefore, the Webster method is used to collect demonstration data, which helps BADQN quickly learn reasonable prior knowledge.

[0048] In some embodiments of the present invention, a second traffic signal control model is obtained by pre-training a first traffic signal control model using preset demonstration data, including but not limited to the following steps: Extract several demonstration sample data from the preset demonstration data.

[0049] The demonstration sample data is input into the first traffic signal control model for model training, and then the first loss data is calculated through a preset composite loss function. The preset composite loss function is constructed using the TD loss function, a supervised marginal classification loss function, and an L2 regularization loss function.

[0050] The network parameters are updated based on the first loss data to obtain the second traffic signal control model.

[0051] In this specific embodiment, the present invention first extracts several demonstration sample data from preset demonstration data and inputs the demonstration sample data into a first traffic signal control model for model training. Then, a first loss data is calculated using a preset composite loss function, and network parameters are updated based on the first loss data to generate a second traffic signal control model. Specifically, unlike related technologies that directly use randomly initialized network parameters (i.e., following a simple Gaussian distribution) to begin interaction, this embodiment first uses an expert demonstration dataset. This involves using pre-set demonstration data to perform supervised training on the network. From a Bayesian inference perspective, this process essentially involves an offline update of the prior distribution of the Bayesian Linear Regression (BLR) layer using expert data. After pre-training, the network weights of the BLR... The distribution is no longer the initial simple Gaussian distribution, but has become a distribution carrying expert policy information. In the subsequent interactive training phase, the agent will perform Thompson sampling based on this optimized distribution, thereby achieving efficient directional exploration. Accordingly, the real purpose of the pre-training phase in this embodiment of the invention is to allow the agent to imitate the demonstrator using the Q function before it begins to interact with the environment. In the pre-training phase, the agent extracts a small batch of samples (demonstration sample data) from the preset demonstration data and updates the network parameters by optimizing three loss functions: TD loss. The algorithm employs a functional framework, including a supervised marginal classification loss function and an L2 regularized loss function. The supervised loss enables the algorithm to learn and imitate the demonstrator, while the TD loss allows the algorithm to learn a self-consistent Q-function, enabling further learning. Accordingly, after pre-training, the agent begins to interact with the environment using its learned policy, updating the network with a mixture of demonstrator data and self-generated data. The TD loss... The formula for calculating the function is shown below:

[0052] Accordingly, supervised loss is crucial for network pre-training. Since the pre-demonstration data only contains a portion of the state space and may not select all possible actions, many state-action pairs are never used. If the network is updated using only the Q-learned TD loss, the agent might choose the maximum Q-value for the next state. The network will then be updated to the maximum Q-value of unobserved state-action pairs, which may be unreasonable, and the network will propagate these values ​​throughout the entire Q-function. To limit the Q-values ​​of unobserved state-action pairs to a reasonable range, this embodiment employs a supervised marginal classification loss, as shown in the following equation:

[0053] Where, in the formula For a marginal function, when hour ,otherwise . For actions in the demonstration data, this loss can limit the Q-value of actions in the demonstration data to a higher range than other actions, and limit the value of unobserved actions to a reasonable range. The loss defines a boundary for undiscovered actions, enhancing the plausibility of the demonstration trajectory. If only this supervised loss is used for pre-training, it does not impose any constraints on the values ​​between consecutive states, and the Q-network will not satisfy the Bellman equation, requiring online policy improvement through TD learning. Furthermore, to prevent the network from overfitting on a relatively small demonstration dataset, this embodiment adds L2 regularization loss to the network's weights and biases. Accordingly, the total loss used for pre-training the network in this embodiment of the invention is given by the formula... The sum of all three losses, By controlling the weights among the losses, supervised loss is not used when interacting with the environment after pre-training, that is... Finally, in this embodiment of the invention, the network parameters of the first traffic signal control model are updated based on the calculated first loss data to obtain the second traffic signal control model.

[0054] In some embodiments of the present invention, a second traffic signal control model is subjected to hybrid interactive training based on preset demonstration data and model experience data to generate a target traffic signal control model, including but not limited to the following steps: Based on preset demonstration data and model experience data, target sample data is obtained by preferential sampling using preset sampling ratio parameters.

[0055] The target sample data is input into the second traffic signal control model for model training, and then the second loss data is calculated.

[0056] The network parameters of the second traffic signal control model are updated using the second loss data to generate the target traffic signal control model.

[0057] In this specific embodiment, the present invention first performs priority sampling based on preset demonstration data and model experience data using a preset sampling ratio parameter. Then, the determined target sample data is input into a second traffic signal control model for model training to calculate second loss data. The network parameters of the model are then updated based on the second loss data to generate the target traffic signal control model. Specifically, in this embodiment, the preset sampling ratio parameter refers to the sampling ratio of the two types of training data, such as setting two separate sampling ratios. and This invention controls the priority sampling of two types of training data to obtain target sample data. By combining preset demonstration data and model experience data for training and updating the second traffic signal control model, this embodiment of the invention ensures the continuous existence and effective guidance of expert knowledge throughout the training cycle, fundamentally alleviating the problem of catastrophic forgetting. Correspondingly, this embodiment of the invention uses a total loss formula... Calculate the second loss data, without using supervised loss, that is... Accordingly, in this embodiment of the invention, the network parameters of the model are updated based on the calculated second loss data to obtain the target traffic signal control model.

[0058] In some embodiments of the present invention, target sample data is obtained by preferential sampling based on preset demonstration data and model experience data using a preset sampling ratio parameter, including but not limited to the following steps: Construct an expert buffer. Preset demo data is stored in the expert buffer.

[0059] Construct a self-generated buffer. Model experience data is dynamically stored in this self-generated buffer.

[0060] The target sample data is obtained by performing mixed priority sampling from the expert buffer and the self-generated buffer according to the preset sampling ratio parameter.

[0061] In this specific embodiment, the present invention constructs an expert buffer and a self-generated buffer respectively, to perform mixed priority sampling from the expert buffer and the self-generated buffer according to a preset sampling ratio parameter, thereby obtaining target sample data. Specifically, the embodiment of the present invention constructs a dual-buffer mixed sampling mechanism by setting an expert buffer and a self-generated buffer. Wherein, as... Figure 3 As shown, this embodiment of the invention establishes an expert buffer for permanently storing expert data and a self-generated buffer that is updated cyclically. It also enforces hybrid sampling during each parameter update. This mechanism ensures the continuous existence and effective guidance of expert knowledge throughout the training cycle from a physical storage perspective, fundamentally mitigating catastrophic forgetting. Accordingly, after pre-training, to ensure the continuous existence and effective guidance of expert knowledge throughout the training cycle, this embodiment employs a dual-buffer hybrid sampling mechanism to train the traffic signal control model and update network parameters. Specifically, this embodiment stores the model-generated experience data in the self-generated buffer. When the self-generated buffer is full, the old data is overwritten, achieving cyclic updates of the self-generated buffer. Correspondingly, to allow the agent to sample both demonstration data and self-generated data simultaneously, this embodiment sets a preset sampling ratio parameter, such as setting two separate sampling ratios. and This controls the priority sampling of the two types of training data, so as to extract the corresponding sample data from the expert buffer and the self-generated buffer to obtain the target sample data.

[0062] The following section provides a detailed introduction and explanation of the solutions in this embodiment of the invention, using specific traffic signal control scenarios as examples: For example, such as Figure 4 As shown, Figure 4This is a schematic diagram of traffic signal control steps provided in an embodiment of the present invention. Specifically, the embodiment first constructs a traffic signal control model based on deep reinforcement learning, including defining the model's state, actions, and reward function. Simultaneously, the embodiment obtains demonstration data from traditional traffic signal control algorithms. For example, the embodiment uses the Webster method as the expert, running the expert policy in a simulation environment to collect demonstration data as pre-training guidance data. Next, the embodiment first extracts sample data from the demonstration data, inputs it into a pre-constructed Bayesian deep Q-learning network model for model pre-training, and calculates a composite loss function containing supervised marginal classification loss and updates the network. After pre-training is completed, the embodiment executes a hybrid interactive training phase, where the model begins to interact with the environment and obtain its current state from the environment. Thompson sampling optimal action In performing the action Then obtain rewards from the environment. and the state at the next moment And update the posterior distribution of the action. Then, embodiments of the present invention store empirical data. to experience replay buffer This refers to a self-generated buffer. Furthermore, embodiments of the present invention utilize an experience replay buffer... In addition, a batch of data is preferentially sampled from the expert data experience replay pool (expert buffer), and the corresponding loss function is calculated and the network is updated. Accordingly, the simulation is stopped and the model parameters are saved after the training reaches the maximum number of rounds, resulting in the target traffic signal control model. Finally, this embodiment of the invention generates a signal timing strategy through the target traffic signal control model for traffic signal control.

[0063] It should be noted that the embodiments of the present invention effectively alleviate the "cold start" problem, greatly improving initial control performance and convergence speed. In related Bayesian DQN methods, network weights are typically initialized with an uninformative simple Gaussian distribution, resulting in the agent being completely unaware of the environment in the early stages of training, requiring a lengthy and inefficient random trial-and-error phase. However, the embodiments of the present invention, by introducing Webster expert demonstration data for pre-training, transform the prior distribution of the Bayesian linear regression layer from an uninformative simple Gaussian distribution into an informative prior carrying domain knowledge. This allows the agent's Q-value distribution to approach a mature expert policy even before interacting with the environment. Related experiments show that the embodiments of the present invention can skip the inefficient random exploration phase, significantly reducing the number of training rounds required for convergence. Furthermore, the embodiments of the present invention effectively alleviate the "value overestimation" problem in imitation learning, ensuring the safety of exploration. Specifically, if expert data is directly introduced into conventional DQN training (only optimizing TD error), the network often generates unreasonable overestimations of the value of actions not selected by the expert (non-observed actions), leading to policy divergence. The embodiments of the present invention, by constructing a specific supervised marginal classification loss function... This loss function forcibly introduces a "safety margin" in the optimization objective, requiring that the Q value of the expert action must be higher than the Q value of any other action by a preset margin. Mathematically, this sets a reasonable upper bound on the value of unobserved actions, effectively constraining the divergence of Q values ​​and ensuring the robustness and safety of the agent when using expert knowledge.

[0064] Meanwhile, this invention effectively reduces the problem of "catastrophic forgetting" and achieves a dynamic balance between expert knowledge and new knowledge. For example, related technologies typically use a single experience replay pool. As training progresses, early effective experience (or expert experience) is constantly overwritten by newly generated data, causing the model to gradually forget prior rules in later stages, potentially leading to performance degradation. This invention innovatively constructs a dual-buffer hybrid sampling mechanism by setting up a dedicated buffer for permanently storing expert data and a self-generated data buffer that is updated cyclically, and by forcibly performing hybrid sampling every time parameters are updated. This isolation and fusion of physical storage and sampling strategies fundamentally ensures the continuous guiding role of expert knowledge throughout its entire lifecycle, achieving long-term stable control. Correspondingly, this invention also improves the exploration efficiency in complex state spaces. This invention retains the advantage of Bayesian DQN in using posterior distribution uncertainty for Thompson sampling. Combined with the aforementioned pre-training mechanism, the agent's exploration behavior is no longer a blind random walk, but a "directed exploration" based on expert knowledge confidence. This mechanism significantly reduces exploration costs while ensuring escape from local optima.

[0065] Please see Figure 5This application also provides a traffic signal control device that can implement the above-described method. The device includes: The first module 210 is used to construct a first traffic signal control model based on a preset deep reinforcement learning model.

[0066] The second module 220 is used to pre-train the first traffic signal control model using preset demonstration data to obtain the second traffic signal control model. The preset demonstration data is obtained through traffic simulation demonstrations using a preset traffic control algorithm.

[0067] The third module 230 is used to dynamically acquire traffic environment status data and input it into the second traffic signal control model to generate model experience data.

[0068] The fourth module 240 is used to perform hybrid interactive training on the second traffic signal control model based on preset demonstration data and model experience data, and generate the target traffic signal control model.

[0069] The fifth module 250 is used to determine the target signal timing strategy through the target traffic signal control model, and then to carry out traffic signal control through the target signal timing strategy.

[0070] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0071] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0072] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0073] Please see Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 310 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 320 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 320 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 320 and is called and executed by the processor 310 using the methods described in the embodiments of this application. Input / output interface 330 is used to realize information input and output; The communication interface 340 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 350 transmits information between various components of the device (e.g., processor 310, memory 320, input / output interface 330, and communication interface 340); The processor 310, memory 320, input / output interface 330 and communication interface 340 are connected to each other within the device via bus 350.

[0074] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0075] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0076] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0077] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0078] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0079] The traffic signal control method, apparatus, electronic device, storage medium, and program product provided in this application embodiment are pre-trained by introducing Webster expert demonstration data. This invention embodiment utilizes mature traffic control rules (such as the Webster method) to train and collect its demonstration data. The demonstration data was used to pre-train the Bayesian network. This allows the agent's posterior distribution to be initialized to near-expert-level policy through supervised learning before formal interaction with the environment, thus skipping the inefficient random exploration phase and achieving a "warm start." Furthermore, this embodiment constructs a specific supervised marginal classification loss function that mandates the Q-value of expert actions must be higher than other actions by a preset margin, thereby setting a reasonable "upper bound" on the value of unobserved actions. This mathematically constrains the divergence of Q-values, ensuring the accuracy of imitation learning. Simultaneously, a dual-buffered mixed sampling mechanism is implemented. By establishing a permanent "expert buffer" for storing expert data and a cyclically updated "self-generated buffer," and enforcing mixed sampling with each parameter update, the physical storage ensures the continuous existence and effective guidance of expert knowledge throughout the training cycle, fundamentally alleviating the catastrophic forgetting problem.

[0080] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0081] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0082] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0084] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0085] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0087] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0089] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A traffic signal control method, characterized in that, The method includes the following steps: The first traffic signal control model is constructed based on a pre-set deep reinforcement learning model; The first traffic signal control model is pre-trained using preset demonstration data to obtain a second traffic signal control model; wherein, the preset demonstration data is obtained through traffic simulation demonstration using a preset traffic control algorithm; Dynamically acquire traffic environment status data and input it into the second traffic signal control model to generate model experience data; The second traffic signal control model is trained using the preset demonstration data and the model experience data to generate the target traffic signal control model. The target traffic signal timing strategy is determined by the target traffic signal control model, and then traffic signal control is performed using the target traffic signal timing strategy.

2. The method according to claim 1, characterized in that, The construction of the first traffic signal control model based on the preset deep reinforcement learning model includes: A traffic signal control architecture is constructed based on a Markov decision model; wherein, the traffic signal control architecture includes state space parameters, action space parameters, state transition probability parameters, reward parameters, and reward discount factors; The first traffic signal control model is constructed based on the traffic signal control architecture combined with a preset deep Q-learning network model; wherein, the preset deep Q-learning network model is obtained by optimizing the dual deep Q-network model using a Bayesian linear regression algorithm.

3. The method according to claim 1, characterized in that, Before performing model pre-training on the first traffic signal control model using the preset demonstration data to obtain the second traffic signal control model, the method further includes: Traffic signal control simulation is performed based on the preset Webster algorithm to generate the preset demonstration data; wherein, the preset demonstration data includes current state data, action data, reward data and next state data.

4. The method according to claim 1, characterized in that, The step of pre-training the first traffic signal control model using preset demonstration data to obtain the second traffic signal control model includes: Extract several demonstration sample data from the preset demonstration data; The demonstration sample data is input into the first traffic signal control model for model training, and then the first loss data is calculated through a preset composite loss function; wherein, the preset composite loss function is constructed by the TD loss function, the supervised marginal classification loss function, and the L2 regularization loss function; The network parameters are updated based on the first loss data to obtain the second traffic signal control model.

5. The method according to claim 1, characterized in that, The step of performing hybrid interactive training on the second traffic signal control model based on the preset demonstration data and the model experience data to generate the target traffic signal control model includes: Based on the preset demonstration data and the model experience data, target sample data is obtained by preferential sampling using a preset sampling ratio parameter; The target sample data is input into the second traffic signal control model for model training, and then the second loss data is calculated. The network parameters of the second traffic signal control model are updated using the second loss data to generate the target traffic signal control model.

6. The method according to claim 5, characterized in that, The step of obtaining target sample data by prioritizing sampling based on the preset demonstration data and the model experience data using a preset sampling ratio parameter includes: Construct an expert buffer; wherein the preset demonstration data is stored in the expert buffer; Construct a self-generated buffer; wherein the model experience data is dynamically stored in the self-generated buffer; The target sample data is obtained by performing mixed priority sampling from the expert buffer and the self-generated buffer according to the preset sampling ratio parameter.

7. A traffic signal control device, characterized in that, The device includes: The first module is used to construct the first traffic signal control model based on a preset deep reinforcement learning model. The second module is used to pre-train the first traffic signal control model using preset demonstration data to obtain a second traffic signal control model; wherein the preset demonstration data is obtained through traffic simulation demonstration using a preset traffic control algorithm; The third module is used to dynamically acquire traffic environment status data and input it into the second traffic signal control model to generate model experience data. The fourth module is used to perform hybrid interactive training on the second traffic signal control model based on the preset demonstration data and the model experience data, and generate a target traffic signal control model. The fifth module is used to determine the target signal timing strategy through the target traffic signal control model, and then to perform traffic signal control through the target signal timing strategy.

8. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images