A large language model control traffic signal method for different types of intersections

Abstract

The application belongs to the technical field of traffic control systems, and in particular relates to a large language model control traffic signal method for different types of intersections, comprising: obtaining a historical intersection environment set; and performing unified state and action representation to form a first matrix; training a reinforcement learning model of proximal policy optimization based on randomly sampled states in the first matrix, and forming a traffic signal control sequence trajectory based on the first matrix; extracting time sequence information contained in state data of the traffic signal control sequence trajectory through a convolutional neural network to form latent space features; processing the latent space features, actions and rewards of the traffic signal control sequence trajectory through a linear layer respectively to obtain input features; and forming prediction features after operation of a fine-tuned large language model that has been trained; the fine-tuned large language model is updated and optimized through an encoder of a loss function and a double simulation metric learning to be trained, and the prediction features include predicted states, predicted actions and predicted rewards.

Description

Technical Field

[0001] This application belongs to the field of traffic control system technology, and specifically relates to a method for controlling traffic signals using a large language model for different types of intersections. Background Technology

[0002] Traffic signal control is a challenging task in the field of intelligent transportation, and its effectiveness directly impacts intersection communication safety and road network efficiency. With increasing traffic congestion across various regions, optimizing traffic signal control schemes has become a widely researched area. Currently, traditional traffic signal control methods often rely on fixed traffic signal timing schemes or manually designed traffic signal control optimization models. However, numerous studies have shown that this optimization approach, dependent on prior knowledge, struggles to adapt well to the complex spatiotemporal variations of road networks, resulting in limited improvements in control effectiveness.

[0003] In recent years, with the further development of technology, traffic signal control methods based on reinforcement learning have provided new solutions and have been widely favored by researchers due to their excellent adaptability to real-time road conditions and good control effects. The main reason for their superior performance is that reinforcement learning starts from dynamic traffic flow data, uses indicators related to traffic efficiency as reward functions, and uses these as optimization objectives to achieve efficient decision-making, thereby enabling real-time interactive feedback with the environment.

[0004] While these methods have achieved state-of-the-art performance in real and synthetic road networks, they are only suitable for specific intersections and difficult to extend to a wider range of intersection configurations. Therefore, a few studies have attempted to train more general reinforcement learning models. However, reinforcement learning-based methods face a series of problems, including low sample efficiency, data shift distribution, and limited generalization ability (especially under high traffic conditions). Recently, emerging large language models have influenced various fields due to their knowledge transfer capabilities and zero-shot learning capabilities, and have been fine-tuned to adapt to different tasks, such as autonomous driving, traffic prediction, and traffic signal control. For traffic signal control, they typically focus on long-tail scenarios such as traffic accidents and priority lanes. Some studies focus on directly utilizing large language models to generate human knowledge for solution generation and recommendation. Summary of the Invention

[0005] This application is proposed based on the aforementioned needs of the prior art. The technical problem to be solved by this application is to provide a large language model control method for traffic signals for different types of intersections to improve the efficiency and accuracy of traffic signal control.

[0006] To address the above problems, the technical solution provided in this application includes:

[0007] A method for controlling traffic signals using a large language model for different types of intersections is provided, comprising: acquiring a historical traffic intersection environment set and forming a first matrix by uniformly representing state and action; training a reinforcement learning model for near-end policy optimization based on randomly sampled states in the first matrix; inputting the entire first matrix into the trained reinforcement learning model to form expert experience data; forming a traffic signal control sequence trajectory based on the expert experience data; extracting time-series information contained in the state data of the traffic signal control sequence trajectory through a convolutional neural network to form latent space features; processing the latent space features, actions, and rewards of the traffic signal control sequence trajectory through linear layers to obtain input features; forming prediction features after the input features are processed by a trained fine-tuned large language model; the fine-tuned large language model is trained by updating and optimizing an encoder using a loss function and dual analog metric learning, and the prediction features include predicted state, predicted action, and predicted reward; acquiring real-time intersection data; and obtaining corresponding real-time prediction data based on the real-time intersection data through the fine-tuned large language model.

[0008] Preferably, the reinforcement learning model for near-end policy optimization based on randomly sampled states in the first matrix includes a policy learning method represented as follows: in, For the expected return of the strategy, Let π be the mathematical expectation, min() be the minimum value function, and π be the expected value. θ (a t |S t This is a new strategy. For the old strategy, A t The reward r is determined based on the average waiting time per vehicle at time t. t The resulting advantage function is clip(), which is used to limit the range of values, and ∈ is the clipping parameter.

[0009] Preferably, the step of extracting time-series information from the state data of traffic signal control sequence trajectories using a convolutional neural network to form latent spatial features, denoted as Z. t =CNN(S′) t ), where Z t For the corresponding latent space features, S′ t Let t be the state data at time t, and CNN(.) denotes a convolutional neural network.

[0010] Preferably, the step of processing latent space features, traffic signal control sequence trajectory actions, and rewards through linear layers to obtain input features includes: obtaining labeled embeddings of state, action, and reward through a state linear layer, an action linear layer, and a reward linear layer, and then embedding the time step linearly into the pos. tAs an absolute position, combined with the embedding of each of the above tags, the input representation of each tag is constructed as follows: in, For the encoded state features, For encoded action features, For the encoded reward feature, a′ t Let r′ be the motion data at time t in the traffic signal control sequence trajectory. t For the reward data at time t in the traffic signal control sequence trajectory, Linear(.) represents the linear layer of the neural network; and... and Stacked together, they form the input features.

[0011] Preferably, the trained fine-tuned large language model includes initializing a learnable low-rank matrix, wherein the learnable matrix includes a first learnable low-rank matrix and a second learnable low-rank matrix, wherein the elements in the initialized first learnable low-rank matrix satisfy a Gaussian distribution, as shown below: Where a ij σ is the element in the i-th row and j-th column of the first learnable low-rank matrix. 2 Let the variance be denoted as b; the elements in the initialized second learnable low-rank matrix are all 0, represented as: b ij =0, where b ij Let i be the element in the i-th row and j-th column of the second learnable low-rank matrix.

[0012] Preferably, the trained fine-tuned large language model includes a multi-layer Transformer module, where the input features are processed through a single Transformer module, and the output is: Where W is the weight, The input features are α, which is a hyperparameter. It is mainly used to adjust the proportion of the LoRA weight matrix in the final model parameters; the output of the L-layer Transformer module is: in, The final features output by the large language model after passing through L layers of Transformer modules. For input features, LLM(.) represents a GPT-2 model with L layers of Transformer modules, and LoRA(.) represents the fine-tuning process after LoRA.

[0013] Preferably, the trained fine-tuned large language model includes the final features being sequentially input into an action prediction network, a state prediction network, and a reward prediction network to obtain prediction features including predicted state, predicted action, and predicted reward. The action prediction network and reward network both include linear layers, and the state prediction network includes linear layers and deconvolutional layers. The prediction features are represented as follows: in, To predict actions, To predict the state, To predict rewards, Linear(.) is a linear function, Tanh(.) is a hyperbolic tangent function, and Deconv(.) is a deconvolution function.

[0014] Preferably, the loss function is expressed as: Where, a′ t This refers to the action data at time t in the traffic signal control sequence trajectory. For the corresponding predicted action, r′ t This refers to the reward data at time t in the traffic signal control sequence trajectory. For the corresponding reward action, S′ t This refers to the state data at time t in the traffic signal control sequence trajectory. This corresponds to the state action.

[0015] Preferably, the similarity between two sets of data is calculated by using dual-simulation to measure different data of the same metric, and this is used to train the encoder. The encoder includes a state linear layer, an action linear layer, and a reward linear layer. When the metric is state data, the behavioral similarity between two sets of state data is represented as follows: Wherein d(s) (1) ,s (2) ) represents state s (1) and s (2) The behavioral similarity between these two states, r(.) represents the reward value corresponding to the current state, and W2 represents the second-order Wasserstein distance; The Gaussian distribution represents the probabilistic dynamics model; the expectation relation obtained by training the encoder is expressed as: in, For the encoder; by using multiple state data pairs, the goal is to minimize the distance between the two analog metric and the latent space. The learning objectives are expressed as: in, Indicates encoder The stopping gradient.

[0016] Compared with existing technologies, this application proposes a general traffic signal control method enhanced by a large language model for intersections with different structures, phases, and flow inputs. This method utilizes the prior knowledge of a pre-trained large language model and adapts it to traffic signal control tasks through fine-tuning. The method directly uses the large language model as an agent, efficiently fulfilling control requirements under different intersection environment configurations and directly inferring the optimal traffic signal configuration, thereby improving control robustness and generalization. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This is a flowchart illustrating the steps of a large language model-based traffic signal control method for different types of intersections according to an embodiment of the present invention.

[0019] Figure 2 Diagram showing the definition of intersection attributes;

[0020] Figure 3 This is a schematic diagram of the signal plan corresponding to the intersection;

[0021] Figure 4 This is a partial representation diagram of the first matrix after unifying the historical intersection environment set;

[0022] Figure 5 This is a schematic diagram representing another part of the first matrix after unifying the historical intersection environment set. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the term "connected" should be interpreted broadly. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0025] Throughout the text, the terms “top,” “bottom,” “above,” “below,” and “on top” refer to the relative positions of components of the device, such as the relative positions of the top and bottom substrates within the device. It is understood that the device is multifunctional and independent of its spatial orientation.

[0026] To facilitate understanding of the embodiments of this application, the following will provide further explanation and description with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this application.

[0027] This embodiment provides a method for controlling traffic signals using a large language model for different types of intersections, such as... Figure 1 As shown.

[0028] The large language model-based traffic signal control method for different types of intersections includes:

[0029] Obtain a set of historical traffic intersection environments and represent them in a unified manner to form the first matrix.

[0030] The acquired historical traffic intersection environment set includes traffic structure, phase, and flow pattern.

[0031] like Figure 2 and Figure 3 As shown, the traffic structure includes T-shaped and cross-shaped planar intersections, and corresponding approach lanes.

[0032] The phase refers to a combination of traffic signals. Each phase consists of one or more non-conflicting traffic movements. Taking a four-phase system as an example, phase 1 includes left turns in the east-west direction. When this phase is set to green, vehicles in this phase are allowed to pass, while vehicles in other phases are restricted. The traffic signals are defined according to traffic movements; a green light indicates that the current movement is permitted, and a red light indicates that the current movement is prohibited. In this embodiment, right turns in different directions are all set to green. Traffic movement refers to the entire process of a vehicle entering a lane, crossing the intersection, and leaving the lane. For a four-way intersection, there are left turns, straight ahead, and right turns in each of the four directions (north, south, east, and west), resulting in 12 traffic movements. A lane refers to a road where vehicles travel in a specific direction. In any intersection, lanes can be divided into entry lanes and exit lanes based on entering and leaving the intersection. The signal plan is used to set the phase sequence and phase timing. Taking a four-phase system as an example, a phase plan is typically executed cyclically, i.e., {(p1,t1),(p2,t2),(p3,t3),(p4,t4),(p1,t1),(p2,t2),…}, as shown. Figure 3 As shown, p and t here represent a phase and its corresponding phase duration.

[0033] The aforementioned flow pattern adjusts the flow rate at different approach lanes under the same flow input at the intersection.

[0034] Unify the data in the historical intersection environment set into a preset state coding space, such as Figure 4 and Figure 5 As shown, the space includes east, west, south, and north entrance lanes, and each entrance lane has left-turn, right-turn, and straight-ahead lanes.

[0035] Specifically, traffic data generated by different traffic movements at a given moment is treated as a state matrix, including occupancy rate, one-hot encoding of straight, left-turn, and right-turn traffic, the number of lanes occupied by that traffic movement, whether the current traffic light is green, and whether the next traffic light is green. If an approach lane or direction is missing, it is filled with 0.

[0036] To better understand the dynamic information of the intersection, the multi-step state S t ={s t-K ,…,s t} is defined as the observed variables in the K steps prior to time t (including the current time). Furthermore, to accommodate different phase patterns at the intersection, a safer method is used for phase switching. Specifically, a pre-defined loop scheme executes different phases sequentially. Therefore, action a... t Defined as hold or switch, i.e., two-vector coding, its function is to control moving signals.

[0037] Based on the randomly sampled states in the first matrix, a reinforcement learning model for near-end policy optimization is trained, and the entire first matrix is ​​input into the trained reinforcement learning model to form expert experience data.

[0038] Specifically, this embodiment uses PPO as the reinforcement learning model, which randomly collects states from the first matrix to train a reinforcement learning traffic signal control network capable of making decisions in different intersection environments. The specific policy learning method is as follows:

[0039]

[0040] in, For the expected return of the strategy, Let π be the mathematical expectation, min() be the minimum value function, and π be the expected value. θ (a t |S t This is a new strategy. For the old strategy, A t The reward r is determined based on the average waiting time per vehicle at time t. t The resulting advantage function is clip(), which is used to limit the range of values, and ∈ is the clipping parameter.

[0041] The trained reinforcement learning model, acting as the decision agent for all intersection environments, re-inputs the data from the first matrix into the reinforcement learning model, thereby outputting expert experience data for different intersection environments, including state data, action data, and reward data.

[0042] Traffic signal control sequence trajectories are generated based on expert experience data.

[0043] Several batches of data are randomly selected from expert experience data to form a traffic signal control sequence trajectory. The traffic signal control sequence trajectory includes action data, state data, and reward data, represented as: τ=(S′1,a′1,r′1,S′2,a′2,r′2…), where S′1 represents the state data at the first moment, a′1 represents the action data at the first moment, r′1 represents the reward data at the first moment, S′2 represents the state data at the second moment, a′2 represents the action data at the second moment, and r′2 represents the reward data at the second moment.

[0044] The convolutional neural network is used to extract time-series information from the state data of traffic signal control sequence trajectories, forming latent spatial features.

[0045] Specifically, it is expressed as follows:

[0046] Z t =CNN(S′) t )

[0047] Among them, Z t For the corresponding latent space features, S′ t Let t be the state data at time t, and CNN(.) denotes a convolutional neural network.

[0048] The input features are obtained by processing latent space features, traffic signal control sequence trajectory actions and rewards through linear layers.

[0049] Specifically, a linear layer is used to obtain embedding labels for state, action, and reward based on latent space features, action data, and reward data in the traffic signal control sequence trajectory, respectively. Furthermore, to determine the position of the label in the sequence trajectory, a linear embedding of the time step pos is applied. t As absolute positions, combined with the embedding of each of the above tags, the input representation of each tag is constructed as follows:

[0050]

[0051] in, For the encoded state features, For encoded action features, For the encoded reward feature, a′ t Let r′ be the action data at time t in the trajectory. t Let t represent the reward data at time t in the trajectory, and Linear(.) represent the linear layer of the neural network.

[0052] Stacking the above results and Form the input matrix, represented as

[0053] The input features are processed by a trained fine-tuned large language model to form predicted features. The fine-tuned large language model is trained by updating and optimizing an encoder through loss function and dual analog metric learning. The predicted features include predicted state, predicted action, and predicted reward.

[0054] The weights W of the pre-trained LLM are frozen using LoRA, and learnable low-rank matrices A and B are injected, where Matrix A and B are then initialized. Specifically, matrix A is randomly initialized so that its elements follow a Gaussian distribution; all elements of matrix B are initialized to 0, as follows:

[0055]

[0056] b ij =0

[0057] Therefore, the output of the GPT-2 model after passing through a Transformer module is:

[0058]

[0059] Where h is the output feature. Given an input matrix, α is a hyperparameter. It is mainly used to adjust the proportion of the LoRA weight matrix in the final model parameters.

[0060] The output of the GPT-2 model after passing through the L-layer Transformer module is:

[0061]

[0062] in, The final features output by the GPT-2 model after passing through the L-layer Transformer module are shown. Let LLM(.) be the input matrix, LLM(.) represents a GPT-2 model with L layers of Transformer modules, and LoRA(.) represents the fine-tuning process after LoRA.

[0063] The final features are then fed into the action prediction network, state prediction network, and reward prediction network to obtain the predicted actions. state and rewards

[0064] The action prediction network and reward network are both linear layers, while the state prediction network consists of linear layers and deconvolutional layers. The predicted actions... state and rewards The specific calculation process is as follows:

[0065]

[0066] The action prediction network uses cross-entropy loss as its loss function to optimize the strategy, as follows:

[0067]

[0068] The state prediction network and reward prediction network use mean squared error as their loss function to estimate the state and reward, as shown in the following formulas:

[0069]

[0070] Therefore, the total loss function is as follows:

[0071]

[0072] To train the network parameters mentioned above using the total loss function, a commonly used optimizer, AdamW, is employed to implement gradient descent, thereby updating all network parameters.

[0073] To further eliminate task-irrelevant information and avoid the influence of spurious correlations between actions and states, thereby enhancing the encoder's learning ability for observed states, this invention employs dual analog metric learning to capture robust latent representations. This introduces an additional self-supervised regularization term to learn a robust encoder. The encoder includes a state linear layer, a dynamic linear layer, and a reward linear layer.

[0074] By using dual-simulation to measure different data of the same metric, the similarity between two data points is calculated and used to train the encoder. When the metric is state data, the behavioral similarity between two state data points is represented as:

[0075]

[0076] Wherein d(s) (1) ,s (2) ) represents state s (1) and s (2) The behavioral similarity between these two states, r(.) represents the reward value corresponding to the current state, and W2 represents the second-order Wasserstein distance; The Gaussian distribution representing the probabilistic dynamics model, s (2) ∈S′ t .

[0077] The expected relation obtained from training the encoder is represented as:

[0078]

[0079] in, For encoder.

[0080] By using multiple state data pairs, the goal is to minimize the distance between the dual simulation metric and the latent space. The learning objectives are expressed as:

[0081]

[0082] in, Indicates encoder The stopping gradient.

[0083] Obtain real-time intersection data.

[0084] Real-time prediction data is obtained by fine-tuning a large language model based on real-time intersection data.

[0085] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for controlling traffic signals using a large language model for different types of intersections, characterized in that, include: Obtain a set of historical traffic intersection environments and represent them in a unified manner to form the first matrix; Based on the randomly sampled states in the first matrix, a reinforcement learning model for near-end policy optimization is trained, and the entire first matrix is ​​input into the trained reinforcement learning model to form expert experience data. Traffic signal control sequence trajectories are generated based on expert experience data; The convolutional neural network is used to extract time-series information from the state data of traffic signal control sequence trajectories, forming latent spatial features. The input features are obtained by processing latent space features, actions and rewards of traffic signal control sequence trajectories through linear layers; The input features are processed by a trained, fine-tuned large language model to form predicted features; this includes: initializing a learnable low-rank matrix, which includes a first learnable low-rank matrix and a second learnable low-rank matrix, wherein the elements in the initialized first learnable low-rank matrix satisfy a Gaussian distribution, as shown below: ,in Let i be the element in the i-th row and j-th column of the first learnable low-rank matrix. The variance is denoted as ; the elements in the initialized second learnable low-rank matrix are all 0, represented as: ,in Let i be the element in the i-th row and j-th column of the second learnable low-rank matrix; The large language model includes multiple Transformer modules. The input features, after passing through one Transformer module, output the following: Where W is the weight, The input features are α, which is a hyperparameter. Mainly used for adjustment The weight matrix represents the proportion of the final model parameters; A and B are learnable low-rank matrices; the output after L layers of Transformer modules is: ,in, The final features output by the large language model after passing through L layers of Transformer modules. For input features, This represents a GPT-2 model with L layers of Transformer modules. This is the fine-tuning process following LoRA; The final features are sequentially input into the action prediction network, state prediction network, and reward prediction network to obtain prediction features including predicted state, predicted action, and predicted reward. The action prediction network and reward network both include linear layers, and the state prediction network includes linear layers and deconvolutional layers. The prediction features are represented as follows: , , ,in, To predict actions, To predict the state, To predict rewards, It is a linear function. It is the hyperbolic tangent function. It is a deconvolution function; The fine-tuned large language model is trained by updating and optimizing the encoder through a loss function and dual-simulated metric learning. The predicted features include predicted state, predicted action, and predicted reward. The loss function is expressed as follows: in, This refers to the action data at time t in the traffic signal control sequence trajectory. For the corresponding predicted action, This refers to the reward data at time t in the traffic signal control sequence trajectory. For the corresponding reward action, This refers to the state data at time t in the traffic signal control sequence trajectory. For the corresponding state action; By using two analog measurements of different data for the same metric, the similarity between the two data points is calculated and used to train the encoder, which includes a state linear layer, an action linear layer, and a reward linear layer. When this indicator is state data, the behavioral similarity between two state data is represented as follows: in, For state and The similarity of behavior between these two states, The table is based on the reward value corresponding to the current state. Represents the second-order Wasserstein distance; The Gaussian distribution represents the probabilistic dynamics model; The expected relation obtained from training the encoder is represented as: in, For the encoder; by using multiple state data pairs, the goal is to minimize the distance between the two analog metric and the latent space. The learning objectives are expressed as: in, Indicates encoder The stopping gradient; Obtain real-time intersection data; Real-time prediction data is obtained by fine-tuning a large language model based on real-time intersection data.

2. The method for controlling traffic signals using a large language model for different types of intersections according to claim 1, characterized in that, The reinforcement learning model for training near-end policy optimization based on randomly sampled states in the first matrix includes a policy learning method represented as follows: in, For the expected return of the strategy, For mathematical expectation, It is a minimum value function. For the new strategy, This is the old strategy. The reward is determined based on the average waiting time per vehicle at time t. The obtained advantage function, This is a shearing function used to limit the range of values. These are the trimming parameters.

3. The method for controlling traffic signals using a large language model for different types of intersections according to claim 1, characterized in that, The time-series information extracted from the state data of traffic signal control sequence trajectories using a convolutional neural network is used to form latent spatial features, represented as follows: in, For the corresponding latent space features, The state data at time t, This represents a convolutional neural network.

4. The method for controlling traffic signals using a large language model for different types of intersections according to claim 3, characterized in that, The input features are obtained by processing latent space features, traffic signal control sequence trajectories, and actions and rewards through linear layers, including: The state, action, and reward labels are embedded through a state linear layer, an action linear layer, and a reward linear layer, respectively, and the time step linear embedding is then applied. As an absolute position, combined with the embedding of each tag, the input representation of each tag is constructed as follows: in, For the encoded state features, For encoded action features, For the encoded reward features, This refers to the action data at time t in the traffic signal control sequence trajectory. This refers to the reward data at time t in the traffic signal control sequence trajectory. Represents a linear layer in a neural network; And , and Stacked together, they form the input features. .

Images