Large language model traffic signal control method for different types of intersections

Abstract

The invention belongs to the technical field of traffic control systems, and particularly relates to a large language model traffic signal control method for different types of intersections, which comprises the following steps: acquiring a historical traffic opening environment set; performing uniform state and action representation to form a first matrix; training a near-end strategy optimized reinforcement learning model based on a randomly sampled state in the first matrix, and forming a traffic signal control sequence track based on the first matrix; time sequence information contained in the state data of the traffic signal control sequence track is extracted through a convolutional neural network, and submerged space features are formed; respectively processing the subsurface space features and actions and rewards of traffic signal control sequence tracks through a linear layer to obtain input features; after operation of the trained fine-tuning large language model, prediction features are formed; the fine-tuned large language model is trained by encoder update and optimization of loss function and dual simulation metric learning, with predictive features including predictive states, predictive actions, and predictive rewards.

Description

Technical Field

[0001] The present application belongs to the technical field of traffic control systems, and specifically relates to a method for controlling traffic signals using a large language model for different types of intersections. Background Art

[0002] Traffic signal control is a challenging task for intelligent transportation systems. Its effectiveness directly impacts intersection communication safety and road network efficiency. With increasing traffic congestion, efficient optimization of traffic signal control schemes has become a hot topic of research. Currently, traditional traffic signal control methods often rely on fixed traffic signal timing schemes or manually designed traffic signal control optimization models. However, numerous studies have shown that this optimization approach, which relies on prior knowledge, struggles to adapt effectively to the complex spatiotemporal variations of road networks, resulting in limited improvements in control effectiveness.

[0003] In recent years, with the further development of technology, traffic signal control methods based on reinforcement learning have provided new solutions and have gained widespread attention among researchers due to their excellent adaptability to real-time road conditions and excellent control effects. Their excellent performance is primarily due to the fact that reinforcement learning uses dynamic traffic flow data as a reward function, using metrics related to traffic efficiency as the optimization goal to achieve efficient decision-making, thereby providing real-time interactive feedback with the environment.

[0004] Although these methods have achieved optimal performance in real and synthetic road networks, they can only be adapted to specific intersections and are difficult to extend to a wider range of intersection configuration conditions. To this end, a few studies have attempted to train more general reinforcement learning models. However, reinforcement learning-based methods face a series of problems such as low sample efficiency, data shift distribution, and limited generalization ability (especially in high-traffic situations). Recently, the emerging large language models have influenced different fields due to their knowledge transfer capabilities and zero-shot learning capabilities, and have been fine-tuned to adapt to different tasks such as autonomous driving, traffic prediction, and traffic signal control. For traffic signal control, they usually focus on long-tail scenarios such as traffic accidents and priority passage. Some studies focus on directly using large language models to generate human knowledge for solution generation and recommendation. Summary of the Invention

[0005] This application is proposed based on the above-mentioned requirements of the prior art. The technical problem to be solved by this application is to provide a method for controlling traffic signals with a large language model for different types of intersections to improve the efficiency and accuracy of traffic signal control.

[0006] In order to solve the above problems, the technical solutions provided in this application include:

[0007] A method for controlling traffic signals using a large language model for different types of intersections is provided, comprising: obtaining a set of historical traffic intersection environments and performing a unified state and action representation to form a first matrix; training a reinforcement learning model for proximal policy optimization based on randomly sampled states in the first matrix; inputting the entire first matrix into the trained reinforcement learning model to form expert experience data; forming a traffic signal control sequence trajectory based on the expert experience data; extracting time series information contained in the state data of the traffic signal control sequence trajectory through a convolutional neural network to form latent space features; processing the latent space features, the actions and rewards of the traffic signal control sequence trajectory through a linear layer to obtain input features; the input features are operated on a trained fine-tuned large language model to form prediction features; the fine-tuned large language model is trained by updating and optimizing an encoder using a loss function and dual-simulation metric learning, and the prediction features include predicted states, predicted actions, and predicted rewards; obtaining real-time intersection data; and obtaining corresponding real-time prediction data based on the real-time intersection data through a fine-tuned large language model.

[0008] Preferably, the reinforcement learning model for training the proximal policy optimization based on the randomly sampled states in the first matrix includes a policy learning method expressed as: in, is the expected return of the strategy, is the mathematical expectation, min() is the minimum function, π θ (a t |S t ) is the new strategy, For the old strategy, A t is the reward r determined based on the average waiting time of each vehicle at time t t The solved advantage function, clip() is the clipping function, which is used to limit the range of values, and ∈ is the clipping parameter.

[0009] Preferably, the time series information contained in the state data of the traffic signal control sequence trajectory is extracted by the convolutional neural network to form a latent space feature, which is expressed as Z t =CNN(S′ t ), where Z t is the corresponding latent space feature, S′ t is the state data at time t, and CNN(.) represents convolutional neural network.

[0010] Preferably, the linear layer is used to process the latent space features, the action and reward of the traffic signal control sequence trajectory respectively to obtain the input features, including: obtaining the label embedding of the state, action and reward respectively through the state linear layer, the action linear layer and the reward linear layer, and embedding the linear embedding of the time step pos tAs the absolute position, combined with the embedding of each token above, the input representation of each token is constructed, which is expressed as: in, is the encoded state feature, is the encoded action feature, is the encoded reward feature, a′ t is the action data at time t in the traffic signal control sequence trajectory, r′ t is the reward data at time t in the traffic signal control sequence trajectory, Linear(.) represents the linear layer of the neural network; and Stacked together to form input features

[0011] Preferably, the trained fine-tuned large language model includes initializing a learnable low-rank matrix, wherein the learnable matrix includes a first learnable low-rank matrix and a second learnable low-rank matrix, wherein the elements in the initialized first learnable low-rank matrix satisfy a Gaussian distribution, which is expressed as: where a ij is the element in row i and column j of the first learnable low-rank matrix, σ 2 is the variance; the elements in the second learnable low-rank matrix after initialization are all 0, expressed as: b ij =0, where b ij is the element in the i-th row and j-th column of the second learnable low-rank matrix.

[0012] Preferably, the trained fine-tuned large language model includes: the large language model includes a multi-layer Transformer module, and the output of the input feature after passing through a layer of Transformer module is: Where W is the weight, is the input feature, α is a hyperparameter, It is mainly used to adjust the proportion of the LoRA weight matrix to the final model parameters; the output of the L-layer Transformer module is: in, is the final feature output by the large language model after the L-layer Transformer module, is the input feature, LLM(.) represents the GPT-2 model with L-layer Transformer modules, and LoRA(.) is the fine-tuning process after LoRA.

[0013] Preferably, the trained fine-tuned large language model includes: the final features are sequentially input into the action prediction network, the state prediction network, and the reward prediction network to obtain prediction features including predicted state, predicted action, and predicted reward, wherein the action prediction network and the reward network both include linear layers, and the state prediction network includes linear layers and deconvolution layers, and the prediction features are expressed as: in, To predict actions, is the predicted state, To predict rewards, Linear(.) is a linear function, Tanh(.) is a hyperbolic tangent function, and Deconv(.) is a deconvolution function.

[0014] Preferably, the loss function is expressed as: Among them, a′ t is the action data at time t in the traffic signal control sequence trajectory, is the corresponding predicted action, r′ t is the reward data at time t in the traffic signal control sequence trajectory, is the corresponding reward action, S′ t is the state data at time t in the traffic signal control sequence trajectory, is the corresponding state action.

[0015] Preferably, the similarity between two data of the same indicator is calculated by dual simulation to train an encoder, wherein the encoder includes a state linear layer, an action linear layer, and a reward linear layer. When the indicator is state data, the behavioral similarity between the two state data is expressed as: Among them, d(s (1) ,s (2) ) is state s (1) and s (2) The behavioral similarity between these two states, r(.) represents the reward value corresponding to the current state, and W2 represents the second-order Wasserstein distance; represents the Gaussian distribution of the probabilistic dynamics model; the expected relationship obtained by training the encoder is expressed as: in, For encoders; through multiple state data pairs, minimize the distance between dual simulation metrics and latent space The learning objectives are expressed as: in, Representation encoder The stopping gradient.

[0016] Compared to existing technologies, this application leverages the prior knowledge of a pre-trained large language model and fine-tunes it to adapt traffic signal control tasks for intersections of varying structures, phases, and traffic inputs. This method proposes a large language model-enhanced general traffic signal control method for different intersection types. This method uses the large language model as an intelligent agent, efficiently meeting control requirements under different intersection configurations and directly inferring the optimal traffic signal configuration, thereby improving control robustness and generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0018] Figure 1 A flowchart of a method for controlling traffic signals using a large language model for different types of intersections according to an embodiment of the present invention;

[0019] Figure 2 Define schematic diagrams for intersection attributes;

[0020] Figure 3 The schematic diagram of the signal plan corresponding to the intersection;

[0021] Figure 4 This is a partial expression diagram of the first matrix after unification based on the historical intersection environment set;

[0022] Figure 5 This is a schematic diagram of another part of the first matrix after unification based on the historical intersection environment set. DETAILED DESCRIPTION

[0023] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] In the description of the embodiments of the present application, it should be noted that unless otherwise clearly specified and limited, the term "connected" should be understood in a broad sense, for example, it can be fixedly connected, or detachably connected, or integrally connected, which can be mechanically connected, or electrically connected, which can be directly connected, or indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above-mentioned term in the present application can be understood according to the specific circumstances.

[0025] The terms "top", "bottom", "above", "under" and "on" used throughout the description are relative positions of components of the device, for example, the relative positions of the top and bottom substrates inside the device. It can be understood that the device is multifunctional, regardless of their orientation in space.

[0026] For the purpose of understanding the embodiments of the present application, the following will be further explained and described with specific embodiments in conjunction with the accompanying drawings, which do not constitute a limitation on the embodiments of the present application.

[0027] The embodiments provide a large language model method for controlling traffic signals of different types of intersections, as shown in Figure 1 .

[0028] The large language model method for controlling traffic signals of different types of intersections comprises:

[0029] A set of historical traffic port environments is obtained, and a unified state and action representation is formed to form a first matrix.

[0030] The set of historical traffic port environments includes traffic structure, phase and flow pattern.

[0031] As shown in Figure 2 and Figure 3 , the traffic structure includes T-shaped and cross-shaped planar intersections, and corresponding entrance roads.

[0032] The phase is a combination of movement signals. Each phase is composed of one or more non-conflicting traffic movements. Taking the four phases as an example, phase 1 includes left turns in the east-west direction. When the phase is set to green, vehicles in this phase are allowed to pass, and vehicles in other phases are restricted. The movement signal is defined according to the traffic movement. The green light indicates that the current movement is allowed, and the red light indicates that the current movement is prohibited. In this embodiment, right turns in different directions are all set to green lights. The traffic movement refers to the entire movement process from the vehicle entering the entry lane, crossing the intersection to leaving the lane. For a four-way intersection, the four directions of east, south, west and north all have left turns, straight going and right turns, that is, 12 traffic movements. The lane refers to the road where vehicles travel in a specific direction. In any intersection, it can be divided into entry lanes and exit lanes according to entering and leaving the intersection. The signal plan is used to set the phase sequence and phase time. Taking four phases as an example, a phase plan is usually executed cyclically, that is, {(p1, t1), (p2, t2), (p3, t3), (p4, t4), (p1, t1), (p2, t2), ...}, such as Figure 3 As shown, p and t here represent a phase and its corresponding phase duration.

[0033] The flow pattern is to adjust the flow of different entrances under the same flow input at the intersection.

[0034] The data in the historical intersection environment collection are unified into a preset state coding space, such as Figure 4 and Figure 5 As shown, the space includes east, west, south and north entrance roads, and each entrance road has left turn, right turn and straight going.

[0035] Specifically, the traffic data generated by different traffic movements at a given moment is represented as a state matrix, including occupancy, one-hot encoding of straight, left, and right turns, the number of lanes occupied by the traffic movement, whether the current movement signal is green, and whether the next movement signal will be green. If an entrance lane or direction is missing, it is filled with zeros.

[0036] In order to better understand the dynamic information of the intersection, the multi-step state S t ={s t-K ,…,s t} is defined as the observation variable of the previous K steps (including the current moment) at time t. In addition, in order to adapt to the different phase modes of the intersection, a safer way is adopted to switch phases. Specifically, a loop scheme is preset to execute different phases in sequence. Therefore, action a t Defined as holding or switching, that is, two-vector encoding, its function is to control the mobile signal.

[0037] Based on the randomly sampled states in the first matrix, a reinforcement learning model for proximal policy optimization is trained, and the entire first matrix is ​​input into the trained reinforcement learning model to form expert experience data.

[0038] Specifically, this embodiment uses PPO as a reinforcement learning model, which randomly collects states in the first matrix to train a reinforcement learning traffic signal control network that can make decisions in different intersection environments. The specific strategy learning method is:

[0039]

[0040] in, is the expected return of the strategy, is the mathematical expectation, min() is the minimum function, π θ (a t |S t ) is the new strategy, For the old strategy, A t is the reward r determined based on the average waiting time of each vehicle at time t t The solved advantage function, clip() is the clipping function, which is used to limit the range of values, and ∈ is the clipping parameter.

[0041] The trained reinforcement learning model, as the decision-making agent for the set of all intersection environments, re-inputs the data of the first matrix into the reinforcement learning model, thereby outputting expert experience data of different intersection environments, including state data, action data, and reward data.

[0042] Form traffic signal control sequence trajectories based on expert experience data.

[0043] A number of batched data are randomly selected from the expert experience data to form a traffic signal control sequence trajectory. The traffic signal control sequence trajectory includes action data, state data and reward data, which is expressed as: τ = (S′1, a′1, r′1, S′2, a′2, r′2…,), where S′1 represents the state data at the first moment, a′1 is the action data at the first moment, r′1 is the reward data at the first moment, S′2 represents the state data at the second moment, a′2 is the action data at the second moment, and r′2 is the reward data at the second moment.

[0044] The time series information contained in the state data of the traffic signal control sequence trajectory is extracted through a convolutional neural network to form latent space features.

[0045] Specifically expressed as:

[0046] Z t =CNN(S′ t )

[0047] Among them, Z t is the corresponding latent space feature, S′ t is the state data at time t, and CNN(.) represents convolutional neural network.

[0048] The latent space features, actions and rewards of the traffic signal control sequence trajectory are processed separately through linear layers to obtain input features.

[0049] Specifically, the linear layer is used to obtain the embedded labels of state, action and reward respectively based on the latent space features, action data and reward data in the traffic signal control sequence trajectory. In addition, in order to determine the position of the label in the sequence trajectory, the linear embedding of the time step is pos t As the absolute position, combined with the embedding of each token above, the input representation of each token is constructed as follows:

[0050]

[0051] in, is the encoded state feature, is the encoded action feature, is the encoded reward feature, a′ t is the action data at time t in the trajectory, r′ t is the reward data at the tth moment in the trajectory, and Linear(.) represents the linear layer of the neural network.

[0052] Stack the above and Form the input matrix, expressed as

[0053] The input features are operated on a trained fine-tuned large language model to form prediction features; the fine-tuned large language model is trained by updating and optimizing an encoder using a loss function and dual-simulation metric learning, and the prediction features include predicted state, predicted action, and predicted reward.

[0054] Freeze the weights W of the pre-trained LLM through LoRA and inject the learnable low-rank matrices A and B, where And initialize matrices A and B. Specifically, initialize matrix A randomly so that the elements in the matrix satisfy the Gaussian distribution; initialize all elements in matrix B to 0, expressed as:

[0055]

[0056] b ij =0

[0057] Therefore, the output of the GPT-2 model after a layer of Transformer module is:

[0058]

[0059] Among them, h is the output feature, is the input matrix, α is a hyperparameter, It is mainly used to adjust the proportion of the LoRA weight matrix to the final model parameters.

[0060] The output of the GPT-2 model after the L-layer Transformer module is:

[0061]

[0062] in, is the final feature output by the GPT-2 model after the L-layer Transformer module, is the input matrix, LLM(.) represents the GPT-2 model with L-layer Transformer modules, and LoRA(.) is the fine-tuning process after LoRA.

[0063] The final features are input into the action prediction network, state prediction network and reward prediction network to obtain the predicted action state and rewards

[0064] Among them, the action prediction network and the reward network are both linear layers, and the state prediction network consists of linear layers and deconvolution layers. state and rewards The specific calculation process is as follows:

[0065]

[0066] The action prediction network uses cross entropy loss as its loss function to optimize the strategy as follows:

[0067]

[0068] The state prediction network and reward prediction network use mean square error as their loss function to estimate the state and reward. The specific formula is as follows:

[0069]

[0070] Therefore, the total loss function is as follows:

[0071]

[0072] In order to train the above-mentioned network parameters through the total loss function, a commonly used optimizer AdamW is used to implement gradient descent to update all network parameters.

[0073] To further eliminate task-irrelevant information, avoid spurious correlations between actions and states, and enhance the encoder's ability to learn the observed state, this paper uses dual-analog metric learning to capture robust latent representations. This introduces an additional self-supervisory regularization term to learn a robust encoder. The encoder comprises a state linear layer, a dynamic linear layer, and a reward linear layer.

[0074] By using dual simulation to measure different data of the same indicator, the similarity between the two data is calculated and used to train the encoder. When the indicator is state data, the behavioral similarity between the two state data is expressed as:

[0075]

[0076] Among them, d(s (1) ,s (2) ) is state s (1) and s (2) The behavioral similarity between these two states, r(.) represents the reward value corresponding to the current state, and W2 represents the second-order Wasserstein distance; represents the Gaussian distribution of the probabilistic dynamics model, s (2) ∈S′ t .

[0077] The expected relationship obtained by training the encoder is expressed as:

[0078]

[0079] in, For the encoder.

[0080] Through multiple state data pairs, minimize the distance between dual simulation metrics and latent space The learning objectives are expressed as:

[0081]

[0082] in, Representation encoder The stopping gradient.

[0083] Get real-time intersection data.

[0084] Based on real-time intersection data, the corresponding real-time prediction data is obtained by fine-tuning the large language model.

[0085] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A method for controlling traffic signals using a large language model for different types of intersections, characterized in that: include: Obtain the historical traffic intersection environment set and perform unified state and action representation to form the first matrix; Based on the randomly sampled states in the first matrix, a reinforcement learning model for proximal policy optimization is trained, and the entire first matrix is ​​input into the trained reinforcement learning model to form expert experience data; Forming traffic signal control sequence trajectories based on expert experience data; The time series information contained in the state data of the traffic signal control sequence trajectory is extracted through a convolutional neural network to form a latent space feature; The latent space features, the actions and rewards of the traffic signal control sequence trajectory are processed separately through the linear layer to obtain the input features; The input features are processed by the trained fine-tuned large language model to form prediction features; The fine-tuned large language model is trained by updating and optimizing an encoder using a loss function and dual-simulation metric learning, and the predicted features include predicted state, predicted action, and predicted reward; Get real-time intersection data; Based on real-time intersection data, the corresponding real-time prediction data is obtained by fine-tuning the large language model.

2. The method for controlling traffic signals using a large language model for different types of intersections according to claim 1 is characterized in that: The reinforcement learning model for training proximal policy optimization based on the randomly sampled states in the first matrix includes a policy learning method expressed as: in, is the expected return of the strategy, is the mathematical expectation, min() is the minimum function, π θ (a t |S t ) is the new strategy, For the old strategy, A t is the reward r determined based on the average waiting time of each vehicle at time t t The solved advantage function, clip() is the clipping function, which is used to limit the range of values, and ∈ is the clipping parameter.

3. The method for controlling traffic signals using a large language model for different types of intersections according to claim 1 is characterized in that: The convolutional neural network is used to extract the time series information contained in the state data of the traffic signal control sequence trajectory to form a latent space feature, which is expressed as Z t =CNN(S′ t ) Among them, Z t is the corresponding latent space feature, S′ t is the state data at time t, and CNN(.) represents convolutional neural network.

4. The method for controlling traffic signals using a large language model for different types of intersections according to claim 3 is characterized in that: The linear layer processes the latent space features, the actions and rewards of the traffic signal control sequence trajectory respectively to obtain input features, including: Through the state linear layer, action linear layer and reward linear layer to obtain the tag embedding of state, action and reward respectively, and the linear embedding of time step pos t As the absolute position, combined with the embedding of each token, the input representation of each token is constructed, which is expressed as: in, is the encoded state feature, is the encoded action feature, is the encoded reward feature, a′ t is the action data at time t in the traffic signal control sequence trajectory, r′ t is the reward data at time t in the traffic signal control sequence trajectory, Linear(.) represents the linear layer of the neural network; and will and Stacked together to form input features 5. The method for controlling traffic signals using a large language model for different types of intersections according to claim 1 is characterized in that: The trained fine-tuned large language model includes: Initialize a learnable low-rank matrix, where the learnable matrix includes a first learnable low-rank matrix and a second learnable low-rank matrix, wherein the elements in the initialized first learnable low-rank matrix satisfy a Gaussian distribution, which is expressed as: where a ij is the element in row i and column j of the first learnable low-rank matrix, σ 2 is the variance; the elements in the second learnable low-rank matrix after initialization are all 0, expressed as: b ij =0, where b ij is the element in the i-th row and j-th column of the second learnable low-rank matrix.

6. The method for controlling traffic signals using a large language model for different types of intersections according to claim 5 is characterized in that: The trained fine-tuned large language model includes: The large language model includes multiple layers of Transformer modules. The output of the input features after passing through one layer of Transformer modules is: Where W is the weight, is the input feature, α is a hyperparameter, It is mainly used to adjust the proportion of the LoRA weight matrix to the final model parameters; the output of the L-layer Transformer module is: in, is the final feature output by the large language model after the L-layer Transformer module, is the input feature, LLM(.) represents the GPT-2 model with L-layer Transformer modules, and LoRA(.) is the fine-tuning process after LoRA.

7. The method for controlling traffic signals using a large language model for different types of intersections according to claim 1 is characterized in that: The trained fine-tuned large language model includes: The final features are sequentially input into the action prediction network, state prediction network, and reward prediction network to obtain prediction features including predicted state, predicted action, and predicted reward. The action prediction network and reward network both include linear layers, and the state prediction network includes linear layers and deconvolution layers. The prediction features are expressed as: in, To predict actions, is the predicted state, To predict rewards, Linear(.) is a linear function, Tanh(.) is a hyperbolic tangent function, and Deconv(.) is a deconvolution function.

8. The method for controlling traffic signals using a large language model for different types of intersections according to claim 1 is characterized in that: The loss function is expressed as: Among them, a′ t is the action data at time t in the traffic signal control sequence trajectory, is the corresponding predicted action, r′ t is the reward data at time t in the traffic signal control sequence trajectory, is the corresponding reward action, S′ t is the state data at time t in the traffic signal control sequence trajectory, is the corresponding state action.

9. The method for controlling traffic signals using a large language model for different types of intersections according to claim 1, characterized in that: By using dual simulation to measure different data of the same indicator, the similarity between the two data is calculated for training an encoder, wherein the encoder includes a state linear layer, an action linear layer, and a reward linear layer; When the indicator is state data, the behavioral similarity between two state data is expressed as: Among them, d(s (1) ,s (2) ) is state s (1) and s (2) The behavioral similarity between these two states, r(.) represents the reward value corresponding to the current state, and W2 represents the second-order Wasserstein distance; Gaussian distribution representing a probabilistic dynamics model; The expected relationship obtained by training the encoder is expressed as: in, For the encoder; Through multiple state data pairs, minimize the distance between dual simulation metrics and latent space The learning objectives are expressed as: in, Representation encoder The stopping gradient.

Images