Fine-tuning auxiliary deep reinforcement learning scheduling method for intelligent traffic lights
By introducing deep reinforcement learning scheduling methods in intelligent traffic light systems, combining CNN and Transformer encoders, and introducing fine-tuned low-rank linear units and unit line-type layers in the model, the calculation capabilities of traditional scheduling methods in complex traffic environments and The problem of insufficient response speed has achieved more efficient traffic light scheduling.
Patent Information
- Application Number
- CN202411316592.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2025-01-10
AI Technical Summary
Traditional traffic light scheduling methods cannot effectively respond to complex traffic environments and dynamic traffic demands, resulting in traffic congestion and waste of resources. Especially during peak hours and large-scale urban transportation systems, the computing power and response speed are insufficient, making it difficult to quickly adapt to variable traffic conditions.
Using a fine-tuning assisted deep reinforcement learning scheduling method for intelligent traffic lights, a basic policy network is built by constructing state space and action space data, combining CNN and Transformer encoders, and on the basis of it, a fine-tuning low-rank linear unit and fine-tuning unit line type are introduced , alternate deep learning and reinforcement learning, optimize model parameters to improve adaptability and efficiency.
It significantly improves the response speed and adaptability of the traffic light scheduling system, avoids local optimal solutions, improves the generalization ability and practical application effect of the model, and provides efficient and intelligent scheduling solutions for complex urban traffic environments.
Smart Images
Figure CN119294472A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent transportation systems, and in particular to a fine-tuning assisted deep reinforcement learning scheduling method for intelligent traffic lights. Background Art
[0002] With the continuous advancement of urbanization, traffic flow has increased significantly worldwide. Traditional traffic light scheduling methods are mostly based on fixed schedules or rule-based control strategies, which mainly rely on preset cycles and simple traffic flow estimates. However, this traditional scheduling method often cannot fully cope with complex traffic environments and dynamic traffic demands, and is prone to traffic congestion, delays, and waste of resources. Especially during peak hours or special circumstances, the limitations of traditional scheduling methods become increasingly apparent, resulting in a significant decrease in the overall efficiency of the transportation system.
[0003] In order to solve these problems, intelligent transportation systems (ITS) came into being. ITS is committed to improving the intelligence level of transportation systems by integrating modern information technology, communication technology and control technology. As an important part of ITS, dynamic traffic light control has become a common improvement measure to adjust the traffic light cycle through real-time monitoring and data analysis. Although many intelligent transportation systems and dynamic traffic light control methods have achieved results in practical applications and can alleviate traffic congestion problems to a certain extent, they are still insufficient in the face of complex and changing traffic environments, especially in large-scale urban transportation systems. Because existing models and algorithms often require a lot of computing resources and the generalization ability of the models is limited, they often face bottlenecks in computing power and response speed, thus limiting the overall efficiency and response ability of the system, making it difficult to quickly adapt to the changing traffic conditions at different intersections and time periods. Therefore, how to improve the response speed and adaptability of the system through effective model fine-tuning has become an urgent problem to be solved. Summary of the invention
[0004] The present invention aims to solve the limitation problem of current traffic light scheduling methods when dealing with complex traffic environments and dynamic traffic demands, and provides a fine-tuning assisted deep reinforcement learning scheduling method for intelligent traffic lights.
[0005] To solve the above problems, the present invention is achieved through the following technical solutions:
[0006] The fine-tuning assisted deep reinforcement learning scheduling method for intelligent traffic lights includes the following steps:
[0007] Step 1: Obtain historical traffic data, and construct state space data and action space data after preprocessing;
[0008] The state space data includes the traffic light state matrix S lights , traffic light state maintenance time matrix T, and vehicle quantity and waiting time matrix W;
[0009] The action space data includes the traffic light scheduling action matrix A and the traffic light action maintenance time vector D;
[0010] Step 2: Construct a basic policy network, which consists of a CNN convolutional neural network, a summation layer, a flattening layer, and a Transformer encoder; the input of the CNN convolutional neural network forms the input of the basic policy network, the input and output of the basic policy network are connected to the input of the summation layer, the output of the summation layer is connected to the input of the flattening layer, the output of the flattening layer is connected to the input of the Transformer encoder, and the output of the Transformer encoder forms the output of the basic policy network;
[0011] Step 3: Use the state space data and action space data constructed in step 1 to perform deep learning on the basic policy network constructed in step 2, comprehensively optimize the model parameters of all layers of the basic policy network through deep learning, and stop iteration when the model parameters converge to obtain the basic policy model;
[0012] Step 4: structurally adjust the basic strategy model obtained in step 3, that is, connect a fine-tuned low-rank linear unit in parallel to each linear layer of the CNN convolutional neural network of the basic strategy network, connect a fine-tuned low-rank linear unit in parallel to the query layer, key layer and value layer of the multi-head attention layer of the Transformer encoder, connect a fine-tuned low-rank linear unit in parallel to each feed-forward linear layer of the Transformer encoder, and connect a fine-tuned unit linear layer in series after the output of the Softmax activation layer of the Transformer encoder to obtain a fine-tuned strategy network;
[0013] Step 5: Use the state space data and action space data constructed in step 1 to perform reinforcement learning on the fine-tuning strategy network constructed in step 4. Optimize the model parameters of all layers of the fine-tuning strategy network through the reinforcement learning stage, and stop the iteration when the number of iterations reaches a preset value to obtain a reinforced and updated fine-tuning strategy model.
[0014] Step 6: Use the state space data and action space data constructed in step 1 to perform deep learning on the enhanced updated fine-tuning strategy model obtained in step 5, optimize the model parameters of each fine-tuning low-rank linear unit and fine-tuning unit linear layer of the enhanced updated fine-tuning strategy model through the deep learning stage, and stop the iteration when the number of iterations reaches a preset value to obtain the deeply updated fine-tuning strategy model;
[0015] Step 7, judging whether the model parameters of the deep updated fine-tuning strategy model obtained in step 6 have converged: if converged, setting the deep updated fine-tuning strategy model as the scheduling strategy model; otherwise, returning to step 5;
[0016] Step 8: Deploy the dispatching strategy model obtained in step 7 to the actual traffic system, and send the collected real-time state space data into the dispatching strategy model to obtain real-time action space data to complete the dispatching of traffic lights.
[0017] In the above scheme, the traffic light state matrix S lights ={s k,i，j},s k,i，j represents the state of the jth traffic light in the i-th direction of intersection k; the traffic light state maintenance time matrix T = {t k，i，j}, t k，i，j represents the maintenance time of the jth traffic light in the i-th direction of intersection k; the vehicle quantity and waiting time matrix W = {w k，i，k},w k，i，k represents the waiting time of vehicles that have not passed the intersection in the kth lane of the i-th direction of intersection k from the previous state cycle to the current state; the traffic light scheduling action matrix A = {a k，i，j}, a k，i，j represents the dispatching action of the jth traffic light in the i-th direction of intersection k;
[0018] The traffic light operation maintenance time vector D = {d k，i，j}, d k，i，j Indicates the time that the j-th traffic light in the i-th direction of intersection k remains in the state after executing the scheduling action.
[0019] The above-mentioned CNN convolutional neural network is composed of N1 convolution and pooling units and a linear layer; wherein N1 is a set positive integer; each convolution and pooling unit is composed of a convolution layer and a pooling layer, the input of the convolution layer forms the input of the convolution and pooling unit, the output of the convolution layer is connected to the input of the pooling layer, and the output of the pooling layer forms the output of the convolution and pooling unit; N1 convolution and pooling units are connected in series, the input of the first convolution and pooling unit forms the input of the CNN convolutional neural network, the output of the last convolution and pooling unit is connected to the input of the linear layer, and the output of the linear layer forms the output of the CNN convolutional neural network.
[0020] The above Transformer encoder consists of an input embedding layer, a position encoding layer, a summation layer, N2 attention and feedforward normalization units, a linear layer and a Softmax activation layer; wherein N2 is a set positive integer; each attention and feedforward normalization unit consists of a multi-head attention layer, 2 normalization layers and 2 feedforward linear layers, the input of the multi-head attention layer forms the input of the attention and feedforward normalization unit, the input and output of the multi-head attention layer are connected to the input of the first normalization layer, the output of the first normalization layer is connected to the input of the first feedforward linear layer, the output of the first feedforward linear layer is connected to the input of the second feedforward linear layer, the output of the second feedforward linear layer and the input of the first feedforward linear layer are connected to the first The input of the two normalization layers, the output of the second normalization layer forms the output of the attention and feedforward normalization unit; the input of the input embedding layer and the input of the position encoding layer form the input of the Transformer encoder, the output of the input embedding layer and the output of the position encoding layer are connected to the input of the summation layer, the output of the summation layer is connected to the input of the first attention and feedforward normalization unit of N2 series-connected attention and feedforward normalization units, the output of the last attention and feedforward normalization unit of N2 series-connected attention and feedforward normalization units is connected to the input of the linear layer, the output of the linear layer is connected to the input of the Softmax activation layer, and the output of the Softmax activation layer forms the output of the Transformer encoder.
[0021] The above-mentioned fine-tuning low-rank linear unit is composed of a first low-rank linear layer and a second low-rank linear layer; the input of the first low-rank linear layer forms the input of the fine-tuning low-rank linear unit, the output of the first low-rank linear layer is connected to the input of the second low-rank linear layer, and the output of the second low-rank linear layer forms the output of the fine-tuning low-rank linear unit; the dimension of the weight matrix of the first low-rank linear layer is d×r, and the initial value of the weight matrix is determined by Gaussian initialization; the dimension of the weight matrix of the second low-rank linear layer is r×d, and the initial value of the weight matrix is determined by zero initialization; wherein, d is the dimension of the weight matrix of the parallel layer, and r is the set rank.
[0022] The dimension of the weight matrix of the above-mentioned fine-tuning unit linear layer is the same as the dimension of the weight matrix of the Softmax activation layer, and the initial value of the weight matrix is the unit matrix.
[0023] Compared with the prior art, the present invention has the following characteristics:
[0024] 1. By introducing fine-tuning low-rank linear units and fine-tuning unit linear layers on the basis of the basic strategy model, it not only ensures that the output results of the model are not affected after the structural changes while retaining the main features of the original model, but also improves the adaptability of the model in complex traffic environments. In addition, the number of parameters is significantly reduced during the model fine-tuning process, the training efficiency is improved, and the convergence speed of the model is accelerated.
[0025] 2. By alternating between reinforcement learning and deep learning fine-tuning processes and combining the advantages of deep learning and reinforcement learning, the intelligent agent is prevented from falling into the local optimal solution in strategy optimization, thereby improving the generalization ability and practical application effect of the model, and providing an efficient and intelligent solution for traffic light scheduling in complex urban traffic environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 This is the overall algorithm flow chart of the fine-tuning assisted deep reinforcement learning scheduling method for intelligent traffic lights.
[0027] Figure 2 Schematic diagram of the structure of the basic strategy network.
[0028] Figure 3 Schematic diagram of the structure of the fine-tuning strategy network.
[0029] Figure 4 Schematic diagram of the structure of fine-tuning multi-head attention unit. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific examples and with reference to the accompanying drawings.
[0031] A fine-tuning assisted deep reinforcement learning scheduling method for intelligent traffic lights, such as Figure 1 As shown, it includes the following steps:
[0032] Step 1: Obtain historical traffic data and construct state space data and action space data after preprocessing.
[0033] Collect historical traffic data from different sources (such as traffic monitoring systems, intelligent traffic management platforms, etc.), which include traffic light state switching and vehicle flow information at different intersections in different time periods. Preprocess the historical traffic light operation data (cleaning, standardization, and formatting, etc.) to ensure data quality and consistency, and provide high-quality basic data for the subsequent preliminary training of the intelligent agent.
[0034] Construct state space data and action space data based on the preprocessed data.
[0035] The state space data is used to represent the traffic lights and lanes at each intersection, which mainly includes the traffic light state matrix S lights , traffic light state maintenance time matrix T, and vehicle quantity and waiting time matrix W.
[0036] Traffic light state matrix S lights ={sk,i，j},s k,i，j Indicates the state of the jth traffic light in the i-th direction of intersection k. k，i，j ∈(0, 1), 0 represents red light (including yellow light) and 1 represents green light.
[0037] Traffic light state maintenance time matrix T = {t k，i，j}, t k,i，j Indicates the maintenance time of the jth traffic light in the i-th direction of intersection k. k，i，j ∈(1, 2, ...), respectively representing the maintenance time (in seconds).
[0038] Vehicle quantity and waiting time matrix W = {w k,i,c},w k,i，k represents the waiting time of vehicles that have not passed the intersection in the kth lane of the i-th direction of intersection k from the previous state cycle to the current state. k，i，c ∈(1, 2, ...), respectively representing the waiting time (in seconds).
[0039] The action space data describes the scheduling operation of traffic lights at each intersection, which mainly includes the traffic light scheduling action matrix A and the maintenance time vector D after the traffic light action.
[0040] Traffic light scheduling action matrix A = {a k,i，j}, a k,i，j represents the dispatching action of the jth traffic light in the i-th direction of intersection k. k,i，j ∈(0, 1, 2), 0 means keeping the current state, 1 means turning to red light, and 2 means turning to green light.
[0041] The traffic light operation maintenance time vector D = {d k，i,j}, d k，i，j Indicates the time that the jth traffic light in the i-th direction of intersection k remains in the same state after executing the scheduling action. k，i，j ∈(1, 2, ...), respectively representing the state maintenance time (in seconds) after the scheduling action is executed.
[0042] The above k∈(1, 2, ...) represent the intersection numbers. For intersections with four directions, i∈(E, S, W, N), E represents east, S represents south, W represents west, and N represents north. For traffic lights with three directions, j∈(L, T, R), L represents left turn, T represents straight ahead, and R represents right turn. c∈(1, 2, ...) represent lane numbers.
[0043] Step 2: Build a basic policy network, which consists of a CNN convolutional neural network, a summation layer, a flattening layer, and a Transformer encoder. The input of the CNN convolutional neural network forms the input of the basic policy network, the input and output of the basic policy network are connected to the input of the summation layer, the output of the summation layer is connected to the input of the flattening layer, the input of the flattening layer is connected to the input of the Transformer encoder, and the output of the Transformer encoder forms the output of the basic policy network.
[0044] Step 2: Build a basic strategy network.
[0045] The purpose of the basic policy network is to learn operational experience from historical traffic data and provide a good starting point for subsequent reinforcement learning. Figure 2 As shown in the figure, the basic policy network consists of a CNN convolutional neural network, a summation layer, a flattening layer, and a Transformer encoder. The input of the CNN convolutional neural network forms the input of the basic policy network, the input and output of the basic policy network are connected to the input of the summation layer, the output of the summation layer is connected to the input of the flattening layer, the output of the flattening layer is connected to the input of the Transformer encoder, and the output of the Transformer encoder forms the output of the basic policy network.
[0046] The CNN convolutional neural network consists of N1 convolution and pooling units and a linear layer. N1 is a set positive integer. Each convolution and pooling unit consists of a convolution layer and a pooling layer. The input of the convolution layer forms the input of the convolution and pooling unit, the output of the convolution layer is connected to the input of the pooling layer, and the output of the pooling layer forms the output of the convolution and pooling unit. N1 convolution and pooling units are connected in series, the input of the first convolution and pooling unit forms the input of the CNN convolutional neural network, the output of the last convolution and pooling unit is connected to the input of the linear layer, and the output of the linear layer forms the output of the CNN convolutional neural network.
[0047] The Transformer encoder consists of an input embedding layer, a position encoding layer, a summation layer, N2 attention and feedforward normalization units, a linear layer, and a Softmax activation layer. N2 is a set positive integer. Each attention and feedforward normalization unit consists of a multi-head attention layer, two normalization layers, and two feedforward linear layers. The input of the multi-head attention layer forms the input of the attention and feedforward normalization unit. The input and output of the multi-head attention layer are connected to the input of the first normalization layer, the output of the first normalization layer is connected to the input of the first feedforward linear layer, the output of the first feedforward linear layer is connected to the input of the second feedforward linear layer, the output of the second feedforward linear layer and the input of the first feedforward linear layer are connected to the input of the second normalization layer, and the output of the second normalization layer forms the output of the attention and feedforward normalization unit. The input of the input embedding layer and the input of the position encoding layer form the input of the Transformer encoder, the output of the input embedding layer and the output of the position encoding layer are connected to the input of the summation layer, the output of the summation layer is connected to the input of the first attention and feedforward normalization unit of N2 series-connected attention and feedforward normalization units, the output of the last attention and feedforward normalization unit of N2 series-connected attention and feedforward normalization units is connected to the input of the linear layer, the output of the linear layer is connected to the input of the Softmax activation layer, and the output of the Softmax activation layer forms the output of the Transformer encoder.
[0048] The CNN convolutional neural network is used to initially extract the features of the traffic status and extract useful spatial information through convolution operations. These features are then converted into a one-dimensional vector form suitable for processing by the Transformer encoder through a flattening layer. The Transformer encoder processes the flattened features and captures the complex correlations and long-distance dependencies between the input features through the self-attention mechanism of its multi-head attention layer. The multi-layer feedforward linear layer structure further enhances the feature extraction capability, enabling it to better understand the relationship between different traffic light states and vehicle distribution. The output processed by the Transformer encoder is mapped to the action space data through a linear layer and a Softmax activation layer.
[0049] Step 3: Use the state space data and action space data constructed in step 1 to perform deep learning on the basic policy network constructed in step 2. Through deep learning, the model parameters (weight matrices) of all layers of the basic policy network are comprehensively optimized, and the iteration is stopped when the model parameters converge to obtain the basic policy model.
[0050] Based on the existing deep learning algorithm, the model parameters of all layers of the basic policy network are continuously optimized to make the predicted action distribution as close as possible to the optimal action distribution in the actual historical data, and the iteration is stopped when the model parameters converge (the model parameters no longer change). As the training progresses, the basic policy network gradually learns how to perform reasonable traffic light operations under various traffic conditions. When performing deep learning, the target loss function Loss is set to the cross entropy of the output action and the data set action.
[0051] Step 4: Structural adjustment is performed on the basic strategy model obtained in step 3 to obtain a fine-tuned strategy network.
[0052] In the basic strategy model, fine-tuning low-rank linear units and fine-tuning unit linear layers are added, and the linear layers in CNN and the Q, K, V layers and feed-forward linear layers of Transformer are structurally adjusted, and the stability of the output results after the structural changes is maintained to improve the flexibility and stability of the model. Figure 3 As shown in Figure 1, the structural adjustment of the basic strategy model specifically includes: connecting a fine-tuned low-rank linear unit in parallel to each linear layer of the CNN convolutional neural network of the basic strategy network, and connecting a fine-tuned low-rank linear unit in parallel to the query layer, key layer and value layer of the multi-head attention layer of the Transformer encoder (such as Figure 4 ), a fine-tuned low-rank linear unit is connected in parallel to each feed-forward linear layer of the Transformer encoder, and a fine-tuned unit linear layer is connected in series after the output of the Softmax activation layer of the Transformer encoder.
[0053] In order to improve the adaptability and training efficiency of the model, a fine-tuning low-rank linear unit is introduced on the basis of the basic strategy model. The fine-tuning low-rank linear unit is composed of two low-rank linear layers (the first low-rank linear layer and the second low-rank linear layer) connected in series, that is, the input of the first low-rank linear layer forms the input of the fine-tuning low-rank linear unit, the output of the first low-rank linear layer is connected to the input of the second low-rank linear layer, and the output of the second low-rank linear layer forms the output of the fine-tuning low-rank linear unit. The weight matrices of the two low-rank linear layers of the fine-tuning low-rank linear unit are denoted as A and B respectively. The weight matrix of the parallel layer is denoted as W0. The adjusted weight matrix W can be expressed as:
[0054] W=W0+ΔW=W0+AB T
[0055] Where, ΔW = AB TIt is the weight update part during fine-tuning. A is a d×r matrix, and B is an r×d matrix, where d is the dimension of W0 and r is the rank, indicating the degree of dimensionality reduction. A and B are initialized with Gaussian initialization and zero initialization, respectively, to ensure the initial training stability of the model. The purpose of connecting the fine-tuned low-rank linear unit in parallel with the parallel layer is to adjust and optimize the original weights by adding additional low-rank parameters to improve its adaptability to complex tasks, thereby improving the performance of the model while keeping the computational complexity controllable.
[0056] When the fine-tuned low-rank linear unit is connected in parallel to the parallel layer (linear layer, query layer, key layer, value layer, feedforward linear layer), the specific operation is: directly connect the input of the fine-tuned low-rank linear unit to the input of the parallel layer, connect the output of the fine-tuned low-rank linear unit and the output of the parallel layer to the input of a summation layer, and then use the output of the summation layer as the output of the fine-tuned low-rank linear unit and the parallel layer.
[0057] In order to enhance the adaptability of the model, a fine-tuning unit linear layer is introduced on the basis of the basic strategy model. The dimension of the weight matrix of the fine-tuning unit linear layer is the same as the dimension of the weight matrix of the Softmax activation layer. The weight matrix of the fine-tuning unit linear layer is initialized to the identity matrix, that is, the element value of the diagonal of the matrix is 1, and the remaining element values are 0, to ensure that the output of the model remains consistent after the structure changes and avoid introducing unnecessary deviations. The main function of the fine-tuning unit linear layer is to maintain the stability of the model and the consistency of the output during the fine-tuning process, to ensure that the main features of the original model are still retained even when the model structure changes.
[0058] When the fine-tuning unit linear layer is connected in series to the layer to be connected in series (Softmax activation layer), the specific operation is: the input of the fine-tuning unit linear layer is connected to the output of the layer to be connected in series, and the output of the fine-tuning unit linear layer is used as the output of the fine-tuning unit linear layer and the layer to be connected in series.
[0059] Step 5: Use the state space data and action space data constructed in step 1 to perform reinforcement learning on the fine-tuning strategy network constructed in step 4. Optimize the model parameters (weight matrix) of all layers of the fine-tuning strategy network through the reinforcement learning stage, and stop the iteration when the number of iterations reaches the preset value to obtain a reinforced and updated fine-tuning strategy model.
[0060] Based on the existing reinforcement learning algorithm, the model parameters of all layers of the fine-tuning strategy network are continuously optimized to make the predicted action distribution as close as possible to the optimal action distribution in the actual historical data, and the iteration is stopped when the number of iterations reaches the preset value (the model parameters are not necessarily converged at this time). As the training progresses, the fine-tuning strategy network further gradually learns how to perform reasonable traffic light operations under various traffic conditions. When performing reinforcement learning, the reward mechanism R is set to the negative value of the average waiting time of the vehicle.
[0061] Step 6: Use the state space data and action space data constructed in step 1 to perform deep learning on the enhanced updated fine-tuning strategy model obtained in step 5. Optimize the model parameters (weight matrix) of each fine-tuning low-rank linear unit and fine-tuning unit linear layer of the enhanced updated fine-tuning strategy model through the deep learning stage, and stop the iteration when the number of iterations reaches the preset value to obtain the deeply updated fine-tuning strategy model.
[0062] Since the basic traffic light scheduling strategy learned by reinforcement learning is optimized based on the features extracted by CNN and the temporal relationship of the Transformer encoder, freezing the parameters of non-newly added structures during deep learning can ensure that these layers maintain their previously learned feature representations and temporal relationships during fine-tuning. The purpose of this approach is to focus on optimizing the performance of the model in a specific traffic environment through fine-tuning without changing the basic feature extraction and temporal relationship modeling capabilities of the original strategy network. At this point, the fine-tuning process will only optimize the newly added structures (each fine-tuned low-rank linear unit and fine-tuned unit linear layer) to refine and strengthen the model's scheduling strategy in specific traffic scenarios.
[0063] Step 6.1: It is necessary to freeze the model parameters of all layers except the newly added structures (each fine-tuned low-rank linear unit and fine-tuned unit linear layer) in the fine-tuning strategy model to ensure that these layers do not change during the fine-tuning process.
[0064] Since the model parameters of the CNN and Transformer layers in the original model have acquired rich feature extraction and pattern recognition capabilities during the previous training process, the main purpose of freezing these parameters is to protect the core structure of the model from interference and ensure that the basic features and knowledge learned can be stable and effective in practical applications. After freezing the parameters of these layers, the model will only update the model parameters of the newly added structure in the subsequent training, and the weights of these layers will not change.
[0065] The specific method of freezing parameters is to set the weight matrices of these layers in the deep learning framework to a non-trainable state. For example, in PyTorch, this can be achieved by setting the requires_grad attribute of each layer to False. In TensorFlow, the trainable attribute of each layer can be set to False.
[0066] Step 6.2: Based on the existing deep learning algorithm, the model parameters of the newly added structure (each fine-tuned low-rank linear unit and fine-tuned unit linear layer) of the fine-tuned strategy model are continuously optimized to make the predicted action distribution as close as possible to the optimal action distribution in the actual historical data, and the iteration is stopped when the number of iterations reaches the preset value (the model parameters are not necessarily converged at this time). As the training progresses, the fine-tuned strategy model that is strengthened and updated further gradually learns how to perform reasonable traffic light operations under various traffic conditions.
[0067] Unlike the deep learning in step 3 and the reinforcement learning in step 5, which require comprehensive learning of the model parameters of all layers, the deep learning in step 6 focuses on the optimization of the newly added structure (each fine-tuned low-rank linear unit and the fine-tuned unit linear layer). The fine-tuning of these layers is mainly aimed at local optimization of the performance of the model in specific traffic scenarios without affecting the CNN and Transformer layers that have been frozen in the original policy network. During fine-tuning training, the feedforward data flow still passes through the original network structure, but at each linear layer, a new low-rank linear layer is inserted in parallel so that the adjusted weights are expressed as W=W0+ΔW, where ΔW=AB T . Matrices A and B are updated during the fine-tuning process through optimization algorithms such as gradient descent. During the fine-tuning process, the weight matrices A and B of the initial fine-tuned low-rank linear units use different initialization strategies: matrix A is usually initialized with Gaussian, while matrix B is initialized to an all-zero matrix, which ensures that ΔW=0 at the beginning, that is, the fine-tuning layer does not interfere with the model. The weight matrix initialized to the fine-tuning unit linear layer is initially a unit matrix to ensure that the output is equal to the input. It is also gradually optimized during fine-tuning training. The initial state of this layer does not change the original output of the model, but during the training process, it will gradually learn adjustments that are beneficial to the scheduling strategy for a specific traffic environment. This design ensures that the model has the ability to flexibly adapt to specific scenarios without losing the basic characteristics of the original strategy network.
[0068] Step 7, determine whether the model parameters of the deep updated fine-tuning strategy model obtained in step 6 converge: if converged, set the current deep updated fine-tuning strategy model as the scheduling strategy model; otherwise, return to step 5.
[0069] In the alternating phase of reinforcement learning and fine-tuning deep learning: first, all parameters of the model are preliminarily trained through reinforcement learning to give the agent preliminary traffic light operation experience. Then, the parameters of CNN and Transformer are fixed, and deep learning fine-tuning of the newly added structural parameters is focused. During the fine-tuning process, the parameters of these newly added layers are optimized in combination with historical data to further improve the performance of the model in specific traffic environments.
[0070] After the deeply updated fine-tuning strategy model converges, it is necessary to evaluate the model's performance to determine whether the model has passed the test. The evaluation mainly simulates a variety of different traffic scenarios and calculates the average vehicle waiting time to measure the model's scheduling strategy effect. If the model performs well in all scenarios and achieves the expected goals, the current deeply updated fine-tuning strategy model will be used as the scheduling strategy model. Otherwise, it is still necessary to return to step 5 to continue learning and training to further optimize the model until the model passes the test. Through this repeated evaluation and training, it is ensured that the model has good adaptability and stability in the actual traffic environment.
[0071] Step 8: Deploy the dispatching strategy model obtained in step 7 to the actual traffic system, and send the collected real-time state space data into the dispatching strategy model to obtain real-time action space data to complete the dispatching of traffic lights.
[0072] Before the scheduling strategy model is deployed to the actual traffic system, the model needs to be converted into a format suitable for the target hardware and software environment, and necessary tests need to be performed to ensure its stability and reliability in real scenarios. After deployment, the model will receive traffic data in real time, dynamically adjust traffic lights according to current traffic conditions, optimize vehicle traffic efficiency, and reduce waiting time. In addition, the model can be continuously optimized in actual applications, that is, by regularly monitoring the performance of the model and adjusting and optimizing it according to changes in the actual traffic environment. By performing long-term fine-tuning of the fine-tuned low-rank linear unit and the fine-tuned unit linear layer, the continuous optimization of the model in actual applications is ensured, enabling it to better cope with dynamic changes in traffic flow.
[0073] It should be noted that although the embodiments of the present invention described above are illustrative, they are not intended to limit the present invention, and therefore the present invention is not limited to the above specific embodiments. Without departing from the principles of the present invention, any other embodiments obtained by those skilled in the art under the guidance of the present invention are deemed to be within the protection of the present invention.
Claims
1. A fine-tuning assisted deep reinforcement learning scheduling method for intelligent traffic lights, characterized by: The steps include: Step 1: Obtain historical traffic data, and construct state space data and action space data after preprocessing; The state space data includes the traffic light state matrix S lights , traffic light state maintenance time matrix T, and vehicle quantity and waiting time matrix W; The action space data includes the traffic light scheduling action matrix A and the traffic light action maintenance time vector D; Step 2: Construct a basic policy network, which consists of a CNN convolutional neural network, a summation layer, a flattening layer, and a Transformer encoder; the input of the CNN convolutional neural network forms the input of the basic policy network, the input and output of the basic policy network are connected to the input of the summation layer, the output of the summation layer is connected to the input of the flattening layer, the output of the flattening layer is connected to the input of the Transformer encoder, and the output of the Transformer encoder forms the output of the basic policy network; Step 3: Use the state space data and action space data constructed in step 1 to perform deep learning on the basic policy network constructed in step 2, comprehensively optimize the model parameters of all layers of the basic policy network through deep learning, and stop iteration when the model parameters converge to obtain the basic policy model; Step 4: structurally adjust the basic strategy model obtained in step 3, that is, connect a fine-tuned low-rank linear unit in parallel to each linear layer of the CNN convolutional neural network of the basic strategy network, connect a fine-tuned low-rank linear unit in parallel to the query layer, key layer and value layer of the multi-head attention layer of the Transformer encoder, connect a fine-tuned low-rank linear unit in parallel to each feed-forward linear layer of the Transformer encoder, and connect a fine-tuned unit linear layer in series after the output of the Softmax activation layer of the Transformer encoder to obtain a fine-tuned strategy network; Step 5: Use the state space data and action space data constructed in step 1 to perform reinforcement learning on the fine-tuning strategy network constructed in step 4. Optimize the model parameters of all layers of the fine-tuning strategy network through the reinforcement learning stage, and stop the iteration when the number of iterations reaches a preset value to obtain a reinforced and updated fine-tuning strategy model. Step 6: Use the state space data and action space data constructed in step 1 to perform deep learning on the enhanced updated fine-tuning strategy model obtained in step 5, optimize the model parameters of each fine-tuning low-rank linear unit and fine-tuning unit linear layer of the enhanced updated fine-tuning strategy model through the deep learning stage, and stop the iteration when the number of iterations reaches a preset value to obtain the deeply updated fine-tuning strategy model; Step 7, judging whether the model parameters of the deep updated fine-tuning strategy model obtained in step 6 have converged: if converged, setting the deep updated fine-tuning strategy model as the scheduling strategy model; otherwise, returning to step 5; Step 8: Deploy the dispatching strategy model obtained in step 7 to the actual traffic system, and send the collected real-time state space data into the dispatching strategy model to obtain real-time action space data to complete the dispatching of traffic lights.
2. The fine-tuning assisted deep reinforcement learning scheduling method for intelligent traffic lights according to claim 1 is characterized in that: Traffic light state matrix S lights ={s k,i,j },s k,i,j Indicates the state of the jth traffic light in the i-th direction of intersection k; Traffic light state maintenance time matrix T = {t k,i,j }, t k,i,j Indicates the maintenance time of the jth traffic light in the i-th direction of intersection k; Vehicle quantity and waiting time matrix W = {w k,i,c },w k,i,k represents the waiting time of vehicles that have not passed through the intersection in the cth lane in the i-th direction of intersection k from the previous state cycle to the current state; Traffic light scheduling action matrix A = {a k,i,j }, a k,i,j represents the dispatching action of the jth traffic light in the i-th direction of intersection k; The traffic light operation maintenance time vector D = {d k,i,j }, d k,i,j Indicates the time that the j-th traffic light in the i-th direction of intersection k remains in the state after executing the scheduling action.
3. The fine-tuning assisted deep reinforcement learning scheduling method for intelligent traffic lights according to claim 1 is characterized in that: CNN convolutional neural network consists of N1 convolution and pooling units and linear layers; where N1 is a set positive integer; Each convolution and pooling unit consists of a convolution layer and a pooling layer. The input of the convolution layer forms the input of the convolution and pooling unit, the output of the convolution layer is connected to the input of the pooling layer, and the output of the pooling layer forms the output of the convolution and pooling unit. N1 convolutions and pooling units are connected in series. The input of the first convolution and pooling unit forms the input of the CNN convolutional neural network. The output of the last convolution and pooling unit is connected to the input of the linear layer, and the output of the linear layer forms the output of the CNN convolutional neural network.
4. The fine-tuning assisted deep reinforcement learning scheduling method for intelligent traffic lights according to claim 1 is characterized in that: The Transformer encoder consists of an input embedding layer, a position encoding layer, a summation layer, N2 attention and feedforward normalization units, a linear layer, and a Softmax activation layer; where N2 is a set positive integer; Each attention and feedforward normalization unit consists of a multi-head attention layer, two normalization layers and two feedforward linear layers. The input of the multi-head attention layer forms the input of the attention and feedforward normalization unit. The input and output of the multi-head attention layer are connected to the input of the first normalization layer. The output of the first normalization layer is connected to the input of the first feedforward linear layer. The output of the first feedforward linear layer is connected to the input of the second feedforward linear layer. The output of the second feedforward linear layer and the input of the first feedforward linear layer are connected to the input of the second normalization layer. The output of the second normalization layer forms the output of the attention and feedforward normalization unit. The input of the input embedding layer and the input of the position encoding layer form the input of the Transformer encoder, the output of the input embedding layer and the output of the position encoding layer are connected to the input of the summation layer, the output of the summation layer is connected to the input of the first attention and feedforward normalization unit of N2 series-connected attention and feedforward normalization units, the output of the last attention and feedforward normalization unit of N2 series-connected attention and feedforward normalization units is connected to the input of the linear layer, the output of the linear layer is connected to the input of the Softmax activation layer, and the output of the Softmax activation layer forms the output of the Transformer encoder.
5. The fine-tuning assisted deep reinforcement learning scheduling method for intelligent traffic lights according to claim 1 is characterized in that: The fine-tuning low-rank linear unit is composed of a first low-rank linear layer and a second low-rank linear layer; the input of the first low-rank linear layer forms the input of the fine-tuning low-rank linear unit, the output of the first low-rank linear layer is connected to the input of the second low-rank linear layer, and the output of the second low-rank linear layer forms the output of the fine-tuning low-rank linear unit; The dimension of the weight matrix of the first low-rank linear layer is d×r, and the initial value of the weight matrix is determined by Gaussian initialization; the dimension of the weight matrix of the second low-rank linear layer is r×d, and the initial value of the weight matrix is determined by zero initialization; where d is the dimension of the weight matrix of the parallel layer, and r is the set rank.
6. The fine-tuning assisted deep reinforcement learning scheduling method for intelligent traffic lights according to claim 1 is characterized in that: The dimension of the weight matrix of the fine-tuned unit linear layer is the same as that of the weight matrix of the Softmax activation layer, and the initial value of the weight matrix is the unit matrix.