A traffic signal control migration method based on multi-expert offline fusion

Abstract

This invention provides a traffic signal control transfer method based on multi-expert offline fusion, belonging to the fields of intelligent transportation and artificial intelligence. The traffic signal control problem is modeled as a partially observable Markov decision process; each intersection in the target road network is controlled by an agent; each agent model includes an encoder, a decoder, and an action value network; the action value network is shared by all agents in the target road network; agent models corresponding to different traffic flow patterns are pre-trained under different traffic flow patterns in the target road network to construct an expert policy pool; offline fusion training is performed on the models in the expert policy pool based on historical data to generate target policies that are applied to the target road network. By constructing an expert pool and combining it with limited historical data in the target domain, dynamic credibility weights are generated using prediction errors, and expert outputs are fused in the action value function space, thereby automatically weakening the influence of unsuitable experts and reducing the risk of negative transfer during the optimization process.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation and artificial intelligence technology, and in particular to a traffic signal control migration method based on multi-expert offline fusion. Background Technology

[0002] In modern urban transportation systems, the control of traffic lights at intersections directly impacts traffic efficiency and travel safety. With the expansion of urban scale and fluctuations in traffic volume, traffic conditions vary significantly at different intersections and at different times. In the field of urban traffic signal control, with the continuous increase in road network scale and traffic flow complexity, multi-agent reinforcement learning has gradually become a hot topic in research and engineering applications, used to achieve adaptive and collaborative intersection timing optimization. However, existing transfer or fine-tuning schemes typically rely on extensive online interactions in the target domain to complete model adaptation. Such large-scale online training in real road networks is not only costly but also accompanied by safety risks and uncontrollable trial-and-error costs; moreover, the target domain often only provides limited historical trajectories or allows only a very small number of short-term samples. More importantly, under conditions of few samples or purely offline conditions, traditional single-source policy selection or online policy distillation methods are prone to problems such as negative transfer, overestimation of the action-value function, and training instability, making it difficult to meet the stringent requirements of safety, stability, and rapid availability for engineering deployment.

[0003] Currently, reinforcement learning-based traffic signal control methods still face numerous challenges in practical applications. First, most methods require continuous data collection and trial-and-error training in simulation environments or on real roads. This is not only time-consuming and labor-intensive but may also introduce safety hazards in real traffic, making it difficult to guarantee reliability. Second, many models are only trained in specific scenarios. For example, traffic flow is dense and volatile during rush hour, while it is relatively sparse and stable during off-peak hours or at night. Furthermore, there are significant differences in lane structure, phase settings, and traffic demand between different intersections. These differences often make control strategies trained in a single scenario difficult to apply directly to new time periods or intersections, easily leading to decreased efficiency or incompatibility.

[0004] Furthermore, most existing transfer learning methods simply "cram" existing models into new environments, ignoring the differences between the source and target environments. This not only easily leads to negative effects but also fails to effectively utilize the experience inherent in multiple existing models. Finally, most current methods rely on online data collection and lack the ability to effectively train and make decisions under purely offline data conditions. Frequently collecting high-quality data in real traffic is both difficult and risky. In practical deployments, large-scale online training not only requires continuous interaction with the environment, consuming significant time and resources, but may also pose safety hazards to real traffic operations due to repeated trial and error. Therefore, traffic managers prefer to utilize existing historical data and models trained in different scenarios to quickly obtain usable signal control strategies for new environments.

[0005] Therefore, there is an urgent need for a traffic signal control migration method based on multi-expert offline fusion. Summary of the Invention

[0006] In view of this, the present invention provides a traffic signal control transfer method based on offline fusion of multiple experts. The transfer learning framework based on offline methods can quickly obtain usable and stable traffic signal control strategies by using only a small amount of short-term sampled historical data of the target domain, through techniques such as expert credibility measurement and fusion of multi-expert action value distribution. This balances safety, efficiency and generalization ability, and is suitable for engineering deployment in actual traffic systems.

[0007] Therefore, the present invention provides the following technical solution: A traffic signal control migration method based on multi-expert offline fusion includes: The traffic signal control problem is modeled as a partially observable Markov decision process; Each intersection in the target road network is controlled by an agent; each agent model includes: an encoder, a decoder, and an action value network; the action value network is shared by all agents in the target road network; Under different traffic flow patterns in the target road network, pre-train intelligent agent models corresponding to different traffic flow patterns, and construct an expert policy pool; Based on historical data, offline fusion training is performed on various models in the expert policy pool to generate target policies that are then applied to the target road network.

[0008] Furthermore, the partially observable Markov decision process includes:

[0009] in, This represents the number of intelligent agents, which is equal to the number of intersections in the traffic network, meaning that each intersection is controlled by one intelligent agent. Represents a finite set of states. This represents the set of observations; at a certain moment, each agent starts from the global state. Obtaining local observations That is, each intelligent agent All observations can be obtained from the global state. ; Represents the set of joint actions, where at any given time, each agent... Based on its own local observation Make decisions and carry out actions The actions taken by all agents constitute a joint action. And it acts on the environment; The system transitions to the next state using a state transition probability function, and receives a reward from environmental feedback. The state transition probability function is as follows:

[0010] in, Represents the state space distribution. For the reward collection; This is a discount factor used to discount future rewards; This represents the state at the next moment.

[0011] Furthermore, the encoder captures the spatial topological relationships between intersections based on a multi-head attention mechanism, mapping local observation information into high-dimensional feature representations; The decoder recursively predicts the evolution of traffic conditions in multiple future steps based on the encoded features. The action value network comprehensively encodes and decodes the Q values ​​of each phase, selects the optimal phase through a greedy strategy, and drives the agent to achieve optimal control of the intersection signal.

[0012] Furthermore, the loss function for pre-training is:

[0013] The time-series difference loss is:

[0014]

[0015] The predicted loss is:

[0016] In the formula, For hyperparameters, , and The parameters corresponding to the encoder, decoder, and motion value network; and These are the parameters of the target network; Where N is the batch size; N is the number of intersections; and The intersections at the current time and observation information from surrounding neighbors, Indicates intersection The action performed by the intelligent agent at the current moment. and Indicates the next moment at the intersection The observation information of the local area and the observation information of its neighbors; For intelligent agents at intersections The optimization objective of the action-value network; For the intersection The action that the intelligent agent will perform in the next moment.

[0017] Furthermore, the offline fusion training of various models in the expert policy pool based on historical data to generate the target policy includes: Any pre-trained strategy from the expert pool is run on the target road network to collect historical datasets; A shared encoder is constructed by weighted averaging of all encoder parameters in the expert pool. Within a batch, the shared encoder extracts the feature representation of the current state, and the independent decoder of each expert model predicts the corresponding next state result based on the feature representation; Calculate the mean Euclidean distance between the actual results and the predicted results within the batch; The fusion weights are calculated based on the mean. The action value distributions of each expert model are weighted and fused based on the fusion weights to serve as the output of the target network. The temporal difference error of the offline fusion training loss function is constructed using the output of the target network; The goal is to achieve stable convergence of network parameters by minimizing the loss function of offline fusion training, and then output the target policy.

[0018] Furthermore, the offline fusion training loss function includes:

[0019] in, To integrate training time-series difference errors, For environmental prediction errors, It is a conservative regularization term in CQL; Let KL divergence be denoted as KL divergence.

[0020] Furthermore, the fusion training temporal difference error includes:

[0021]

[0022] in, The first in the batch The temporal difference objective for the nth sample; for the nth sample... Sample , To perform the action The instant reward obtained afterward; To perform the action Then, the state of the environment at the next moment is observed; For state observation All possible actions to be performed; This is the value function of expert actions after fusion.

[0023] Furthermore, the CQL conservative regularization term includes:

[0024] in, It is a hyperparameter that controls the strength of conservatism.

[0025] Advantages and positive effects of the present invention: This invention constructs an expert pool by pre-training multiple expert models on different road segments or at different time periods. It then combines this pool with limited historical data from the target domain and utilizes prediction errors to generate dynamic credibility weights. These weights are then fused within the action-value function space, automatically mitigating the influence of unsuitable experts and reducing negative transfer risk during optimization. This method enables the target policy to achieve stable and efficient learning even with limited access to offline data from the target domain. The overall approach reduces reliance on online interaction with the target domain while improving cross-time-period adaptability and convergence stability. It boasts advantages such as strong security, high efficiency, and good engineering feasibility, providing a practical solution for the application of reinforcement learning in high-risk, highly constrained traffic scenarios.

[0026] The offline fusion phase in this method can improve robustness, reduce risk costs, and enhance generalization ability: When faced with data noise or distribution drift, the value of a single expert's actions may be distorted, while fusion mechanisms can buffer biases through collective wisdom, avoiding drastic fluctuations in the learning objective. Since the temporal difference objective is entirely derived from offline data and expert knowledge, no additional online interaction is required to construct a reasonable training signal, fundamentally avoiding the trial-and-error risks in real traffic environments. Different experts often possess adaptation experience for different time periods and intersections; fusion can maximize the exploitation of this complementarity, thereby enabling the target strategy to have generalization performance across time periods and scenarios. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a framework diagram of the traffic signal control migration method based on multi-expert offline fusion in an embodiment of the present invention; Figure 2 This is a framework diagram of the offline fusion stage in an embodiment of the present invention; Figure 3 This is a graph showing the overall performance data on the Jinan dataset in this embodiment of the invention; Figure 4 This is a graph showing the overall performance data on the Hangzhou dataset in this embodiment of the invention; Figure 5 This is the training and testing curve on the Jinan dataset in this embodiment of the invention; Figure 6 This is the training and testing curve on the Hangzhou dataset in this embodiment of the invention; Figure 7 This is a comparison diagram of ablation experiments in the embodiments of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] This invention provides a traffic signal control migration method based on multi-expert offline fusion. It formalizes the traffic signal control task, defines core elements such as state, action, and reward, and clarifies the optimization objective by leveraging the characteristics of multi-agent interaction. By combining source domain expert knowledge with target domain historical data, it achieves a low-risk, rapidly adaptable traffic signal control scheme.

[0032] 1. In a traffic network, there are multiple intersections. Coordinated control of traffic lights at each intersection can effectively alleviate traffic congestion and improve traffic efficiency. The task of multi-intersection traffic signal control can be viewed as a multi-agent collaborative problem, modeled as a Partially Observable Markov Decision Process (POMDP):

[0033] in, This represents the number of intelligent agents, which is equal to the number of intersections in the traffic network, meaning that each intersection is controlled by one intelligent agent. Represents a finite set of states. This represents the set of observations; at a certain moment, each agent starts from the global state. Obtaining local observations That is, each intelligent agent All observations can be obtained from the global state. ; Represents the set of joint actions, where at any given time, each agent... Based on its own local observation Make decisions and carry out actions The actions taken by all agents constitute a joint action. And it acts on the environment.

[0034] The system transitions to the next state using a state transition probability function, and receives a reward from environmental feedback. State transition probability function:

[0035] in, Represents the state space distribution. For the reward set. This is a discount factor used to discount future rewards; This represents the state at the next moment.

[0036] 2. Define a univariate in a Markov decision process: 1) Define observation information: When an intelligent agent interacts with its environment, it needs to make decisions based on the environment's state information. In traffic signal control tasks, the global state information is usually vast and complex; therefore, the agent needs to extract concise and effective observation information from it. The selection of observations should be based on the real environment, ensuring easy access so that the agent can more accurately understand the environment and make reasonable decisions. Therefore, for an agent at a given moment... The observation information is as follows:

[0037] in, The phase signal representing the current moment at the intersection can be directly obtained by observing the state of the traffic lights at the corresponding moment. Enter the lane at the current moment The feature vectors of waiting vehicle queues can be obtained through Discrete Traffic State Encoding (DTSE).

[0038] 2) Define the action: When the next green light signal begins, a specific phase is directly selected as the green light signal. At this point, the number of actions corresponds to the number of selectable phases. The action space includes four phases, which are mutually exclusive. Among them, action 0 indicates a left turn in the east-west direction (WL-EL), action 1 indicates a left turn in the north-south direction (SL-NL), action 2 indicates going straight in the east-west direction (WT-ET), and action 3 indicates going straight in the north-south direction (NT-ST).

[0039] 3) Define rewards: To minimize vehicle travel time in the system, this can be achieved by reducing the length of the waiting queue at each intersection's entry lane. Therefore, the reward is defined as:

[0040] in, express Time Crossing Lane The length of the waiting queue. The smaller this value, the shorter the waiting time for vehicles at the intersection, and the higher the reward.

[0041] 3. Construct an intelligent agent model and build an expert policy pool through pre-training in multi-traffic scenarios.

[0042] To improve the adaptability and generalization ability of traffic signal control strategies in multiple scenarios, an expert strategy pool is constructed through pre-training in multi-traffic scenarios.

[0043] Heterogeneous intelligent agent model, including: encoder, decoder and action value network; Encoder: Based on a multi-head attention mechanism, it captures the spatial topological relationships between intersections and maps local observation information into high-dimensional feature representations with structure awareness capabilities; Decoder: Based on the encoded features, it recursively predicts the evolution of traffic conditions in multiple future steps, constructs an environmental dynamics model, and provides forward-looking information for decision-making; Action Value Network: The policy network integrates encoding and decoding to output the Q value of each phase. It selects the optimal phase through a greedy strategy to drive the agent to achieve optimal control of the intersection signal.

[0044] During the pre-training phase, multiple heterogeneous agent models are trained for different traffic flow patterns under the same road network. By jointly optimizing temporal difference error and state prediction error, decision-making ability and environmental modeling ability are improved simultaneously. The overall loss function includes temporal difference error and prediction error. Temporal difference error is used to optimize the action value function and ensure the consistency of value estimation; prediction error is used to measure the difference between the decoder output and the actual observation, thereby improving the model's ability to model environmental dynamics.

[0045] After pre-training, the model performance is comprehensively evaluated based on metrics such as average passage time, throughput, and queue length. The best-performing models are then selected to build an expert policy pool. This pool contains multiple pre-trained agent models with multi-scenario generalization capabilities, providing a rich policy foundation for transfer learning in subsequent target tasks.

[0046] The total loss function of the agent model during pre-training is defined as:

[0047] The time-series difference loss is:

[0048]

[0049] The predicted loss is:

[0050] In the formula, For hyperparameters, , and The parameters corresponding to the encoder, decoder, and motion value network; and These are the parameters of the target network; Where N is the batch size; N is the number of intersections; and The intersections at the current time and observation information from surrounding neighbors, Indicates intersection The action performed by the intelligent agent at the current moment. and Indicates the next moment at the intersection The observation information of the local area and the observation information of its neighbors; For intelligent agents at intersections The optimization objective of the action-value network; For the intersection The action that the intelligent agent will perform in the next moment.

[0051] 4. Offline fusion training based on historical data: After obtaining a pre-trained pool of expert policies, high-performance target policies can be safely and stably trained using a small amount of historical data collected from the target environment, with little or no online interaction, through expert evaluation and knowledge fusion.

[0052] 1) Expert prediction error measurement: The historical dataset D was collected by running any pre-trained policy from the expert pool in the target environment for a short time, and contains transition tuples. Because different expert encoders represent the state space differently, directly comparing prediction results is easily affected by inconsistencies in feature scale and representation methods.

[0053] A shared encoder is constructed by weighting and averaging the parameters of all encoders in the expert pool. This design is used to capture the common structure of the state space and unify the feature representation space. This design ensures that predictions from different experts can be compared in the same representation domain, enhancing the stability and interpretability of the metric.

[0054] During training, a mini-batch is randomly sampled from the experience replay pool each time. For the k-th sample... First, the observation-action pair Input the data into expert model i to obtain the expert's predicted next state. .

[0055] A shared encoder is used to map the predicted results to the actual results:

[0056]

[0057] in, To share the encoder's prediction of the next state of the i-th expert model, This is the actual result of the shared encoder for the current state.

[0058] The difference between the predicted results and the actual results of the shared encoder is calculated using Euclidean distance:

[0059] This error characterizes the prediction accuracy of expert i on sample j. To obtain the overall batch-level performance, the sample errors are aggregated within the mini-batch:

[0060] in, B This represents the mini-batch size. A smaller value indicates a higher degree of fit between the expert and the current data distribution. Finally, a negative softmax operation is used to convert the batch error of each expert into normalized weights.

[0061] in, This reflects the credibility of expert i in the current batch and will be used as a dynamic weight in subsequent action value fusion processes. By repeating this process in each mini-batch, the system can continuously update expert weights, thereby achieving data-driven adaptive expert selection.

[0062] 2) Fusion of multi-expert action value distribution: After the similarity weights are calculated, the action value distributions of each expert model can be weighted and fused.

[0063] For each sample in the mini-batch The input is fed into the encoder-action value network (Encoder-Q net) of each expert model. The action value network, based on the duel-style DQN architecture, outputs the value of each expert's actions for that observation across all possible actions. After obtaining the action value distributions of N experts, the aforementioned Softmax weights are used... We weight each element according to its distribution dimension to form the fused expert action value function:

[0064] Based on the aforementioned fusion Q function, for the first in the batch TD target constructed from samples :

[0065] Unlike traditional target network estimation, here... It is itself a product of knowledge transfer: it not only incorporates the experience of multiple experts in the source domain, but also adaptively adjusts the weights based on the similarity of the target domain data. Therefore, each iteration is equivalent to reorganizing a data-driven prior that best fits the target domain, which is then used to replace the estimation of a single network.

[0066] 3) Joint optimization mechanism: After calculating and fusing weights and constructing the target action value, multi-objective joint optimization is performed to further improve the stability and performance of offline reinforcement learning. Multiple loss functions are constructed for both the action value network and the policy network, and then uniformly optimized and updated. The loss function mainly consists of four parts: temporal difference error, environment prediction error, CQL conservative regularization term, and KL divergence. Stable convergence of network parameters is achieved by jointly minimizing these losses.

[0067] During the offline training phase, the loss function still consists of two parts: the target action error (TD error) and the prediction error, and its form is basically the same as in the pre-training phase. The target action value in the TD error is no longer directly generated by the target action value network, but is obtained through the fusion of multi-expert action value functions, thus introducing expert knowledge into the optimization process and improving the reliability of target estimation. Its loss function... Defined as:

[0068] To enhance the safety and robustness of offline reinforcement learning, a conservative regularization term from Conservative Q-Learning (CQL) is introduced to suppress the risk of unreasonable overestimation of state-action pairs in the action-value network and limit policy bias. The CQL loss is defined as:

[0069] in, It is a hyperparameter that controls the strength of conservatism; During the policy network update process, it is necessary to maximize the expected return of the policy under the action-value network, while also aiming for the policy initialization to more closely resemble expert behavior to reduce instability in the early stages of training. To this end, KL divergence is incorporated as a constraint term to measure the target policy. With expert strategy The distribution differences under the same conditions. Its KL-constrained loss term is:

[0070] in, The normal intensity coefficient, This is an offline experience dataset. This item guides... It converges to the expert policy distribution more quickly in the early stages, improving training stability.

[0071] The total loss function, integrating all its components, is expressed as follows:

[0072] To enhance training stability, a soft update strategy is employed during training to iteratively update the target action value network. This involves smoothly merging the current action value network parameters with historical target network parameters proportionally, mitigating training oscillations caused by drastic policy changes. This multi-objective optimization mechanism provides reliable support for the transfer and deployment of the model in complex traffic environments.

[0073] 5. After offline fusion training is completed, the target policy is applied to the target domain environment without further online calibration. This is used to verify the applicability of the offline policy in real traffic environments.

[0074] The trained strategy is tested in the target environment for a limited number of rounds, and its performance is evaluated by collecting metrics such as average passage time and queue length. Unlike traditional methods that require extensive online fine-tuning, this method's strategy is already deployable offline. Therefore, online validation does not involve additional parameter updates or incur new training overhead, thus ensuring the method's safety and low-cost advantages. Example Experiments were conducted using CityFlow, a microscopic traffic simulation platform for urban roads. To fully verify the method's adaptability in diverse traffic scenarios, two representative real-world urban road networks were selected for testing. Each road network scenario was configured with multiple datasets of different traffic flow distributions to simulate dynamic traffic conditions over one hour, thereby achieving a comprehensive evaluation of the method's generalization performance.

[0075] 1. Comparative Experiment This experiment uses the road networks of two real cities as test scenarios: Jinan Road Network: This road network includes 12 intersections and 72 road connecting sections. It is distributed in a grid pattern, with a regular structure, strong controllability and representativeness, and is suitable for training and evaluation of multi-agent traffic signal control tasks.

[0076] Hangzhou Road Network: This road network is constructed based on actual road planning, covering 16 signalized intersections and 112 road segments, forming a large urban traffic sub-network with a more complex road connection structure and dynamic traffic flow characteristics.

[0077] In the experiment, various traffic flow scenarios were designed on the real-world road networks of Jinan and Hangzhou to construct a multi-source to target task transfer test environment. In the Jinan road network, three different traffic flow data distributions were used, numbered jn1, jn2, and jn3. jn1 and jn2 were used as source task data, and jn3 as the target transfer task. In the Hangzhou road network, four traffic flow data distributions were used, hz1, hz2, hz3, and hz4. hz1, hz2, and hz4 were used for training the source policy, and hz3 was the target transfer task. This design verifies the effectiveness of the proposed method in multi-source knowledge integration and transfer generalization.

[0078] Average travel time was used as the primary evaluation metric to quantify the impact of signal control strategies on traffic flow. In comparative experiments, CoLight, MPLight, and AttendLight were selected as online reinforcement learning algorithms, while BCQ and CQL were selected as offline reinforcement learning algorithms. The transferability and generalization performance of the proposed methods under different traffic flow conditions were compared and evaluated.

[0079] The method of this invention was applied to two typical road network environments, jn3 and hz3, and compared with existing methods. Figure 3 and Figure 4 As can be seen, the average travel time of the method of this invention is significantly lower than that of existing methods in both road network environments, demonstrating superior traffic signal control performance. Especially in complex road network scenarios, the method of this invention consistently maintains optimal performance, indicating its good generalization ability and stability in different environments.

[0080] Figure 5 and Figure 6 To train and test the curves, the proposed method significantly outperformed the traditional offline reinforcement learning algorithms BCQ and CQL under different traffic flow scenarios in two real road networks, jn3 and hz3. In the jn3 scenario, the proposed algorithm demonstrated excellent and stable performance from the beginning of training, with the average travel time consistently remaining at a minimum level of around 400; while the BCQ algorithm's curve fluctuated wildly, and the CQL algorithm, despite gradual optimization, still exhibited significant oscillations. In the hz3 scenario, the advantages of the proposed method were even more pronounced. Its strategy reached optimal performance in the early stages of training (epoch=2) and maintained this excellent level, demonstrating outstanding "out-of-the-box" performance; the comparative algorithms BCQ and CQL required numerous iterations to converge slowly, and their final performance was far inferior to that of the proposed method.

[0081] 2. Ablation test By using the controlled variable method, the dynamic weight calculation mechanism (denoted as O3L-WDW), the regularization constraint in the multi-objective joint loss function (denoted as O3L-WR), and the multi-expert knowledge reuse strategy (denoted as O3L-WME) were removed in sequence. Under the same hyperparameters and test environment, the contribution of each module to the overall performance was quantitatively analyzed with average travel time as the core indicator.

[0082] like Figure 7 As shown, based on the ablation experiments in the jn3 scenario, the complete method exhibits the best performance, with its travel time continuously decreasing during training and eventually stabilizing at a low level. In contrast, each ablation model shows significant performance deficiencies: O3L-WDW has limited optimization effects, with its travel time consistently exceeding the benchmark; O3L-WR exhibits drastic performance fluctuations and is extremely unstable during training; and O3L-WME shows optimization stagnation. The experimental results fully demonstrate that the dynamic weight mechanism is key to achieving accurate knowledge fusion, regularization constraints are the foundation for ensuring training stability, and the multi-expert fusion strategy is the core of improving model performance; these three are indispensable components of this method.

[0083] In summary, the expert-policy-assisted multi-agent offline transfer reinforcement learning method proposed in this invention significantly improves training efficiency and generalization performance in complex traffic environments by fully utilizing existing expert knowledge to guide policy learning without requiring online interaction. Validation in real-world traffic tasks across different regions and durations demonstrates excellent adaptability and stability, exhibiting strong engineering applicability and potential for widespread adoption.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A traffic signal control migration method based on multi-expert offline fusion, characterized in that, include: The traffic signal control problem is modeled as a partially observable Markov decision process; Each intersection in the target road network is controlled by an intelligent agent; Each agent model includes: an encoder, a decoder, and an action value network; the action value network is shared by all agents in the target road network; Under different traffic flow patterns in the target road network, pre-train intelligent agent models corresponding to different traffic flow patterns, and construct an expert policy pool; Based on historical data, offline fusion training is performed on various models in the expert policy pool to generate target policies that are then applied to the target road network.

2. The method according to claim 1, characterized in that, The partially observable Markov decision process includes: in, This represents the number of intelligent agents, which is equal to the number of intersections in the traffic network, meaning that each intersection is controlled by one intelligent agent. Represents a finite set of states. This represents the set of observations; at a certain moment, each agent starts from the global state. Obtaining local observations That is, each intelligent agent All observations can be obtained from the global state. ; Represents the set of joint actions, where at any given time, each agent... Based on its own local observation Make decisions and carry out actions The actions taken by all agents constitute a joint action. And it acts on the environment; The system transitions to the next state using a state transition probability function, and receives a reward from environmental feedback. The state transition probability function is as follows: in, Represents the state space distribution. For the reward collection; This is a discount factor used to discount future rewards; This represents the state at the next moment.

3. The method according to claim 1, characterized in that, The encoder captures the spatial topological relationships between intersections based on a multi-head attention mechanism, mapping local observation information into high-dimensional feature representations; The decoder recursively predicts the evolution of traffic conditions in multiple future steps based on the encoded features. The action value network comprehensively encodes and decodes the Q values ​​of each phase, selects the optimal phase through a greedy strategy, and drives the agent to achieve optimal control of the intersection signal.

4. The method according to claim 1, characterized in that, The loss function for the pre-training is: The time-series difference loss is: The predicted loss is: In the formula, For hyperparameters, , and The parameters corresponding to the encoder, decoder, and motion value network; and These are the parameters of the target network; Where N is the batch size; N is the number of intersections; and The intersections at the current time and observation information from surrounding neighbors, Indicates intersection The action performed by the intelligent agent at the current moment. and Indicates the next moment at the intersection The observation information of the local area and the observation information of its neighbors; For intelligent agents at intersections The optimization objective of the action-value network; For the intersection The action that the intelligent agent will perform in the next moment.

5. The method according to claim 1, characterized in that, The offline fusion training of models in the expert policy pool based on historical data to generate the target policy includes: Any pre-trained strategy from the expert pool is run on the target road network to collect historical datasets; A shared encoder is constructed by weighted averaging of all encoder parameters in the expert pool. Within a batch, the shared encoder extracts the feature representation of the current state, and the independent decoder of each expert model predicts the corresponding next state result based on the feature representation; Calculate the mean Euclidean distance between the actual results and the predicted results within the batch; The fusion weights are calculated based on the mean. The action value distributions of each expert model are weighted and fused based on the fusion weights to serve as the output of the target network. The temporal difference error of the offline fusion training loss function is constructed using the output of the target network; The goal is to achieve stable convergence of network parameters by minimizing the loss function of offline fusion training, and then output the target policy.

6. The method according to claim 1, characterized in that, The offline fusion training loss function includes: in, To integrate training time-series difference errors, For environmental prediction errors, It is a conservative regularization term in CQL; Let KL divergence be denoted as KL divergence.

7. The method according to claim 6, characterized in that, The fusion training temporal difference error includes: in, The first in the batch The temporal difference objective for the nth sample; for the nth sample... Sample , To perform the action The instant reward obtained afterward; To perform the action Then, the state of the environment at the next moment is observed; For state observation All possible actions to be performed; This is the value function of expert actions after fusion.

8. The method according to claim 6, characterized in that, The CQL conservative regularization terms include: in, It is a hyperparameter that controls the strength of conservatism.

Images