A world model based regional boundary control method and related device

Abstract

The application discloses a regional boundary control method based on a world model and related equipment. A world model for a regional macroscopic traffic system is constructed, the model learns a hidden space representation method of a macroscopic traffic state, and a future evolution process of the regional traffic state is deduced in the hidden space, the model has a continuous multi-step rolling deduction capability; then, based on a reinforcement learning and a strategy optimization mechanism, operation effects of a candidate boundary control sequence in a future time domain are evaluated, so that prospective boundary regulation and control facing a future state are realized; and an online adaptive updating mechanism based on a prediction error is introduced, so that robustness, generalization capability and engineering deployment feasibility in a complex traffic environment are improved. The application constructs an integrated closed-loop technical scheme of dynamic modeling, rolling deduction, strategy evaluation and online correction for a regional traffic control scene, so that boundary control decision can be driven from a static threshold to a future state, and the application can be widely applied to the technical field of intelligent traffic control.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic control technology, and in particular to a regional boundary control method and related equipment based on a world model. Background Technology

[0002] With the continuous growth of urban traffic demand, urban road networks are increasingly exhibiting characteristics such as high density, strong coupling, strong random disturbances, and significant nonlinearity. Traditional local optimization methods targeting single intersections or road segments, while improving traffic efficiency in local time periods, often fail to consider the overall operational status of large-scale road networks, easily leading to problems such as increased local traffic flow and exacerbated regional congestion. Macro-level traffic state modeling and boundary control at the regional scale have become an important research direction in intelligent traffic control. The macro-basic graph is a classic macro-model describing the relationship between the cumulative number of vehicles in a region and its output capacity. Its core idea is to control the inflow flow at the regional boundary to maintain the cumulative number of vehicles in the region near the efficient operating range, thereby suppressing oversaturation congestion within the region. Based on this approach, academia and industry have developed several technical routes, including regional boundary control based on macro-basic graphs, model predictive control based on explicit traffic flow models, and traffic control based on reinforcement learning.

[0003] In existing technologies, the first type of approach is a regional boundary control method based on macroscopic basic maps and model predictive control. This type of method typically first calibrates the regional macroscopic basic map based on historical or simulation data, and then solves for the release ratio, boundary signal timing, or access rate at boundary entrances based on optimal cumulative volume, congestion thresholds, or vehicle conservation relationships. This type of method has advantages such as clear physical meaning and a clear engineering implementation path, but it relies heavily on pre-constructed static macroscopic relationships and struggles to accurately characterize the dynamic impact of control inputs on future traffic conditions under complex demand disturbances. Its technical focus lies in macroscopic basic map modeling, operational situation awareness, and regional equilibrium management. The second type of approach is a boundary control method based on explicit traffic flow models and reinforcement learning, which completes boundary adjustment by combining regional dynamic conservation relationships with policy learning. The third type of approach is a traffic state prediction or model-driven reinforcement learning approach, which learns traffic evolution patterns through recurrent neural networks, long short-term memory networks, Transformers, or environmental inference models, and then assists in control decisions.

[0004] The shortcomings of existing technologies are mainly reflected in the following aspects. First, control methods based on macroscopic basic maps are better at describing the static or quasi-static relationship between regional cumulative output and network output, but they are difficult to explicitly characterize the dynamic process of how control inputs affect the evolution of future traffic states. Therefore, in scenarios with strong disturbances, non-steady states, and rapid demand fluctuations, they often fail to provide sufficiently accurate basis for forward-looking boundary control. Second, although predictive control methods based on explicit traffic flow models have clear physical mechanisms, model parameter identification is complex, and they are sensitive to regional division, demand estimation, and model structure assumptions. When applied to complex large-scale road networks, they often suffer from high modeling costs and insufficient adaptability. Third, although model-free reinforcement learning methods can learn complex nonlinear control strategies, they usually require a large number of environmental interaction samples, resulting in high training costs, and are prone to performance fluctuations when traffic demand distribution or operating environment changes. Fourth, in existing schemes, the prediction module and control module are often isolated from each other; prediction is only used for state estimation, and control is only executed based on current observations. Summary of the Invention

[0005] The main objective of this invention is to propose a method, apparatus, electronic device, storage medium, and program product for regional boundary control based on a world model, aiming to solve at least one problem in the prior art.

[0006] To achieve the above objectives, one aspect of this invention proposes a regional boundary control method based on a world model, the method comprising: Acquire historical operational data of the regional transportation system and construct a state transition sample set that includes the macroscopic state of the previous moment, boundary control inputs, and the macroscopic state of the next moment. The world model is jointly trained using a state transition sample set so that it learns the mapping relationship between the macroscopic state and the latent space, and learns the dynamic evolution relationship from the macroscopic state and boundary control input in the previous moment to the macroscopic state in the next moment in the latent space; the world model includes an encoder, an inference module and a decoder. Use the current time as the target time; Obtain the real macroscopic traffic state at the target time and map it into target latent space variables using an encoder; Based on the real macro-level traffic conditions, candidate boundary control sequences are generated using a reinforcement learning policy network. Based on the target latent space variables and candidate boundary control sequences, the future macro-state of traffic for multiple control cycles is obtained by using the inference module and decoder to perform rolling inference. Based on the future macro-level traffic conditions, the cumulative benefits of each candidate boundary control sequence are obtained by evaluating the evaluation network. Execute the target boundary control action based on the candidate boundary control sequence with the highest cumulative revenue, take the next time of the target time as the target time, return to execute the step of obtaining the real macro traffic state at the target time, and continue to perform regional boundary control; The prediction error is constructed based on the future macro-state of traffic and the actual macro-state of traffic at the same moment. The world model is then corrected based on the prediction error triggering an update mechanism.

[0007] In some embodiments, joint training of a preset world model using a state transition sample set includes the following steps: Input the previous macroscopic state from the state transition sample set into the encoder, process the output to train the latent space variables; The action embedding network is used to map the boundary control input into action feature vectors. The action feature vectors are concatenated with the training latent space variables to form a fusion vector. The historical sequence is constructed based on the fusion vectors of multiple consecutive time points. The historical sequence is then input into the inference module, which processes and outputs the latent space variables for the next time point. Input the hidden space variables of the next time step into the decoder, process the output, and train and infer the macroscopic state. A loss function is constructed based on the macroscopic state generated during training and the macroscopic state at the next time step. The parameters of the encoder, the inference module, and the decoder are jointly optimized using the loss function, and then the parameters are frozen.

[0008] In some embodiments, a loss function is constructed based on the macroscopic state derived from training and the macroscopic state at the next time step, including the following steps: The prediction loss and multi-step rolling simulation loss are constructed based on the macro state of training and the macro state of the next moment. Based on the macro-state of training and the macro-state of the next moment, a data consistency loss is constructed through traffic flow statistical consistency constraints. The prediction loss, data consistency loss, and multi-step rolling inference loss are weighted and fused to obtain the total loss function of the model.

[0009] In some embodiments, candidate boundary control sequences are generated using a reinforcement learning policy network based on real-world traffic conditions, including the following steps: Input the real-world traffic macro-state into the reinforcement learning policy network and process the output action distribution; Using the action distribution as the initial sampling distribution, multiple sets of candidate boundary control sequences are generated in the rolling simulation time domain using the cross-entropy optimization method; Each candidate boundary control sequence includes a series of action vectors for multiple future control cycles, and each action vector is obtained based on a preset boundary flow regulation constraint.

[0010] In some embodiments, each set of candidate boundary control sequences includes continuous action vectors for multiple future control cycles. Based on the target latent space variables and the candidate boundary control sequences, the future macroscopic traffic state for multiple future control cycles is obtained by rolling inference using an inference module and a decoder, including the following steps: For each set of candidate boundary control sequences, starting from the target latent space variables, iteratively perform the following operations: The target latent space variables are used as candidate latent space variables, and the action vector of the first control cycle in the candidate boundary control sequence is used as the candidate action vector. The candidate latent space variables are concatenated with the candidate action vectors and then input into the inference module to predict the predicted latent space variables for the next control cycle. The predicted latent space variables are input into the decoder to obtain the future macro-level traffic conditions for the corresponding control period. The predicted latent space variables are used as candidate latent space variables, and the action vector of the next control cycle in the candidate boundary control sequence is used as the candidate action vector. The process of concatenating the candidate latent space variables with the candidate action vector is repeated until the future macroscopic state of traffic for multiple control cycles is obtained.

[0011] In some embodiments, the reinforcement learning policy network generates candidate boundary control sequences using a cross-entropy optimization method. Based on the future macro-level traffic conditions, an evaluation network is used to assess the cumulative benefit of each candidate boundary control sequence, including the following steps: For each set of candidate boundary control sequences, the reward value for each time step of each control cycle is constructed using a preset reward function based on the future macro-level traffic conditions of the corresponding multiple control cycles. The reward function is determined by a weighted sum of regional travel completion rate, regional congestion penalty, boundary queuing penalty, and control action change rate penalty. The cumulative reward value of the corresponding candidate boundary control sequence is obtained by summing the reward values ​​of all control cycles of the same group of candidate boundary control sequences. The candidate boundary control sequences with the highest cumulative returns are used as elite samples, and the cross-entropy sampling distribution of the cross-entropy optimization method is updated using these elite samples. The candidate boundary control sequences with the highest cumulative returns are determined based on the final cumulative returns after multiple iterations of optimization.

[0012] In some embodiments, the world model is modified based on a prediction error-triggered update mechanism, including the following steps: When the prediction error exceeds a preset threshold, a trigger update is executed; or, if no trigger update is generated for a preset number of consecutive control cycles, a periodic update is executed. In response to triggered or periodic updates, the parameters of the encoder and decoder are frozen, while the parameters of the inference module and the parameters of the online adaptation layer at the input of the inference module are updated. The parameter update adopts a preset small step parameter update mechanism, and the parameter update is also configured with an experience playback mechanism and an elastic weight consolidation mechanism.

[0013] To achieve the above objectives, another aspect of the present invention proposes a region boundary control device based on a world model, the device comprising: The first module is used to acquire historical operational data of the regional transportation system and construct a state transition sample set that includes the macroscopic state of the previous moment, boundary control inputs, and the macroscopic state of the next moment. The second module is used to jointly train a pre-defined world model using a state transition sample set, so that the world model learns the mapping relationship between macroscopic states and latent space, and learns the dynamic evolution relationship from the macroscopic state and boundary control input of the previous moment to the macroscopic state of the next moment in the latent space; the world model includes an encoder, an inference module and a decoder. The third module is used to set the current time as the target time. The fourth module is used to obtain the real macroscopic traffic state at the target time and map it into target latent space variables using an encoder; The fifth module is used to generate candidate boundary control sequences based on real traffic macro-states using a reinforcement learning policy network. The sixth module is used to obtain the future macro-level traffic conditions for multiple control cycles by using the inference module and decoder to infer the target latent space variables and candidate boundary control sequences. The seventh module is used to evaluate the cumulative benefits of each candidate boundary control sequence based on the future macro-level traffic conditions using an evaluation network. The eighth module is used to execute the target boundary control action based on the candidate boundary control sequence with the highest cumulative return, take the next time step of the target time as the target time, return to execute the operation of the fourth module, and continue to perform regional boundary control. The ninth module is used to construct prediction errors based on the future macro-level traffic conditions and the actual macro-level traffic conditions at the same time. Based on the prediction error, an update mechanism is triggered to correct the parameters of the world model.

[0014] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method.

[0015] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0016] To achieve the above objectives, another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0017] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a regional boundary control method, device, electronic device, storage medium, and program product based on a world model. This solution acquires historical operational data of a regional traffic system to construct a state transition sample set containing the macroscopic state of the previous moment, boundary control input, and the macroscopic state of the next moment; it uses the state transition sample set to jointly train a preset world model, enabling the world model to learn the mapping relationship between the macroscopic state and the latent space, and to learn the dynamic evolution relationship from the macroscopic state and boundary control input of the previous moment to the macroscopic state of the next moment in the latent space; wherein, the world model includes an encoder, an inference module, and a decoder; the current moment is taken as the target moment; the actual traffic macroscopic state at the target moment is acquired and mapped to the target latent space using the encoder. Spatial variables; Based on the real traffic macro-state, candidate boundary control sequences are generated using a reinforcement learning policy network; Based on the target latent spatial variables and candidate boundary control sequences, the future traffic macro-state for multiple control cycles is obtained through rolling inference using an inference module and decoder; Based on the future traffic macro-state, the cumulative benefit of each candidate boundary control sequence is evaluated using an evaluation network; The target boundary control action is executed based on the candidate boundary control sequence with the highest cumulative benefit, and the next time step of the target time is taken as the target time. The process then returns to the step of obtaining the real traffic macro-state at the target time, and regional boundary control is continuously performed; A prediction error is constructed based on the future traffic macro-state and the real traffic macro-state at the same time, and the world model is parameter-corrected based on the prediction error-triggered update mechanism. This invention constructs a world model for a regional macro-level transportation system, mapping the current macro-state, external demand, and boundary control inputs to the future state evolution process, enabling the system to perform continuous multi-step rolling deduction. Then, based on reinforcement learning and policy optimization mechanisms, the performance of candidate boundary control sequences in the future time domain is evaluated, thereby achieving forward-looking boundary control for future states. Simultaneously, an online adaptive update mechanism based on prediction errors is introduced to mitigate performance degradation caused by model mismatch, demand drift, and scenario changes, improving the system's robustness, generalization ability, and engineering deployment feasibility in complex traffic environments. Specifically, this invention does not simply improve traffic prediction accuracy or learn boundary control strategies in isolation; rather, it constructs an integrated closed-loop technical solution for regional traffic control scenarios, encompassing dynamic modeling, rolling deduction, policy evaluation, and online correction, enabling boundary control decisions to shift from static threshold-driven to future state-driven. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of an implementation environment for a world model-based regional boundary control method provided in an embodiment of the present invention; Figure 2This is a flowchart illustrating the region boundary control method based on a world model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the observation evolution process and training backpropagation process of the traffic macro-world model provided in this embodiment of the invention; Figure 4 This is a schematic diagram illustrating the process architecture principle of the traffic macro-world model and reinforcement learning boundary control provided in this embodiment of the invention; Figure 5 This is a schematic diagram of the structure of the region boundary control device based on the world model provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0020] It is understood that the terms "first," "second," etc., used in this invention may be used to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of embodiments of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to determination," or "in the event of a determination."

[0021] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0022] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this invention is for descriptive purposes only and is not intended to limit the invention.

[0023] To facilitate understanding of the technical content of the present invention, the meanings of the terms related to the technical features that may be involved in the present invention will be explained first: Macroscopic Fundamental Diagram (MFD): A macroscopic model describing the relationship between the cumulative number of vehicles and the network operating efficiency in a transportation network; it is used here only as background information and is not intended as the basis for the control and solution of this invention. World Model: A predictive model that learns the laws governing the evolution of a system's state through historical data; Perimeter control: A method of regulating regional traffic conditions by controlling the inflow of traffic into a region at its boundaries; Trip Completion Rate: The percentage of vehicles that complete a trip within a given time period; Among related technologies, although existing publicly available technologies involve elements such as macroscopic basic maps, regional boundary control, traffic state prediction, and reinforcement learning control, most solutions still handle state prediction and control decisions separately, or rely on preset static relationships, fixed macroscopic equations, or manually set dynamic rules for control. A technical solution has not yet been formed that takes the world model as the core and integrates macroscopic state encoding, control input participation in the extrapolation, multi-step rolling extrapolation, strategy evaluation, and online error adaptive correction into the same closed-loop framework.

[0024] In view of this, this invention provides a regional boundary control method and related equipment based on a world model. This method acquires historical operational data of the regional traffic system to construct a state transition sample set containing the macroscopic state of the previous moment, boundary control input, and the macroscopic state of the next moment. The state transition sample set is used to jointly train a preset world model, enabling the world model to learn the mapping relationship between the macroscopic state and the latent space, and to learn the dynamic evolution relationship from the macroscopic state and boundary control input of the previous moment to the macroscopic state of the next moment in the latent space. The world model includes an encoder, an inference module, and a decoder. The current moment is used as the target moment. The actual traffic macroscopic state at the target moment is acquired and mapped to target latent space variables using the encoder. Based on the actual traffic macroscopic state... Based on the macroscopic state, a reinforcement learning policy network is used to generate candidate boundary control sequences. Using the target latent space variables and the candidate boundary control sequences, a rolling inference module and decoder are used to deduce the future traffic macroscopic state for multiple control cycles. Based on the future traffic macroscopic state, an evaluation network is used to assess the cumulative benefit of each candidate boundary control sequence. The target boundary control action is executed based on the candidate boundary control sequence with the highest cumulative benefit. The next time step after the target time is taken as the target time, and the process returns to obtain the actual traffic macroscopic state at the target time, continuously performing regional boundary control. A prediction error is constructed based on the future traffic macroscopic state and the actual traffic macroscopic state at the same time. An update mechanism is triggered based on the prediction error to correct the parameters of the world model. This invention constructs a world model for a regional macro-level transportation system, mapping the current macro-state, external demand, and boundary control inputs to the future state evolution process, enabling the system to perform continuous multi-step rolling deduction. Then, based on reinforcement learning and policy optimization mechanisms, the performance of candidate boundary control sequences in the future time domain is evaluated, thereby achieving forward-looking boundary control for future states. Simultaneously, an online adaptive update mechanism based on prediction errors is introduced to mitigate performance degradation caused by model mismatch, demand drift, and scenario changes, improving the system's robustness, generalization ability, and engineering deployment feasibility in complex traffic environments. Specifically, this invention does not simply improve traffic prediction accuracy or learn boundary control strategies in isolation; rather, it constructs an integrated closed-loop technical solution for regional traffic control scenarios, encompassing dynamic modeling, rolling deduction, policy evaluation, and online correction, enabling boundary control decisions to shift from static threshold-driven to future state-driven.

[0025] It is understood that the region boundary control method based on a world model provided by this invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet, laptop, or desktop computer, but it is not limited to these.

[0026] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0027] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0028] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0029] Terminal 102 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0030] For example, based on Figure 1The implementation environment shown in this embodiment of the invention provides a region boundary control method based on a world model. The following description uses the application of this region boundary control method based on a world model in server 101 as an example. It can be understood that this region boundary control method based on a world model can also be applied to terminal 102.

[0031] Reference Figure 2 , Figure 2 This is an optional flowchart of a region boundary control method based on a world model provided in an embodiment of the present invention. The executing entity of this region boundary control method based on a world model can be any of the aforementioned computer devices (including servers or terminals). Figure 2 The method may include, but is not limited to, steps S100 to S900.

[0032] Step S100: Obtain historical operation data of the regional transportation system and construct a state transition sample set containing the macro state of the previous moment, boundary control input, and macro state of the next moment; For example, in some specific implementations, during the offline modeling stage, historical operational data of the regional traffic system is first collected, and state transition samples are constructed according to a unified control cycle. The historical operational data consists of signal control data, demand data, accumulated data, and operational performance data. Signal control data includes phase duration, cycle length, green light ratio, release rate, and equivalent flow restriction parameters at each boundary entrance. Demand data includes external arrival demand, directional entry demand, time-based demand intensity, and demand disturbance information. Accumulated data includes the cumulative number of vehicles in the region, the cumulative number of vehicles in each zone, boundary queuing volume, boundary inflow volume, and boundary outflow volume. Operational performance data includes the travel completion rate, average delay, average speed, and congestion indicators for each region. After the above multi-source traffic data is synchronized, aligned, normalized, and anomaly corrected at the same time granularity, it constitutes the input basis for the macroscopic world model of traffic. The world model uses historically observed state transition relationships as the learning object, without requiring pre-calibration of static macroscopic relationships or pre-defined fixed-form traffic dynamics equations. Let the first... The macroscopic traffic state at any given moment is Boundary control input is The sample is then represented as:

[0033] Among them, the macro-level traffic conditions It is composed of cumulative vehicle count, boundary flow, demand intensity, signal control parameters, and operational performance indicators, specifically written as:

[0034] in, This indicates the cumulative number of vehicles in a region or zone. Indicates the inflow at the boundary. Indicates the outflow at the boundary. Indicates the intensity of demand. Indicates signal control parameters, Indicates operational performance indicators; control inputs Defined as the equivalent flow restriction coefficient vector of each boundary inlet within the current control cycle. In this way, the historical traffic operation process is organized into a unified set of state-action-next state time-series samples, providing standardized data support for subsequent training of macroscopic traffic models.

[0035] Step S200: Use the state transition sample set to jointly train the preset world model so that the world model learns the mapping relationship between the macro state and the latent space, and learns the dynamic evolution relationship from the macro state and boundary control input of the previous moment to the macro state of the next moment in the latent space. The world model includes an encoder, a deduction module, and a decoder; It should be noted that, in some embodiments, the joint training of a preset world model using a state transition sample set may include the following steps: inputting the previous moment's macroscopic state from the state transition sample set into the encoder, processing and outputting training latent space variables; mapping the boundary control input into action feature vectors using an action embedding network, and concatenating the action feature vectors with the training latent space variables into a fusion vector; constructing a historical sequence based on the fusion vectors from multiple consecutive moments, inputting the historical sequence into the inference module, processing and outputting the next moment's latent space variables; inputting the next moment's latent space variables into the decoder, processing and outputting the training inferred macroscopic state; constructing a loss function based on the training inferred macroscopic state and the next moment's macroscopic state, using the loss function to jointly optimize the parameters of the encoder, inference module, and decoder, and then freezing the parameters.

[0036] For example, in some specific embodiments, in constructing a macroscopic world model of traffic, this invention employs a neural network structure combining an encoder, an inference module, and a decoder to model the macroscopic dynamics of the traffic system. Specifically, the encoder... It is implemented using a multilayer perceptron network, whose input is the current traffic macroscopic state vector. The output is the latent variables in the hidden space. The encoder consists of a three-layer fully connected network. The first layer has an output dimension of 256, the second layer has an output dimension of 128, and the third layer has an output dimension of 64. Each hidden layer uses the ReLU activation function, and layer normalization is introduced after a linear transformation. After being mapped by the encoder, the current macroscopic traffic observation state is compressed into a 64-dimensional deterministic real-valued vector. ,Right now:

[0037] and:

[0038] In this invention, latent variables The latent variables are explicitly defined as deterministic vectors, rather than probability distribution parameters. They do not incorporate Gaussian mean or variance representations, nor do they employ reparameterized sampling mechanisms. The latent variable dimension is fixed at 64, determined through a joint evaluation of one-step prediction error, multi-step rolling error, and online inference latency during offline training, ensuring a balance between state representation capability and online computational efficiency.

[0039] Inference module This invention describes the temporal evolution of the latent space state under the influence of control inputs. In this invention, the derivation module is implemented using a Transformer network. First, the control inputs at each time step are... Input the action embedding network to obtain action feature vectors Then, the hidden variables at each time point are... With the corresponding action feature vector The features are concatenated to form a fusion vector. Subsequently, the fused vectors from multiple consecutive time points are used to construct the historical input sequence:

[0040] After being overlaid with position encoding, the data is input into the Transformer inference unit. A multi-head self-attention mechanism and a feedforward network are used to model the temporal dependency of the macroscopic traffic state under control. The latent variables in the hidden space at the next time step are then obtained through the output mapping layer. ,Right now:

[0041] Therefore, the control input in this invention With latent variables The combination method is limited to first mapping through an action embedding network, then forming a temporal fusion feature sequence through vector concatenation, and inputting it into a Transformer for temporal inference. This structure allows the impact of boundary control actions on future traffic evolution to be directly reflected in the latent space state transition process.

[0042] decoder Used to store the latent variables in the hidden space at the next time step. The system reverts to a macroscopic traffic observation. The decoder also employs a multilayer perceptron network with an input dimension of 64, a first fully connected layer with an output dimension of 128, a second fully connected layer with an output dimension of 256, and a final output layer dimension consistent with the macroscopic traffic state vector dimension, outputting the predicted state for the next time step. ,Right now:

[0043] via encoder Deduction Module and decoder By employing a serial structure, this invention establishes a complete dynamic evolutionary relationship from the current traffic observation state and control input to the prediction of future traffic observation states. This dynamic mapping relationship is learned by the world model based on state transition samples and is used to support the multi-step rolling evaluation of subsequent candidate control sequences, without relying on pre-given fixed macroscopic equations or artificially set dynamic expressions as the basis for state evolution.

[0044] During the training of the macroscopic world model of transportation, historical real state transition sequences are used as supervision signals to jointly optimize the parameters of the encoder, inference module and decoder through backpropagation.

[0045] It should be noted that, in some embodiments, constructing a loss function based on the macroscopic state in training and the macroscopic state at the next moment may include the following steps: constructing a prediction loss and a multi-step rolling deduction loss based on the macroscopic state in training and the macroscopic state at the next moment; constructing a data consistency loss based on the macroscopic state in training and the macroscopic state at the next moment through traffic flow statistical consistency constraints; and weighting and fusing the prediction loss, the data consistency loss, and the multi-step rolling deduction loss to obtain the total loss function of the model.

[0046] For example, in some specific implementations, the actual traffic state at the next moment is assumed to be... The predicted state is The basic prediction loss of the model is defined as:

[0047] To enhance the stability of the prediction results in terms of traffic business semantics, this invention further introduces a traffic flow statistical consistency constraint. This constraint is used to ensure that the output of the world model remains consistent with the actual observations in terms of the cumulative vehicle number change trend, boundary inflow and outflow statistical characteristics, demand intensity changes, and operational performance indicators. It participates in loss calculation as a soft constraint during the training phase, does not constitute a preset fixed macroscopic evolution equation, and does not require deduction according to artificially set dynamic expressions.

[0048] Based on the aforementioned statistical consistency characteristics, a data consistency loss term is constructed. This data consistency loss measures the deviation between the predicted state and the actual state in the traffic statistical feature space. Consistency characteristics may include cumulative changes in vehicle numbers, boundary inflow and outflow statistics, changes in demand intensity, operational performance indicators, and combinations thereof.

[0049] Meanwhile, to suppress error accumulation and state drift during rolling simulation, this invention further introduces multi-step rolling consistency loss during the training phase. Specifically, within each training window, simulations are continuously performed starting from the true initial state. Step by step, a supervisory constraint is imposed on the prediction result at each step, and the multi-step rolling loss is defined as:

[0050] in, These are weighting coefficients that decrease with the prediction step size. The total loss function of the model is expressed as:

[0051] in, and The weights are preset parameters or parameters adjusted based on the validation set. They are used to balance the training objectives between one-step prediction, multi-step rolling deduction, and statistical consistency constraints, and do not represent the physical coefficients in the macroscopic evolution equation of fixed traffic.

[0052] To further reduce the distribution bias between the training and online inference phases, this invention employs a planned sampling mechanism during the world model training process. Specifically, in the initial training phase, the input to the inference module is primarily latent variables encoded from the previous moment's actual observed states; as training progresses, the proportion of the model's own predicted latent variables participating in subsequent rolling inferences is gradually increased, exposing the inference module to the continuous state transition process under the model's predicted distribution during the training phase. Through this mechanism, the traffic macro-world model, in its execution... During step-by-step rolling simulation, it can effectively suppress error amplification and state drift, and improve the stability of multi-step prediction and the reliability of online decision-making.

[0053] Step S300: Set the current time as the target time; Step S400: Obtain the real macroscopic traffic state at the target time and map it to the target latent space variable using the encoder; For example, in some specific implementations, during the online operation phase, the system collects real-time data on the current macroscopic traffic conditions. The data is then input into a pre-trained macroscopic traffic model for state encoding to obtain the current latent variables in the hidden space. The encoder is frozen during the online control phase, and its parameters remain unchanged; the current latent space variable (i.e., the target latent space variable) obtained by the encoder mapping is a deterministic real-valued vector, without introducing probability distribution parameters.

[0054] Step S500: Based on the real traffic macro-state, generate candidate boundary control sequences using a reinforcement learning policy network; It should be noted that, in some embodiments, generating candidate boundary control sequences based on real traffic macro-states using a reinforcement learning policy network may include the following steps: inputting the real traffic macro-states into the reinforcement learning policy network and processing the output action distribution; using the action distribution as the initial sampling distribution and using the cross-entropy optimization method to generate multiple sets of candidate boundary control sequences in the rolling extrapolation time domain; wherein, each set of candidate boundary control sequences includes continuous action vectors for multiple future control cycles, and each action vector is obtained based on preset boundary flow regulation constraints.

[0055] For example, in some specific implementations, the boundary control decision-making process is initiated through a reinforcement learning module (reinforcement learning policy network). The reinforcement learning module does not directly output a single control action, but rather outputs a parameterized policy distribution of the boundary control actions at the current moment. Specifically, the policy network adopts a three-layer, multi-layer perceptron structure, with the current macroscopic traffic state as the input. The hidden layer dimensions are 256 and 128, respectively, and the output is the action mean vector and the log-standard deviation vector, thus defining the action distribution:

[0056] This action distribution is directly used as the initial sampling distribution for generating candidate boundary control actions. Therefore, the function of the reinforcement learning module is explicitly limited to outputting the policy prior for candidate control search, rather than directly providing a single, definitive action.

[0057] In the process of generating candidate control sequences, this invention employs a combination of cross-entropy optimization and policy distribution initialization to complete the search for future control sequences. Specifically, the mean and covariance of the actions output by the reinforcement learning policy network are first used as the initial sampling distribution, and then the distribution is used in the rolling extrapolation time domain. Generate within each control cycle Groups of candidate control sequences, each group of candidate control sequences is represented as follows:

[0058] Each action vector satisfies the boundary flow regulation constraint and is restricted to the legal control range through projection mapping.

[0059] In some preferred embodiments, the present invention employs a PPO mechanism to update the parameters of the policy network and the evaluation network during the training of the reinforcement learning module itself. The policy network loss uses a pruning policy optimization objective.

[0060] in:

[0061] The network loss is evaluated using the state value regression error:

[0062] The overall loss function consists of policy loss, value loss, and policy entropy regularization term:

[0063] The trajectory data used for training the reinforcement learning module comes from both real traffic data and imaginary trajectories generated through rolling world model simulations. Real traffic trajectories reflect the actual effectiveness of boundary control strategies in real traffic systems, while imaginary trajectories are obtained by online simulations of candidate control sequences using the world model, expanding the policy evaluation sample and improving search efficiency. The two types of trajectories are mixed in a fixed ratio into the PPO training batch, ensuring that reinforcement learning policy updates maintain consistency with real traffic operations while possessing sufficient sample utilization efficiency and policy exploration capabilities.

[0064] Step S600: Based on the target latent space variables and candidate boundary control sequences, the future macroscopic traffic state for multiple control cycles is obtained by using the inference module and decoder to perform rolling inference. It should be noted that each set of candidate boundary control sequences includes continuous action vectors for multiple future control cycles. In some embodiments, step S600 may include the following steps: For each set of candidate boundary control sequences, starting from the target latent space variable, iteratively perform the following operations: use the target latent space variable as a candidate latent space variable, and use the action vector of the first control cycle in the candidate boundary control sequence as a candidate action vector; concatenate the candidate latent space variable and the candidate action vector, and then input them into the inference module to predict the predicted latent space variable for the next control cycle; input the predicted latent space variable into the decoder to decode the future traffic macro state for the corresponding control cycle; use the predicted latent space variable as a candidate latent space variable, and use the action vector of the next control cycle in the candidate boundary control sequence as a candidate action vector, and return to execute the step of concatenating the candidate latent space variable and the candidate action vector, until the future traffic macro state for multiple future control cycles is obtained.

[0065] For example, in some specific implementations, the world model, for each set of candidate control sequences, starts from the current traffic state and continuously predicts the future according to the encoding-deduction-decoding process (the specific logical principle is the same as the process logic of the specific implementation in the offline modeling stage, and will not be repeated here). By analyzing the macroscopic state of traffic, we obtain the regional travel completion rate, cumulative number of vehicles, boundary queue volume, and regional congestion level corresponding to several possible control strategies, thus forming the process of the macroscopic world model of traffic in control.

[0066] Step S700: Based on the future macro-state of traffic, use the evaluation network to assess the cumulative benefits of each candidate boundary control sequence; It should be noted that the reinforcement learning policy network uses the cross-entropy optimization method to generate candidate boundary control sequences. In some embodiments, step S700 may include the following steps: for each group of candidate boundary control sequences, based on the future macro-traffic state of its corresponding future multiple control cycles, construct the reward value for each time step of each control cycle using a preset reward function; wherein, the reward function is determined based on the weighted sum of regional travel completion rate, regional congestion penalty, boundary queuing penalty, and control action change rate penalty; accumulate the reward values ​​of all control cycles of the same group of candidate boundary control sequences to obtain the cumulative revenue of the corresponding candidate boundary control sequences; use the predetermined number of candidate boundary control sequences with the highest cumulative revenue as elite samples, and use the elite samples to update the cross-entropy sampling distribution of the cross-entropy optimization method; wherein, the candidate boundary control sequence with the highest cumulative revenue is determined based on the final cumulative revenue result after multiple consecutive iterations of optimization.

[0067] For example, in some specific implementations, during the evaluation of candidate control sequences, the evaluation model in the reinforcement learning module calculates the cumulative reward for each candidate control sequence. The evaluation model adopts a three-layer multilayer perceptron structure, with the input being the macroscopic traffic state and the output being a state value estimate. The reward function is defined as:

[0068] in, Indicates the regional travel completion rate. This indicates penalties for regional congestion. Indicates the boundary queuing penalty item. This indicates a penalty term for the rate of change of control action. , , and The weighting coefficients are fixed. For any candidate control sequence Its future time-domain cumulative return is expressed as:

[0069] in, The discount factor is used. The system selects the top 8 candidate sequences with the highest cumulative returns as elite samples, and uses these elite samples to update the cross-entropy sampling distribution for the next round, performing three consecutive rounds of iterative optimization. Finally, the candidate control sequence with the highest cumulative return is selected as the optimal sequence.

[0070] Step S800: Execute the target boundary control action based on the candidate boundary control sequence with the highest cumulative revenue, take the next time step of the target time as the target time, return to the step of obtaining the real traffic macro state at the target time, and continue to perform regional boundary control. For example, in some specific implementations, the candidate control sequence with the highest cumulative benefit is selected as the optimal sequence. Then, based on this optimal sequence, the boundary control action to be executed (i.e., the target boundary control action) is determined and the action is executed. After the boundary control action is executed, the system continues to collect the real traffic observation values ​​at the next time step. Iterative processes are performed to continuously control the region boundaries.

[0071] Step S900: Construct a prediction error based on the future macro-state of traffic and the actual macro-state of traffic at the same time. Based on the prediction error, trigger an update mechanism to correct the parameters of the world model. It should be noted that in some embodiments, the parameter correction of the world model based on the prediction error trigger update mechanism may include the following steps: when the prediction error exceeds a preset threshold, a trigger update is executed, or when no trigger update is generated for a preset number of consecutive control cycles, a periodic update is executed; in response to the trigger update or periodic update, the parameters of the encoder and decoder are frozen, and the parameters of the inference module and the online adaptation layer parameters at the input of the inference module are updated; wherein, the parameter update adopts a preset small step parameter update mechanism, and the parameter update is also configured with an experience playback mechanism and an elastic weight consolidation mechanism.

[0072] For example, in some specific implementations, after the boundary control action is executed, the system continues to collect the real traffic observation values ​​at the next moment. and the results of the world model prediction By comparing the results, the prediction error is obtained:

[0073] This invention employs an online update mechanism that combines error triggering with periodic updates. When the normalized prediction error satisfies... When an error update is triggered, an incremental update of the world model is immediately executed; if no error update is triggered for 12 consecutive control cycles, a periodic update is performed. Threshold The 95th percentile of the normalized root mean square error of the one-step prediction on the offline validation set is taken, and in this embodiment of the invention, it is fixed at 0.10. During the online update process, not all parameters of the world model are modified; only the inference module is updated. The parameters and the online adaptation layer parameters set at the input of the inference module, encoder With decoder The update process remains frozen during the online phase. This approach ensures the stability of the state representation space and the observation reconstruction mapping, while enabling the model to make targeted corrections for dynamic drift caused by fluctuations in traffic demand, changes in signal timing, and sudden disturbances.

[0074] The online update of the inference module parameters is written as:

[0075] To prevent catastrophic forgetting during continuous online updates, this invention introduces an experience replay mechanism, a flexible weight consolidation mechanism, and a small-step parameter update mechanism during the online update phase. The experience replay mechanism maintains a fixed-capacity sample cache, ensuring that each online update simultaneously includes representative offline samples and the latest online samples, guaranteeing that the model retains its memory of historical typical traffic evolution patterns while learning new traffic patterns. The flexible weight consolidation mechanism adds a parameter offset constraint term based on Fisher information to the online loss function, limiting the deviation of key parameters from the optimal parameter points after offline training. Its regularization term is expressed as:

[0076] in, These are reference parameters after offline training. The Fisher information weighting coefficients are used. The small step size parameter update mechanism fixes the online learning rate to one-twentieth of the offline training learning rate, ensuring that model parameter adjustments always maintain a gradual characteristic. Therefore, the total loss for online updates is written as:

[0077] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0078] This invention relates to the field of intelligent traffic control and traffic state prediction technology, specifically to a boundary control method based on a world model for macroscopic traffic state prediction and reinforcement learning. The method of this invention addresses regional traffic boundary regulation scenarios, constructing a unified macroscopic traffic modeling and control decision-making framework around the coupling relationship between observed states, control inputs, latent space states, state evolution, control evaluation, and error feedback within the traffic system. This method uses a world model as its core, learning the evolution of traffic states under boundary control inputs through a data-driven approach. It does not pre-set fixed macroscopic equations, does not rely on manually defined dynamic expressions, and does not use static relationships of basic macroscopic diagrams as the basis for control solutions. Simultaneously, it uses the observation evolution process and training backpropagation process of the macroscopic traffic world model as the modeling foundation, unifying and coupling the reinforcement learning model's process in boundary control with the macroscopic traffic world model's process in control. This integrates macroscopic traffic state prediction, candidate control sequence evaluation, boundary control execution, and online error feedback updates into a single closed-loop technical system, thereby achieving efficient and stable boundary flow regulation in complex traffic networks. The specific implementation of this invention can be achieved through the following process: like Figure 3As shown, in the offline modeling phase, historical operational data of the regional traffic system is first collected, and state transition samples are constructed according to a unified control cycle. The historical operational data consists of signal control data, demand data, accumulated data, and operational performance data. Signal control data includes phase duration, cycle length, green light ratio, release rate, and equivalent flow restriction parameters at each boundary entrance. Demand data includes external arrival demand, directional entry demand, time-based demand intensity, and demand disturbance information. Accumulated data includes the cumulative number of vehicles in the region, the cumulative number of vehicles in each zone, boundary queuing volume, boundary inflow volume, and boundary outflow volume. Operational performance data includes the travel completion rate, average delay, average speed, and congestion indicators for each region. After the above multi-source traffic data are synchronized, aligned, normalized, and anomaly corrected at the same time granularity, they form the input basis for the macroscopic traffic world model. Let the first... The macroscopic traffic state at any given moment is Boundary control input is The sample is then represented as:

[0079] Among them, the macro-level traffic conditions It is composed of cumulative vehicle count, boundary flow, demand intensity, signal control parameters, and operational performance indicators, specifically written as:

[0080] in, This indicates the cumulative number of vehicles in a region or zone. Indicates the inflow at the boundary. Indicates the outflow at the boundary. Indicates the intensity of demand. Indicates signal control parameters, Indicates operational performance indicators; control inputs Defined as the equivalent flow restriction coefficient vector of each boundary inlet within the current control cycle. In this way, the historical traffic operation process is organized into a unified set of state-action-next state time-series samples, providing standardized data support for subsequent training of macroscopic traffic models.

[0081] In constructing a macroscopic world model of transportation, this invention employs a neural network structure combining an encoder, an inference module, and a decoder to model the macroscopic dynamics of the transportation system. Specifically, the encoder... It is implemented using a multilayer perceptron network, whose input is the current traffic macroscopic state vector. The output is the latent variables in the hidden space. The encoder consists of a three-layer fully connected network. The first layer has an output dimension of 256, the second layer has an output dimension of 128, and the third layer has an output dimension of 64. Each hidden layer uses the ReLU activation function, and layer normalization is introduced after a linear transformation. After being mapped by the encoder, the current macroscopic traffic observation state is compressed into a 64-dimensional deterministic real-valued vector. ,Right now:

[0082] and:

[0083] In this invention, latent variables The latent variables are explicitly defined as deterministic vectors, rather than probability distribution parameters. They do not incorporate Gaussian mean or variance representations, nor do they employ reparameterized sampling mechanisms. The latent variable dimension is fixed at 64, determined through a joint evaluation of one-step prediction error, multi-step rolling error, and online inference latency during offline training, ensuring a balance between state representation capability and online computational efficiency.

[0084] The derivation module F is used to describe the temporal evolution of the latent space state under the influence of control inputs. In this invention, the derivation module is implemented using a Transformer network. First, the control inputs at each time step are... Input the action embedding network to obtain action feature vectors Then, the hidden variables at each time point are... With the corresponding action feature vector The features are concatenated to form a fusion vector. Subsequently, the fused vectors from multiple consecutive time points are used to construct the historical input sequence:

[0085] After being overlaid with position encoding, the data is input into the Transformer inference unit. A multi-head self-attention mechanism and a feedforward network are used to model the temporal dependency of the macroscopic traffic state under control. The latent variables in the hidden space at the next time step are then obtained through the output mapping layer. ,Right now:

[0086] Therefore, the control input in this invention With latent variables The combination method is limited to first mapping through an action embedding network, then forming a temporal fusion feature sequence through vector concatenation, and inputting it into a Transformer for temporal inference. This structure allows the impact of boundary control actions on future traffic evolution to be directly reflected in the latent space state transition process.

[0087] decoder Used to store the latent variables in the hidden space at the next time step. The system reverts to a macroscopic traffic observation. The decoder also employs a multilayer perceptron network with an input dimension of 64, a first fully connected layer with an output dimension of 128, a second fully connected layer with an output dimension of 256, and a final output layer dimension consistent with the macroscopic traffic state vector dimension, outputting the predicted state for the next time step. ,Right now:

[0088] via encoder Deduction Module and decoder By employing a serial structure, this invention establishes a complete dynamic evolutionary relationship from the current traffic observation state and control inputs to the prediction of future traffic observation states. This mapping is learned by the world model from data, without relying on preset fixed macroscopic equations or manual dynamic expressions, forming the observational evolution process of the traffic macroscopic world model.

[0089] During the training of the macroscopic traffic world model, historical real-world state transition sequences are used as supervision signals to jointly optimize the parameters of the encoder, inference module, and decoder through backpropagation. Let the actual next-time traffic state be... The predicted state is The basic prediction loss of the model is defined as:

[0090] To enhance the stability of the prediction results in terms of traffic business semantics, this invention further introduces a traffic flow statistical consistency constraint. This constraint is used to ensure that the output of the world model remains consistent with the actual observations in terms of the cumulative vehicle number change trend, boundary inflow and outflow statistical characteristics, demand intensity changes, and operational performance indicators. It participates in loss calculation as a soft constraint during the training phase, does not constitute a preset fixed macroscopic evolution equation, and does not require deduction according to artificially set dynamic expressions.

[0091] Based on the aforementioned statistical consistency characteristics, a data consistency loss term is constructed. This data consistency loss measures the deviation between the predicted state and the actual state in the traffic statistical feature space; the traffic statistical features may include cumulative vehicle number changes, boundary inflow and outflow statistics, demand intensity changes, operational performance indicators, and combinations thereof.

[0092] Meanwhile, to suppress error accumulation and state drift during rolling simulation, this invention further introduces multi-step rolling consistency loss during the training phase. Specifically, within each training window, simulations are continuously performed starting from the true initial state. Step by step, a supervisory constraint is imposed on the prediction result at each step, and the multi-step rolling loss is defined as:

[0093] in, These are weighting coefficients that decrease with the prediction step size. The total loss function of the model is expressed as:

[0094] The weights of each loss term are preset parameters or parameters adjusted based on the validation set. They are used to balance the training objectives between one-step prediction, multi-step rolling deduction, and statistical consistency constraints, and do not represent the physical coefficients in the macroscopic evolution equation of fixed traffic.

[0095] To further reduce the distribution bias between the training and online inference phases, this invention employs a planned sampling mechanism during the world model training process. Specifically, in the initial training phase, the input to the inference module is primarily latent variables encoded from the previous moment's actual observed states; as training progresses, the proportion of the model's own predicted latent variables participating in subsequent rolling inferences is gradually increased, exposing the inference module to the continuous state transition process under the model's predicted distribution during the training phase. Through this mechanism, the traffic macro-world model, in its execution... During step-by-step rolling simulation, it can effectively suppress error amplification and state drift, and improve the stability of multi-step prediction and the reliability of online decision-making.

[0096] like Figure 4 As shown, during the online operation phase, the system collects real-time macroscopic traffic data at the current moment. The data is then input into a pre-trained macroscopic traffic model for state encoding to obtain the current latent variables in the hidden space. Building upon this, the reinforcement learning module initiates the boundary control decision-making process. Instead of directly outputting a single control action, the reinforcement learning module outputs a parameterized policy distribution of the current boundary control action. Specifically, the policy network employs a three-layer, multi-layer perceptron structure, with the current macroscopic traffic state as input. The hidden layer dimensions are 256 and 128, respectively, and the output is the action mean vector and the log-standard deviation vector, thus defining the action distribution:

[0097] This action distribution is directly used as the initial sampling distribution for generating candidate boundary control actions. Therefore, the function of the reinforcement learning module is explicitly limited to outputting the policy prior for candidate control search, rather than directly providing a single, definitive action.

[0098] In the process of generating candidate control sequences, this invention employs a combination of cross-entropy optimization and policy distribution initialization to complete the search for future control sequences. Specifically, the mean and covariance of the actions output by the reinforcement learning policy network are first used as the initial sampling distribution, and then the distribution is used in the rolling extrapolation time domain. Generate within each control cycle Groups of candidate control sequences, each group of candidate control sequences is represented as follows:

[0099] Each action vector satisfies boundary flow control constraints and is limited to the legal control range through projection mapping. Subsequently, for each set of candidate control sequences, the world model continuously predicts the future based on the current traffic state, following an encoding-deduction-decoding process. By analyzing the macroscopic traffic conditions, we obtain the regional travel completion rate, cumulative number of vehicles, boundary queue length, and regional congestion level corresponding to several possible control strategies, thus forming the process of implementing the macroscopic traffic world model in control. This deduction step is a prerequisite for evaluating candidate controls; otherwise, reinforcement learning can only conduct local trials based on current observations and cannot compare future time-domain benefits.

[0100] During the evaluation of candidate control sequences, the evaluation model in the reinforcement learning module calculates the cumulative reward for each candidate control sequence. The evaluation model employs a three-layer multilayer perceptron structure, with the input being the macroscopic traffic state and the output being a state value estimate. The reward function is defined as:

[0101] in, Indicates the regional travel completion rate. This indicates penalties for regional congestion. Indicates the boundary queuing penalty item. This indicates a penalty term for the rate of change of control action. , , and The weighting coefficients are fixed. For any candidate control sequence Its future time-domain cumulative return is expressed as:

[0102] in, The discount factor is used. The system selects the top 8 candidate sequences with the highest cumulative returns as elite samples, and uses these elite samples to update the cross-entropy sampling distribution for the next round, performing three consecutive rounds of iterative optimization. Finally, the candidate control sequence with the highest cumulative return is selected as the optimal sequence, and the first control action corresponding to the current time step in that sequence is executed, forming a complete control process from the evaluation model and strategy selection model to the optimal decision.

[0103] Regarding the training of the reinforcement learning module itself, this invention employs a PPO mechanism to update the parameters of the policy network and the evaluation network. The policy network loss uses a pruning policy optimization objective.

[0104] in:

[0105] The network loss is evaluated using the state value regression error:

[0106] The overall loss function consists of policy loss, value loss, and policy entropy regularization term:

[0107] The trajectory data used for training the reinforcement learning module comes from both real traffic data and imaginary trajectories generated through rolling world model simulations. Real traffic trajectories reflect the actual effectiveness of boundary control strategies in real traffic systems, while imaginary trajectories are obtained by online simulations of candidate control sequences using the world model, expanding the policy evaluation sample and improving search efficiency. The two types of trajectories are mixed in a fixed ratio into the PPO training batch, ensuring that reinforcement learning policy updates maintain consistency with real traffic operations while possessing sufficient sample utilization efficiency and policy exploration capabilities.

[0108] After the boundary control action is executed, the system continues to collect real traffic observations for the next moment. and the results of the world model prediction By comparing the results, the prediction error is obtained:

[0109] This invention employs an online update mechanism that combines error triggering with periodic updates. When the normalized prediction error satisfies... When an error update is triggered, an incremental update of the world model is immediately executed; if no error update is triggered for 12 consecutive control cycles, a periodic update is performed. Threshold The 95th percentile of the normalized root mean square error of the one-step prediction on the offline validation set is taken, and in this embodiment of the invention, it is fixed at 0.10. During the online update process, not all parameters of the world model are modified; only the inference module is updated. The parameters and the online adaptation layer parameters set at the input of the inference module, encoder With decoder The update process remains frozen during the online phase. This approach ensures the stability of the state representation space and the observation reconstruction mapping, while enabling the model to make targeted corrections for dynamic drift caused by fluctuations in traffic demand, changes in signal timing, and sudden disturbances.

[0110] The online update of the inference module parameters is written as:

[0111] To prevent catastrophic forgetting during continuous online updates, this invention introduces an experience replay mechanism, a flexible weight consolidation mechanism, and a small-step parameter update mechanism during the online update phase. The experience replay mechanism maintains a fixed-capacity sample cache, ensuring that each online update simultaneously includes representative offline samples and the latest online samples, guaranteeing that the model retains its memory of historical typical traffic evolution patterns while learning new traffic patterns. The flexible weight consolidation mechanism adds a parameter offset constraint term based on Fisher information to the online loss function, limiting the deviation of key parameters from the optimal parameter points after offline training. Its regularization term is expressed as:

[0112] in, These are reference parameters after offline training. The Fisher information weighting coefficients are used. The small step size parameter update mechanism fixes the online learning rate to one-twentieth of the offline training learning rate, ensuring that model parameter adjustments always maintain a gradual characteristic. Therefore, the total loss for online updates is written as:

[0113] In summary, this invention incorporates traffic control data, demand data, accumulated data, and travel completion rate into the macroscopic traffic state description. It deeply couples a macroscopic traffic world model comprised of encoders, inference modules, and decoders with a reinforcement learning-driven candidate control sequence optimization mechanism. This forms a complete closed-loop control method that starts from the current traffic observation state, proceeds through latent variable modeling in the latent space, predicts future traffic evolution, evaluates candidate boundary control sequences, executes optimal decisions, and updates online via error triggering. This method simultaneously enhances the forward-looking perception of future state changes, the goal-oriented nature of control action generation, the stability of multi-step rolling inference, and the system's online adaptability in non-stationary traffic environments in regional traffic boundary control scenarios, thereby achieving higher travel completion rates, lower congestion levels, and superior boundary control performance.

[0114] Compared to existing regional boundary control schemes based on macroscopic fundamental maps, this invention no longer relies on a single static macroscopic relationship, fixed threshold, or empirical rules to regulate boundary flow. Instead, it explicitly learns the dynamic evolution of the traffic macroscopic state under control input through a world model. Therefore, it can predict and compare the future effects of different candidate control sequences under conditions of significant demand fluctuations and system non-steady-state operation. Specifically, this invention unifies state prediction, strategy evaluation, and boundary control into a single closed loop, elevating the control basis from static interval discrimination to future state projection results. MFD is only used as a comparative example in the background art.

[0115] Compared to reinforcement learning-based regional boundary control schemes, this invention does not rely entirely on trial-and-error training through extensive environmental interactions. Instead, it uses a world model to predict and then make decisions, continuously refining the model based on real observations. Therefore, it has significant advantages in sample utilization efficiency, policy search efficiency, and long-term benefit evaluation capabilities. In this closed loop, the world model not only undertakes the prediction task but also supports the comparison of candidate control sequences and online correction, making it an indispensable core module for achieving forward-looking boundary control. This invention further enhances the model's adaptability to demand distribution drift and scenario changes through an online error adaptive update mechanism, making it more suitable for continuous deployment in real, complex transportation networks.

[0116] This invention combines the regional perspective of macro-traffic control, the dynamic expression capability of data-driven models, and the forward-looking decision-making capability of strategy optimization methods. It can improve the stability, robustness, and practical application value of regional boundary control while ensuring the feasibility of engineering projects.

[0117] Compared with existing technologies, this invention provides at least the following innovative means and achieves related beneficial effects: (1) Modeling method for traffic world model based on the "observation-deduction-prediction" structure: This invention proposes a world modeling method for macroscopic traffic systems. Its core lies in employing an "observation-deduction-prediction" structure for unified modeling of the traffic system. Specifically, the observation module maps macroscopic traffic state information to a low-dimensional latent space representation to extract key features of the traffic system; the deduction module depicts the evolution of traffic states under control inputs, realizing the modeling of the dynamic changes in the traffic system; and the prediction module restores the latent space states to macroscopic indicators in the traffic observation space, including trends in traffic accumulation and trip completion rates. Through this structure, this invention achieves a unified modeling approach from "current traffic state and control inputs" to "future traffic state." Unlike traditional methods based on static macroscopic basic graphs or simple regression models, this invention uses a world model for data-driven learning of traffic state evolution patterns, thereby more accurately depicting the dynamic changes of complex traffic systems.

[0118] (2) Traffic state extrapolation modeling mechanism involving control inputs: This invention introduces control input as a crucial variable in the inference module during traffic system modeling, ensuring that the evolution of traffic states depends not only on the current state but also directly on the control strategy. Specifically, during the inference process, the current latent space state and the boundary flow control input are used together as model inputs to predict the traffic state at the next moment, thereby establishing a dynamic evolutionary relationship of "state-control-state". This mechanism enables the model to characterize the impact of control behavior on the evolution of the traffic system, overcoming the limitation of traditional methods that only describe the correlation between states.

[0119] (3) Multi-step rolling deduction and control strategy evaluation method based on world model: This invention utilizes the extrapolation capabilities of a world model to perform multi-step rolling simulations of candidate control strategies, thereby achieving continuous assessment of future traffic conditions. Within each control cycle, multiple candidate control sequences are generated, and the traffic conditions under different control sequences are simulated step-by-step based on the world model, calculating the corresponding travel completion rate and congestion level. By comparing the cumulative effects of different strategies, the optimal control strategy is selected and the current control action is executed. This method achieves decision optimization based on future traffic evolution trends, significantly improving the foresight of control strategies.

[0120] (4) Boundary control method combining world model and reinforcement learning: This invention combines world models with reinforcement learning methods to construct a boundary control framework that coordinates prediction and control. In this framework, reinforcement learning is used to generate candidate control policies, while the world model is used to evaluate the future evolution of the traffic system under different policies, thereby guiding policy optimization. This method differs from traditional model-free reinforcement learning methods by utilizing model prediction information to improve policy search efficiency. It also differs from static model-based control methods by achieving integrated design of prediction and control.

[0121] (5) Online adaptive model update mechanism based on prediction error: To address the non-stationarity issues inherent in traffic systems during actual operation, this invention proposes an online adaptive update mechanism based on prediction error. During system operation, real-time traffic observation data is acquired and compared with model prediction results to calculate the prediction error. The model parameters are then dynamically adjusted based on this error. This mechanism enables the model to continuously adapt to changes in traffic demand and random disturbances, improving prediction accuracy and ensuring the stability and effectiveness of the control strategy in complex environments.

[0122] (6) Dynamic modeling and control methods that do not rely on macroscopic basic graphs: This invention introduces a world model to replace the traditional static macroscopic basic diagram, enabling dynamic modeling of the unsteady-state processes of a transportation system. Compared to the macroscopic basic diagram that relies on historical data fitting, the model constructed in this invention can describe the dynamic evolution of the transportation system under different control inputs, overcoming the problems of difficulty in calibrating the macroscopic basic diagram and the inability to observe dynamic changes. This allows the boundary control method to maintain good adaptability under conditions of demand fluctuations, signal timing changes, and sudden disturbances.

[0123] (7) Traffic control methods that support the migration from simulation to reality: This invention employs a combination of offline training and online adaptive updating, enabling the model to learn the dynamic laws of traffic systems in a simulation environment and continuously correct itself based on real-time data during actual operation, thereby achieving effective transfer from the simulation environment to real traffic systems. This method reduces reliance on manual modeling and fixed parameter calibration, improves the model's generalization ability and engineering application feasibility, and is suitable for boundary control problems in complex traffic scenarios.

[0124] like Figure 5 As shown, this embodiment of the invention also provides a region boundary control device 900 based on a world model, which can implement the above-described method. This device may include: The first module 901 is used to acquire historical operational data of the regional transportation system and construct a state transition sample set that includes the macro state of the previous moment, the boundary control input, and the macro state of the next moment. The second module 902 is used to jointly train a preset world model using a state transition sample set, so that the world model learns the mapping relationship between the macroscopic state and the latent space, and learns the dynamic evolution relationship from the macroscopic state and boundary control input of the previous moment to the macroscopic state of the next moment in the latent space; wherein, the world model includes an encoder, an inference module and a decoder. The third module 903 is used to set the current time as the target time; The fourth module, 904, is used to obtain the real macroscopic traffic state at the target time and map it into the target latent space variable using the encoder. The fifth module, 905, is used to generate candidate boundary control sequences based on real-world traffic conditions using a reinforcement learning policy network. The sixth module, 906, is used to obtain the future macro-level traffic conditions for multiple control cycles by using the inference module and decoder to infer the target latent space variables and candidate boundary control sequences. Module 7, 907, is used to evaluate the cumulative benefits of each candidate boundary control sequence based on the future macro-level traffic conditions using an evaluation network. The eighth module 908 is used to execute the target boundary control action based on the candidate boundary control sequence with the highest cumulative profit, take the next time step of the target time as the target time, return to execute the operation of the fourth module, and continue to perform regional boundary control; Module 909 is used to construct prediction errors based on the future macro-state of traffic and the actual macro-state of traffic at the same time. Based on the prediction error, an update mechanism is triggered to correct the parameters of the world model.

[0125] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0126] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0127] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0128] like Figure 6 As shown, Figure 6 The hardware structure of an electronic device 1000 according to another embodiment is illustrated. The electronic device 1000 includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (aSIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RaM). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0129] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0131] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0132] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0133] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0134] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0135] This invention provides a method, apparatus, electronic device, storage medium, and program product for regional boundary control based on a world model. It acquires historical operational data of a regional traffic system to construct a state transition sample set containing the macroscopic state of the previous moment, boundary control input, and the macroscopic state of the next moment. The state transition sample set is used to jointly train a preset world model, enabling the world model to learn the mapping relationship between macroscopic states and latent space, and to learn the dynamic evolution relationship from the macroscopic state and boundary control input of the previous moment to the macroscopic state of the next moment in the latent space. The world model includes an encoder, an inference module, and a decoder. The current moment is used as the target moment. The actual traffic macroscopic state at the target moment is acquired and mapped to target latent space variables using the encoder. Based on... Based on the real traffic macro-state, a reinforcement learning policy network is used to generate candidate boundary control sequences. Using the target latent space variables and the candidate boundary control sequences, a rolling inference module and decoder are used to deduce the future traffic macro-state for multiple control cycles. Based on the future traffic macro-state, an evaluation network is used to assess the cumulative benefit of each candidate boundary control sequence. The target boundary control action is executed based on the candidate boundary control sequence with the highest cumulative benefit. The next time step after the target time is taken as the target time, and the process returns to obtain the real traffic macro-state at the target time, continuously performing regional boundary control. A prediction error is constructed based on the future traffic macro-state and the real traffic macro-state at the same time. An update mechanism is triggered based on the prediction error to correct the parameters of the world model. This invention does not rely on MFD-style static relationships, fixed macroscopic evolution equations, or manual dynamics as control criteria. Instead, it constructs a world model for regional macroscopic traffic systems, mapping the current macroscopic state, external demand, and boundary control inputs to the future state evolution process, enabling the model to perform continuous multi-step rolling deductions. Furthermore, based on reinforcement learning and policy optimization mechanisms, the operational effectiveness of candidate boundary control sequences in the future time domain is evaluated, thereby achieving forward-looking boundary control for future states. Simultaneously, an online adaptive update mechanism based on prediction errors is introduced to mitigate performance degradation caused by model mismatch, demand drift, and scenario changes, improving the system's robustness, generalization ability, and engineering deployment feasibility in complex traffic environments. Specifically, this invention does not simply improve traffic prediction accuracy or learn boundary control strategies in isolation; rather, it constructs an integrated closed-loop technical solution for regional traffic control scenarios, encompassing dynamic modeling, rolling deduction, policy evaluation, and online correction, enabling boundary control decisions to shift from being driven by static thresholds to being driven by future states.

[0136] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0137] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0138] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0140] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A regional boundary control method based on a world model, characterized in that, The method includes the following steps: Acquire historical operational data of the regional transportation system and construct a state transition sample set that includes the macroscopic state of the previous moment, boundary control inputs, and the macroscopic state of the next moment. The world model is jointly trained using the state transition sample set so that the world model learns the mapping relationship between the macroscopic state and the latent space, and learns the dynamic evolution relationship from the previous macroscopic state and the boundary control input to the next macroscopic state in the latent space; wherein, the world model includes an encoder, an inference module and a decoder; Use the current time as the target time; The actual macroscopic traffic state at the target time is obtained and mapped to the target latent space variable using the encoder. Based on the actual macroscopic traffic conditions, a candidate boundary control sequence is generated using a reinforcement learning policy network. Based on the target latent space variables and the candidate boundary control sequence, the future macroscopic traffic state for multiple control cycles is obtained by using the inference module and the decoder in a rolling inference. Based on the future macro-state of traffic, the cumulative benefits of each of the candidate boundary control sequences are evaluated using an evaluation network; Execute the target boundary control action based on the candidate boundary control sequence with the highest cumulative revenue, take the next time of the target time as the target time, return to the step of obtaining the real traffic macro state at the target time, and continue to perform regional boundary control; A prediction error is constructed based on the future macro-state of traffic and the actual macro-state of traffic at the same time. An update mechanism is triggered based on the prediction error to correct the parameters of the world model. Each set of candidate boundary control sequences includes continuous action vectors for multiple future control cycles. The process of using the inference module and the decoder to perform rolling inference to obtain the future macroscopic traffic state for multiple future control cycles based on the target latent space variables and the candidate boundary control sequences includes the following steps: For each set of candidate boundary control sequences, starting from the target latent space variables, iteratively perform the following operations: The target latent space variable is used as a candidate latent space variable, and the action vector of the first control cycle in the candidate boundary control sequence is used as a candidate action vector. The candidate latent space variables are concatenated with the candidate action vectors and then input into the inference module to predict the predicted latent space variables for the next control cycle. The predicted latent space variables are input into the decoder to obtain the future macroscopic traffic state for the corresponding control period. The predicted latent space variable is used as the candidate latent space variable, and the action vector of the next control cycle in the candidate boundary control sequence is used as the candidate action vector. The step of concatenating the candidate latent space variable with the candidate action vector is returned to be executed until the future traffic macro state of multiple future control cycles is obtained. The reinforcement learning policy network generates the candidate boundary control sequences using a cross-entropy optimization method. The cumulative benefit of each candidate boundary control sequence is evaluated using an evaluation network based on the future macro-level traffic conditions, and includes the following steps: For each set of candidate boundary control sequences, a reward value for each time step of each control cycle is constructed using a preset reward function based on the future traffic macro-state of the corresponding multiple future control cycles. The reward function is determined by a weighted sum of regional travel completion rate, regional congestion penalty, boundary queuing penalty, and control action change rate penalty. The reward values ​​of all control cycles of the same group of candidate boundary control sequences are summed to obtain the cumulative revenue of the corresponding candidate boundary control sequence. The predetermined number of candidate boundary control sequences with the highest cumulative returns are used as elite samples, and the cross-entropy sampling distribution of the cross-entropy optimization method is updated using the elite samples; wherein, the candidate boundary control sequences with the highest cumulative returns are determined based on the final cumulative return results after multiple consecutive iterations of optimization; The parameter correction of the world model based on the prediction error-triggered update mechanism includes the following steps: When the prediction error exceeds a preset threshold, a triggered update is executed; or, if the triggered update is not generated for a preset number of consecutive control cycles, a periodic update is executed. In response to the triggered update or the periodic update, the parameters of the encoder and the decoder are frozen, and the parameters of the inference module and the online adaptation layer parameters of the input end of the inference module are updated. The parameter update adopts a preset small step parameter update mechanism, and the parameter update is also configured with an experience playback mechanism and an elastic weight consolidation mechanism.

2. The method according to claim 1, characterized in that, The joint training of the preset world model using the state transition sample set includes the following steps: The previous macroscopic state in the state transition sample set is input into the encoder, and the encoder is processed to output training latent space variables. The boundary control input is mapped to an action feature vector using an action embedding network, and the action feature vector is concatenated with the training latent space variables to form a fusion vector; A historical sequence is constructed based on the fusion vectors at multiple consecutive time points. The historical sequence is then input into the inference module, which processes and outputs the latent space variables for the next time point. The hidden space variables of the next time step are input into the decoder, and the output is used to train and infer the macroscopic state. A loss function is constructed based on the macroscopic state of the training and the macroscopic state of the next moment. The parameters of the encoder, the inference module and the decoder are jointly optimized using the loss function, and then the parameters are frozen.

3. The method according to claim 2, characterized in that, The construction of the loss function based on the macroscopic state derived from the training and the macroscopic state at the next moment includes the following steps: Based on the macroscopic state of the training simulation and the macroscopic state of the next moment, a prediction loss and a multi-step rolling simulation loss are constructed. Based on the macroscopic state derived from the training and the macroscopic state at the next moment, a data consistency loss is constructed through traffic flow statistical consistency constraints. The prediction loss, the data consistency loss, and the multi-step rolling inference loss are weighted and fused to obtain the total model loss function.

4. The method according to claim 1, characterized in that, The process of generating candidate boundary control sequences based on the actual macroscopic traffic conditions using a reinforcement learning policy network includes the following steps: The real traffic macro-state is input into the reinforcement learning policy network, and the output action distribution is processed. Using the action distribution as the initial sampling distribution, multiple sets of the candidate boundary control sequences are generated in the rolling simulation time domain using the cross-entropy optimization method; Each candidate boundary control sequence includes a series of action vectors for multiple future control cycles, and each action vector is obtained based on a preset boundary flow regulation constraint.

5. A regional boundary control device based on a world model, characterized in that, The device includes: The first module is used to acquire historical operational data of the regional transportation system and construct a state transition sample set that includes the macroscopic state of the previous moment, boundary control inputs, and the macroscopic state of the next moment. The second module is used to jointly train a preset world model using the state transition sample set, so that the world model learns the mapping relationship between the macroscopic state and the latent space, and learns the dynamic evolution relationship from the previous macroscopic state and the boundary control input to the next macroscopic state in the latent space; wherein, the world model includes an encoder, an inference module and a decoder. The third module is used to set the current time as the target time. The fourth module is used to obtain the real macroscopic traffic state at the target time and map it into target latent space variables using the encoder; The fifth module is used to generate candidate boundary control sequences based on the actual macroscopic traffic conditions using a reinforcement learning policy network. The sixth module is used to obtain the future macroscopic traffic state for multiple control cycles by using the inference module and the decoder to perform rolling inference based on the target latent space variables and the candidate boundary control sequence. The seventh module is used to evaluate the cumulative benefits of each of the candidate boundary control sequences based on the future macro-state of traffic using an evaluation network. The eighth module is used to execute the target boundary control action based on the candidate boundary control sequence with the highest cumulative benefit, take the next time of the target time as the target time, return to execute the operation of the fourth module, and continue to perform regional boundary control; The ninth module is used to construct a prediction error based on the future macro-state of traffic and the actual macro-state of traffic at the same time, and to correct the parameters of the world model based on the prediction error triggering an update mechanism. Each set of candidate boundary control sequences includes continuous action vectors for multiple future control cycles. The process of using the inference module and the decoder to perform rolling inference to obtain the future macroscopic traffic state for multiple future control cycles based on the target latent space variables and the candidate boundary control sequences includes the following steps: For each set of candidate boundary control sequences, starting from the target latent space variables, iteratively perform the following operations: The target latent space variable is used as a candidate latent space variable, and the action vector of the first control cycle in the candidate boundary control sequence is used as a candidate action vector. The candidate latent space variables are concatenated with the candidate action vectors and then input into the inference module to predict the predicted latent space variables for the next control cycle. The predicted latent space variables are input into the decoder to obtain the future macroscopic traffic state for the corresponding control period. The predicted latent space variable is used as the candidate latent space variable, and the action vector of the next control cycle in the candidate boundary control sequence is used as the candidate action vector. The step of concatenating the candidate latent space variable with the candidate action vector is returned to be executed until the future traffic macro state of multiple future control cycles is obtained. The reinforcement learning policy network generates the candidate boundary control sequences using a cross-entropy optimization method. The cumulative benefit of each candidate boundary control sequence is evaluated using an evaluation network based on the future macro-level traffic conditions, and includes the following steps: For each set of candidate boundary control sequences, a reward value for each time step of each control cycle is constructed using a preset reward function based on the future traffic macro-state of the corresponding multiple future control cycles. The reward function is determined by a weighted sum of regional travel completion rate, regional congestion penalty, boundary queuing penalty, and control action change rate penalty. The reward values ​​of all control cycles of the same group of candidate boundary control sequences are summed to obtain the cumulative revenue of the corresponding candidate boundary control sequence. The predetermined number of candidate boundary control sequences with the highest cumulative returns are used as elite samples, and the cross-entropy sampling distribution of the cross-entropy optimization method is updated using the elite samples; wherein, the candidate boundary control sequences with the highest cumulative returns are determined based on the final cumulative return results after multiple consecutive iterations of optimization; The parameter correction of the world model based on the prediction error-triggered update mechanism includes the following steps: When the prediction error exceeds a preset threshold, a triggered update is executed; or, if the triggered update is not generated for a preset number of consecutive control cycles, a periodic update is executed. In response to the triggered update or the periodic update, the parameters of the encoder and the decoder are frozen, and the parameters of the inference module and the online adaptation layer parameters of the input end of the inference module are updated. The parameter update adopts a preset small step parameter update mechanism, and the parameter update is also configured with an experience playback mechanism and an elastic weight consolidation mechanism.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 4.

7. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 4.

Images