Distributed optimization method for road network traffic signal based on hierarchical reinforcement learning

Abstract

The application relates to the field of wisdom tourism and intelligent traffic integration, and discloses a layered reinforcement learning-based distributed optimization method for road network traffic signals, which comprises the following steps: a multi-modal scenic spot traffic semantic knowledge graph is constructed, and an implicit passenger flow conduction structure is reconstructed by utilizing mutual information; a layered control architecture is adopted; a macroscopic scheduling logic operator is generated by a neural symbol reasoning in an upper-layer agent, and is converted into a continuous semantic context vector via a Gumbel-Softmax differentiable interface; a reference dredging strategy is matched based on a scenic spot flow fingerprint library in a lower-layer agent, and a residual network is combined to adaptively fine-tune the actions of traffic signals or control facilities; and an extreme passenger flow virtual sample is generated by counterfactual reasoning to perform hybrid training. The application solves the problems of cold start control difficulty and poor adaptability to sudden peaks in a tourism scene, and realizes the dynamic balance of scenic spot touring efficiency and traffic order.

Description

Technical Field

[0001] This invention relates to the field of smart tourism and intelligent transportation integration technology, specifically a distributed optimization method for road network traffic signals based on hierarchical reinforcement learning. Background Technology

[0002] With the booming development of the cultural tourism industry, popular scenic spots face enormous pressure from both visitor flow and traffic during holidays and peak tourist seasons. The scenic area's transportation system not only includes the introduction of external vehicles but also involves internal shuttle buses, private cars, and high-density pedestrian movement, exhibiting strong spatiotemporal dynamics and complexity. Existing scenic area traffic management methods mainly rely on manual experience or isolated signal control systems, often only optimizing single intersections and lacking an understanding of the coupling relationship between "scenic spot popularity" and "road traffic."

[0003] However, existing technologies have significant limitations in applications within the cultural and tourism sector. First, traditional methods neglect the implicit, long-range impact of scenic spot semantic information on traffic flow. For example, overflowing crowds at a core scenic spot often trigger cascading congestion at distant parking lots or intersections via non-physically connected paths. Static topology-based control methods cannot capture this causal transmission based on visitor behavior.

[0004] Secondly, existing deep learning models lack interpretability. Macro-level scheduling instructions for scenic areas, such as "interlateral flow control" and "balancing the load of attractions," are difficult to translate into specific seconds of micro-level signals through end-to-end methods, resulting in a disconnect between management intentions and execution actions.

[0005] Furthermore, scenic area traffic exhibits significant "tidal effects" and "suddenness," making it difficult for historical data to cover all extreme scenarios. Purely data-driven methods face extremely high safety risks during the cold start phase and struggle to cope with sudden large-scale visitor flows with a long tail distribution, failing to meet the high standards of safety and experience required by smart scenic areas. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a distributed optimization method for road network traffic signals based on hierarchical reinforcement learning. This method solves the technical problems of existing scenic area management systems, such as a lack of cross-domain causal reasoning capabilities regarding "tourism-traffic," a disconnect between macro-level scheduling and micro-level execution, and insufficient policy robustness under extreme peak visitor flow and cold start scenarios.

[0007] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of the present invention provides a distributed optimization method for road network traffic signals based on hierarchical reinforcement learning.

[0008] This invention constructs a multimodal scenic area traffic semantic knowledge graph and dynamically reconstructs the implicit passenger flow transmission structure between attractions and intersections using mutual information. It employs a layered architecture, with upper-layer agents reasoning and generating macro-level logic such as "regional collaborative flow restriction," while lower-layer agents adaptively execute based on a scenic area flow fingerprint database and residual network. Counterfactual reasoning is used to simulate virtual extreme congestion or accident scenarios for enhanced training. This invention achieves a shift in scenic area traffic management from "passive guidance" to "active causal control," effectively improving visitor traffic efficiency and safety.

[0009] This method constructs a traffic knowledge graph with self-evolution capabilities and utilizes a hierarchical architecture to achieve closed-loop control from "semantic intent reasoning" to "adaptive execution," specifically including the following steps: First, an initial knowledge graph of multimodal traffic semantics is constructed. A heterogeneous graph structure describing the basic state of the road network is established, and this heterogeneous graph structure contains a set of nodes. Sum of edges The set of nodes Specifically, it includes three types of heterogeneous entities: physical entities, including intersections, lanes, and detector nodes; semantic entities, including points of interest (POIs) and functional area nodes around the road network; and event entities, including traffic congestion levels, weather conditions, and special event nodes. The edge set... It includes topological relationships representing physical connections, semantic relationships representing spatial affiliation, and dynamic relationships representing state influence.

[0010] Secondly, based on the feedback signals from reinforcement learning, the initial multimodal traffic semantic knowledge graph is dynamically reconstructed to generate a real-time evolving knowledge graph containing implicit causal edges. This step aims to mine causal relationships between non-physically connected intersections. Specifically, it calculates the causal relationships between any two non-adjacent nodes. , Causal correlation score of traffic state sequences between :

[0011] in, Represents the node state vector. For a time window. When When the threshold is exceeded, implicit causal edges are generated in the graph. Simultaneously, temporal difference errors from reinforcement learning are introduced. As a feedback signal, the weight of the opposite edge Perform adaptive updates:

[0012] in The learning rate is the structural learning rate. If the existence of an implicit causal edge fails to reduce the prediction error, the weight of that edge is reduced, thereby achieving dynamic evolution of the graph structure.

[0013] Next, the upper-level management agent performs neural symbolic reasoning based on the real-time evolutionary knowledge graph to generate discrete logic operators representing high-order control strategies. The upper-level management agent then uses a relational graph convolutional network to aggregate heterogeneous neighbor information and update the node embedding representations. :

[0014] in For nodes In relationship The neighbor set. Based on global state embedding, the upper-level management agent selects from a predefined set of logical operators. We select logical operators, each of which corresponds to a specific macro-level traffic control strategy objective.

[0015] Subsequently, the discrete logic operators are transformed into continuous differentiable vectors through a differentiable logic interface and passed to the lower-level execution agent. To achieve end-to-end gradient propagation, this step utilizes the Gumbel-Softmax technique to sample the selection probability distribution of the logic operators, generating continuous one-hot approximation vectors. :

[0016] in For noise terms, This represents the temperature coefficient. Furthermore, it is achieved through a learnable embedding matrix... Mapped to semantic embedding vector This is passed as a differentiable vector to the lower layer. During this process, the gradient signals of the lower-layer agent are allowed to pass through this interface to update the parameters of the upper layer. To ensure that the behavior of the lower layer conforms to the intention of the upper layer, a semantic alignment loss function is constructed. Calculate the state transition distribution of the lower level Ideal distribution corresponding to the upper-level logic operator Kullback-Leibler divergence between:

[0017] Then, the lower-level execution agent, based on the current local traffic state of the intersection, combines a pre-set traffic flow pattern fingerprint database with the received differentiable vector to generate the final residual action to control the traffic signal. The lower-level execution agent first extracts the current local state features. Calculate its relationship with each basic strategy prototype in the fingerprint database. Based on the similarity, select the baseline strategy corresponding to the best matching prototype. :

[0018] Generating motion adjustment amounts using residual neural networks final action It is composed of the base strategy action and the adjustment amount:

[0019] Furthermore, the method of the present invention also includes an enhanced training step based on counterfactual reasoning. The semantic attributes of entities are perturbed in the real-time evolutionary knowledge graph to generate counterfactual initial states; forward inference is performed using the learned environment model to generate counterfactual trajectories. The system employs a hybrid optimization objective function. Training was performed using reinforcement learning loss based on real data. Semantic alignment loss and counterfactual consistency loss :

[0020] This step improves the generalization ability of the agent in sparse and extreme scenarios by generating virtual samples.

[0021] A second aspect of the present invention provides an electronic device. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method described in the first aspect.

[0022] This invention provides a distributed optimization method for road network traffic signals based on hierarchical reinforcement learning. It has the following beneficial effects: 1. Breaking through physical topological limitations, accurately capturing implicit traffic transmission mechanisms. This invention abandons the limitations of traditional methods that rely solely on static road network geographical connections for modeling, and creatively introduces a dynamic reconstruction mechanism of causal structures based on reinforced feedback. The system can automatically identify and establish "implicit causal edges" between non-physically connected intersections based on the temporal mutual information of traffic flow and prediction error feedback. This means that when congestion in a certain area affects distant intersections "remotely" due to complex traffic flow weaving, the evolutionary map constructed by this invention can instantly capture this nonlinear transmission relationship, thereby achieving preemptive collaborative control before congestion physically spreads, significantly improving the road network's perception depth of complex flow patterns.

[0023] 2. Breaking down the training barriers between symbolic logic and neural networks to achieve interpretable end-to-end optimization. This invention cleverly solves the problem of the inability to jointly train upper-layer discrete symbolic reasoning and lower-layer continuous neural network control in hierarchical architectures by constructing a differentiable logic interface. Utilizing Gumbel-Softmax relaxation and semantic embedding techniques, this invention transforms high-order policies into continuous vectors that can propagate gradients, allowing the execution feedback from the lower layers to directly propagate back to the upper-layer inference engine. This not only endows deep learning models with scarce logical interpretability but also ensures the consistency between macroscopic decision-making and microscopic execution in terms of mathematical optimization objectives, avoiding the policy fragmentation phenomenon common in traditional hierarchical methods.

[0024] 3. By integrating expert experience and data-driven approaches, this invention addresses control instability during the cold start period. Relying on the pain point of pure reinforcement learning methods easily leading to traffic paralysis due to random exploration in the early training stages, this invention designs a hybrid execution mechanism of "flow fingerprint matching + residual correction." The lower-level agent does not start from scratch through trial and error, but instead prioritizes matching pre-set expert policy prototypes based on real-time flow patterns, learning only the fine-tuning amount for the current specific scenario through the residual network. This design ensures that the system possesses expert-level baseline traffic capability from the outset, while retaining the flexibility to continuously evolve with data accumulation, greatly enhancing the system's security and robustness in actual deployment.

[0025] 4. To enhance the semantic consistency of intentions between upper and lower layers and avoid the trap of local optima, this invention introduces a semantic alignment loss function based on KL divergence. At the mathematical level, this invention mathematically forces the behavioral distribution of lower-layer agents to closely approximate the expected distribution of upper-layer logical operators. This mechanism effectively prevents distributed agents from deviating from the overall regional control objective when pursuing the maximization of local intersection traffic efficiency. It establishes a rigid "command-execution" contract, ensuring that macro-level tactical intentions can be faithfully "translated" into specific phase actions by the lower-level intersections.

[0026] 5. By leveraging counterfactual reasoning to generate virtual samples, this invention enhances generalization ability in extreme scenarios. Addressing the scarcity of long-tail samples in the real world, such as severe accidents and extreme weather, this invention utilizes the causal reasoning capabilities of knowledge graphs to construct a counterfactual data augmentation module. The system can proactively infer virtual scenarios such as "how the road network state would evolve if an accident occurred" and inject the generated counterfactual trajectories into the training process. This proactive mechanism enables the agent to make reasonable emergency responses based on reasoning logic even in extreme conditions it has never experienced before, significantly improving the model's adaptability and generalization level to unexpected events. Attached Figure Description

[0027] Figure 1 This is the main flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the cloud-edge-device collaborative system architecture of the present invention; Figure 3 This is a diagram of the evolutionary neural symbol hierarchical control framework of the present invention; Figure 4 This is a schematic diagram illustrating the construction and evolution of the multimodal traffic semantic knowledge graph of the present invention; Figure 5 This is the execution logic diagram of the present invention based on flow fingerprint matching and residual correction; Figure 6 This is a flowchart of the data augmentation training process based on counterfactual reasoning of the present invention; Figure 7 This is a block diagram of the hardware structure of the electronic device of the present invention. Detailed Implementation

[0028] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Please see the appendix Figures 1 to 7 This invention provides a distributed optimization method for road network traffic signals based on hierarchical reinforcement learning, including the following steps: First, step S1 is executed to construct an initial knowledge graph of multimodal traffic semantics. Unlike traditional methods that only abstract the road network into a simple node-edge topology, this embodiment constructs a heterogeneous graph structure containing physical attributes, semantic information, and dynamic events, serving as the underlying memory and reasoning foundation of the system. The knowledge graph at the initial moment is defined as follows: ,in Represents a set of nodes. Represents the set of edges Represents a set of attribute features.

[0030] The set of nodes Specifically, it consists of three types of heterogeneous entities, represented as follows: .in, This refers to a set of physical entities, encompassing controlled intersections, connecting road segments, and induction coil or radar detector nodes deployed on the road surface. These nodes form the backbone of the road network and bear the physical constraints of vehicle passage. This represents a set of semantic entities, encompassing points of interest (POIs) and functional areas surrounding the road network, such as "top-tier hospital," "CBD core area," "primary and secondary schools," or "large sports stadium." The purpose of introducing such nodes is to give the system the ability to understand the sources of traffic flow. For example, the "hospital" node is often associated with high-priority ambulance traffic needs and stable, 24 / 7 traffic flow, while the "sports stadium" node is associated with sudden tidal traffic flows. This represents a set of event entities used to characterize non-physical environmental states, including but not limited to virtual nodes such as "rainstorm weather," "road construction," "traffic accident," or "traffic control." These nodes affect the traffic efficiency of physical intersections through dynamic connections.

[0031] The set of edges Various relationships between nodes are defined and formalized as triples. ,in , For nodes, For relation types. In this embodiment, the set of relation types. It includes at least the following three types: The first type is a physical connection relationship, which exists in... Internally, it represents the actual road connection between intersections, with edge weights initialized to normalized values ​​of road segment length or lane number; the second type is spatial affiliation, existing within... and The first type is the connection between nodes, such as an intersection node connected to a "commercial area" node, indicating that the intersection is influenced by the traffic attraction and characteristics generated by the commercial area; the second type is the state influence relationship, which exists in... and Between them, for example, the "water accumulation" event node is connected to a specific road segment node, indicating that the environmental factor inhibits the traffic capacity of the road segment.

[0032] For the graph structure constructed above, the system performs set... The node attributes in the data are initialized and embedded. For each node... First, obtain its original feature vector. For physical nodes, It includes static attributes and a dynamically perceived initial state; for semantic nodes, Includes one-hot encoding for its functional classification and historical popularity index; for event nodes, This includes predicted event severity and duration. Subsequently, a learnable linear projection layer maps the original features of different dimensions to a unified-dimensional hidden state space, resulting in the initial node embeddings. :

[0033] in, and These are the projection weight matrices and bias vectors set for different node types (physical, semantic, event), ensuring that heterogeneous information can be uniformly encoded and processed. This step constructs... It is not static and unchanging, but rather serves as the initial state input for the causal structure evolution and neural symbolic reasoning in subsequent steps. By introducing semantics and event entities, the system can not only perceive that "traffic is increasing," but also understand "why it is increasing" at the reasoning level, thus laying the data foundation for the subsequent generation of interpretable control strategies.

[0034] After constructing the initial multimodal traffic semantic knowledge graph, this embodiment further executes step S2, dynamically reconstructing the causal structure of the knowledge graph based on the feedback signal of reinforcement learning. Traditional traffic control methods are often limited to fixed physical road network topologies, making it difficult to capture the complex traffic flow transmission effects between non-directly connected intersections. For example, overflow from an upstream intersection may cause congestion in a non-adjacent downstream area within minutes. To address this, this invention introduces a dynamic evolution mechanism driven by both data and models, enabling the structure of the knowledge graph to be adjusted in real time according to changes in traffic conditions.

[0035] Specifically, the system first initiates a causal correlation mining process to discover implicit traffic transmission paths beyond the physical topology. During operation, the system collects real-time data on any two non-physically adjacent nodes in the road network. and Traffic status time series. A sliding time window is set. Calculate nodes within this window state sequence With nodes Lag time The subsequent state sequence Mutual information between them is used as a causal association score. The formula for calculating this score is as follows:

[0036] in, Represents a node At any moment The state vector, This represents the probability distribution function. When the calculated causal association score... Exceeding the preset significance threshold At that time, the system determines the node. For nodes There is a significant implicit influence, which in turn generates a directed implicit causal edge between these two nodes. The newly generated edge is added to the current edge set. This allows subsequent inference calculations to directly utilize this cross-space correlation information without needing to pass through multiple physical nodes, thereby reducing information attenuation and noise interference.

[0037] Building upon the generation of implicit connections, this embodiment further introduces an adaptive edge weight adjustment mechanism based on reinforcement learning feedback. Each edge in the graph (including physical edges and implicit causal edges) is assigned a dynamic weight. This weight represents the importance of information transmission between nodes. To ensure that the graph structure not only conforms to statistical laws but also directly serves the final control objective, the system utilizes the temporal difference error (TD-Error) generated by the lower-level executive agent during training. The weights are updated as a feedback signal. TD-Error reflects the accuracy of the current system's prediction of future states. If the existence of an edge helps the agent to more accurately evaluate the value of the current state, then the weight of that edge should be strengthened.

[0038] The specific weight updates follow the gradient descent principle, aiming to minimize prediction error. Weights In the The update rule for each moment is expressed as follows:

[0039] in, The structure learning rate is used to control the step size of weight updates; This represents the partial derivative of the squared-side weights of the TD-Error, which is calculated in the graph neural network using the backpropagation algorithm. The function is used to limit the weights to a preset minimum value. and maximum value To prevent weight divergence or excessive decay, this mechanism ensures that redundant edges, incorrectly added during the causal discovery process but actually useless for control decisions, have their weights gradually reduced. Nearby paths are removed during regular graph pruning; conversely, critical paths essential for decision-making are preserved and enhanced. This dynamic reconstruction mechanism transforms the knowledge graph from a static data container into a dynamic reasoning model capable of self-optimization and adapting to environmental changes, effectively solving the problem of structural rigidity in traditional methods when dealing with sudden changes in traffic patterns.

[0040] In step S3, the upper-level management agent performs neural symbolic reasoning based on the real-time evolutionary knowledge graph to generate discrete logic operators representing high-order control strategies. In this embodiment, the upper-level management agent no longer directly outputs specific signal phase timing parameters through a black-box neural network, but instead acts as a "traffic commander," generating macro-level strategic instructions based on the global situation. To effectively process the evolutionary graph containing multiple relation types, the upper-level management agent uses a relational graph convolutional network as a state encoder. This encoder can distinguish the differentiated contributions of different edge types, such as physical connections, semantic affiliation, and causal influence, to the node state, thereby preserving the heterogeneous semantic structure of the road network during feature aggregation.

[0041] Specifically, for any node in the evolutionary knowledge graph In the neural network, Hidden state of a layer The update is performed by aggregating information about neighboring nodes of all relation types. This aggregation update process follows the mathematical model below:

[0042] In this formula, Represents the set of all relation types in the graph; Represents a node In a specific relationship The set of neighboring nodes under a relation, for example, if the relation If it is an "implicit causal effect", then Includes those nodes that have no physical connection but are related to the node Upstream nodes that have a significant congestion transmission effect; It is a normalization constant used to balance the influence of nodes with different degrees, and is usually set to a value of 1. ; It is for relationships The learned hierarchical weight matrix means that the system has learned different feature transformation rules for physical transmission and semantic influence; It is a self-circulating weight matrix used to retain the historical information of the nodes themselves; For non-linear activation functions, the ReLU function is typically used. Through multiple stacked layers of this type of convolution operation, nodes... The final embedding vector not only encodes its own local state, but also incorporates multi-hop global context information passed through explicit physical paths and implicit causal paths.

[0043] After completing the feature update of the graph nodes, the system aggregates the final layer embeddings of all physical intersection nodes through a global readout function to generate a global state vector that can represent the current macroscopic situation of the entire road network. This global state vector is then fed into the policy network of the upper-level management agent. In the middle, the policy network does not output continuous values, but is responsible for processing a predefined set of discrete logic operators. Select the most suitable control logic from the set of logic operators. It includes several high-order strategies with clear physical meanings, such as the "maximize main road green wave bandwidth" operator to improve the travel experience under low saturation, the "regional boundary interception" operator to restrict the entry of external traffic when the core area is congested, and the "balanced regional queue length" operator to prevent local deadlock.

[0044] The policy network maps the global state vector using a multilayer perceptron (MLP) and outputs the selection probability distribution for each logic operator. The calculation process for this probability distribution is as follows:

[0045] in, This indicates the current overall road network situation. Next, select the first one. Logical operators The probability of all operators is 1; Softmax is a normalized exponential function that ensures the sum of probabilities of all operators is 1. This process realizes the mapping from high-dimensional, complex road network perception states to low-dimensional, sparse, and interpretable symbolic logic. In this way, the upper-layer agent establishes the global control objective within the current time window. This objective is not a rigid rule, but an optimal strategic intent inferred from the real-time road network evolution state, providing clear semantic guidance for the specific execution of the lower-layer agent.

[0046] This embodiment further executes step S4, transforming the discrete logic operator into a continuous differentiable vector through a differentiable logic interface and passing it to the lower-level execution agent. In traditional hierarchical control architectures, upper-level decisions are typically passed to lower layers in the form of hard-coded discrete instructions. This non-differentiable transmission method blocks the backpropagation of gradients between neural network layers, causing upper and lower layer models to be trained independently, making it difficult to achieve global optimum. To overcome this deficiency, this invention designs a soft logic interface based on reparameterization techniques, realizing a smooth mapping from discrete symbol space to continuous vector space, and establishing a semantic alignment mechanism to constrain lower-level behavior.

[0047] Specifically, to enable the discrete logic operator selection process to participate in end-to-end gradient updates, this embodiment employs the Gumbel-Softmax relaxation technique to process the probability distribution of the upper-layer policy network output. Instead of directly performing hard sampling, the system generates a continuous one-hot approximation vector. The generation process of this vector follows the formula below:

[0048] In this formula, The output of the upper-layer policy network is the first The original selection probabilities of each logical operator; It is a noise term that is independently sampled from the standard Gumbel(0,1) distribution. This noise is introduced to simulate the randomness of the sampling. This is a temperature coefficient used to control the smoothness of the distribution. When When it approaches 0, Approximating the discrete one-heat vector; when When the value is large, the distribution tends to be uniform. During training, the system uses an annealing strategy to gradually reduce the value. The values ​​are such that the forward propagation process approximates discrete selection, while the backward propagation process uses continuous gradients for parameter updates.

[0049] Obtain the one-hot approximation vector Then, the system uses a learnable semantic embedding matrix. Map it to a dense semantic context vector Each column vector of the embedding matrix corresponds to a representation of a logical operator in a high-dimensional semantic space. This representation is not fixed but is continuously adjusted during training to more accurately encode the operator's influence on the lower-level policy. The mapping process is represented as follows:

[0050] The semantic context vector This is then passed as conditional input to the lower-level agent. During the backpropagation phase, the gradient signal generated by the lower-level agent based on environmental feedback can be passed through the chain rule... and It penetrates to the upper-level policy network, thereby directly guiding the upper layer on how to adjust the selection probability of logical operators to obtain higher long-term returns.

[0051] To prevent lower-level agents from deviating from the macroscopic intent specified by the upper layer when optimizing local rewards, this invention further introduces a semantic alignment loss function. The system pre-defines each logical operator. An ideal state transition distribution is defined. This distribution describes the physical evolution of the intersection state after executing the logical operator. During training, the system mandates that the lower-level execution agents generate the actual state transition distribution. This approximates the ideal distribution. The semantic alignment loss is obtained by calculating the Kullback-Leibler divergence (KL divergence) between the two:

[0052] in, It is a vector The The nth component represents the nth component. The activation weights of each operator; The KL divergence operation represents the distance between two probability distributions; This represents the expectation based on the current distribution of experience playback data. By minimizing this loss function, the system constructs a mathematical contract mechanism to ensure that the micro-actions of the lower-level agents strictly conform to the macro-strategic deployments of the upper-level agents in terms of statistical characteristics, thereby achieving true neural symbolic cooperative control.

[0053] This embodiment further executes step S5, where the lower-level execution agent, based on the local traffic state of the current intersection, combines a pre-set traffic flow pattern fingerprint database with the received differentiable vector to generate the final residual action to control the traffic signal. In traditional traffic control systems, rule-based methods struggle to cope with complex and ever-changing traffic fluctuations, while pure end-to-end reinforcement learning methods often face risks such as excessive exploration space, poor cold-start performance, and even traffic paralysis in the early stages of training. Therefore, this invention employs a hybrid intelligent control architecture of "prototype matching + residual correction," aiming to simultaneously balance the robustness of expert experience with the adaptability of deep learning.

[0054] Specifically, the lower-level executing agent first obtains the local state of the current intersection through roadside sensing devices. And map it into a high-dimensional feature vector. The system has a pre-installed traffic flow pattern fingerprint database. Each fingerprint prototype This represents a typical traffic flow distribution pattern, and each prototype is associated with a set of validated expert rule strategies. In each control cycle, the agent calculates the cosine similarity between the current local state features and all fingerprint prototypes in the database to identify the closest flow pattern for the current scene. Best Matching Prototype Index The calculation formula is as follows:

[0055] in, This represents the vector dot product operation. This represents the L2 norm of the vector. This step effectively performs a rapid classification based on unsupervised pattern recognition, selecting a safe and reasonably efficient baseline strategy for the current control task. This means that even if the neural network has not yet converged or a communication failure occurs, the system can still maintain the basic operation of the intersection based on this baseline strategy, ensuring the system's safety as a fallback.

[0056] After determining the baseline policy, the lower-level executive agent does not directly adopt its output. Instead, it uses a lightweight residual neural network to fine-tune the baseline action. This residual network receives the current local state. and semantic context vectors from the upper-level management agent As input, output a motion adjustment amount. This adjustment reflects the phase-time offset required in the current specific micro-scenario to respond to the macro-level intent and compensate for the limitations of general expert rules. The generation process of the residual action is represented as follows:

[0057] In this formula, The nonlinear mapping function represents the residual neural network; the Tanh activation function restricts the network output to the interval [−1, 1]. It is a configurable physical constraint constant used to limit the maximum adjustment range and prevent the neural network from outputting overly aggressive actions.

[0058] Finally, the lower-level executing agent superimposes the actions generated by the baseline strategy with the residual adjustment to obtain the final control actions actually issued to the signal controller. The synthesis formula is as follows:

[0059] in, It is the basic timing scheme recommended by the selected expert rules under the current conditions; It is a confidence factor that decays dynamically with the training process, especially in the early stages of training. A smaller value makes the system more inclined to a conservative expert strategy; as training progresses, Gradually increasing the size allows reinforcement learning models to play a greater dominant role; and These represent the minimum green light time and maximum cycle duration allowed by the traffic signal, respectively, as physical hard constraints. The function ensures that the final output value strictly falls within this safe range. Through this residual adaptive control mechanism, the system not only utilizes prior knowledge to accelerate convergence, but also possesses the evolutionary ability to break through rule constraints and search for optimal solutions in complex dynamic environments.

[0060] This embodiment further executes a counterfactual reasoning-based augmentation training step and uses a hybrid optimization objective function to update the system as a whole. In real-world traffic scenarios, long-tail events such as extreme congestion, major traffic accidents, or severe weather have extremely low probabilities, making it difficult for the system to collect sufficient samples to train the agent's ability to cope with such critical situations. To overcome the generalization bottleneck caused by data sparsity, this invention utilizes the causal reasoning capabilities of the aforementioned constructed multimodal traffic semantic knowledge graph to proactively generate virtual counterfactual trajectories, thereby augmenting the training data.

[0061] Specifically, the system first samples a state from a pool of real historical experience replays. And using perturbation functions The semantic attributes of entities in this state are modified in a targeted manner to generate a counterfactual initial state. For example, a perturbation function can forcibly change a key intersection node that was originally in a "smooth" state to "severely congested," or change a weather attribute from "sunny" to "heavy rain." This process is not a simple superposition of numerical noise, but a structured transformation based on graph semantic logic. Subsequently, the system calls a pre-trained environmental dynamics model. ,by Starting from this point, a multi-step forward deduction is performed. In each step, the agent generates an action based on the current counterfactual state, and the environment model predicts the next state and reward, thus generating a complete counterfactual trajectory. :

[0062] in, This refers to the time step in the simulation. It is the next state predicted by the environmental model; These are the policy outputs of the agent in a virtual state. These counterfactual trajectories simulate parallel worlds of "how the system should evolve if some extreme situation occurs," and are stored in the experience pool for subsequent training.

[0063] To optimize system performance in real-world scenarios and enhance its robustness in counterfactual scenarios, this embodiment constructs a hybrid total loss function with multiple constraints. This function not only focuses on maximizing the cumulative reward, but also emphasizes the semantic consistency between upper and lower layer policies and the policy entropy constraint in unseen scenarios. The mathematical expression of the total loss function is as follows:

[0064] In this formula, Based on real historical trajectory The standard reinforcement learning loss term is calculated to maximize the expected cumulative reward; It is the semantic alignment loss term defined in the preceding steps, used to constrain the lower-level executing agent to conform to the logical intent of the upper-level managing agent; and These are hyperparameters used to balance the contribution weights of each loss term.

[0065] The third term in the formula It is a counterfactual consistency loss specifically designed for counterfactual trajectories. Considering that agents are prone to policy collapse or outputting erroneous actions with high confidence in virtual extreme states, this loss term usually includes a maximum entropy regularization component, which aims to encourage agents to remain exploratory in uncertain counterfactual environments and prevent premature convergence to suboptimal solutions. The specific form is defined as follows:

[0066] in, The first part describes the Shannon entropy, which represents the policy distribution. By maximizing this entropy, the system forces the agent to retain multiple possible coping strategies when faced with unfamiliar counterfactual states, rather than blindly relying on its own abilities. The second part focuses on the value function. The consistency constraint ensures that the evaluation value of the value network in the virtual state matches the total return deduced from the environmental model. Through this hybrid optimization mechanism, the present invention can significantly improve the survivability and resilience of the system under complex and sudden operating conditions by utilizing virtual trial and error in the absence of real extreme samples.

[0067] This embodiment provides an electronic device designed to serve as the physical carrier for executing the aforementioned distributed optimization method for road network traffic signals based on hierarchical reinforcement learning. Depending on the specific application scenario, this electronic device can be configured as a roadside computing unit, a regional edge server, or a cloud-based central control platform. As shown in the figure, the electronic device mainly includes a processor, memory, communication interfaces, and a system bus connecting the various components at the hardware level. The system bus serves as a data transmission channel, ensuring high-speed flow of control signals, address commands, and status data between the components.

[0068] In the specific hardware configuration, the processor is not limited to a traditional central processing unit. To meet the large-scale parallel computing requirements of the relational graph convolutional network and reinforcement learning policy network involved in this invention, the processor preferably adopts a heterogeneous computing architecture that includes a graphics processing unit, a tensor processor, or a dedicated neural network processor. The processor is configured to execute computer program instructions stored in memory. By loading and processing these instructions, the processor can instantiate upper-level management agents and lower-level execution agents, specifically performing all logical operation steps, including constructing a multimodal traffic semantic knowledge graph, calculating causal association scores, performing neural symbolic reasoning, and generating residual control actions.

[0069] The memory, serving as a non-volatile computer-readable storage medium, is used to persistently store the operating system, applications, and key data structures generated during the implementation of this invention. Specifically, the memory is divided into dedicated data partitions to store the real-time evolving traffic knowledge graph data structure. Pre-set traffic flow pattern fingerprint database It also includes a counterfactual experience replay pool. This storage medium is connected to the processor via a high-speed I / O interface, supporting high-throughput state vector read and write operations. This ensures that the retrieval and sampling of massive historical trajectory data is completed within millisecond-level control cycles, thereby supporting the online training process based on counterfactual reasoning.

[0070] The communication interface is a key component for realizing distributed collaborative control, supporting wired Ethernet, fiber optic communication, or 5G / V2X wireless communication protocols. In the layered architecture, this communication interface is responsible for establishing low-latency data links between physically isolated but logically coupled devices. Specifically, when the electronic device operates as a roadside unit, the communication interface is responsible for receiving raw sensing data from the road surface detector in real time and uploading the calculated local state features to the upper-layer nodes; when the electronic device operates as a regional edge server, the communication interface is responsible for broadcasting the semantic context vector processed by the differentiable logic interface to the lower-layer nodes. And gradient update signals. This hardware architecture based on high-speed communication links ensures that the gradient flow can propagate back across physical device boundaries, achieving end-to-end joint optimization of the entire system.

[0071] Furthermore, embodiments of the present invention also provide a computer-readable storage medium on which a computer program is stored. The computer program is designed as an ordered sequence of instructions, which, when executed in the execution environment of a computer or processor, causes the computer or processor to perform the various steps of the distributed optimization method for road network traffic signals based on hierarchical reinforcement learning as described in the foregoing embodiments. This storage medium includes, but is not limited to, flash drives, hard disk drives, optical disks, or any other form of magnetic or optical storage device, ensuring the portability and reproducibility of the technical solution.

[0072] This embodiment further provides a computer-readable storage medium storing a computer program. When executed by a processor, this program not only implements the steps described in the above embodiments but also persistently stores the core data structures unique to this invention. Specifically, the storage medium retains the dynamic adjacency matrix and feature embedding table of the multimodal traffic semantic knowledge graph, as well as a traffic flow pattern fingerprint database pre-set through cluster analysis. When the system starts, the processor reads these initialization parameters from the storage medium to construct a digital twin base with causal reasoning capabilities. The computer program adopts a modular design, logically divided into a graph evolution module, a neural symbolic reasoning module, and a residual control module. These modules can be centrally deployed in a single high-performance server, or they can be split and distributed between edge computing nodes and roadside terminals, achieving data synchronization and gradient transfer through cross-process communication mechanisms.

[0073] The computer-readable storage medium mentioned in this invention may include, but is not limited to, random access memory, read-only memory, erasable programmable read-only memory, portable compact disk read-only memory, or optical storage devices. When executed, the instruction sequence stored on this medium enables the computer device to perform the following key operation: calculating the causal correlation score between non-adjacent intersections using a mutual information formula. The weights of graph edges are updated based on backpropagation of temporal difference error. The Gumbel-Softmax interface relaxes discrete logical operators into continuous semantic context vectors. and based on hybrid loss function End-to-end joint training is performed on the entire network. These instructions ensure that the system can continuously learn from environmental feedback during long-term operation, achieving a smooth evolution from "rule-driven" to "data-driven".

[0074] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the embodiments of devices, electronic devices, and storage media, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. Those skilled in the art should understand that the hierarchical reinforcement learning-based architecture proposed in this invention is not limited to traditional traffic light signal control; its core neural symbolic reasoning and causal graph reconstruction mechanisms are also applicable to generalized traffic flow optimization scenarios such as ramp control, variable lane scheduling, and vehicle speed guidance in vehicle-to-everything (V2X) environments. Without departing from the core logic of this invention, adjustments to the number of network layers, replacement of activation functions, or fine-tuning of hyperparameters are all obvious variations of this technical solution.

[0075] Although the embodiments in this specification are described in detail using road traffic lights as an example, those skilled in the art should understand that the technical solution is also applicable to vehicle and pedestrian traffic control facilities within enclosed scenic areas and resorts. The "intersection" can correspond to traffic nodes or entrances and exits of attractions within the scenic area, and the "signal" can correspond to traffic indicator lights or gate control signals.

[0076] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A distributed optimization method for road network traffic signals based on hierarchical reinforcement learning, characterized in that, Includes the following steps: Step S1: Construct a multimodal scenic area traffic semantic initial knowledge graph, which includes heterogeneous entities describing the scenic area's road network structure, functional areas, and dynamic events, as well as the relationships between these entities. Step S2: Based on the feedback signal of reinforcement learning, the initial knowledge graph of multimodal scenic area traffic semantics is dynamically reconstructed according to causal structure to generate a real-time evolutionary knowledge graph containing implicit passenger flow transmission edges; Step S3: Utilize the upper-level management agent to perform neural symbolic reasoning based on the real-time evolutionary knowledge graph to generate discrete logic operators representing high-order passenger flow control strategies; Step S4: Convert the discrete logic operator into a continuous differentiable vector through a differentiable logic interface and pass it to the lower-level execution agent; Step S5: The lower-level execution agent generates the final residual action based on the local flow status of the current control node, combined with the pre-set scenic area passenger flow pattern fingerprint database and the received differentiable vector, to control traffic signals or passenger flow gate equipment.

2. The distributed optimization method for road network traffic signals based on hierarchical reinforcement learning according to claim 1, characterized in that, In step S1, the heterogeneous entity includes: Physical entities include scenic area road intersections, parking lot entrances and exits, attraction gates and detector nodes; Semantic entities include core attraction POIs, dining and rest areas, and performance area nodes; Event entities include attraction crowd level, weather conditions, performance times, and sudden crowd control events; The relationships include topological relationships representing road connections, semantic relationships representing the affiliation of attractions, and dynamic relationships representing environmental impacts.

3. The distributed optimization method for road network traffic signals based on hierarchical reinforcement learning according to claim 1, characterized in that, In step S2, the dynamic reconstruction of the causal structure specifically includes: Calculate the causal correlation score of traffic state sequences between any two non-physically connected nodes; When the causal association score exceeds a preset threshold, an implicit causal edge is generated between the two nodes; the weight of the implicit causal edge is adaptively adjusted according to the temporal difference error (TD-Error) during the system's training process. If the existence of an implicit causal edge fails to reduce the time-series difference error, then the weight of that edge is reduced.

4. The distributed optimization method for road network traffic signals based on hierarchical reinforcement learning according to claim 1, characterized in that, In step S3, the neural symbolic reasoning specifically includes: The features of heterogeneous neighbor nodes in the real-time evolutionary knowledge graph are aggregated using a relational graph convolutional network to obtain the global state embedding; The policy network of the upper-level management agent outputs the selection probability distribution of each logical operator from a predefined set of logical operators based on the global state embedding. Each logical operator in the set of logical operators corresponds to a specific macro-level traffic control strategy objective.

5. The distributed optimization method for road network traffic signals based on hierarchical reinforcement learning according to claim 1, characterized in that, In step S4, the processing procedure of the differentiable logic interface specifically includes: The selection probability distribution of the logic operator is sampled using the Gumbel-Softmax relaxation technique to generate a continuous one-hot approximation vector. The generation process of this vector follows the formula below: ； in, This indicates the current overall road network situation. Below, the output of the upper-layer policy network is the first... Logical operators The original choice probability, For noise terms, For temperature coefficient, when When it approaches 0, the uniquely hot approximation vector Approximates the discrete one-hot vector; The one-hot approximation vector is obtained through a learnable embedding matrix. The mapping is performed as a semantic embedding vector, which serves as the differentiable vector. During backpropagation, gradient signals from lower-level executive agents are allowed to penetrate the differentiable logic interface to update the parameters of upper-level management agents.

6. The distributed optimization method for road network traffic signals based on hierarchical reinforcement learning according to claim 1, characterized in that, The method also includes constructing a semantic alignment loss function: Calculate the Kullback-Leibler divergence between the state transition distribution of the lower-level agent after executing an action and the predefined ideal state transition distribution corresponding to the selected logic operator in the upper layer; The Kullback-Leibler divergence is added as a semantic alignment loss term to the total loss function to constrain the behavior of the lower-level executive agent to conform to the logical intent of the upper-level management agent.

7. The distributed optimization method for road network traffic signals based on hierarchical reinforcement learning according to claim 1, characterized in that, In step S5, the action of generating the final residual specifically includes: Extract the local passenger flow or vehicle flow status features of the current node and calculate its similarity with the prototypes of each basic diversion strategy in the pre-set scenic area passenger flow pattern fingerprint database. Select the prototype of the basic strategy with the highest similarity as the current benchmark strategy; Using a residual neural network, with the local flow state and the differentiable vector as input, the output is an adjustment amount for the control duration or release frequency; the action generated by the baseline strategy is superimposed with the adjustment amount to obtain the final residual control action.

8. The distributed optimization method for road network traffic signals based on hierarchical reinforcement learning according to claim 1, characterized in that, The method also includes an augmented training step based on counterfactual reasoning: In the real-time evolutionary knowledge graph, the semantic attributes of entities are perturbed to generate counterfactual initial states; The learned environment model is used to perform forward extrapolation on the counterfactual initial state to generate a counterfactual trajectory that includes virtual states and virtual actions; The counterfactual trajectory is stored in the experience replay pool for mixed training by the upper-level management agent and the lower-level execution agent.

9. The distributed optimization method for road network traffic signals based on hierarchical reinforcement learning according to claim 8, characterized in that, The overall optimization objective function used in the hybrid training includes: Reinforcement learning loss term based on real historical data; Semantic alignment loss term based on logical intent consistency; And a counterfactual consistency loss term based on the counterfactual trajectory, used to constrain the agent's policy entropy in counterfactual scenarios.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 9.

Images