Reinforcement learning intelligent traffic signal control method and system based on state-driven search enhancement

Abstract

The application discloses a kind of reinforcement learning intelligent traffic signal control method and system based on state-driven search enhancement. The method comprises the following steps: collecting real-time traffic flow state information of traffic environment and constructing global traffic state vector, retrieving knowledge base to obtain relevant knowledge fragments and weighted aggregation to obtain comprehensive knowledge representation vector, constructing a fusion decision model that integrates search enhancement and Markov decision mechanism, training and converging the policy network after generating the enhanced state vector, implementing traffic signal control based on the converged policy network, improving the generalization adaptation capability, anti-interference capability and operation stability of the signal control strategy for various traffic scenarios, effectively dealing with the random fluctuations and sudden traffic conditions of traffic flow, achieving traceability and explainability of control decision through knowledge retrieval and prior knowledge fusion, effectively improving the efficiency of traffic signal control, reducing intersection delay and congestion incidence, and providing reliable support for intelligent collaborative management and control of road network.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation and artificial intelligence decision-making technology, and in particular to a reinforcement learning-based intelligent traffic signal control method and system based on state-driven retrieval enhancement. Background Technology

[0002] Currently, with the rapid growth of urban traffic volume and the relative scarcity of road resources, traditional fixed-time control and inductive control methods are no longer sufficient to meet the dynamic scheduling needs of modern urban intelligent transportation. In recent years, intelligent traffic signal control based on artificial intelligence has gradually become a research hotspot. Among them, reinforcement learning, due to its ability to automatically learn optimal strategies through continuous interaction with the environment, has been widely applied in the field of traffic signal timing optimization. By constructing Markov decision models, intelligent agents can dynamically adjust traffic light timings according to real-time traffic conditions, thereby effectively alleviating traffic congestion and improving traffic efficiency.

[0003] However, existing reinforcement learning-based traffic signal control methods still suffer from several technical bottlenecks. First, decision interpretability is insufficient. Most current reinforcement learning models rely on empirical data for policy learning, lacking explicit modeling of traffic regulations and emergency knowledge. This results in a black-box policy generation process, limiting the reliable application of the models in safety-critical scenarios. Second, generalization and robustness are inadequate. Reinforcement learning models typically rely on simulations or historical data in specific scenarios for training. When faced with sudden traffic events or atypical traffic patterns, they often struggle to transfer and adapt, exhibiting poor generalization ability and robustness. Furthermore, knowledge utilization is low. Some existing research attempts to incorporate traffic knowledge into the reinforcement learning process, but these efforts mostly remain at the level of model pre-training, feature enhancement, or reward function design, without modeling the dynamic retrieval and selection of knowledge as an optimizable decision variable for the agent. This prevents the agent from flexibly utilizing external knowledge according to changes in traffic conditions, limiting the proactive role of knowledge in the decision-making process. Summary of the Invention

[0004] The main objective of this invention is to provide a reinforcement learning-based intelligent traffic signal control method and system based on state-driven retrieval enhancement, which aims to solve the technical problems of insufficient interpretability of decision-making, insufficient generalization and robustness, and low knowledge utilization in existing intelligent traffic control systems, resulting in low efficiency and accuracy of traffic signal control.

[0005] To achieve the above objectives, this invention provides a reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement. The method is applied to agents in a reinforcement learning environment, where multiple agents are configured. Each agent is equipped with a policy network, which is a parameterized graph neural network structure. Each agent is configured to control the traffic signal at a traffic intersection. The method includes the following steps:

[0006] Collect real-time traffic flow status information of the traffic environment, and construct a global traffic state vector based on the real-time traffic flow status information;

[0007] Based on the global traffic state vector retrieval knowledge base, a set of knowledge fragments related to the current traffic state is obtained, and the vectors of each knowledge fragment in the set of knowledge fragments are weighted and aggregated to obtain a comprehensive knowledge representation vector;

[0008] A fusion decision model is constructed based on the retrieval enhancement mechanism and the Markov decision mechanism. The fusion decision model is a five-tuple structure, which includes a state space, an action space, a state transition probability function, an immediate reward function, and a discount factor. The state space includes the traffic environment state corresponding to the global traffic state vector and the knowledge retrieval state corresponding to the comprehensive knowledge representation vector.

[0009] The global traffic state vector is concatenated with the comprehensive knowledge representation vector to generate an enhanced state vector. Based on the enhanced state vector, the policy network is trained and converged using the fusion decision model and the near-end policy optimization algorithm.

[0010] Traffic signal control strategies are generated based on the converged strategy network, and traffic signal control is performed based on the traffic signal control strategies.

[0011] Optionally, the step of collecting real-time traffic flow state information of the traffic environment and constructing a global traffic state vector based on the real-time traffic flow state information includes:

[0012] Obtain road network structure data for the target area, extract road node information and roadside information from the road network structure data, and generate a directed topology graph of the road network;

[0013] The directed topology graph of the road network is imported into the traffic simulation platform to generate a multi-intersection traffic simulation model.

[0014] The multi-intersection traffic simulation model collects real-time traffic flow status information corresponding to each control intersection in the traffic environment and generates an intersection state vector corresponding to each control intersection. The real-time traffic flow status information includes real-time traffic flow characteristics and real-time signal phase characteristics. The intersection state vector includes traffic volume, average speed, lane density, average queue length, and current signal phase.

[0015] The intersection state vectors corresponding to all controlled intersections are concatenated to generate a global traffic state vector.

[0016] Optionally, the step of retrieving a knowledge base based on the global traffic state vector to obtain a set of knowledge fragments related to the current traffic state, and then weighting and aggregating the vectors of each knowledge fragment in the set to obtain a comprehensive knowledge representation vector, includes:

[0017] The knowledge fragments in the knowledge base are encoded into vector representations by a semantic embedding model to form a knowledge vector space. The knowledge base includes a traffic regulations database, an emergency plan database, and an accident case database.

[0018] The global traffic state vector is converted into a structured retrieval request using a mapping function;

[0019] The structured search request is analyzed based on a policy network to determine whether to perform a search action.

[0020] If the retrieval action is performed, the knowledge vector space is searched to obtain a set of knowledge fragments related to the current traffic state;

[0021] Calculate the similarity between the global traffic state vector and each knowledge fragment vector in the knowledge fragment set, and generate the knowledge weight coefficient corresponding to each knowledge fragment vector based on the similarity.

[0022] Based on the knowledge weight coefficients, the vectors of each knowledge segment in the knowledge segment set are weighted and aggregated to obtain a comprehensive knowledge representation vector.

[0023] Optionally, the state space of the fusion decision model is defined as:

[0024]

[0025] in, Representing the state space, The traffic environment status perceived by the intelligent agent includes traffic flow, average speed, lane density, average queue length, and current signal phase at each intersection. The knowledge retrieval state perceived by the intelligent agent includes the current knowledge confidence, retrieval action history, knowledge relevance score, and support.

[0026] The action space of the fusion decision model is defined as follows:

[0027]

[0028]

[0029] in, Represents the action space. This represents the knowledge retrieval action space. This indicates that no search action will be triggered. This indicates a search of the traffic regulations database. This indicates a search of the emergency response plan database. This indicates a search of the accident case database. This indicates the traffic signal control action space, which includes the phase switching and timing adjustment of traffic lights;

[0030] The policy function of the policy network is defined as follows:

[0031]

[0032] in, This represents the policy parameters of the policy network. Indicates the policy network in time Joint decision-making actions based on traffic environment status and knowledge retrieval status output;

[0033] The reward function of the fusion decision model is defined as:

[0034]

[0035] in, Indicates the total reward. Incentives related to traffic efficiency. Rewards related to the effectiveness of knowledge retrieval. and The weighting coefficients are used to comprehensively measure the instantaneous decision quality of the agent at each time step t.

[0036] The overall optimization objective during the training and convergence process of the policy network is to maximize the expected cumulative reward in the future, as shown in the following formula:

[0037]

[0038] in, Indicates the discount factor. This represents the maximum step size in the time domain for a single round of interactive decision-making. Indicates the first intelligent agent Instant rewards for each time step Indicates that the agent is policy-based The mathematical expectation of the cumulative reward corresponding to the discounts of all decision trajectories when interacting with the environment.

[0039] Optionally, the step of concatenating the global traffic state vector with the comprehensive knowledge representation vector to generate an enhanced state vector, and training and converging the policy network based on the enhanced state vector using the fusion decision model and the near-end policy optimization algorithm, includes:

[0040] The global traffic state vector and the comprehensive knowledge representation vector are concatenated to generate an enhanced state vector;

[0041] The enhanced state vector is input into the policy network to generate corresponding execution actions, and the immediate reward from the reinforcement learning environment and the state data at the next time step are obtained. The immediate reward is defined as:

[0042]

[0043]

[0044]

[0045]

[0046] in, Indicators representing traffic efficiency. Indicates safety indicators, Indicators representing knowledge consistency , and These represent the reward weight coefficients, which are adaptively adjusted during training using a Pareto optimization strategy. This represents the weighting coefficient for the traffic flow throughput sub-item. This represents the weighting factor for the vehicle delay sub-item. Indicates the first The actual average traffic flow throughput of the controlled intersection within the decision-making time step. This indicates the saturation traffic flow at the intersection. Indicates the first The average delay time for vehicles at controlled intersections within a decision-making time step. This indicates the maximum permissible delay time limit preset at the controlled intersection. Indicates the first Negative statistics on traffic conflicts at controlled intersections within a single decision-making timeframe. This indicates the maximum preset traffic conflict value at the controlled intersection. This represents the semantic embedding vector of the decision action output by the agent. This represents the semantic embedding vector of knowledge suggestions retrieved by the agent from the knowledge base;

[0047] An experience sample set is constructed based on the enhanced state vectors, executed actions, immediate rewards, and state data of the next moment generated by multi-round environmental interactions.

[0048] Based on the empirical sample set, the network parameters of the policy network and the parameters of the action value function are updated through the fusion decision model and the proximal policy optimization algorithm.

[0049] Iteratively execute environment interaction and parameter update operations until the policy network meets the preset convergence conditions, thus completing the training and convergence of the policy network.

[0050] Optionally, the action value function is updated by minimizing the Bellman residual, as shown in the following formula:

[0051]

[0052] in, The Bellman residual loss term represents the action-value function. Represents the trainable parameters of the value network. Let the action value function be... This represents the augmented state vector at time step t. This represents the state of the agent at the next moment after it performs an action. This represents the action performed by the agent at time step t. Indicates the optional action for the next moment. Represents the set of empirical samples. Represents the mathematical expectation operation;

[0053] The policy function of the policy network is optimized by maximizing the expected reward and introducing an interpretability loss term and a knowledge inhibition regularization term, as shown in the following formula:

[0054]

[0055] in, This is the optimization function for the policy function. This represents the output value of the action value function. The policy network is state-based. The optimal action to output. This is an interpretability loss term, used to strengthen the semantic correspondence between knowledge retrieval results and action decisions. This is a knowledge suppression regularization term used to prevent the agent from becoming overly reliant on the knowledge base during training. The weighting coefficients for the interpretability loss term. The weighting coefficients for knowledge suppression regularization terms.

[0056] Optionally, the knowledge suppression regularization term penalizes the agent's retrieval actions using the following formula, as shown below:

[0057]

[0058] in, This indicates the revised instant reward. This represents the coefficient of the knowledge suppression regularization term. This represents the degree of execution of the retrieval action, which is a binary variable taking the value 0 or 1. The degree of execution of the retrieval action represents the agent's performance in the [missing information - likely a specific action or event]. The retrieval behavior at each decision-making time step.

[0059] Furthermore, to achieve the above objectives, this invention also proposes a reinforcement learning-based intelligent traffic signal control system with state-driven retrieval enhancement. The system applies the reinforcement learning-based intelligent traffic signal control method with state-driven retrieval enhancement described above, and includes:

[0060] The information acquisition module is used to collect real-time traffic flow status information of the traffic environment and construct a global traffic state vector based on the real-time traffic flow status information.

[0061] The knowledge retrieval enhancement module is used to retrieve the knowledge base based on the global traffic state vector, obtain a set of knowledge fragments related to the current traffic state, and perform weighted aggregation on the knowledge fragment vectors in the knowledge fragment set to obtain a comprehensive knowledge representation vector.

[0062] The decision fusion module is used to construct a fusion decision model based on the retrieval enhancement mechanism and the Markov decision mechanism. The fusion decision model is a five-tuple structure, which includes a state space, an action space, a state transition probability function, an immediate reward function, and a discount factor. The state space includes the traffic environment state corresponding to the global traffic state vector and the knowledge retrieval state corresponding to the comprehensive knowledge representation vector.

[0063] The reinforcement learning module is used to concatenate the global traffic state vector with the comprehensive knowledge representation vector to generate an enhanced state vector, and based on the enhanced state vector, to train and converge the policy network through the fusion decision model and the near-end policy optimization algorithm.

[0064] The traffic signal control module is used to generate a traffic signal control strategy based on the converged strategy network, and to perform traffic signal control based on the traffic signal control strategy.

[0065] Furthermore, to achieve the above objectives, this application also proposes a reinforcement learning-based intelligent traffic signal control device with state-driven retrieval enhancement. The device includes: a memory, a processor, and a reinforcement learning-based intelligent traffic signal control program stored in the memory. The processor is used to run the reinforcement learning-based intelligent traffic signal control program, and the computer program is configured to implement the steps of the reinforcement learning-based intelligent traffic signal control method with state-driven retrieval enhancement as described above.

[0066] In addition, to achieve the above objectives, this application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement as described above.

[0067] This invention achieves comprehensive perception of the traffic environment by collecting traffic status data in real time and constructing a global state vector. Through knowledge base retrieval and knowledge aggregation, it overcomes the limitations of real-time data and improves the rationality of decision-making. By integrating decision-making models and near-end policy optimization algorithms, it ensures stable training and rapid convergence of the policy network. The converged policy network generates control strategies and executes control, achieving precise regulation of traffic signals. On the one hand, through global traffic state modeling and multi-intersection collaborative simulation, it enhances the generalization and adaptability, anti-interference ability, and operational stability of the signal control strategy for various traffic scenarios, effectively addressing random fluctuations in traffic flow and sudden traffic situations. On the other hand, through knowledge retrieval and prior knowledge fusion, it makes the control decision logic traceable and explainable, avoiding the black-box decision-making risks of purely data-driven models, while improving the compliance and rationality of the control strategy. This effectively improves the efficiency of traffic signal control, reduces intersection delays and congestion rates, and provides reliable technical support for the intelligent collaborative management and control of urban road networks. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a schematic diagram of the structure of a state-driven retrieval-enhanced reinforcement learning-based intelligent traffic signal control device in the hardware operating environment of the embodiment of the present invention.

[0070] Figure 2 This is a flowchart illustrating the first embodiment of the reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement of the present invention.

[0071] Figure 3 This is a flowchart illustrating the second embodiment of the reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement of the present invention.

[0072] Figure 4 This is a structural block diagram of the first embodiment of the reinforcement learning-based intelligent traffic signal control system based on state-driven retrieval enhancement of the present invention.

[0073] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0074] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0075] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a state-driven retrieval-enhanced reinforcement learning-based intelligent traffic signal control device, which is part of the hardware operating environment of an embodiment of the present invention.

[0076] like Figure 1 As shown, the reinforcement learning-based intelligent traffic signal control device with state-driven retrieval enhancement may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; the user interface 1003 may also include standard wired and wireless interfaces. The network interface 1004 may optionally include standard wired and wireless interfaces (such as Wireless-Fidelity (Wi-Fi) interfaces). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also optionally be a storage system independent of the aforementioned processor 1001.

[0077] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the reinforcement learning-based intelligent traffic signal control device based on state-driven retrieval enhancement, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0078] like Figure 1As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a reinforcement learning intelligent traffic signal control program.

[0079] exist Figure 1 In the state-driven retrieval-enhanced reinforcement learning intelligent traffic signal control device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the state-driven retrieval-enhanced reinforcement learning intelligent traffic signal control device of the present invention can be set in the state-driven retrieval-enhanced reinforcement learning intelligent traffic signal control device. The state-driven retrieval-enhanced reinforcement learning intelligent traffic signal control device calls the reinforcement learning intelligent traffic signal control program stored in the memory 1005 through the processor 1001 and executes the state-driven retrieval-enhanced reinforcement learning intelligent traffic signal control method provided in the embodiment of the present invention.

[0080] This invention provides a reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement of the present invention.

[0081] In this embodiment, the method is applied to agents in a reinforcement learning environment. The reinforcement learning environment is configured with multiple agents, each equipped with a policy network. The policy network is a parameterized graph neural network structure. Each agent is configured to control traffic signals at a traffic intersection. The reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement includes the following steps:

[0082] Step S10: Collect real-time traffic flow status information of the traffic environment, and construct a global traffic state vector based on the real-time traffic flow status information.

[0083] It should be noted that this embodiment is applied to intelligent traffic signal control. It utilizes the SUMO simulation platform to collect real-time traffic flow state information at intersections and constructs a structured state vector. Based on this state, a retrieval request is generated, dynamically acquiring relevant knowledge fragments from a knowledge base containing traffic regulations, emergency plans, and accident cases. The agent autonomously selects whether to retrieve knowledge and the retrieval category based on the current traffic state, and fuses the retrieval results with state features after semantic embedding, inputting the results into a policy network for joint decision-making. Furthermore, to achieve unified modeling of knowledge retrieval and signal control, this invention formalizes the Retrieval Enhancement Generation (RAG) process as a Markov Decision Process (MDP), defines a multi-class retrieval action space and reward function structure, and uses the reinforcement learning PPO algorithm to solve for the optimal joint policy, realizing state-driven knowledge retrieval and traffic signal control optimization.

[0084] It should be understood that the executing entity of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a terminal electronic device capable of realizing the above functions. The following description uses a reinforcement learning-based intelligent traffic signal control device (hereinafter referred to as the control device) based on state-driven retrieval enhancement as an example to illustrate this embodiment and the following embodiments.

[0085] It should be noted that the global traffic state vector can be a one-dimensional or multi-dimensional numerical vector that is transformed from the collected scattered real-time traffic flow data through standardization, feature fusion and other processing, and is used to uniformly represent the overall traffic state of the entire control area.

[0086] It should be noted that the reinforcement learning environment can be a virtual or semi-virtual environment that simulates traffic operation scenarios, providing an interactive scenario for the intelligent agent, and can provide feedback on changes in traffic status and the effectiveness of control behavior, supporting the training and decision-making of the intelligent agent.

[0087] In this embodiment, an intelligent agent refers to an algorithm module with autonomous decision-making capabilities. Each intelligent agent corresponds to a traffic intersection and is responsible for the perception, decision-making, and control of traffic signals at that intersection, optimizing control strategies through interaction with the environment.

[0088] The policy network refers to the core network in an intelligent agent used to generate control policies. It adopts a parameterized graph neural network structure and can learn the mapping relationship between traffic states and optimal control actions through training, so as to output control instructions that conform to the current traffic state.

[0089] Parameterized graph neural network structures capture the relationships between nodes (such as traffic intersections and vehicles) and edges (such as inter-intersection connections and vehicle flow directions) through trainable parameters. They are used to process complex relational data such as traffic networks and can effectively mine the collaborative features between traffic nodes.

[0090] In some embodiments, the control device employs distributed data acquisition methods, deploying traffic detectors (such as video detectors, loop detectors, and radar detectors) at the entrance lanes, exit lanes, and center of each traffic intersection to collect data such as traffic flow, vehicle speed, vehicle queue length, vehicle type, and current traffic light phase in real time. The collected data is preprocessed to remove outliers and missing values, and data of different dimensions are uniformly mapped to the same interval through standardization. Based on the preprocessed data, core features (such as the proportion of traffic flow in each direction, the average queue length, and the variance of vehicle speed) are extracted, and all features are concatenated in a preset order to construct a global traffic state vector, ensuring that the vector can fully cover the traffic operation characteristics within the control area.

[0091] Furthermore, in order to provide high-quality environmental support for subsequent strategy optimization, step S10 may include:

[0092] Step S101: Obtain road network structure data of the target area, extract road node information and road edge information from the road network structure data, and generate a directed topology graph of the road network.

[0093] It should be noted that the target area refers to the traffic area under control, which can be defined according to actual needs (such as the road network around a certain urban area or business district), and is the boundary of the traffic signal control range.

[0094] Road network structure data can be various types of data that describe the distribution, connection relationships, and road attributes of roads within a target area, including road names, road grades, number of lanes, intersection locations, and road connection methods.

[0095] Road node information refers to the clearly identified key points in the road network structure, mainly including traffic intersections, road junctions, and road start and end points. These are the units used to construct the road network topology map. Road edge information describes the connection relationship between two road nodes, including road length, number of lanes, traffic direction, speed limits, etc., and is used to characterize the traffic association between nodes.

[0096] A directed topology graph of a road network is a graphical structure constructed with road nodes as vertices and road edges as directed edges. The direction of the directed edges corresponds to the direction of traffic on the road, which can intuitively reflect the connection relationship and traffic logic of the road network within the target area.

[0097] Step S102: Import the directed topology graph of the road network into the traffic simulation platform to generate a multi-intersection traffic simulation model;

[0098] Step S103: Using the multi-intersection traffic simulation model, collect real-time traffic flow status information corresponding to each control intersection in the traffic environment, and generate an intersection state vector corresponding to each control intersection. The real-time traffic flow status information includes real-time traffic flow characteristics and real-time signal phase characteristics. The intersection state vector includes traffic volume, average speed, lane density, average queue length, and current signal phase.

[0099] Step S104: Concatenate the intersection state vectors corresponding to all controlled intersections to generate a global traffic state vector.

[0100] In the specific implementation, the control device acquires road network structure data of a city through OSM, extracts road node and edge information, and represents it as a directed graph. This topology graph is then imported into the simulation environment using the SUMO platform to generate a dynamically interactive multi-intersection traffic model. In the simulation scenario, multiple sets of traffic participants are defined:

[0101]

[0102] in, This refers to the group of traffic participants: cars represent small passenger vehicles, buses represent medium and large public transport vehicles, pedestrians represent pedestrians, and bikes represent two-wheeled non-motorized vehicles.

[0103] And based on the average arrival rate of various transportation entities Define traffic flow input:

[0104]

[0105] in, Indicates the time interval The number of traffic bodies entering the road network. This represents the average arrival rate for the corresponding type.

[0106] During SUMO simulation, each control intersection The state vector consists of the following features:

[0107]

[0108] in: Indicates flow rate. Indicates average speed. Indicates lane density, This represents the average queue length. Indicates the current signal phase.

[0109] The status of all intersections can be concatenated to form the global system status:

[0110] This state serves as the input to the reinforcement learning agent and the knowledge retrieval module, providing environmental support for subsequent decision optimization.

[0111] Step S20: Based on the global traffic state vector, retrieve the knowledge base to obtain a set of knowledge fragments related to the current traffic state, and perform weighted aggregation on the knowledge fragment vectors in the knowledge fragment set to obtain a comprehensive knowledge representation vector.

[0112] It should be noted that the knowledge base can be a database that stores knowledge related to the transportation field, including historical traffic status data, control experience of typical traffic scenarios, traffic signal control rules, traffic congestion management cases, etc., to provide prior knowledge support for current decision-making.

[0113] The knowledge fragment set can be a collection of several knowledge fragments that are highly similar to the current global traffic state vector by searching the knowledge base. Each knowledge fragment corresponds to relevant knowledge of a similar traffic scenario (such as control strategies and state change patterns).

[0114] Knowledge fragment vectors are numerical vectors generated from text, data, and other information in knowledge fragments through encoding algorithms (such as word embedding and feature extraction). These vectors are used for similarity calculation and fusion processing with global traffic state vectors.

[0115] A comprehensive knowledge representation vector can be a numerical vector that integrates the core information of all relevant knowledge fragments, combining prior knowledge with the current real-time traffic state to enrich the dimension and depth of state representation.

[0116] It is understandable that this embodiment achieves a preliminary fusion of prior knowledge and real-time traffic conditions. By retrieving experiential knowledge of similar scenarios from the knowledge base, it compensates for the limitations of real-time traffic data (such as insufficient data on sudden traffic scenarios). Through weighted aggregation, core knowledge is filtered out to avoid interference from redundant knowledge. The generated comprehensive knowledge representation vector can enrich the representation information of traffic conditions, provide more comprehensive input for the subsequent construction of decision-making models, and improve the rationality and reliability of decisions.

[0117] In some embodiments, the control device uses a preset similarity calculation algorithm (such as cosine similarity, Euclidean distance, etc.) to input the constructed global traffic state vector as a search keyword into the knowledge base for retrieval; it filters out knowledge fragments with similarity higher than a preset threshold to form a knowledge fragment set; it uses an encoding model to convert each knowledge fragment in the set into a knowledge fragment vector, and determines the weight of each knowledge fragment vector through the similarity calculation results (the higher the similarity, the greater the weight); it aggregates all knowledge fragment vectors using a weighted summation method to obtain a comprehensive knowledge representation vector, ensuring that the aggregated vector can highlight the prior knowledge most relevant to the current traffic state.

[0118] Step S30: Construct a fusion decision model based on the retrieval enhancement mechanism and the Markov decision mechanism.

[0119] It should be noted that the retrieval enhancement mechanism refers to the mechanism of integrating prior knowledge retrieved from the knowledge base into the decision-making process. By calling prior knowledge, it makes up for the lack of real-time data in reinforcement learning and improves the generalization ability and decision efficiency of the decision model.

[0120] Markov decision mechanisms are decision mechanisms based on the Markov property (the current state is only related to the previous state and actions, and is not related to earlier states). They are used to simplify the decision-making process in dynamic decision-making scenarios of traffic signal control and focus on the relationship between the current state and subsequent actions.

[0121] It should be noted that the fusion decision model refers to a decision framework that integrates retrieval enhancement mechanisms and Markov decision mechanisms. Defined in the form of a quintuple, it is used to standardize the decision logic of an intelligent agent and achieve deep integration of prior knowledge and real-time decision-making. The quintuple structure includes a state space, an action space, a state transition probability function, an immediate reward function, and a discount factor. The state space contains the traffic environment state corresponding to the global traffic state vector and the knowledge retrieval state corresponding to the comprehensive knowledge representation vector.

[0122] The state space refers to the set of all possible states in a decision-making model. In this embodiment, it includes two parts: traffic environment state (represented by a global traffic state vector) and knowledge retrieval state (represented by a comprehensive knowledge representation vector), which comprehensively covers the state information required for decision-making.

[0123] Action space refers to the set of all control actions that an intelligent agent can execute, corresponding to different control methods of traffic signals (such as phase switching, phase duration adjustment, etc.).

[0124] State transition probability function: describes the probability that an agent will transition from the current state to the next state after performing a certain action, reflecting the changing pattern of traffic state with control actions.

[0125] An immediate reward function is a function used to evaluate the immediate effect produced by an agent after performing a certain action. By setting reward rules (such as reducing congestion and improving traffic efficiency as positive rewards, and aggravating congestion as negative rewards), the agent is guided to learn the optimal control action.

[0126] The discount factor is used to adjust the weight of future rewards in the current decision. The larger the discount factor, the more the agent values ​​long-term future rewards, and vice versa. It is used to balance short-term traffic efficiency and long-term traffic optimization.

[0127] It is understood that this embodiment constructs a standardized and efficient decision-making framework, which clarifies the input (state space), output (action space), and decision-making logic (state transition, reward mechanism) of the agent's decision-making. By integrating the retrieval enhancement mechanism and the Markov decision-making mechanism, it achieves a deep combination of prior knowledge and real-time dynamic decision-making, solving the problems of weak generalization ability and lack of experience support in traditional reinforcement learning decision-making. The five-tuple structure makes the decision-making process more interpretable.

[0128] In some embodiments, the control device combines the core logic of retrieval enhancement mechanisms and Markov decision mechanisms to construct a fusion decision model with a five-tuple structure: First, a state space is defined, integrating the traffic environment state corresponding to the global traffic state vector with the knowledge retrieval state corresponding to the comprehensive knowledge representation vector, clarifying all possible decision input states; second, an action space is defined, sorting out all possible actions of traffic signal control (such as different phase combinations and different phase durations) to form a standardized action set; then, based on historical traffic data and knowledge fragments, a state transition probability function is fitted to describe the influence of actions on the state; an immediate reward function is designed, using traffic efficiency, queue length, vehicle delay, etc., as core evaluation indicators, and setting positive and negative reward rules; according to the long-term optimization needs of traffic control, a reasonable discount factor is set to balance immediate and long-term rewards; finally, the above five elements are integrated to form a complete fusion decision model, ensuring that the model can take into account both prior knowledge and real-time dynamic decision-making needs.

[0129] Furthermore, to enhance the stability and rationality of the decision-making model, thereby improving the efficiency and accuracy of traffic signal control, in one embodiment, the control device constructs a fusion decision-making model based on Retrieval-Augmented Generation (RAG) and Markov Decision Process (MDP), namely RAG-MDP, to achieve joint optimization of traffic control and knowledge retrieval. This process is defined as follows:

[0130]

[0131] in, Representing the state space, Represents the action space. Let be the state transition probability function. For instant reward function, This is the discount factor. The specific design is as follows:

[0132] The state space of the fusion decision model is defined as follows:

[0133]

[0134] in, Representing the state space, The traffic environment state perceived by the intelligent agent includes physical characteristics such as traffic flow, average speed, lane density, average queue length, and current signal phase at each intersection. The knowledge retrieval state is perceived by the intelligent agent, and the knowledge retrieval state includes indicators such as current knowledge confidence, retrieval action history, knowledge relevance score and support.

[0135] The action space of the fusion decision model is defined as follows:

[0136]

[0137]

[0138] in, Represents the action space. This represents the knowledge retrieval action space. This indicates that no search action will be triggered. This indicates a search of the traffic regulations database. This indicates a search of the emergency response plan database. This indicates a search of the accident case database. This indicates the traffic signal control action space, which includes the phase switching and timing adjustment of traffic lights;

[0139] The policy function of the policy network is defined as follows:

[0140]

[0141] in, The policy parameters of the policy network are parameterized by the GNN. Indicates the policy network in time Joint decision-making actions based on traffic environment status and knowledge retrieval status output;

[0142] The reward function of the fusion decision model is defined as:

[0143]

[0144] in, Indicates the total reward. Incentives related to traffic efficiency. Rewards related to the effectiveness of knowledge retrieval. and Indicates the weighting coefficient ( The reward function is used to comprehensively measure the instantaneous decision quality of the agent at each time step t;

[0145] The overall optimization objective during the training and convergence process of the policy network is to maximize the expected cumulative reward in the future, as shown in the following formula:

[0146]

[0147] in, Indicates the discount factor. This represents the maximum step size in the time domain for a single round of interactive decision-making. Indicates the first intelligent agent Instant rewards for each time step Indicates that the agent is policy-based The mathematical expectation of the cumulative reward corresponding to the discounts of all decision trajectories when interacting with the environment.

[0148] By achieving this goal, the system can jointly optimize the balance between traffic efficiency and knowledge utilization, enabling the agent to autonomously explore control strategies and supplement decisions with external knowledge when necessary, thereby achieving efficient, stable and interpretable control in complex and dynamic traffic environments.

[0149] Step S40: Concatenate the global traffic state vector with the comprehensive knowledge representation vector to generate an enhanced state vector, and train and converge the policy network based on the enhanced state vector using the fusion decision model and the near-end policy optimization algorithm.

[0150] It should be noted that the augmented state vector is a new vector formed by concatenating the global traffic state vector and the comprehensive knowledge representation vector according to the feature dimension. It integrates real-time traffic state information and prior knowledge information and is the core input for training the policy network.

[0151] The Proximal Policy Optimization (PPO) algorithm is used to improve the stability and convergence speed of policy training by limiting the magnitude of policy updates and avoiding gradient explosion or gradient vanishing during the update process. In this embodiment, it is used for policy network training.

[0152] Understandably, this embodiment provides a more comprehensive and representative input to the policy network by concatenating the global traffic state vector with the comprehensive knowledge representation vector to form an enhanced state vector, thereby improving the network's adaptability to complex traffic scenarios. Training with a near-end policy optimization algorithm effectively solves the problems of unstable gradients and slow convergence speed in traditional reinforcement learning training, accelerating the convergence speed of the policy network while ensuring training stability. The converged policy network can initially possess the ability to generate optimal control policies.

[0153] In some embodiments, the agent concatenates the global traffic state vector with the comprehensive knowledge representation vector to generate an enhanced state vector; inputs the enhanced state vector into the policy network to generate corresponding actions, and obtains immediate rewards from environmental feedback and state data for the next time step; constructs an experience sample set based on the enhanced state vector, actions, immediate rewards, and state data for the next time step generated through multiple rounds of interaction; updates the network parameters of the policy network and the parameters of the action value function based on the experience sample set; iteratively executes the interaction and parameter update operations until the policy network meets the preset convergence conditions, thus completing the training and convergence of the policy network.

[0154] Furthermore, to ensure that the agent simultaneously optimizes traffic efficiency, safety, and decision interpretability during the learning process, step S40 may include:

[0155] Step S401: Concatenate the global traffic state vector with the comprehensive knowledge representation vector to generate an enhanced state vector;

[0156] Step S402: Input the enhanced state vector into the policy network to generate the corresponding execution action and obtain the immediate reward and state data of the reinforcement learning environment feedback.

[0157] Step S403: Construct an experience sample set based on the enhanced state vector, executed actions, immediate rewards, and state data of the next moment generated by multi-round environmental interactions;

[0158] Step S404: Based on the empirical sample set, update the network parameters of the policy network and the parameters of the action value function through the fusion decision model and the proximal policy optimization algorithm;

[0159] Step S405: Iteratively execute environment interaction and parameter update operations until the policy network meets the preset convergence conditions, thus completing the training and convergence of the policy network.

[0160] Understandably, to ensure that the agent simultaneously optimizes traffic efficiency, safety, and decision interpretability during the learning process, a multi-objective reward feedback mechanism is introduced. The agent at each time step... The theoretical framework, implementation, and design for obtaining immediate rewards are as follows:

[0161]

[0162] Traffic efficiency indicators are calculated by weighting factors such as average vehicle throughput per unit time, average delay time, and average queue length.

[0163]

[0164] The safety index is inversely normalized from negative factors such as the number of red light violations, the probability of conflict incidents, and the braking rate.

[0165]

[0166] The knowledge consistency index measures the degree of consistency between the agent's current decision and the results retrieved from the knowledge base, and is defined as follows:

[0167]

[0168] To balance the three objectives, a Pareto optimization strategy is adopted, which adaptively adjusts the weights during training to achieve dynamic multi-objective equilibrium. , and These represent the reward weight coefficients, which are adaptively adjusted during training using a Pareto optimization strategy. This represents the weighting coefficient for the traffic flow throughput sub-item. This represents the weighting factor for the vehicle delay sub-item. Indicates the first The actual average traffic flow throughput of the controlled intersection within the decision-making time step. This indicates the saturation traffic flow at the intersection. Indicates the first The average delay time for vehicles at controlled intersections within a decision-making time step. This indicates the maximum permissible delay time limit preset at the controlled intersection. Indicates the first Negative statistics on traffic conflicts at controlled intersections within a single decision-making timeframe. This indicates the maximum preset traffic conflict value at the controlled intersection. This represents the semantic embedding vector of the decision action output by the agent. This represents the semantic embedding vector of knowledge suggestions retrieved by the agent from the knowledge base.

[0169] Furthermore, to prevent over-reliance on the knowledge base from leading to a decline in exploration capabilities, a knowledge suppression regularization term is introduced. This term penalizes the agent's retrieval actions using the following formula:

[0170]

[0171] in, This indicates the revised instant reward. This represents the coefficient of the knowledge suppression regularization term, which is usually a small positive number. This represents the degree of execution of the retrieval action, which is a binary variable taking the value 0 or 1, where 0 indicates no retrieval and 1 indicates strong retrieval. The degree of execution of the retrieval action indicates the level of execution of the action by the agent in the [missing information - likely a specific action or event]. The retrieval behavior at each decision time step refers to the extent to which the agent performs a retrieval action at the current moment.

[0172] Further, after completing state-driven knowledge retrieval and fusion, the process moves to model training and policy convergence. This step uses reinforcement learning as its core, optimizing the agent's decision-making strategy through multiple rounds of interactive iteration. The agent... The state at any given moment is jointly defined by the state of the environment and the knowledge-enhanced representation:

[0173]

[0174] in, express The global traffic environment status at any given moment. This represents the set of knowledge fragments obtained through state-driven retrieval. This represents a function that integrates state and knowledge.

[0175] The agent, based on the policy network Select Action or Environmental return rewards Next state Through multiple rounds of interaction with the environment, a set of experience samples is constructed:

[0176]

[0177] And use this set of experiences to analyze the value function. With policy function Joint optimization was carried out.

[0178] The value function is updated by minimizing the Bellman residual:

[0179]

[0180] in, The Bellman residual loss term represents the action-value function. Represents the trainable parameters of the value network. Let the action value function be... This represents the augmented state vector at time step t. This represents the state of the agent at the next moment after it performs an action. This represents the action performed by the agent at time step t. Indicates the optional action for the next moment. Represents the set of empirical samples. This represents the mathematical expectation operation.

[0181] The policy function is then optimized by maximizing the expected reward and introducing interpretability and knowledge constraints:

[0182]

[0183] in, This is the optimization function for the policy function. This represents the output value of the action value function. The policy network is state-based. The optimal action to output. This is an interpretability loss term, used to strengthen the semantic correspondence between knowledge retrieval results and action decisions. This is a knowledge suppression regularization term used to prevent the agent from becoming overly reliant on the knowledge base during training. The weighting coefficients for the interpretability loss term. The weighting coefficients for knowledge suppression regularization terms.

[0184] Step S50: Generate a traffic signal control strategy based on the converged strategy network, and perform traffic signal control based on the traffic signal control strategy.

[0185] It should be noted that the traffic signal control strategy can be output by the converged strategy network and is a specific traffic signal control scheme for the current traffic state. For example, it may include core control parameters such as the sequence of signal light phase switching, the duration of each phase, and the timing of switching.

[0186] It is understood that this embodiment realizes intelligent and adaptive regulation of traffic signal control. The generated control strategy can accurately match the current traffic state, effectively alleviate traffic congestion, reduce vehicle delays, and improve intersection efficiency. The control process does not require manual intervention, reducing the cost and error of manual control. At the same time, it can adaptively adjust according to the dynamic changes in traffic state, improving the flexibility and adaptability of traffic signal control and ensuring the orderly operation of traffic.

[0187] In some embodiments, the control device concatenates the real-time collected and processed global traffic state vector with the retrieved and aggregated comprehensive knowledge representation vector to generate an enhanced state vector, which is then input into the converged policy network. Based on the input enhanced state vector and the mapping relationship learned during training, the policy network outputs the optimal traffic signal control strategy for the current traffic state (specifying the duration, switching order, and activation timing of each phase). The control strategy is then converted into standardized control commands, which are sent to the traffic light control devices at each intersection via the communication module. The devices execute traffic signal control operations according to the commands. Simultaneously, real-time traffic state data after control is collected and fed back to the policy network for subsequent fine-tuning and optimization, ensuring that the control strategy can adapt to the dynamic changes in traffic conditions.

[0188] In practical implementation, after model training converges, the system enters the actual decision-making and performance evaluation phase. For any given time... When the agent receives the environmental state Knowledge-enhanced input At that time, the policy network generates interpretable decision outputs and presents the decision explanation results in the form of text visualization.

[0189] The system's performance is comprehensively evaluated from three dimensions: traffic efficiency, safety, and interpretability. Key indicators include: average delay time, average queue length, average throughput, and decision-knowledge consistency. By comparing the system with baseline models such as traditional timed control, adaptive control, and simple RL control, the improvements in overall performance and interpretable decision quality are verified. Finally, the system outputs include: the agent's final policy, interpretable decision logs, performance indicators, and visualization analysis results.

[0190] This embodiment achieves comprehensive perception of the traffic environment by collecting traffic status data in real time and constructing a global state vector. Knowledge base retrieval and aggregation compensate for the limitations of real-time data, improving the rationality of decision-making. The fusion of decision-making models and near-end policy optimization algorithms ensures stable training and rapid convergence of the policy network. The converged policy network generates control strategies and executes control, achieving precise regulation of traffic signals. On one hand, global traffic state modeling and multi-intersection collaborative simulation enhance the generalization and adaptability, anti-interference ability, and operational stability of the signal control strategy for various traffic scenarios, effectively addressing random fluctuations in traffic flow and sudden traffic situations. On the other hand, knowledge retrieval and prior knowledge fusion make the control decision logic traceable and explainable, avoiding the black-box decision-making risks of purely data-driven models. This also improves the compliance and rationality of the control strategy, effectively increasing the efficiency of traffic signal control, reducing intersection delays and congestion rates, and providing reliable technical support for the intelligent collaborative management and control of urban road networks.

[0191] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement of the present invention.

[0192] Based on the first embodiment described above, in this embodiment, step S20 further includes:

[0193] Step S201: Encode each knowledge fragment in the knowledge base into a vector representation using a semantic embedding model to form a knowledge vector space.

[0194] It should be noted that semantic embedding models are deep learning models used to transform unstructured knowledge such as text and data into structured numerical vectors. They can capture the semantic information and core features of knowledge fragments and realize the computable representation of knowledge.

[0195] A knowledge fragment refers to the smallest unit of knowledge in a knowledge base that has independent semantics and can be extracted separately, such as a traffic regulation clause, an emergency response procedure, or a fragment of an accident handling case.

[0196] The knowledge vector space is a high-dimensional space composed of vector representations of all knowledge fragments. Each point in the space corresponds to a knowledge fragment vector, and the distance between vectors reflects the semantic similarity of knowledge fragments.

[0197] It should be noted that the knowledge base includes a traffic regulations database, an emergency response plan database, and an accident case database.

[0198] The traffic regulations database stores laws, regulations, rules, and signal control standards related to the traffic field, providing a compliance basis for traffic signal control.

[0199] The emergency response plan database stores emergency response plans, procedures, and control strategies for various traffic emergencies (such as congestion, accidents, and severe weather), providing support for decision-making in emergency traffic situations.

[0200] The accident case database stores historical traffic accident scene information, handling process, control measures, and effect feedback, providing experience and reference for signal control in similar accident scenarios.

[0201] Step S202: Convert the global traffic state vector into a structured retrieval request using a mapping function.

[0202] It should be noted that the mapping function is used to convert the global traffic state vector (high-dimensional numerical vector) into a structured request that conforms to the knowledge vector space retrieval specification. It can adapt the state vector to the retrieval format and ensure the accuracy and efficiency of the retrieval.

[0203] A structured retrieval request can be a retrieval instruction with a clear retrieval target. It is transformed from a global traffic state vector and contains information such as the core features of traffic state and the retrieval scope. It can be directly used for retrieval in the knowledge vector space.

[0204] In some embodiments, the control device generates a linear mapping function, presets mapping parameters and format rules, takes a global traffic state vector (including features such as traffic flow, queue length, and signal phase) as input, and adjusts the dimensions and filters features of the vector through the mapping function, retaining core features related to knowledge retrieval (such as congestion level, signal phase state, and traffic scenario type); converts the adjusted feature vector into a structured format that conforms to the knowledge vector space retrieval specification, clarifies the retrieval scope (such as prioritizing the retrieval of emergency plan databases and accident case databases), generates a structured retrieval request, and ensures that the request can accurately match relevant knowledge fragment vectors in the knowledge vector space, avoiding retrieval bias.

[0205] Step S203: Analyze the structured search request based on the policy network to determine whether to perform a search action.

[0206] In some embodiments, the control device inputs the generated structured retrieval request into the policy network in the current training phase. The policy network analyzes the complexity of the current traffic state (such as whether it is a normal off-peak state, or whether there are abnormal situations such as congestion or accidents) based on the learned mapping relationship between traffic states and retrieval requests. A preset retrieval judgment threshold is set. If the policy network analyzes that the current traffic state is complex and conventional control strategies cannot be adapted (such as sudden congestion or rare accident scenarios), then it is determined to execute the retrieval action. If the current traffic state is a normal scenario and can be directly controlled by existing strategies, then it is determined not to execute the retrieval action to avoid invalid retrieval consuming computing resources.

[0207] Step S204: If the retrieval action is performed, the knowledge vector space is searched to obtain a set of knowledge fragments related to the current traffic state.

[0208] In some embodiments, if the policy network determines to perform a retrieval action, it converts the structured retrieval request into a retrieval vector, uses a cosine similarity algorithm to calculate the similarity between the retrieval vector and all knowledge fragment vectors in the knowledge vector space; a preset similarity threshold is set to filter out knowledge fragment vectors with similarity higher than the threshold, and the corresponding original knowledge fragment is restored through the mapping relationship between the vector and the original knowledge fragment; all restored knowledge fragments are integrated to form a set of knowledge fragments related to the current traffic state. For example, if the current traffic state is "a sudden rear-end collision at the intersection causing congestion", then similar accident handling cases in the accident case database, congestion mitigation plans in the emergency plan database, and relevant traffic regulations in the traffic regulations database are retrieved to form a set of knowledge fragments.

[0209] Step S205: Calculate the similarity between the global traffic state vector and each knowledge fragment vector in the knowledge fragment set, and generate the knowledge weight coefficient corresponding to each knowledge fragment vector based on the similarity.

[0210] It should be noted that the knowledge weight coefficient can be a weight value assigned to each knowledge fragment vector based on the similarity calculation results. It is used to characterize the importance of the knowledge fragment to the current traffic signal control decision. The weight coefficient is positively correlated with the similarity.

[0211] Step S206: Based on the knowledge weight coefficient, perform weighted aggregation on each knowledge segment vector in the knowledge segment set to obtain a comprehensive knowledge representation vector.

[0212] In its implementation, the control device first constructs an external knowledge base, which comprises three subsets: a traffic regulations database. Emergency Response Plan Database With accident case library Its overall definition is:

[0213]

[0214] Each knowledge segment After semantic embedding model Encoded as a vector representation:

[0215]

[0216] Forming a knowledge vector space , This indicates that the database can be dynamically updated periodically to adapt to new knowledge.

[0217] Secondly, the obtained global traffic state vector Through mapping function Transform status information into a structured search request:

[0218]

[0219] in, Implemented using a graph neural network (GNN). The output includes semantic tags, status summary, and search category fields.

[0220] Meanwhile, in order to improve retrieval efficiency and reduce retrieval costs, and to determine whether the agent needs to retrieve external knowledge and which type of knowledge to retrieve, this system designs the retrieval action space as follows:

[0221]

[0222] in, This indicates that no search action will be triggered; This indicates a search of the traffic regulations database; This indicates a search of the emergency response plan database. This indicates a search of the accident case database.

[0223] Subsequently, the intelligent agent in the knowledge base Local execution is performed on each type of knowledge base The system dynamically retrieves a set of knowledge fragments most relevant to the current traffic condition.

[0224] ,

[0225] in Indicates the first Vector representation of knowledge fragments.

[0226] Next, the global traffic status will be updated. The similarity between each knowledge fragment and the knowledge fragment is defined in a cross-modal semantic space, and the cosine similarity is calculated. The similarity is then normalized using softmax to obtain probabilistic weight coefficients. The agent continues to perform weighted aggregation operations, compressing multiple knowledge fragments into a single comprehensive knowledge representation vector:

[0227] ,

[0228] in, A vector represents comprehensive knowledge. Indicates the first The weight of each knowledge fragment.

[0229] Finally, the global traffic state vector With aggregated knowledge representation By concatenating the data, an enhanced state input can be formed:

[0230]

[0231] Augmented State Vector As the input state of the extended Markov decision process, it integrates environmental dynamics information and knowledge semantic information, providing context and external knowledge support for the joint decision-making of intelligent agents.

[0232] This embodiment encodes knowledge into vectors using a semantic embedding model, constructing a knowledge vector space and achieving knowledge computability. It transforms retrieval requests through mapping functions to ensure retrieval format adaptation; it judges retrieval actions through a policy network, improving operational efficiency; it obtains a set of relevant knowledge fragments through retrieval, compensating for the limitations of real-time data; it highlights core knowledge through similarity calculation and weight allocation; and it generates a comprehensive knowledge representation vector through weighted aggregation, achieving effective fusion of prior knowledge. Overall, this embodiment ensures accurate matching and deep fusion of prior knowledge and real-time traffic conditions, improving the efficiency and accuracy of knowledge retrieval and fusion, and providing comprehensive and reliable prior knowledge support for decision optimization.

[0233] Furthermore, this embodiment of the invention also proposes a computer-readable storage medium storing a reinforcement learning intelligent traffic signal control program. When the reinforcement learning intelligent traffic signal control program is executed by a processor, it implements the steps of the reinforcement learning intelligent traffic signal control method based on state-driven retrieval enhancement as described above.

[0234] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0235] The aforementioned computer-readable storage medium may be included in a state-driven retrieval-enhanced reinforcement learning intelligent traffic signal control device; or it may exist independently and not assembled into a state-driven retrieval-enhanced reinforcement learning intelligent traffic signal control device.

[0236] Furthermore, this invention also proposes a computer program product, including a reinforcement learning intelligent traffic signal control program, which, when executed by a processor, implements the steps of the state-driven retrieval-enhanced reinforcement learning intelligent traffic signal control method described above.

[0237] The specific implementation of the computer program product of the present invention is basically the same as the embodiments of the reinforcement learning intelligent traffic signal control method based on state-driven retrieval enhancement described above, and will not be repeated here.

[0238] Reference Figure 4 , Figure 4 This is a structural block diagram of the first embodiment of the reinforcement learning-based intelligent traffic signal control system based on state-driven retrieval enhancement of the present invention.

[0239] like Figure 4 As shown, the reinforcement learning-based intelligent traffic signal control system based on state-driven retrieval enhancement proposed in this embodiment of the invention includes:

[0240] The information acquisition module 10 is used to collect real-time traffic flow status information of the traffic environment and construct a global traffic state vector based on the real-time traffic flow status information.

[0241] The knowledge retrieval enhancement module 20 is used to retrieve the knowledge base based on the global traffic state vector, obtain a set of knowledge fragments related to the current traffic state, and perform weighted aggregation on the vectors of each knowledge fragment in the set of knowledge fragments to obtain a comprehensive knowledge representation vector.

[0242] The decision fusion module 30 is used to construct a fusion decision model based on the retrieval enhancement mechanism and the Markov decision mechanism. The fusion decision model is a five-tuple structure, which includes a state space, an action space, a state transition probability function, an immediate reward function, and a discount factor. The state space includes the traffic environment state corresponding to the global traffic state vector and the knowledge retrieval state corresponding to the comprehensive knowledge representation vector.

[0243] The reinforcement learning module 40 is used to concatenate the global traffic state vector with the comprehensive knowledge representation vector to generate an enhanced state vector, and to train and converge the policy network based on the enhanced state vector through the fusion decision model and the near-end policy optimization algorithm.

[0244] The traffic signal control module 50 is used to generate a traffic signal control strategy based on the converged strategy network, and to perform traffic signal control based on the traffic signal control strategy.

[0245] This embodiment achieves comprehensive perception of the traffic environment by collecting traffic status data in real time and constructing a global state vector. Knowledge base retrieval and aggregation compensate for the limitations of real-time data, improving the rationality of decision-making. The fusion of decision-making models and near-end policy optimization algorithms ensures stable training and rapid convergence of the policy network. The converged policy network generates control strategies and executes control, achieving precise regulation of traffic signals. On one hand, global traffic state modeling and multi-intersection collaborative simulation enhance the generalization and adaptability, anti-interference ability, and operational stability of the signal control strategy for various traffic scenarios, effectively addressing random fluctuations in traffic flow and sudden traffic situations. On the other hand, knowledge retrieval and prior knowledge fusion make the control decision logic traceable and explainable, avoiding the black-box decision-making risks of purely data-driven models. This also improves the compliance and rationality of the control strategy, effectively increasing the efficiency of traffic signal control, reducing intersection delays and congestion rates, and providing reliable technical support for the intelligent collaborative management and control of urban road networks.

[0246] The state-driven retrieval-enhanced reinforcement learning-based intelligent traffic signal control system provided in this application employs the state-driven retrieval-enhanced reinforcement learning-based intelligent traffic signal control method described in the above embodiments, and can solve the technical problems of state-driven retrieval-enhanced reinforcement learning-based intelligent traffic signal control. Compared with the prior art, the beneficial effects of the state-driven retrieval-enhanced reinforcement learning-based intelligent traffic signal control system provided in this application are the same as those of the state-driven retrieval-enhanced reinforcement learning-based intelligent traffic signal control method provided in the above embodiments, and other technical features in the state-driven retrieval-enhanced reinforcement learning-based intelligent traffic signal control system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0247] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0248] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0249] In addition, for technical details not described in detail in this embodiment, please refer to the reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement provided in any embodiment of the present invention, which will not be repeated here.

[0250] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0251] It should be noted that the user information (including but not limited to user device information, user personal information, user location information, user behavior information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0252] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0253] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0254] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement, characterized in that, The method is applied to agents in a reinforcement learning environment, wherein multiple agents are configured in the reinforcement learning environment, each agent is configured with a policy network, the policy network being a parameterized graph neural network structure, and each agent is configured to control traffic signals at a traffic intersection. The method includes: Collect real-time traffic flow status information of the traffic environment, and construct a global traffic state vector based on the real-time traffic flow status information; Based on the global traffic state vector retrieval knowledge base, a set of knowledge fragments related to the current traffic state is obtained, and the vectors of each knowledge fragment in the set of knowledge fragments are weighted and aggregated to obtain a comprehensive knowledge representation vector; A fusion decision model is constructed based on the retrieval enhancement mechanism and the Markov decision mechanism. The fusion decision model is a five-tuple structure, which includes a state space, an action space, a state transition probability function, an immediate reward function, and a discount factor. The state space includes the traffic environment state corresponding to the global traffic state vector and the knowledge retrieval state corresponding to the comprehensive knowledge representation vector. The global traffic state vector is concatenated with the comprehensive knowledge representation vector to generate an enhanced state vector. Based on the enhanced state vector, the policy network is trained and converged using the fusion decision model and the near-end policy optimization algorithm. A traffic signal control strategy is generated based on the converged strategy network, and traffic signal control is performed based on the traffic signal control strategy. The state space of the fusion decision model is defined as follows: in, Representing the state space, The traffic environment status perceived by the intelligent agent includes traffic flow, average speed, lane density, average queue length, and current signal phase at each intersection. The knowledge retrieval state perceived by the intelligent agent includes the current knowledge confidence, retrieval action history, knowledge relevance score, and support. The action space of the fusion decision model is defined as follows: in, Represents the action space. This represents the knowledge retrieval action space. This indicates that no search action will be triggered. This indicates a search of the traffic regulations database. This indicates a search of the emergency response plan database. This indicates a search of the accident case database. This indicates the traffic signal control action space, which includes the phase switching and timing adjustment of traffic lights; The policy function of the policy network is defined as follows: in, This represents the policy parameters of the policy network. Indicates the policy network in time Joint decision-making actions based on traffic environment status and knowledge retrieval status output; The reward function of the fusion decision model is defined as: in, Indicates the total reward. Incentives related to traffic efficiency. Rewards related to the effectiveness of knowledge retrieval. and The weighting coefficients are used to comprehensively measure the instantaneous decision quality of the agent at each time step t. The overall optimization objective during the training and convergence process of the policy network is to maximize the expected cumulative reward in the future, as shown in the following formula: in, Indicates the discount factor. This represents the maximum step size in the time domain for a single round of interactive decision-making. Indicates the first intelligent agent Instant rewards for each time step Indicates that the agent is policy-based The mathematical expectation of the cumulative reward corresponding to the discounts of all decision trajectories when interacting with the environment.

2. The reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement as described in claim 1, characterized in that, The process of collecting real-time traffic flow state information of the traffic environment and constructing a global traffic state vector based on the real-time traffic flow state information includes: Obtain road network structure data for the target area, extract road node information and roadside information from the road network structure data, and generate a directed topology graph of the road network; The directed topology graph of the road network is imported into the traffic simulation platform to generate a multi-intersection traffic simulation model. The multi-intersection traffic simulation model collects real-time traffic flow status information corresponding to each control intersection in the traffic environment and generates an intersection state vector corresponding to each control intersection. The real-time traffic flow status information includes real-time traffic flow characteristics and real-time signal phase characteristics. The intersection state vector includes traffic volume, average speed, lane density, average queue length, and current signal phase. The intersection state vectors corresponding to all controlled intersections are concatenated to generate a global traffic state vector.

3. The reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement as described in claim 1, characterized in that, The process involves retrieving a knowledge base based on the global traffic state vector, obtaining a set of knowledge fragments related to the current traffic state, and weighted aggregation of the vectors of each knowledge fragment in the set to obtain a comprehensive knowledge representation vector, including: The knowledge fragments in the knowledge base are encoded into vector representations by a semantic embedding model to form a knowledge vector space. The knowledge base includes a traffic regulations database, an emergency plan database, and an accident case database. The global traffic state vector is converted into a structured retrieval request using a mapping function; The structured search request is analyzed based on a policy network to determine whether to perform a search action. If the retrieval action is performed, the knowledge vector space is searched to obtain a set of knowledge fragments related to the current traffic state; Calculate the similarity between the global traffic state vector and each knowledge fragment vector in the knowledge fragment set, and generate the knowledge weight coefficient corresponding to each knowledge fragment vector based on the similarity. Based on the knowledge weight coefficients, the vectors of each knowledge segment in the knowledge segment set are weighted and aggregated to obtain a comprehensive knowledge representation vector.

4. The reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement as described in claim 1, characterized in that, The step of concatenating the global traffic state vector with the comprehensive knowledge representation vector to generate an enhanced state vector, and then training and converging the policy network based on the enhanced state vector using the fusion decision model and the near-end policy optimization algorithm, includes: The global traffic state vector and the comprehensive knowledge representation vector are concatenated to generate an enhanced state vector; The enhanced state vector is input into the policy network to generate corresponding execution actions, and the immediate reward from the reinforcement learning environment and the state data at the next time step are obtained. The immediate reward is defined as: in, Indicators representing traffic efficiency. Indicates safety indicators, Indicators representing knowledge consistency , and These represent the reward weight coefficients, which are adaptively adjusted during training using a Pareto optimization strategy. This represents the weighting coefficient for the traffic flow throughput sub-item. This represents the weighting factor for the vehicle delay sub-item. Indicates the first The actual average traffic flow throughput of the controlled intersection within the decision-making time step. This indicates the saturation traffic flow at the intersection. Indicates the first The average delay time for vehicles at controlled intersections within a decision-making time step. This indicates the maximum permissible delay time limit preset at the controlled intersection. Indicates the first Negative statistics on traffic conflicts at controlled intersections within a single decision-making timeframe. This indicates the maximum preset traffic conflict value at the controlled intersection. This represents the semantic embedding vector of the decision action output by the agent. This represents the semantic embedding vector of knowledge suggestions retrieved by the agent from the knowledge base; An experience sample set is constructed based on the enhanced state vectors, executed actions, immediate rewards, and state data of the next moment generated by multi-round environmental interactions. Based on the empirical sample set, the network parameters of the policy network and the parameters of the action value function are updated through the fusion decision model and the proximal policy optimization algorithm. Iteratively execute environment interaction and parameter update operations until the policy network meets the preset convergence conditions, thus completing the training and convergence of the policy network.

5. The reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement as described in claim 4, characterized in that, The action value function is updated by minimizing the Bellman residual, as shown in the following formula: in, The Bellman residual loss term represents the action-value function. Represents the trainable parameters of the value network. Let the action value function be... This represents the augmented state vector at time step t. This represents the state of the agent at the next moment after it performs an action. This represents the action performed by the agent at time step t. Indicates the optional action for the next moment. Represents the set of empirical samples. This represents the mathematical expectation operation; The policy function of the policy network is optimized by maximizing the expected reward and introducing an interpretability loss term and a knowledge inhibition regularization term, as shown in the following formula: in, This is the optimization function for the policy function. This represents the output value of the action value function. The policy network is state-based. The optimal action to output. This is an interpretability loss term, used to strengthen the semantic correspondence between knowledge retrieval results and action decisions. This is a knowledge suppression regularization term used to prevent the agent from becoming overly reliant on the knowledge base during training. The weighting coefficients for the interpretability loss term. The weighting coefficients for knowledge suppression regularization terms.

6. The reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement as described in claim 5, characterized in that, The knowledge suppression regularization term penalizes the agent's retrieval actions using the following formula, as shown below: in, This indicates the revised instant reward. This represents the coefficient of the knowledge suppression regularization term. This represents the degree of execution of the retrieval action, which is a binary variable taking the value 0 or 1. The degree of execution of the retrieval action represents the agent's performance in the [missing information - likely a specific action or event]. The retrieval behavior at each decision-making time step.

7. A reinforcement learning-based intelligent traffic signal control system with state-driven retrieval enhancement, characterized in that, The system employs the reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement as described in any one of claims 1 to 6, and the system comprises: The information acquisition module is used to collect real-time traffic flow status information of the traffic environment and construct a global traffic state vector based on the real-time traffic flow status information. The knowledge retrieval enhancement module is used to retrieve the knowledge base based on the global traffic state vector, obtain a set of knowledge fragments related to the current traffic state, and perform weighted aggregation on the knowledge fragment vectors in the knowledge fragment set to obtain a comprehensive knowledge representation vector. The decision fusion module is used to construct a fusion decision model based on the retrieval enhancement mechanism and the Markov decision mechanism. The fusion decision model is a five-tuple structure, which includes a state space, an action space, a state transition probability function, an immediate reward function, and a discount factor. The state space includes the traffic environment state corresponding to the global traffic state vector and the knowledge retrieval state corresponding to the comprehensive knowledge representation vector. The reinforcement learning module is used to concatenate the global traffic state vector with the comprehensive knowledge representation vector to generate an enhanced state vector, and based on the enhanced state vector, to train and converge the policy network through the fusion decision model and the near-end policy optimization algorithm. The traffic signal control module is used to generate a traffic signal control strategy based on the converged strategy network, and to perform traffic signal control based on the traffic signal control strategy.

8. A reinforcement learning-based intelligent traffic signal control device with state-driven retrieval enhancement, characterized in that, The reinforcement learning-based intelligent traffic signal control device based on state-driven retrieval enhancement includes: a memory, a processor, and a reinforcement learning-based intelligent traffic signal control program stored in the memory. The processor is used to run the reinforcement learning-based intelligent traffic signal control program, which is configured to implement the reinforcement learning-based intelligent traffic signal control method based on state-driven retrieval enhancement as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a reinforcement learning intelligent traffic signal control program, which, when executed by a processor, implements the reinforcement learning intelligent traffic signal control method based on state-driven retrieval enhancement as described in any one of claims 1 to 6.

Images