Traffic signal lamp cooperative control method based on multi-agent reinforcement learning and related device

Abstract

The invention discloses a traffic signal lamp cooperative control method based on multi-agent reinforcement learning and a related device. The method comprises the following steps: acquiring a global traffic state of multiple intersections; inputting the global traffic state processed by the preset prompt into a preset large language model, and outputting a high-level planning embedding set; performing optimization processing on the task relationship among the high-level planning embedding in the high-level planning embedding set through a self-attention mechanism to obtain global target task embedding; generating agent state embedding according to a local traffic state observed by each agent, and performing embedding fusion on global target task embedding and agent state embedding through a cross self-attention mechanism to obtain team state embedding corresponding to each agent; based on team state embedding, traffic signal lamp cooperative control actions are output through intelligent agents, so that the information interaction and cooperation efficiency between the intelligent agents is enhanced, and a global efficient traffic signal control strategy is provided in a complex dynamic traffic scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of traffic signal control, in particular to a traffic signal cooperative control method based on multi-agent reinforcement learning and related device. BACKGROUND

[0002] With the increasing complexity of urban traffic systems, traffic congestion has become a major problem faced by major cities, especially at the bottleneck area of the traffic network, the intersection. In recent years, using multi-agent reinforcement learning (MARL) for adaptive traffic signal control has become one of the research hotspots for alleviating traffic congestion. Traffic signal control is a challenging and important problem, the core goal of which is to minimize travel time by optimizing vehicle flow at intersections. Improper signal timing not only affects travel efficiency, but in severe cases can even cause the traffic system to collapse, increasing the risk of traffic accidents. Therefore, how to optimize signal timing and improve traffic efficiency at intersections has become a key research problem in the field of intelligent transportation. Although the existing multi-agent traffic signal control based on classical reinforcement learning algorithms can learn dynamic strategies, their performance depends on manual feature engineering, and the fixed network structure limits scalability, making it prone to slow or unstable convergence in high-dimensional state and action spaces. In the face of complex dynamic traffic scenarios, the multi-agent control of traffic signals is inefficient. SUMMARY

[0003] The present application provides a traffic signal cooperative control method based on multi-agent reinforcement learning and related device, which solves the technical problem of low multi-agent collaboration efficiency in existing multi-agent traffic signal control tasks.

[0004] The traffic signal cooperative control method based on multi-agent reinforcement learning provided by the present application comprises:

[0005] The multi-agent subsystem obtains global traffic information of multiple intersections and converts it into global traffic states;

[0006] The global traffic states processed by the preset prompt are input into the preset large language model for task decision processing, and a high-level planning embedding set is output; the high-level planning embedding set contains K high-level planning embeddings;

[0007] The task relationship between each high-level planning embedding in the high-level planning embedding set is optimized by a self-attention mechanism to obtain a global target task embedding;

[0008] The local traffic state observed by each agent generates an agent state embedding, and the global target task embedding and the agent state embedding are embedded and fused through a cross self-attention mechanism to obtain a team state embedding corresponding to each agent;

[0009] Based on the team state embedding, the agent outputs a traffic signal lamp cooperative control action.

[0010] Optionally, the global traffic state includes traffic flow states and signal states of multiple intersections; the traffic flow state of each intersection is spliced from a density state matrix, a speed state matrix, a left turn information matrix, and a right turn information matrix.

[0011] Optionally, the step of inputting the global traffic state processed by the preset prompt into the preset large language model for task decision processing and outputting a high-level planning embedding set includes:

[0012] Inputting the global traffic state processed by the preset prompt into the preset large language model;

[0013] In the preset large language model, the global traffic state processed by the preset prompt is converted into an initial traffic flow embedding vector and an initial signal embedding vector through an embedding layer;

[0014] The number of embedding vectors is dynamically adjusted according to the initial traffic flow embedding vector and the initial signal embedding vector;

[0015] Based on the number of embedding vectors, the target traffic flow embedding vector and the target signal embedding vector are determined according to the initial traffic flow embedding vector and the initial signal embedding vector;

[0016] The target traffic flow embedding vector and the target signal embedding vector are decision reasoned through a multi-layer self-attention mechanism to output a high-level planning embedding set.

[0017] Optionally, the step of optimizing the task relationship between each high-level planning embedding in the high-level planning embedding set through a self-attention mechanism to obtain a global target task embedding includes:

[0018] Based on the self-attention mechanism, each high-level planning embedding is linearly converted into a corresponding query vector, a key vector, and a value vector;

[0019] According to the task similarity of the query vector, the key vector, and the value vector between each high-level planning embedding, a weighted task corresponding to each agent is calculated;

[0020] According to the high-level planning embedding and the weighted task, a global target task embedding corresponding to each agent is calculated.

[0021] Optionally, the embedding fusion process of the global target task embedding and the agent state embedding is specifically:

[0022]

[0023] In the formula: represents a cross attention mechanism process, represents a global target task embedding of the agent i, represents an agent state embedding of the agent j, represents the dimension of the key vector;

[0024]

[0025] In the formula: represents a team state embedding of the agent i.

[0026] Optionally, based on the team state embedding, the step of outputting a traffic signal lamp cooperative control action by the agent includes:

[0027] Decoupling and projecting all team state embeddings through a dimension layer to decompose into multiple subspace features;

[0028] Inputting the subspace features into the policy head network of the corresponding agent to output the traffic signal lamp cooperative control action.

[0029] The application also provides a traffic signal lamp cooperative control device based on multi-agent reinforcement learning, which comprises:

[0030] An acquisition module is configured to acquire global traffic information of multiple intersections through a multi-agent subsystem and convert the global traffic information into global traffic states;

[0031] A high-level planning embedding output module is configured to input the global traffic states processed by a preset prompt into a preset large language model for task decision processing, and output a high-level planning embedding set; the high-level planning embedding set comprises K high-level planning embeddings;

[0032] A global target task embedding generation module is configured to optimize the task relationship between each high-level planning embedding in the high-level planning embedding set through a self-attention mechanism to obtain a global target task embedding;

[0033] A team state embedding generation module is configured to generate an agent state embedding from a local traffic state observed by each agent, and fuse the global target task embedding and the agent state embedding through a cross self-attention mechanism to obtain a team state embedding corresponding to each agent;

[0034] A traffic signal lamp cooperative control action execution module is configured to output a traffic signal lamp cooperative control action based on the team state embedding through the agent.

[0035] The application further provides a computer device comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the traffic signal coordination control method according to any one of the above.

[0036] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program / instruction, and the computer program is executed by a processor to implement the steps of the traffic signal coordination control method according to any one of the above.

[0037] The application further provides a computer program product, comprising a computer program / instruction, and the computer program is executed by a processor to implement the steps of the traffic signal coordination control method according to any one of the above.

[0038] From the above technical solutions, the application has the following advantages:

[0039] The application provides a traffic signal coordination control method based on multi-agent reinforcement learning and related devices, wherein the method comprises: acquiring global traffic information of multiple intersections through a multi-agent subsystem and converting the global traffic information into a global traffic state; inputting the global traffic state processed by a preset prompt into a preset large language model for task decision processing, and outputting a high-level planning embedding set; the high-level planning embedding set comprises K high-level planning embeddings; the task relationship between each high-level planning embedding in the high-level planning embedding set is optimized through a self-attention mechanism to obtain a global target task embedding; an agent state embedding is generated from a local traffic state observed by each agent, and the global target task embedding and the agent state embedding are embedded and fused through a cross self-attention mechanism to obtain a team state embedding corresponding to each agent; and based on the team state embedding, an agent outputs a traffic signal coordination control action.

[0040] In the application, a preset large language model is introduced, knowledge transfer is combined with multi-agent strategy learning, the information interaction and collaboration efficiency between agents are enhanced, strategy guidance is provided for the agents, the expression ability of the model for high-dimensional states and action spaces is improved through an attention mechanism, and the training process and convergence speed are accelerated, so that a global efficient traffic signal control strategy is provided in a complex dynamic traffic scene, and the technical problem of low multi-agent collaboration efficiency in the existing multi-agent traffic signal control task is solved. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0042] Figure 1 A step flow chart of a traffic signal lamp cooperative control method based on multi-agent reinforcement learning provided for an embodiment of the present application is shown in the figure.

[0043] Figure 2 A framework diagram of a traffic signal lamp cooperative control system based on multi-agent reinforcement learning provided for an embodiment of the present application is shown in the figure.

[0044] Figure 3 An execution flow chart of a top-level task planning module based on a large language model provided for an embodiment of the present application is shown in the figure.

[0045] Figure 4 An execution flow chart of a multi-agent reinforcement learning network architecture module based on Transformer provided for an embodiment of the present application is shown in the figure.

[0046] Figure 5 A multi-intersection signal simulation environment based on SUMO provided for an example of the present application is shown in the figure.

[0047] Figure 6 A generation flow chart of global target task embedding based on a self-attention mechanism provided for an embodiment of the present application is shown in the figure.

[0048] Figure 7 A generation flow chart of team state embedding based on a cross-attention mechanism provided for an embodiment of the present application is shown in the figure.

[0049] Figure 8 A simulation schematic diagram for realizing cooperative traffic signal scheduling under different test environments provided for an example of the present application is shown in the figure.

[0050] Figure 9 A structural block diagram of a traffic signal lamp cooperative control device based on multi-agent reinforcement learning provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0051] The embodiments of the present application provide a traffic signal lamp cooperative control method and related device based on multi-agent reinforcement learning, which are used to solve the technical problem of low multi-agent cooperation efficiency in the existing multi-agent traffic signal control task.

[0052] In order to make the application purpose, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the embodiments described below are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0053] The main process of the current multi-agent traffic signal control includes collection and preprocessing of traffic flow data, modeling of state and action space, and multi-agent strategy learning and deployment based on optimization algorithm. The existing schemes mainly include the following five categories: (1) fixed time control method is adopted, fixed green, yellow and red light time is designed through historical traffic flow data, and is executed in a preset cycle, which is suitable for simple scenes with relatively stable traffic flow; (2) inductive control method is adopted, traffic flow information is collected in real time by using magnetic coil or camera sensor, and signal light time is dynamically adjusted to adapt to changing traffic demand; (3) adaptive control based algorithm is adopted, signal light timing is dynamically adjusted through optimization technology and real-time data analysis to maximize road traffic efficiency; (4) classical reinforcement learning based algorithm is adopted, traffic signal optimization control is carried out through designing fixed network structure and feature engineering; (5) multi-agent algorithm based on deep learning is introduced, the cooperation relationship among multi-agents is modeled and the optimal strategy is learned through convolution network or fully connected network.

[0054] The traditional fixed time control method cannot adapt to the change of real-time traffic flow, which easily leads to congestion in peak period and wastes resources in low traffic, and is difficult to cope with dynamic and complex traffic environment. Although the inductive control method can adjust the signal light time in real time, it is strongly dependent on the sensor and is easily disturbed by environmental factors, and at the same time, it can only be locally optimized, lacks global perspective, and is difficult to play a role in multi-intersection cooperation. In contrast, the adaptive control method improves flexibility by dynamic adjustment, but its implementation process is complex, and has high requirements for real-time data quality and computing power, and the reaction may lag in high dynamic or non-stationary environment. Further, the classical reinforcement learning based algorithm can learn dynamic strategy, but its performance depends on manual feature engineering, and the fixed network structure limits the scalability, which easily causes slow convergence or instability in high-dimensional state and action space.

[0055] Therefore, the present application provides a traffic signal lamp cooperative control method based on multi-agent reinforcement learning, which is applied to a traffic signal lamp cooperative control system based on multi-agent reinforcement learning (MARL), and aims to solve the problems of reward sparsity, dimension disaster and slow cold start in multi-agent traffic signal control. Please refer to Figure 2The overall framework of the traffic signal coordination control system mainly consists of a top-level task planning module based on a large language model (LLM) and a multi-agent reinforcement learning network architecture module based on a Transformer. The traffic signal coordination control system also includes a multi-agent subsystem. The multiple agents (traffic signal controllers) in the multi-agent subsystem correspond one-to-one to multiple intersections. Each agent can obtain the traffic state of the corresponding intersection and output a coordinated control action corresponding to the traffic state. Each agent includes an evaluation network, an action network, and a policy head network.

[0056] 1. Top-level task planning module based on a large language model (LLM)

[0057] This module performs high-level task planning through an LLM to provide global strategy guidance for each agent. Specifically, this module is responsible for: scene understanding and task decomposition: converting real-time data of traffic signal control (such as traffic flow, signal state, etc.) into natural language and using an LLM to analyze and generate hierarchical task objectives. Strategy guidance generation: based on the task representation (LMEmbs) output by the LLM, providing specific signal control strategy suggestions for each agent to ensure global traffic coordination. Real-time calibration and feedback mechanism: based on the feedback after the agent executes, adjusting and optimizing the strategy in real time to respond to dynamic traffic flow changes.

[0058] The purpose of this module is to provide top-level task planning through an LLM to optimize task allocation and collaboration strategies in the multi-agent subsystem, especially in a multi-agent reinforcement learning (MARL) system, to address issues such as sparse rewards and slow cold start. For complex traffic signal control tasks, the invention utilizes the semantic reasoning and task generation capabilities of an LLM to improve the learning efficiency and strategy optimization speed of the multi-agent subsystem through efficient task planning and dynamic adjustment.

[0059] 2. Multi-agent reinforcement learning network architecture module based on a Transformer

[0060] The module is used for optimizing state space and action space representation of the multi-agent subsystem through a Transformer network structure, and solves the dimension disaster problem. The specific implementation steps include: fusing the local observation of each agent, the state information of neighbor agents, and global environment data, and inputting into the Transformer network. The self-attention mechanism of the Transformer is used to capture the complex interaction and global dependence relationship between agents, so as to improve the collaboration ability of the system. The module adopts an optimized MAPPO policy network, and the PPO (Proximal Policy Optimization) and the Transformer are combined in the architecture. The self-attention mechanism is used to solve the calculation complexity problem of the traditional MAPPO algorithm in a large-scale agent system, and the policy update is optimized.

[0061] The two modules form a closed-loop optimization system through the bidirectional data flow of strategy guidance-execution feedback, so as to improve the overall efficiency and learning ability of the multi-agent traffic signal control system.

[0062] Specifically, refer to Figure 1 The application provides a traffic signal lamp cooperative control method based on multi-agent reinforcement learning.

[0063] In step 101, the multi-agent subsystem is used to acquire global traffic information of multiple intersections and convert the global traffic information into global traffic states.

[0064] It should be noted that the input data of the traffic signal lamp cooperative control system is specifically the global traffic states of multiple intersections, and the global traffic states include current traffic flow states and signal states of multiple intersections. The traffic flow state of each intersection is composed of a position state matrix, a speed state matrix, a left turning information matrix, and a right turning information matrix.

[0065] For example, as shown in Figure 5 It is assumed that there are N=4 intersections in the traffic signal lamp cooperative control system, and the numbers of the intersections are {1, 2, 3, 4} respectively, and each intersection has four entrances in the east, west, south, and north directions. Since there is little difference between the low-resolution state input and the high-resolution state input in the output result, and the low-resolution input can effectively reduce the calculation complexity of the model, and considering the technical limitations in actual application, especially the positioning accuracy cannot reach the lane level, in this example, the vehicle information on the entrance roads of the four intersections acquired by the multi-agent subsystem is directly used as the state variable, and then the entrance roads are discretized.

[0066] Specifically, the four incoming roads of each intersection are divided into m segments with length d, where m = r / d, r is the radius range of vehicle information collection. Then the vehicle density and average speed in each segment are calculated, forming two vectors with length n, which are the vehicle position density vector and the vehicle average speed vector , respectively, for each road. , where corresponds to the east, west, south and north directions. The road is divided into multiple regions, and the left turn state vector and the right turn state vector are obtained by collecting the turn signal status of each vehicle in each region through the Internet of Vehicles.

[0067] For example, in calculating the vehicle position density, it is assumed that there are 3 vehicles in the first road surface region, and the maximum vehicle capacity of the region is 6. The vehicle density is the number of vehicles / the maximum number of vehicles that can be accommodated in the region, so the vehicle density of the first road surface region is 0.5. Assuming that the road speed limit is , the vehicle speed of the left lane of the first road surface is , and the vehicle speed of the right turn lane and the straight lane is 0, so the vehicle average speed in the first road surface region is:

[0068]

[0069] where n is the total number of vehicles in the region, is the road speed limit, represents the speed of the ith vehicle in the region. Thus, the vehicle average speed in the first road surface region is 0.133 In the first road surface region, the number of vehicles with the left turn signal on is 2, i.e. corresponds to 2, and the number of vehicles with the right turn signal on is 1, i.e. corresponds to 1.

[0070] For example, the traffic flow state of the first intersection road is , which is represented as a four-dimensional tensor composed of the density state matrix , the speed state matrix , the left turn information matrix and the right turn information matrix , where , , contains the position, speed and turning intention of the surrounding vehicles. ​

[0071] In addition, the signal states of multiple intersections The signal phase states of the previous time step that can be observed by the multi-agent subsystem are generated by one-hot encoding. Then for the first intersection, its traffic state can be represented as And the traffic states of the remaining intersections can be established in the same way, respectively , , The global traffic state of the four intersections in the final example is represented as It can also be represented as .

[0072] Step 102, input the global traffic state processed by the preset prompt into the preset large language model for task decision processing, and output a set of high-level planning embeddings; the set of high-level planning embeddings contains K high-level planning embeddings.

[0073] This step specifically includes:

[0074] Sub-step 1021, input the global traffic state processed by the preset prompt into the preset large language model.

[0075] In order to enable the preset LLM to understand the multi-intersection signal control problem, the input data also needs to be preprocessed by the prompt. Wherein the prompt is a formatted text input, which not only provides a specific task background, but also guides the LLM how to parse the input data.

[0076] After the traffic flow state and signal state data are preprocessed by the prompt, they can be combined into format data that meets the input requirements of the LLM. The format is usually composed of a structured text, which contains task description, input data, and problem to be solved, as follows:

[0077] (1) Task description: Add a brief task description in front of the input data to inform the model that the current task is to perform signal control planning for multiple intersections. This part of the prompt instructs the model to understand the background and goal of the task.

[0078] For example: According to the traffic flow and signal state, generate the signal light control strategy for each intersection. The traffic flow information includes traffic volume, traffic density, speed, etc., and the signal state information includes signal light period, phase, and signal state.

[0079] (2) Input data: Format the data of traffic flow state and signal state into the structure that conforms to LLM. For example, the traffic flow data can include traffic volume, speed, queue length, etc., and the signal state data includes cycle, phase, signal light state, etc. Convert these information into structured text representation, as shown in Table 1.

[0080] Table 1 Input data corresponding to intersection 1

[0081]

[0082] (3) Problem to be solved: Clearly define the problem requirements so that the LLM generates high-level planning decisions for each intersection, i.e., signal control strategies.

[0083] For example: Ask the model to generate K high-level planning embeddings, each corresponding to a signal control strategy for an intersection.

[0084] Substep 1022, in the preset large language model, convert the global traffic state processed by the preset prompt into an initial traffic flow embedding vector and an initial signal embedding vector through an embedding layer.

[0085] Wherein, the embedding layer uses tokenization (word segmentation) and embedding (word embedding) to convert the global traffic state processed by the preset prompt into a digital representation, encoding each input feature. It is equivalent to converting the global traffic state processed by the preset prompt into a vector representation. Therefore, when the traffic flow state and signal state are processed by the embedding layer, two groups of embedding vectors are obtained and i.e.

[0086]

[0087]

[0088] In the formula: represents the initial traffic flow embedding vector, represents the initial traffic flow embedding of the nth intersection, represents the initial signal embedding vector, represents the initial signal embedding of the nth intersection.

[0089] Substep 1023, dynamically adjust the number of embedding vectors according to the initial traffic flow embedding vector and the initial signal embedding vector.

[0090] In actual traffic control tasks, traffic flow and signal state are constantly changing, so the number and content of the subsequently generated high-level planning embeddings (LM Embs) should also be dynamically adjusted, which can better adapt to the complexity and real-time changes of the traffic environment, thereby improving the response ability and control effect of the system. In order to cope with different modes and complex situations of traffic flow, the system will dynamically adjust the number of embedding vectors according to the complexity of the current traffic flow state and signal control state.

[0091] wherein the complexity of the system is determined by the initial traffic flow embedding vector and the initial signal embedding vector, that is:

[0092]

[0093] In the formula: represents the complexity, represents the complexity measurement function. When the complexity C increases, the system needs more embedding vectors to represent and process diversified traffic situations, at which time the number of embedding vectors K will increase; on the contrary, when the complexity decreases, K will decrease. In this way, K is no longer fixed, but is flexibly adjusted according to the current state and demand of traffic.

[0094] wherein the number of embedding vectors is dynamically adjusted as follows:

[0095]

[0096] In the formula: represents the mapping function, and are adjustment parameters, and can be set according to experiments or daily experience, represents the floor operation. In this way, the number of embedding vectors can be flexibly adjusted according to the changes of the traffic environment.

[0097] Sub-step 1024, based on the number of embedding vectors, determining target traffic flow embedding vectors and target signal embedding vectors according to the initial traffic flow embedding vectors and the initial signal embedding vectors.

[0098] It should be noted that assuming that the initial traffic flow embedding vectors and the initial signal embedding vectors correspond to N intersections, in the case of K N, the traffic state information of the N intersections can be combined into the traffic state information corresponding to the K control sub-regions, such as combining the intersections with similar traffic flow modes into a group, so as to determine the target traffic flow embedding vectors and the target signal embedding vectors, and then taking the target traffic flow embedding vectors and the target signal embedding vectors as the input features of the initial self-attention layer.

[0099] In sub-step 1025, the target traffic flow embedding vector and the target signal embedding vector are subjected to decision reasoning through a multi-layer self-attention mechanism, and a high-level planning embedding set is output.

[0100] In this sub-step, a self-attention mechanism is used to learn the mutual relationship between each intersection. The self-attention mechanism can consider the information of other intersections when processing the features of each intersection, ensuring that the model captures the influence between different intersections in the traffic network. The processing process of the self-attention mechanism of the initial self-attention layer is as follows:

[0101]

[0102] In the formula: are the query vector, key vector, and value vector of the input feature , respectively, is the dimension of the key vector; is the attention output, representing the importance of intersection i in the global context.

[0103] Through the multi-layer self-attention mechanism, the LLM converts each intersection's input feature vector into its corresponding high-level planning embedding . These embeddings represent high-level decision information for signal control strategies, mainly including signal phase arrangement and optimization of signal cycle and duration for intersections. Finally, the high-level planning embedding for each intersection can be represented as:

[0104]

[0105] In the formula: is the reasoning process of the LLM, which will generate corresponding high-level planning decision embeddings based on the input features .

[0106] According to the complexity of the traffic network, the LLM generates K high-level planning embeddings, where K corresponds to the number of intersection categories. The final output of the high-level planning embedding set is:

[0107]

[0108] The preset LLM can learn the mutual influence between different intersections based on the input data of traffic flow and signal state through the self-attention mechanism, generating high-level planning embeddings. In the process of dynamically adjusting K, the preset LLM can adaptively adjust the number of signal control strategies according to the complexity of the traffic network, thereby achieving more accurate and efficient multi-intersection signal control.

[0109] As Figure 3As shown, the LLM-based top-level task planning module converts information such as traffic flow and traffic signal state into natural language descriptions through semantic understanding of the traffic signal control scene, and inputs the descriptions into the LLM to generate high-level planning embeddings (LMEmbs). In this example, the LLM uses a large language model Qwen1.5B to infer, and the high-level planning embeddings obtained by the LLM are used to provide strategy guidance for each agent (traffic signal controller) through hierarchical task decomposition, avoiding single local optimization in traditional methods and improving global collaboration efficiency. Secondly, the LLM decomposes the global task (such as "optimize regional traffic flow") into specific tasks for each agent (such as "adjust the red-green light cycle"). The specific process includes scene understanding, task decomposition, strategy generation, real-time calibration, and finally outputs structured strategy guidance (for example, "regional green wave band coordination"). At the same time, the execution results of the agents in the multi-agent subsystem are fed back to the LLM, which adjusts the strategy according to the feedback to form a closed-loop optimization process. For example, the global traffic state can be updated every few seconds to adjust the high-level planning embeddings, ensuring that the task planning is continuously optimized as the environment changes.

[0110] In step 103, the task relationships between the high-level planning embeddings in the set of high-level planning embeddings are optimized through a self-attention mechanism to obtain a global target task embedding.

[0111] Please refer to Figure 6 This step includes:

[0112] In sub-step 1031, each high-level planning embedding is linearly converted into a query vector, a key vector, and a value vector corresponding to each agent based on the self-attention mechanism.

[0113] In the LLM-based top-level task planning module, the final output set of high-level planning embeddings is Then, the multi-agent reinforcement learning network architecture module based on the Transformer optimizes the network structure by introducing the Transformer architecture. The self-attention mechanism in the Transformer architecture allows the model to dynamically adjust the weights when generating the planning embeddings for each agent based on the relationships between the planning embeddings, thereby optimizing the quality of the planning embeddings. In the self-attention mechanism, each planning embedding is weighted based on the calculation of query (Query), key (Key), and value (Value).

[0114] For each agent i, the high-level planning embedding is processed as a query (Query), a key (Key), and a value (Value). Specifically, each high-level planning embedding The query vector (Query), the key vector (Key) and the value vector (Value) are generated by linear transformation:

[0115]

[0116] In the formula: and are the linear transformation matrices corresponding to the query vector, the key vector and the value vector respectively, is the high-level planning embedding (Language Model Emb) of the agent i.

[0117] In substep 1032, the weighted task corresponding to each agent is calculated according to the task similarity of the query vector, the key vector and the value vector between each high-level planning embedding.

[0118] In this embodiment, the attention weight is generated by calculating the similarity between the query vector and the key vector between different high-level planning embeddings based on the self-attention mechanism, so as to weight each high-level planning embedding. The calculation formula of the attention mechanism is:

[0119]

[0120] wherein, is the dimension of the key vector, denotes the query vector of the agent i, denotes the key vector of the agent j, denotes the similarity between the high-level planning embedding of the agent i and the high-level planning embedding of the agent j , that is, the attention weight.

[0121] The attention weight is normalized by the softmax function, and the finally calculated weighted task is represented as:

[0122]

[0123] wherein, denotes the weighted task of the agent i, is the value vector corresponding to the high-level planning embedding , and the integrated language model embedding (Integrate Language Model Emb) is obtained after weighting.

[0124] In substep 1033, the global target task embedding corresponding to each agent is calculated according to the high-level planning embedding and the weighted task.

[0125] Through the calculation of the above self-attention mechanism, the high-level task planning​ will be combined into an integrated language model embedding (Integrate Language ModelEmb), which contains the task information of all agents and optimizes the dependencies between tasks. Among them, the i-th global target task embedding in the integrated language model embedding The calculation formula is:

[0126]

[0127] The integrated language model embedding integrates the task goals and information of all agents, and the integrated language model embedding will serve as the input of the subsequent cross-attention mechanism.

[0128] In step 104, the agent state embedding is generated based on the local traffic state observed by each agent. The global target task embedding and the agent state embedding are embedded and fused through the cross self-attention mechanism to obtain the team state embedding corresponding to each agent.

[0129] See also Figure 7 , integrating language model embedding and agent state embedding (Agent Stat Embs) to generate team state embedding (Team State Emb), thereby providing global collaborative strategy optimization information.

[0130] It is understandable that in addition to receiving the task representation (LM Embs), each agent also observes its local environment (such as traffic flow, signal light status, etc.) and generates an agent state embedding (Agent State Embed). The state embedding of each agent i can be expressed as:

[0131]

[0132] in, is the traffic flow observed by agent i, It is the current state of the traffic light. Other characteristics include road conditions, emergency situations, etc.

[0133] After obtaining the integrated language model embedding and agent state embedding, the two are then fused through the cross-attention mechanism to generate a team state embedding (Team State Embed), which represents the joint state of the entire multi-agent. The goal of the cross-attention mechanism is to combine the integrated language model embedding (Integrate LM Embedding) and the agent state embedding (Agent State Embed) to provide a state representation for each agent with global collaborative information. The calculation formula of the cross-attention mechanism is:

[0134]

[0135] wherein, represents the cross-attention mechanism process, is the global target task embedding of the agent i.

[0136] The mutual influence between agents is calculated through cross-attention to generate a globally optimized team state embedding:

[0137]

[0138] wherein, represents the team state embedding of the agent i. Through the cross-attention mechanism, the global state information and the local state information of the agent are effectively fused, thereby generating a team state embedding TeamState from a global perspective.

[0139] Step 105, based on the team state embedding, outputting traffic signal coordination control actions through the agent.

[0140] In this step, first, all team state embeddings are projected through a dimension decoupling layer to decompose into multiple subspace features; then the subspace features are input into the corresponding agent's policy head network to output traffic signal coordination control actions, dynamically achieving optimal coordination regulation of complex traffic scenarios.

[0141] In traditional multi-agent subsystems, each agent usually has an independent feature extraction module, which not only increases the computational cost but also leads to parameter redundancy. In the present invention, in order to reduce the computational load and improve the training efficiency, a shared bottom feature extraction network is adopted. By sharing the bottom feature extraction module, unnecessary parameter duplication is reduced. Specifically, the bottom feature extraction part of the system (such as the embedding layer and the initial self-attention layer mentioned above) can adopt shared parameters, thereby reducing the computational complexity and improving the computational efficiency. These shared bottom parameters include: wherein, The set of shared parameters includes the parameters of the embedding layer (Embedding Layer) and the attention layer (Attention Layer).

[0142] By sharing the parameters of these bottom feature extraction modules, the system not only reduces the computational overhead during training, but also improves the generalization ability of the model, because all agents are learning in the same bottom feature space, thereby reducing the risk of overfitting. After the shared embedding layer and attention layer, the output shared bottom features In this way, all agents can share the same network structure and parameters in the feature extraction stage, greatly reducing the computational overhead.

[0143] To avoid the interference of features between different agents, the application designs a dimension decoupling projection layer to decompose the input high-dimensional feature space into multiple independent subspaces, so as to ensure that the state of each agent can be optimized independently. Specifically, assuming that there is an input state feature D containing the input state features of all agents, that is, containing the team state embedding TeamState. Through the dimension decoupling projection layer, the input state feature can be decomposed into multiple independent subspaces:

[0144]

[0145] wherein, is the independent subspace of agent i, and each subspace is a vector. Through this dimension decoupling manner, the features of each agent are independently processed, thereby avoiding the information interference between different agents. The formula of the dimension decoupling projection operation is:

[0146]

[0147] The strategy generation of each agent needs to be optimized independently according to the task target and the environment state. Therefore, each agent has an independent strategy head network for generating the strategy output of the agent, that is:

[0148]

[0149] In this way, the input feature of each agent will enter the independent strategy head network of the agent to generate the final strategy , and the action network outputs the traffic signal lamp cooperative control action, while the evaluation network evaluates the action value, and the policy gradient update guides the optimization of the strategy head network to optimize the action distribution--finally realizing the cooperative optimization of the control of the signal lamps of each intersection.

[0150] As Figure 4 ​As shown, the Transformer-based multi-agent reinforcement learning network architecture module improves the learning efficiency and decision quality of the multi-agent subsystem through effective feature fusion and task collaboration. First, the self-attention mechanism is used to optimize the high-level planning embedding (Language Model Emb) generated by the large language model to generate an integrated language model embedding (Integrate Language Model Emb) to provide high-level task guidance for each agent. Then, the cross-attention mechanism is used to process the integrated language model embedding and the state embedding (Agent State Emb) of each agent to generate a team state embedding (Team State Emb) containing global collaborative information, which reflects the global state of the entire system and the interaction between individuals.

[0151] The purpose of the present application is to solve the problems of dimension disaster and low multi-agent collaboration efficiency in the multi-agent traffic signal control task. The present application optimizes the MAPPO (Multi-Agent Proximal Policy Optimization) algorithm based on the Transformer network structure, improves the model's expression ability for high-dimensional state and action space through the attention mechanism, and accelerates the training process and convergence speed. In addition, the pre-training knowledge of the preset large language model is introduced, and the knowledge transfer is combined with multi-agent policy learning to enhance the information interaction and collaboration efficiency between agents, so as to realize efficient traffic signal control strategy in complex dynamic traffic scenarios. This method is suitable for multi-intersection and complex traffic network, and provides an effective solution for global optimization in dynamic traffic environment.

[0152] The traffic signal lamp cooperative control method based on multi-agent reinforcement learning provided by the present application has the following advantages:

[0153] (1) The present application introduces a large language model (LLM) for task planning, solves the reward sparsity problem in multi-agent reinforcement learning (MARL), and accelerates the policy cold start and policy cold start.

[0154] (2) By using the Transformer self-attention mechanism to optimize the network structure, the network structure of the MAPPO algorithm is optimized, the dimension disaster problem in the multi-agent system is effectively solved, and the modeling ability of complex interactions is enhanced, and the calculation efficiency and performance are improved.

[0155] (3) Combined with the attention mechanism and the reasoning ability of the large model, the high-dimensional state and action space of the multi-agent in the complex dynamic traffic scene are optimized, so as to enhance the information interaction and collaboration efficiency between multi-agents.

[0156] Please refer toFigure 8 The present example also provides simulation of cooperative traffic signal scheduling in different test environments for controlling traffic signals at 4 or 6 intersections on a main road (corresponding to the scales of 4 and 6 agents, respectively) to achieve cooperative traffic signal scheduling to achieve the goal of optimal control.

[0157] Figure 8 The red curve in corresponds to the reward curve of the cooperative signal traffic control scheme using the traditional MAPPO algorithm, Figure 8 The blue curve in corresponds to the reward curve of the cooperative signal traffic control scheme using the present application, Figure 8 (a) in corresponds to 4 agents, Figure 8 (b) in corresponds to 6 agents. From Figure 8 It can be seen from that compared with the traditional scheme, the present application significantly improves the convergence speed and overall performance. The present application introduces a network structure based on Transformer, which enhances the expression ability of the model and reduces the complexity of the state-action space, solves the dimension disaster problem, thereby speeding up the training speed and improving the learning efficiency. In addition, a large language model (LLM) is introduced for task planning, which utilizes its advanced reasoning and planning capabilities to achieve better coordination and cooperation among agents. These improvements collectively contribute to a more stable and optimized strategy, enabling the algorithm to perform well in complex multi-agent tasks such as traffic signal control, ultimately achieving the goal of minimizing travel time and improving traffic efficiency.

[0158] Please refer to Figure 9 The present application also provides a traffic signal lamp cooperative control device based on multi-agent reinforcement learning, the device comprising:

[0159] The acquisition module 201 is configured to acquire global traffic information of multiple intersections through a multi-agent subsystem and convert the global traffic information into global traffic states;

[0160] The high-level planning embedding output module 202 is configured to input the global traffic states processed by the preset prompt into a preset large language model for task decision processing, and output a high-level planning embedding set; the high-level planning embedding set includes K high-level planning embeddings;

[0161] The global target task embedding generation module 203 is configured to optimize the task relationship between each high-level planning embedding in the high-level planning embedding set through a self-attention mechanism, and obtain a global target task embedding;

[0162] The team state embedding generation module 204 is configured to generate an agent state embedding from a local traffic state observed by each agent, and to embed and fuse the global target task embedding and the agent state embedding through a cross self-attention mechanism to obtain a team state embedding corresponding to each agent.

[0163] The traffic signal cooperative control action execution module 205 is configured to output a traffic signal cooperative control action based on the team state embedding.

[0164] The application further provides a computer device, including a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the traffic signal cooperative control method according to any one of the above.

[0165] The application further provides a computer readable storage medium, which stores a computer program / instruction, and the computer program is executed by a processor to implement the steps of the traffic signal cooperative control method according to any one of the above.

[0166] The application further provides a computer program product, which includes a computer program / instruction, and the computer program is executed by a processor to implement the steps of the traffic signal cooperative control method according to any one of the above.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, and will not be described here.

[0168] In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner for actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0169] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0170] In addition, each functional unit in the various embodiments of the application can be integrated into one processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0171] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the entire or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiment methods of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0172] The above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A traffic light collaborative control method based on multi-agent reinforcement learning, characterized in that: Methods include: Obtain global traffic information of multiple intersections through the multi-agent subsystem and convert it into global traffic status; The global traffic state after the preset prompt processing is input into the preset large language model for task decision processing, and the high-level planning embedding set is output; the high-level planning embedding set contains K high-level planning embeddings; The task relationship between each high-level planning embedding in the high-level planning embedding set is optimized through the self-attention mechanism to obtain the global target task embedding; The agent state embedding is generated from the local traffic state observed by each agent. The global target task embedding and the agent state embedding are embedded and fused through the cross self-attention mechanism to obtain the team state embedding corresponding to each agent. Based on the team state embedding, the traffic light collaborative control actions are output by the intelligent agent.

2. The traffic light coordinated control method according to claim 1, characterized in that: The global traffic state includes the traffic flow state and signal state of multiple intersections; the traffic flow state of each intersection is composed of the density state matrix, speed state matrix, left turn information matrix and right turn information matrix.

3. The traffic light coordinated control method according to claim 1, characterized in that: The global traffic state after the preset prompt processing is input into the preset large language model for task decision processing, and the steps of outputting the high-level planning embedding set include: Input the global traffic status processed by the preset prompt into the preset large language model; In the preset large language model, the global traffic state after the preset prompt processing is converted into the initial traffic flow embedding vector and the initial signal embedding vector through the embedding layer; Dynamically adjust the number of embedding vectors according to the initial traffic flow embedding vector and the initial signal embedding vector; Based on the number of embedding vectors, determining the target traffic flow embedding vector and the target signal embedding vector according to the initial traffic flow embedding vector and the initial signal embedding vector; Through a multi-layer self-attention mechanism, decision reasoning is performed on the target traffic flow embedding vector and the target signal embedding vector, and a high-level planning embedding set is output.

4. The traffic light coordinated control method according to claim 1, characterized in that: The steps of optimizing the task relationships between high-level plan embeddings within the high-level plan embedding set through the self-attention mechanism to obtain the global target task embedding include: Based on the self-attention mechanism, each high-level plan embedding is linearly converted into the corresponding query vector, key vector and value vector; Based on the task similarity of query vectors, key vectors, and value vectors between the embeddings of each high-level plan, the weighted tasks corresponding to each agent are calculated; The global target task embedding corresponding to each agent is obtained based on the high-level planning embedding and weighted task calculation.

5. The traffic light coordinated control method according to claim 1, characterized in that: The embedding fusion process of global target task embedding and agent state embedding is as follows: Where: represents the cross attention mechanism process, represents the global target task embedding of agent i, represents the agent state embedding of agent j, represents the dimension of the key vector; Where: represents the team state embedding of agent i.

6. The traffic light coordinated control method according to claim 1, characterized in that: Based on the team state embedding, the steps for the agent to output the traffic light collaborative control action include: All team state embeddings are decomposed into multiple subspace features through a dimensionally decoupled projection layer; The subspace features are input into the corresponding agent's strategy head network to output the traffic light collaborative control action.

7. A traffic light collaborative control device based on multi-agent reinforcement learning, characterized in that: The device includes: The acquisition module is used to obtain the global traffic information of multiple intersections through the multi-agent subsystem and convert it into the global traffic state; The high-level planning embedding output module is used to input the global traffic state after the preset prompt processing into the preset large language model for task decision processing and output the high-level planning embedding set; The high-level planning embedding set contains K high-level planning embeddings; The global target task embedding generation module is used to optimize the task relationship between each high-level plan embedding in the high-level plan embedding set through the self-attention mechanism to obtain the global target task embedding; The team state embedding generation module is used to generate agent state embeddings based on the local traffic state observed by each agent. The global target task embedding and agent state embedding are embedded and fused through the cross self-attention mechanism to obtain the team state embedding corresponding to each agent. The traffic light collaborative control action execution module is used to output traffic light collaborative control actions through the intelligent agent based on team state embedding.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the traffic light coordinated control method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the traffic light coordinated control method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program is executed by a processor, the steps of the traffic light coordinated control method according to any one of claims 1 to 6 are implemented.

Images