Regional traffic optimization control method and system based on multi-agent reinforcement learning

Abstract

The present disclosure provides a regional traffic optimization control method and system based on multi-agent reinforcement learning, which relates to the technical field of regional traffic control. The single-agent action value network adopts centralized training and distributed execution, including: taking the local state observations of all intersections as input, inputting them into the action value network of the single-agent, using a multi-head attention mechanism to assign weights to the importance of each intersection at a certain time T in the traffic area, using a super network to fuse the high-dimensional data generated by the multi-head attention mechanism, distributing the value of each action, selecting the action corresponding to the maximum value, and deciding the optimal action of each intersection under the global situation, so as to achieve optimal control of regional traffic.

Description

Technical Field

[0001] The present disclosure relates to the technical field of regional traffic control, and in particular to a regional traffic optimization control method and system based on multi-agent reinforcement learning. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Traffic congestion not only increases fuel consumption and exhaust emissions, but also increases the incidence of traffic accidents. Solving the problem of traffic congestion has high practical value and theoretical significance. Generally speaking, there are two ways to solve the problem of traffic congestion. One is to increase the construction of transportation infrastructure, and the other is to develop advanced traffic control systems. In cities where land resources are limited, simply increasing transportation infrastructure cannot solve the problem. Therefore, more attention should be paid to the priority development of advanced traffic control systems. At present, the mainstream traffic control systems include TRANSYT, SCOOT and SCAT. These control systems optimize the timing of traffic lights based on traditional traffic parameters, rough estimates based on modeling or expert experience, which can alleviate traffic congestion to a certain extent. However, due to the high nonlinearity of traffic control problems, the effect of regional control based on traditional modeling and expert experience systems often fails to meet expectations.

[0004] In recent years, deep reinforcement learning has become a hot topic. Deep reinforcement learning is data-driven and does not require modeling. It combines the perception ability of deep learning and the decision-making ability of reinforcement learning. Through end-to-end learning, it can achieve direct and efficient control from raw input to output. At present, deep reinforcement learning has been widely used in games, robot control, machine vision and other fields, showing strong efficiency and accuracy.

[0005] However, traffic control problems have their own uniqueness. At present, traffic area control methods using reinforcement learning mostly use memoryless networks, such as CNN, to extract features, but there are still the following problems:

[0006] 1) Unable to accurately capture timing information.

[0007] 2) At the same time, most area controls do not distinguish between critical intersections and non-critical intersections, ignoring the particularity of traffic control issues, which to a certain extent leads to a decline in control effectiveness. Summary of the invention

[0008] In order to solve the above problems, the present invention proposes a regional traffic optimization control method and system based on multi-agent reinforcement learning. Through the action value network of multi-agents with GRU and MLP components, the time series changes and traffic road characteristics are extracted. Through the multi-head attention mechanism, the key intersections in different situations are found and different attentions are allocated. The execution process adopts distributed execution. According to the output, the value of each action is determined, and the action corresponding to the maximum value is selected, so as to decide the optimal action for each intersection considering the global situation.

[0009] According to some embodiments, the present disclosure adopts the following technical solutions:

[0010] The regional traffic optimization control method based on multi-agent reinforcement learning includes:

[0011] Obtain traffic data for each intersection in the traffic area and define multi-agent reinforcement learning parameters;

[0012] Construct a single-agent action value network, deploy the agent network to the corresponding intersections in sequence, obtain the local state observation of each intersection and input it into the action value network, and use the action corresponding to the output maximum action value to control execution;

[0013] Among them, the single-agent action value network adopts centralized training and distributed execution, including: taking the local state observations of all intersections as input, inputting them into the action value network of the single-agent, using the multi-head attention mechanism to assign weights to the importance of each intersection at a certain time T in the traffic area, using the super network to fuse the high-dimensional data generated by the multi-head attention mechanism, distributing the output of the value of each action, selecting the action corresponding to the maximum value, and deciding the optimal action of each intersection in the global situation to achieve optimal control of regional traffic.

[0014] According to some embodiments, the present disclosure adopts the following technical solutions:

[0015] Regional traffic optimization control system based on multi-agent reinforcement learning, including:

[0016] The data acquisition module is used to obtain the traffic data of each intersection in the traffic area and define the multi-agent reinforcement learning parameters;

[0017] The action output module is used to construct a single-agent action value network, deploy the agent network to the corresponding intersections in sequence, obtain the local state observation of each intersection and input it into the action value network, and use the action corresponding to the output maximum action value to control execution;

[0018] Among them, the single-agent action value network adopts centralized training and distributed execution, including: taking the local state observations of all intersections as input, inputting them into the action value network of the single-agent, using the multi-head attention mechanism to assign weights to the importance of each intersection at a certain time T in the traffic area, using the super network to fuse the high-dimensional data generated by the multi-head attention mechanism, distributing the output of the value of each action, selecting the action corresponding to the maximum value, and deciding the optimal action of each intersection in the global situation to achieve optimal control of regional traffic.

[0019] According to some embodiments, the present disclosure adopts the following technical solutions:

[0020] A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the regional traffic optimization control method based on multi-agent reinforcement learning is implemented.

[0021] According to some embodiments, the present disclosure adopts the following technical solutions:

[0022] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the regional traffic optimization control method based on multi-agent reinforcement learning.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The present invention provides a regional traffic optimization control method based on multi-agent reinforcement learning. The invention constructs a single-agent action value network through GRU, MLP, etc. to extract time sequence changes and traffic road characteristics. Through the multi-head attention mechanism, key intersections in different situations are found, and different attention is allocated to different intersections. Then, centralized training is performed using the hypernetwork-associated global reward function to complete the determination of the action value network parameters. The execution process adopts distributed execution. The local observations of each intersection (i.e., intersection number, traffic light timing and road vehicle characteristics) are used as input and input into the action value network of the trained single-agent. According to the output, the value of each action is determined, and the action corresponding to the maximum value is selected, so as to decide the optimal action of each intersection under the consideration of the global situation. When the corresponding actions of each agent are executed by the signal control machine, the regional joint control is completed, and the optimal control of regional traffic is achieved. This method solves the problems of incomplete perception of traffic status and signal timing by traditional traffic signal control systems, inefficient and inaccurate signal control strategies, etc. to a certain extent. At the same time, it also solves to a certain extent the attention allocation between key intersections and other non-key intersections, as well as the problem of balanced coordination between the local optimum of a single intersection and the global optimum of regional intersection groups. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0026] Figure 1 is a diagrammatic illustration of the phase and yellow light phase sequence of an embodiment of the present disclosure;

[0027] Figure 2 A schematic diagram of the vehicle information division area in the local observation state of the intersection according to an embodiment of the present disclosure;

[0028] Figure 3 It is a local observation schematic diagram of an embodiment of the present disclosure;

[0029] Figure 4 The global joint action of the traffic area of ​​the embodiment of the present disclosure is a linear combination diagram of the actions of each intersection;

[0030] Figure 5 A rough schematic diagram of the architecture of the entire network of an embodiment of the present disclosure;

[0031] Figure 6 Schematic diagram of the network training process of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0033] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0034] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0035] Example 1

[0036] In one embodiment of the present disclosure, a method for optimizing regional traffic control based on multi-agent reinforcement learning is provided, comprising:

[0037] Obtain traffic data for each intersection in the traffic area and define multi-agent reinforcement learning parameters;

[0038] Construct a single-agent action value network, deploy the agent network to the corresponding intersections in sequence, obtain the local state observation of each intersection and input it into the action value network, and use the action corresponding to the output maximum action value to control execution;

[0039] Among them, the single-agent action value network adopts centralized training and distributed execution, including: taking the local state observations of all intersections as input, inputting them into the action value network of the single-agent, using the multi-head attention mechanism to assign weights to the importance of each intersection at a certain time T in the traffic area, using the super network to fuse the high-dimensional data generated by the multi-head attention mechanism, distributing the output of the value of each action, selecting the action corresponding to the maximum value, and deciding the optimal action of each intersection in the global situation to achieve optimal control of regional traffic.

[0040] As an embodiment, the specific implementation process of a regional traffic optimization control method based on multi-agent reinforcement learning disclosed in the present invention includes:

[0041] Step 1: Define the global state S in multi-agent reinforcement learning t , local observation t , joint action A t , local agent action a ti and the reward function R t , specifically including the following sub-steps (t represents the time, i represents the i-th agent):

[0042] Step 1.1 defines the state space S, which includes the following steps:

[0043] 1.1.1 Define phases and their order. Consider only 4-phase intersections, and the selected intersection is a 4x4 traffic area (16 traffic lights). The phases of each intersection are {North-South straight right turn, North-South left turn, East-West straight right turn, East-West left turn}, and the phase sequence is similar to the yellow light phase sequence diagram as shown in the figure Figure 1 shown.

[0044] 1.1.2 Define the vehicle information of the local observation state of each single agent. Divide each lane of each intersection at time T into different areas based on the parking line and the entrance. Count the number of vehicles in each area according to the phase sequence. Divide them into two categories: the first category is the number of vehicles that have parked and queued, and the other category is the number of vehicles moving on the lane but not yet parked and queued. Figure 2 shown.

[0045] The local state can be obtained as follows: [0, 0, 2, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], where two groups represent the vehicle information of a lane. For example, 5 represents that there are 5 vehicles that have stopped and are waiting to pass in the second lane from north to south, and the latter position is 0, which means that there are no moving vehicles in this lane. (Leaving vehicles do not belong to the entrance lane and are not counted). Since there are 3 lanes, the dimension of the vehicle information is: 1*3*4*2=1*24. After obtaining the local state description of the number of vehicles in a single lane, all lanes can be inferred by analogy.

[0046] 1.1.3 Define the phase information of the local observation state of each single agent. There are two types of phase information here, the first is the phase information of itself, and the second is the phase information of the adjacent intersections. Generally speaking, there are four intersections connected to an intersection, so the dimension of the phase information is 5. The phase is represented by 1 to 8. Odd phases represent all yellow lights or all red lights, and even phases represent the green light phase sequence. For edge intersections, the surrounding intersections do not exist, and the corresponding phase value is set to -1.

[0047] 1.1.4 Construct the local observation state matrix o of the intersection at time t t With the global observation state S t . Definition r = [intersection number, road vehicle information at time t, 0, phase information at time t]. A total of 24+16+5+1=46 dimensions. And so on to the entire regional intersection group. The local observation diagram is as follows Figure 3 shown.

[0048] After obtaining the local state observation matrices of all intersections, at time t, the global state matrix S of the regional road network (n intersections) can be spliced ​​from these local state matrices, omitting some redundant information t .

[0049] S t ←[o t1 , o t2 , o t3 ……o tn ].

[0050] Step 1.2 Define single agent action a ti Joint action with multiple agents A t .

[0051] Definition i ∈{c0, c1}, c0 means that the intersection keeps the same phase at time t, c1 means that the signal phase of the intersection changes at time t, and i represents the i-th intersection. i Represents the action of the i-th intersection. In the subsequent output of the value network, through:

[0052]

[0053] To select an action, qc0 refers to the action value network's valuation of executing action c0 at time t. The global joint action of the entire traffic area is defined as a linear combination of the actions of each intersection, such as Figure 4 As shown, that is:

[0054] A t ={a t1 , a t2 , a t3 ……a tn}.

[0055] Among them, a ti Represents the action of the i-th intersection at time t.

[0056] Step 1.3 Define the reward function

[0057] In this disclosure, in order to balance various factors at traffic intersections, such as queue length, delay time, and traffic emissions, the global reward function of the multi-agent is defined as the weighted sum of multiple indicators:

[0058] R t =β1△w t +β2L t +β3△E t ,

[0059] Where △w is the delay time difference, Q is the global queue length, △E is the traffic emission difference before and after the action, βi is the weight.

[0060] The specific calculation formula is:

[0061]

[0062]

[0063]

[0064] Where CNT is the intersection number, N represents N intersections, the i-th lane of the CNT-th intersection, j represents the vehicle number of the lane, and vsp(v(t)) refers to the emission generated at time t when the speed is v. nij ) is the representation of the symbolic function, which means that when the speed of the jth vehicle in the nth intersection, the ith lane is less than or equal to 0, the value is 0, otherwise it is 1.

[0065] At the same time, according to the value decomposition method, the action value function of a single agent and the global action function Q At The relationship is:

[0066]

[0067] Among them, F is a function mapping relationship, which can be represented by the neural network built later. t is the global linkage action in (1.2). Using the global reward function and the offline temporal difference method, we can perform training updates according to the following formula:

[0068]

[0069] Among them, α is a constant between 0 and 1, which plays a role in smoothing updates. γ is called the discount factor, which mainly measures the impact of future conditions on the present. In this method, we set γ to 0.85 and α to 0.5.

[0070] Step 2: Construct an action value training network for multi-agent deep reinforcement learning, which includes the following steps:

[0071] Step 2.1 Build a single agent action value network. This network controls the traffic of a single intersection in the area. The network consists of MLP (multi-layer perceptron) + GRU (recurrent neural network) + MLP (multi-layer perceptron). The input is 46-dimensional local observation data and 46-dimensional hidden layer data, and the output is a 2-dimensional action value value. Since this method is implemented in a 4*4 road network, the global output is 2*16=32 dimensions. The GRU network has sequential memory. When the content in step 1 is composed <St , A t , R t , S t+1 …>When the sequence is input, not only the current information of the traffic condition can be extracted, but also the time series information of the evolution of the traffic condition can be extracted, so as to achieve more accurate value estimation and determine more efficient and accurate traffic light actions.

[0072] Step 2.2 uses the multi-head attention mechanism to assign weights to the importance of each intersection in the traffic area at time T. The value network (value) of the multi-head attention mechanism is the action value estimate in step 2.1. The inputs of the query network (query) and the key network (key) are S t , o t1 . Both query and key are composed of a basic 2-layer perceptron network, and the network activation function is RELU. The final attention weight is obtained by fusing value, key, and query through the softmax method. Using the attention mechanism, higher attention can be given to key intersections. Focus on optimizing these intersections to make traffic control more targeted. Furthermore, the multi-head attention mechanism can avoid the problems of single-layer attention mechanism being too affected by initial parameters, unreasonable attention allocation, and imbalance. In training practice, it is found that too high a dimension not only cannot improve efficiency, but will prolong the training time, so a 4-head attention mechanism is selected. The value network (value) of the multi-head attention mechanism is input through the action value values ​​output by each single agent, and the query network (query) and key network (key) are input through global observation and local observation. They are all composed of a basic 2-layer perceptron network, and the inter-layer network activation function is RELU. The outputs of the value network (value), key network (key), and query network (query) are fused through the softmax method to obtain the final attention weight.

[0073] Step 2.3 Build a super network to fuse the high-dimensional (four-dimensional) data generated by the multi-head attention mechanism to obtain the overall value. Combined with the time series difference method for training. The super network consists of two layers of networks, each layer of which consists of two layers of perceptrons, and the network output is used as weight and offset. The activation function is RELU. During the training process, it is observed that the loss decreases and the reward value increases. When the two are finally stable and unchanged, the final distributed execution action value network is obtained.

[0074] Step 2.4 builds an offline training experience pool. The experience pool is the most prominent feature of offline reinforcement learning. Using the experience pool can significantly improve sample utilization efficiency and training efficiency. The experience pool dynamically stores the data generated during the simulation.<S0,A0,R0,S1,...,end> The complete state action reward sequence from the beginning to the end. This method sets the experience pool size to 200, that is, a maximum of 200 simulation trajectories are allowed to exist (each simulation trajectory contains about 170 <state-action-value-next state-end marker> single steps). When 201 data are generated, the oldest data will be expelled. When used for training, a part of the continuous sequence is randomly extracted, and a segment of length 32 is randomly extracted from the sequence, and the single-step temporal difference method is used. The rough schematic diagram of the entire network architecture is as follows Figure 5 As shown, the target action value network is finally obtained.

[0075] Step 3: Centralized training, training the action value network, such as Figure 6 As shown, it specifically includes the following steps:

[0076] Step 3.1 collects flow data of each intersection in the actual traffic area.

[0077] Step 3.2 Pre-filling process. Use NETEDIT software to draw a road network map, enter the data in SUMO, and perform a signal control operation on the entire area every 9 seconds. The control operation of each intersection is to define the action, that is, change the phase to the next phase or remain unchanged. In this process, the action value network is not used for decision-making, but the traditional traffic control method is used. Record the sequence generated by the state, action, reward and other data. Put the generated sequence fragments into the constructed experience pool until the sequence trajectory reaches 5 (the initial 5 rounds of simulation neural network are not trained and no decision is made).

[0078] Step 3.2 Pre-training process: When the number of complete trajectories in the experience pool reaches a certain number (here set to 10, i.e., 10 rounds of simulation are completed), training begins with a frequency of one training per 10 decisions.

[0079] Step 3.2: When the experience pool reaches 15, terminate the pre-training process and stop using traditional traffic control methods. Each agent makes its own decision using the action value network. While executing actions, it continuously trains the action value network using existing trajectory segments. Observe the reward curve until it stops rising, and then obtain the action value network that can be used for traffic area control.

[0080] Step 4: Use the trained action value network to control traffic signals, which includes the following parts:

[0081] Step 4.1, deploy the agent network to the corresponding intersection in sequence, that is, the signal control machine at intersection 1 corresponds to the first agent action value network that has been trained.

[0082] Step 4.2: Every 9 seconds, the queue counter and the cloud are used to share the road conditions of each intersection. The queue counter is mainly used to obtain the vehicle queue situation at its own intersection, and the cloud shares the phase information of each intersection. After obtaining the data, a local state matrix is ​​formed according to the defined rules.

[0083] Step 4.3, obtain the local state matrix in 4.2, use it as input, input it into the trained action value network, obtain the action value set output by the action value network, and select the action corresponding to the maximum value according to the maximum value. Notify the signal machine to execute the action. Perform the above operation for each intersection, and the signal control process is completed.

[0084] Example 2

[0085] In one embodiment of the present disclosure, a regional traffic optimization control system based on multi-agent reinforcement learning is provided, comprising:

[0086] The data acquisition module is used to obtain the traffic data of each intersection in the traffic area and define the multi-agent reinforcement learning parameters;

[0087] The action output module is used to construct a single-agent action value network, deploy the agent network to the corresponding intersections in sequence, obtain the local state observation of each intersection and input it into the action value network, and use the action corresponding to the output maximum action value to control execution;

[0088] Among them, the single-agent action value network adopts centralized training and distributed execution, including: taking the local state observations of all intersections as input, inputting them into the action value network of the single-agent, using the multi-head attention mechanism to assign weights to the importance of each intersection at a certain time T in the traffic area, using the super network to fuse the high-dimensional data generated by the multi-head attention mechanism, distributing the output of the value of each action, selecting the action corresponding to the maximum value, and deciding the optimal action of each intersection in the global situation to achieve optimal control of regional traffic.

[0089] Example 3

[0090] A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the regional traffic optimization control method based on multi-agent reinforcement learning is implemented.

[0091] Example 4

[0092] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the regional traffic optimization control method based on multi-agent reinforcement learning.

[0093] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0095] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A regional traffic optimization control method based on multi-agent reinforcement learning, characterized in that: include: Obtain traffic data for each intersection in the traffic area and define multi-agent reinforcement learning parameters; The definition of multi-agent reinforcement learning parameters includes: Define the global observation state matrix, local observation state matrix, joint action, single agent action and reward function in multi-agent reinforcement learning; define the local observation state matrix = [intersection number, number of road vehicles at time t, 0, phase information at time t]; Construct a single-agent action value network, deploy the agent network to the corresponding intersections in sequence, obtain the local observation state matrix of each intersection and input it into the action value network, use the action corresponding to the maximum action value output to control execution, and define the single-agent action Joint action with multiple agents ,definition { , }, It means that the phase of the intersection remains unchanged at time t. Indicates the signal phase change of the intersection at time t, i represents the i-th intersection, represents the action of the ith intersection; in the output of the subsequent value network, through: , { ， } To select an action, Refers to the action value network for execution Action valuation, at time t, defines the global joint action of the entire traffic area as a linear combination of the actions of each intersection, that is: = { , , …… } in, represents the action of the i-th intersection at time t; Among them, the single-agent action value network in the multi-agent reinforcement learning specifically includes: building a single-agent action value network, a single-agent action value network controls the traffic of a single-point intersection in a certain area, the single-agent action value network is composed of a multi-layer perceptron MLP, a recurrent neural network GRU and a multi-layer perceptron MLP, the input is multi-dimensional local observation data and hidden layer data, and the output is a two-dimensional action value value; the single-agent action value network adopts centralized training and distributed execution, including: taking the local state observations of all intersections as input, inputting them into the action value network of the single agent, using a multi-head attention mechanism to assign weights to the importance of each intersection at a certain time T in the traffic area, using a super network to fuse the high-dimensional data generated by the multi-head attention mechanism, distributing the value of each action, selecting the action corresponding to the maximum value, and deciding the optimal action of each intersection under the global situation, so as to achieve optimal control of regional traffic; The multi-head attention mechanism is used to assign weights to the importance of each intersection in the traffic area at time T, including: The value network of the multi-head attention mechanism is input through the action value values ​​output by each single agent, and the query network and key network are input through global observation and local observation. They are both composed of a basic 2-layer perceptron network, and the inter-layer network activation function is RELU. The outputs of the value network, key network, and query network are fused through the softmax method to obtain the final attention weight.

2. The regional traffic optimization control method based on multi-agent reinforcement learning as claimed in claim 1, characterized in that: Defining the state space under the global state includes: defining the phase and order, defining the vehicle information of the local observation state of each single agent, defining the phase information of the local observation state of each single agent, and constructing the local observation state matrix and the global observation state matrix of a certain intersection at time t.

3. The regional traffic optimization control method based on multi-agent reinforcement learning as claimed in claim 2 is characterized in that: The reward function is defined as the global reward function of multiple agents as the weighted sum of multiple indicators. According to the value decomposition method, the action value function and the global action function of the single agent are obtained, and the global joint action is updated using the global reward function and the offline temporal difference method.

4. The regional traffic optimization control method based on multi-agent reinforcement learning as claimed in claim 1, characterized in that: A hypernetwork is used to fuse the high-dimensional data generated by the multi-head attention mechanism to obtain the overall value. The hypernetwork consists of two layers of networks, each of which consists of two layers of perceptrons. The network output is used as weight and offset, and RELU is used as the activation function.

5. The regional traffic optimization control method based on multi-agent reinforcement learning as claimed in claim 1, characterized in that: Centralized training of single-agent action-value networks is implemented in an offline training experience pool, which dynamically stores the complete state-action-reward sequence from the beginning to the end of the simulation process.

6. Regional traffic optimization control system based on multi-agent reinforcement learning, characterized by: include: The data acquisition module is used to obtain the traffic data of each intersection in the traffic area and define the multi-agent reinforcement learning parameters; The definition of multi-agent reinforcement learning parameters includes: Define the global observation state matrix, local observation state matrix, joint action, single agent action and reward function in multi-agent reinforcement learning; define the local observation state matrix = [intersection number, number of road vehicles at time t, 0, phase information at time t]; The action output module is used to build a single-agent action value network, deploy the agent network to the corresponding intersection in sequence, obtain the local observation state matrix of each intersection and input it into the action value network, use the action corresponding to the output maximum action value to control execution, and define the single-agent action Joint action with multiple agents ,definition { , }, It means that the phase of the intersection remains unchanged at time t. Indicates the signal phase change of the intersection at time t, i represents the i-th intersection, represents the action of the ith intersection; in the output of the subsequent value network, through: , { ， } To select an action, Refers to the action value network for execution Action valuation, at time t, defines the global joint action of the entire traffic area as a linear combination of the actions of each intersection, that is: = { , , …… } in, represents the action of the i-th intersection at time t; Among them, the single-agent action value network in the multi-agent reinforcement learning specifically includes: building a single-agent action value network, a single-agent action value network controls the traffic of a single-point intersection in a certain area, the single-agent action value network is composed of a multi-layer perceptron MLP, a recurrent neural network GRU and a multi-layer perceptron MLP, the input is multi-dimensional local observation data and hidden layer data, and the output is a two-dimensional action value value; the single-agent action value network adopts centralized training and distributed execution, including: taking the local state observations of all intersections as input, inputting them into the action value network of the single agent, using a multi-head attention mechanism to assign weights to the importance of each intersection at a certain time T in the traffic area, using a super network to fuse the high-dimensional data generated by the multi-head attention mechanism, distributing the value of each action, selecting the action corresponding to the maximum value, and deciding the optimal action of each intersection under the global situation, so as to achieve optimal control of regional traffic; The multi-head attention mechanism is used to assign weights to the importance of each intersection in the traffic area at time T, including: The value network of the multi-head attention mechanism is input through the action value values ​​output by each single agent, and the query network and key network are input through global observation and local observation. They are both composed of a basic 2-layer perceptron network, and the inter-layer network activation function is RELU. The outputs of the value network, key network, and query network are fused through the softmax method to obtain the final attention weight.

7. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the regional traffic optimization control method based on multi-agent reinforcement learning as described in any one of claims 1-5 is implemented.

8. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the regional traffic optimization control method based on multi-agent reinforcement learning as described in any one of claims 1-5.

Images