Multi-agent reinforcement learning method based on graph attention mechanism
By integrating spatiotemporal information and optimizing traffic signal control decisions through a multi-agent reinforcement learning method based on graph attention mechanism, the problem of insufficient coordination in existing methods is solved, and more efficient traffic light control is achieved.
Patent Information
- Application Number
- CN202411481546.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing multi-agent reinforcement learning methods fail to fully utilize spatiotemporal correlations, resulting in insufficient coordination in traffic signal control.
We employ a multi-agent reinforcement learning method based on graph attention mechanism. By extracting features through a multilayer perceptron and combining temporal convolutional coding and graph attention layers, we integrate spatiotemporal information to predict action value and optimize decision-making.
It improves the decision-making accuracy of multi-agent reinforcement learning and enhances the efficiency and coordination of traffic light control.
Smart Images

Figure CN119005292A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a multi-agent reinforcement learning method based on a graph attention mechanism. Background Art
[0002] As one of the more classic methods in robotics, reinforcement learning, especially deep reinforcement learning, has been sought after by many researchers in recent years. Deep reinforcement learning methods have made significant breakthroughs in single-agent and multi-agent control. The main idea of reinforcement learning is that the agent continuously iterates in the environment through trial and error to maximize the reward and achieve the goal. Reinforcement learning algorithms can be divided into value-based and policy-based according to the action selection method. The value-based method learns the optimal value function in each state and selects the corresponding action according to the maximum value function. The policy-based method directly uses the real trajectory data to solve the optimal strategy and updates the strategy parameters by optimizing the cumulative discounted return. The Actor-Critic algorithm combines the two methods mentioned above. The Actor selects actions according to the probability in the strategy, and the value function of the Critic judges the update strategy, while the Critic updates the value function in a single step through the TD error. Graph neural networks can accurately and efficiently express the relationship between entities and the characteristics of entities and have been applied in many fields. By combining the local structure of the graph and the characteristics of neighboring nodes, it performs well in the field of multi-agents. The graph attention network captures the weights of neighbor nodes through the attention mechanism, performs weighted summation of neighbor node features, and assigns different weights to different neighbor nodes, so that neighbor nodes that are more relevant to the central node obtain higher weights.
[0003] In summary, the coordinated control in multi-agent reinforcement learning methods usually considers cooperation with other agents in the neighborhood. In order to promote the coordination of traffic signal control, graph neural networks can be used to extract the temporal and spatial influences of neighboring agents on the target agent. However, existing studies usually use these two pieces of information separately and do not fully utilize the temporal and spatial correlation to learn better strategies. The graph-based attention mechanism can organically combine the historical state with the current state to form a dynamic state, thereby achieving effective integration of temporal and spatial information and making full use of the joint relationship of temporal and spatial information. Summary of the invention
[0004] In view of this, the present invention provides a multi-agent reinforcement learning method based on graph attention mechanism.
[0005] The present invention discloses a multi-agent reinforcement learning method based on a graph attention mechanism, which comprises: Step 1: Extract features from the current state of the agent using a multi-layer perceptron to obtain a feature vector ; Step 2: The state sequence of the current agent's historical time period is subjected to time convolution encoding, pooling, and splicing to obtain the time domain features ; Step 3: Transform the feature vector and time domain characteristics Input graph attention layer to get the result ; Step 4: Transform the feature vector Input to the graph attention layer to get the result ; Step 5: and The concatenated input is fed into a deep Q-learning network to predict the value of each action, and the action with the maximum action value is selected as the current action of the agent.
[0006] Furthermore, the step 1 comprises: Step 11: Define the name of each agent, define the agents as the 1st, 2nd, 3rd, ... Nth agents, collect the state of the i-th agent at the current time t and define it as ; Step 12: Input into the four-layer multi-layer perceptron to extract the feature vector and get ; The step 12 comprises: Step 121: Input to the first layer of perceptron MLP, and use the following formula to get the vector ;
[0007] in, is the weight matrix, is the bias vector, is the ReLU activation function, Features extracted by the first layer of perceptron; Step 122: Input to the second layer perceptron MLP, and use the following formula to get the vector ;
[0008] in, is the weight matrix, is the bias vector, is the ReLU activation function, Features extracted by the second layer perceptron; Step 123: Enter the third layer of perceptron MLP and use the following formula to get ;
[0009] in, is the weight matrix, is the bias vector, is the ReLU activation function, Features extracted by the third layer perceptron; Step 124: Input to the fourth layer perceptron MLP, and use the following formula to get ;
[0010] in, is the weight matrix, is the bias vector, is the ReLU activation function, is the feature vector.
[0011] Furthermore, the step 2 comprises: Step 21: The state sequence of the agent corresponding to the time interval T backward for the ith agent is ;Will The tensor sequence is obtained by encoding and activating the temporal convolutional network ; Step 22: Tensor Sequence After global average pooling and global maximum pooling, the tensor sequence is obtained and , the tensor sequence and Splicing, after splicing, is encoded through the fully connected layer to obtain the time domain features .
[0012] Furthermore, the step 21 comprises: The state sequence The tensor sequence is obtained by encoding and activating the temporal convolutional network , TCN is a layer of temporal convolutional network, is the ReLU activation function; The tensor sequence The tensor sequence is obtained by encoding and activating the temporal convolutional network .
[0013] Further, the step 22 comprises: For a sequence of tensors Perform global average pooling and global maximum pooling to obtain the following formula:
[0014] in, is the global maximum pooling, is global average pooling; The processing results and Splicing, that is:
[0015] in, Indicates splicing processing; Use the fully connected layer to process the results Encoding to obtain time domain features ,Right now:
[0016] Among them, Dense represents the fully connected layer.
[0017] Furthermore, the step 3 comprises: Step 31: Time Domain Features As the key words in the three elements of the attention mechanism of the graph attention layer, As a query method, calculate the zoom point ; Step 32: Get the attention score through the softmax function ; Step 33: Use multi-head attention mechanism and execute in parallel The attention function of different linear mappings is used, and the neighborhood influences of all different versions are summed and averaged to finally obtain the spatiotemporal features. .
[0018] Further, in step 31, the scaling point is calculated by the following formula: :
[0019] in, Represents the size of hidden neurons in the attention mechanism; In step 32, the attention score is obtained by the following formula :
[0020] Among them, softmax is the activation function, Representing an Agent Neighboring agents; In step 33, the spatiotemporal features are obtained by the following formula: :
[0021] in, is the number of attention functions for different linear mappings, is the attention function, and m is a positive integer.
[0022] Further, step 4 comprises: Step 41: As a query method, calculate the zoom point : Step 42: Get the attention score through the softmax function : Step 43: Use multi-head attention mechanism and execute in parallel The attention function of different linear mappings is used, and the neighborhood influences of all different versions are summed and averaged to finally obtain the spatiotemporal features. .
[0023] Further, in step 41, the scaling point is calculated by the following formula: :
[0024] in, Represents the size of hidden neurons in the attention mechanism; In step 42, the attention score is obtained by the following formula :
[0025] in, Representing an Agent Neighboring agents; In step 43, the spatiotemporal features are obtained by the following formula: :
[0026] in, is the number of attention functions for different linear mappings, is the attention function, and m is a positive integer.
[0027] Furthermore, the step 5 comprises: Step 51: and Splicing, that is:
[0028] in, is the splicing feature, Indicates splicing processing; Step 52: Input into a Q-learning network composed of a deep multi-layer perceptron network to predict the value of each action ;
[0029] in, is the state of the i-th agent at the current time t, represents the fully connected layer; Step 53: From the value of the action Select the largest value and record the action number of that value , represents the largest action value among the action values, The action number representing the value of this action.
[0030] Due to the adoption of the above technical solution, the present invention has the following advantages: the present invention can greatly improve the decision-making accuracy of multi-agent reinforcement learning in terms of time and space, thereby improving the control efficiency of traffic lights. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0032] Figure 1 Schematic diagram of a time domain convolutional network according to an embodiment of the present invention; Figure 2 A flowchart of a multi-agent reinforcement learning method based on a graph attention mechanism according to an embodiment of the present invention. DETAILED DESCRIPTION
[0033] The present invention is further described in conjunction with the accompanying drawings and embodiments, and the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present invention.
[0034] See also Figure 2 , the present invention provides an embodiment of a multi-agent reinforcement learning method based on a graph attention mechanism, which comprises the following steps; S1. Define the name of each agent, define the agents as the 1st, 2nd, 3rd…Nth agents, collect the state of each agent at each moment and define it as .
[0035] S2. Collect the state sequence of an agent. Define the time period that an agent needs to collect as T. Go back a certain time interval for the agent to obtain the state sequence of the agent: .
[0036] S3, collect the state of the i-th agent at time t Input into the four-layer multi-layer perceptron to extract the feature vector .
[0037] S4, the state sequence After encoding and activation by two layers of time convolutional networks, the tensor sequence is obtained , and then transformed into a tensor sequence through global average pooling and global maximum pooling , , concatenate the two tensors and encode the time domain features through the fully connected layer .
[0038] S5. The feature vector and time domain characteristics Input to the graph attention layer The spatiotemporal characteristics of the results are obtained .
[0039] S6. The feature vector Input to the graph attention layer The results are obtained .
[0040] S7. and Concatenate and input into a deep Q-learning network to predict the value of each action, and then select the action with the maximum action value as the result.
[0041] S3, collect the state of the i-th agent at time t Input into the four-layer multi-layer perceptron to extract the feature vector Specifically: S3.1: Input to the first layer of perceptron MLP, and use the following formula to get the vector .
[0042]
[0043] , are the weight matrix and bias vector, is the ReLU activation function, Features extracted by the first layer of perceptron.
[0044] S3.2: Input to the second layer perceptron MLP, and use the following formula to get the vector , the formula is as follows;
[0045] is the weight matrix, is the bias vector, is the ReLU activation function, Features extracted by the second layer perceptron.
[0046] S3.3: Enter the third layer of perceptron MLP and use the following formula to get , the calculation formula is as follows.
[0047]
[0048] is the weight matrix, is the bias vector, is the ReLU activation function, Features extracted by the third layer perceptron.
[0049] S3.4: Input to the fourth layer perceptron MLP, and use the following formula to get , the calculation formula is as follows.
[0050]
[0051] , are the weight matrix and bias vector, is the ReLU activation function, is the feature vector.
[0052] S4, the state sequence After encoding and activation by two layers of time convolutional networks, the tensor sequence is obtained , and then transformed into a tensor sequence through global average pooling and global maximum pooling , , concatenate the two tensors and encode the time domain features through the fully connected layer Specifically: S4.1: Sequence the state After a layer of temporal convolutional network, the tensor sequence is encoded and activated , the structure of the temporal convolutional neural network is as follows Figure 1 As shown in FIG. 1 , there is a residual block and four hidden layers. The expansion coefficients corresponding to the four hidden layers are exponential powers of 2, which are 1, 2, 4, and 8 respectively.
[0053]
[0054] TCN is a layer of temporal convolutional network. is the ReLU activation function.
[0055] S4.2: Convert tensor sequences Then pass through a layer of time convolutional network for encoding and activation to obtain a tensor sequence , the structure of the temporal convolutional neural network is as follows Figure 1 As shown in FIG. 1 , there is a residual block and four hidden layers. The expansion coefficients corresponding to the four hidden layers are exponential powers of 2, which are 1, 2, 4, and 8 respectively.
[0056]
[0057] TCN is a layer of temporal convolutional network. is the ReLU activation function.
[0058] S4.3: Tensor Sequence Perform global average pooling and global maximum pooling for processing:
[0059] is the global maximum pooling, is global average pooling.
[0060] S4.4: Process the results and Splicing.
[0061]
[0062] Indicates splicing processing.
[0063] S4.5: Process the results Use the fully connected layer to encode it to obtain the time domain features .
[0064]
[0065] Dense represents a fully connected layer.
[0066] S5. The feature vector and time domain characteristics Input to the graph attention layer The spatiotemporal characteristics of the results are obtained Specifically: S5.1: Time Domain Information As the key word in the three elements of the attention mechanism of the graph attention layer, the implicit representation of the current state mapping As a query method, calculate the zoom point :
[0067] in, Represents the size of hidden neurons in the attention mechanism.
[0068] S5.2: Get the attention score through the softmax function .
[0069]
[0070] Among them, softmax is the activation function, Representing an Agent neighboring agents.
[0071] S5.3: Use multi-head attention mechanism and execute in parallel The attention function of different linear mappings is used, and the neighborhood influences of all different versions are summed and averaged to finally obtain the spatiotemporal features. .
[0072] in, is the number of attention functions for different linear mappings, is the attention function, and m is a positive integer.
[0073] S6. The feature vector Input to the graph attention layer The results are obtained Specifically: S6.1: Implicit representation mapped to current state As a query method, calculate the zoom point :
[0074] in, Represents the size of hidden neurons in the attention mechanism.
[0075] S6.2: Get the attention score through the softmax function .
[0076]
[0077] S6.3: Use multi-head attention mechanism and execute in parallel The attention function of different linear mappings is used, and the neighborhood influences of all different versions are summed and averaged to finally obtain the spatiotemporal features. . S7. and The concatenation is input into a Q-learning network composed of a deep fully connected layer network to predict the value of each action, and then the action with the maximum action value is selected as the result. Specifically: S7.1: and Splicing.
[0078]
[0079] Indicates splicing processing.
[0080] S7.2: Results Input into a Q-learning network composed of a deep multi-layer perceptron network to predict the value of each action .
[0081] Represents a fully connected layer.
[0082] S7.3: Select the largest value from the action values and record the action number of the action value.
[0083]
[0084] in, represents the largest action value among the action values, The action number representing the value of this action.
[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A multi-agent reinforcement learning method based on graph attention mechanism, characterized in that: include: Step 1: Extract features from the current state of the agent using a multi-layer perceptron. Get the feature vector ; Step 2: The state sequence of the current agent's historical time period is subjected to time convolution encoding, pooling, and splicing to obtain the time domain features ; Step 3: Transform the feature vector and time domain characteristics Input graph attention layer to get the result ; Step 4: Transform the feature vector Input to the graph attention layer to get the result ; Step 5: and The concatenated input is fed into a deep Q-learning network to predict the value of each action, and the action with the maximum action value is selected as the current action of the agent.
2. The method according to claim 1, characterized in that: The step 1 comprises: Step 11: Define the name of each agent, define the agents as the 1st, 2nd, 3rd, ... Nth agents, collect the state of the i-th agent at the current time t and define it as ; Step 12: Input into the four-layer multi-layer perceptron to extract the feature vector and get the feature vector ; The step 12 comprises: Step 121: Input to the first layer of perceptron MLP, and use the following formula to get the vector ; in, is the weight matrix, is the bias vector, is the ReLU activation function, Features extracted by the first layer of perceptron; Step 122: Input to the second layer perceptron MLP, and use the following formula to get the vector ; in, is the weight matrix, is the bias vector, is the ReLU activation function, Features extracted by the second layer perceptron; Step 123: Enter the third layer of perceptron MLP and use the following formula to get ; in, is the weight matrix, is the bias vector, is the ReLU activation function, Features extracted by the third layer perceptron; Step 124: Input to the fourth layer perceptron MLP, and use the following formula to get ; in, is the weight matrix, is the bias vector, is the ReLU activation function, is the feature vector.
3. The method according to claim 1, characterized in that: The step 2 comprises: Step 21: The state sequence of the agent corresponding to the time interval T backward for the ith agent is ;Will The tensor sequence is obtained by encoding and activating the temporal convolutional network ; Step 22: Tensor Sequence After global average pooling and global maximum pooling, the tensor sequence is obtained and , the tensor sequence and Splicing, after splicing, is encoded through the fully connected layer to obtain the time domain features .
4. The method according to claim 3, characterized in that The step 21 comprises: The state sequence The tensor sequence is obtained by encoding and activating the temporal convolutional network , TCN is a layer of temporal convolutional network, is the ReLU activation function; The tensor sequence The tensor sequence is obtained by encoding and activating the temporal convolutional network .
5. The method according to claim 3 or 4, characterized in that: The step 22 comprises: For a sequence of tensors Perform global average pooling and global maximum pooling to obtain the following formula: in, is the global maximum pooling, is global average pooling; The processing results and Splicing, that is: in, Indicates splicing processing; Use the fully connected layer to process the results Encoding to obtain time domain features ,Right now: Among them, Dense represents the fully connected layer.
6. The method according to claim 1, characterized in that The step 3 comprises: Step 31: Time Domain Features As the key words in the three elements of the attention mechanism of the graph attention layer, As a query method, calculate the zoom point ; Step 32: Get the attention score through the softmax function ; Step 33: Use multi-head attention mechanism and execute in parallel The attention function of different linear mappings is used, and the neighborhood influences of all different versions are summed and averaged to finally obtain the spatiotemporal features. .
7. The method according to claim 6, characterized in that In step 31, the scaling point is calculated by the following formula : in, Represents the size of hidden neurons in the attention mechanism; In step 32, the attention score is obtained by the following formula : Among them, softmax is the activation function, Representing an Agent Neighboring agents; In step 33, the spatiotemporal features are obtained by the following formula: : in, is the number of attention functions for different linear mappings, is the attention function, and m is a positive integer.
8. The method according to claim 1, characterized in that The step 4 comprises: Step 41: As a query method, calculate the zoom point : Step 42: Get the attention score through the softmax function : Step 43: Use multi-head attention mechanism and execute in parallel The attention function of different linear mappings is used, and the neighborhood influences of all different versions are summed and averaged to finally obtain the spatiotemporal features. .
9. The method according to claim 8, characterized in that In step 41, the scaling point is calculated by the following formula : in, Represents the size of hidden neurons in the attention mechanism; In step 42, the attention score is obtained by the following formula : in, Representing an Agent Neighboring agents; In step 43, the spatiotemporal features are obtained by the following formula: : in, is the number of attention functions for different linear mappings, is the attention function, and m is a positive integer.
10. The method according to claim 1, characterized in that The step 5 comprises: Step 51: and Splicing, that is: in, is the splicing feature, Indicates splicing processing; Step 52: Input into a Q-learning network composed of a deep multi-layer perceptron network to predict the value of each action ; in, is the state of the i-th agent at the current time t, represents the fully connected layer; Step 53: From the value of the action Select the largest value and record the action number of that value , represents the largest action value among the action values, The action number representing the value of this action.
