Multi-agent reinforcement learning method and device, electronic device, and storage medium
By generating hypergraph structures and calculating message weights, and fusing communication information of heterogeneous agents, the problem of low information exchange efficiency in heterogeneous agent collaboration is solved, and more efficient information interaction and action guidance is achieved.
Patent Information
- Application Number
- CN202410189439.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-20
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-02-20
AI Technical Summary
The prior art fails to fully consider the dynamic nature and connection correlation of isomeric agents, making it difficult for isomeric agents to learn useful collaboration strategies and inefficient information exchange.
By generating a hypergraph structure, the message weight and correlation between multiple agents are calculated, and communication information is fused to guide the agent's actions, achieving more complex information interaction and dependency capture.
It improves the efficiency of information exchange between heterogeneous agents, captures potential high-order interactions and dependencies, overcomes functional differences, and achieves more accurate information fusion and action guidance.
Smart Images

Figure CN118052272B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure belongs to the field of hypergraph representation learning technology, and more specifically, relates to a multi-agent reinforcement learning method and device, electronic equipment, and storage medium. Background Art
[0002] Homogeneous agents refer to agents with the same functions and capabilities, which perform similar tasks and behaviors in collaborative tasks. Heterogeneous agents refer to agents with different functions, capabilities or characteristics in collaborative tasks. Different agents may have different sensors, actuators, or different knowledge, experience and decision-making strategies. In homogeneous agent collaboration, information exchange can be more simplified and unified because the agents have similar functions and capabilities. In heterogeneous agent collaboration, due to the functional differences between agents, information exchange may require more complex processing, and different types of agents may use different communication methods, transmission protocols or encoding rules. In the current existing technology, the dynamic nature of communication and the correlation between agent connections have not been fully considered. Message understanding and fusion between heterogeneous agents is still a problem to be solved, which makes it difficult for agents to learn useful collaborative strategies. Therefore, a multi-agent reinforcement learning method and device, electronic device, and storage medium are urgently needed. Summary of the invention
[0003] The purpose of the present disclosure is to provide a multi-agent reinforcement learning method and device, electronic device, and storage medium to improve the processing and integration capabilities of heterogeneous multi-agents on heterogeneous information.
[0004] A first aspect of the embodiments of the present disclosure provides a multi-agent reinforcement learning method, comprising:
[0005] Based on the observation information of each agent in the multi-agent system, multiple hyperedges are generated. Each hyperedge connects multiple agents, and the multiple hyperedges form a hypergraph structure. The hypergraph structure is a communication structure corresponding to the multi-agent system.
[0006] The message weights between multiple agents are calculated based on the observation information of each agent.
[0007] The correlation degree corresponding to the multi-agent system is generated based on the communication structure and the message weight.
[0008] The communication information of each intelligent agent is fused based on the correlation degree to obtain fused communication information corresponding to each intelligent agent. The fused communication information is used to guide the action of the corresponding intelligent agent.
[0009] A second aspect of the embodiments of the present disclosure provides a multi-agent reinforcement learning device, comprising:
[0010] A hypergraph generation module, for generating a plurality of hyperedges based on observation information of each agent in a multi-agent system; each hyperedge connects a plurality of agents, and the plurality of hyperedges form a hypergraph structure; the hypergraph structure is a communication structure corresponding to the multi-agent system;
[0011] The message weight calculation module is used to calculate the message weights between multiple agents based on the observation information of each agent;
[0012] A correlation generation module, used to generate a correlation corresponding to the multi-agent system based on the communication structure and the message weight;
[0013] The information fusion module is used to fuse the communication information of each intelligent agent based on the correlation degree to obtain the fused communication information corresponding to each intelligent agent; the fused communication information is used to guide the action of the corresponding intelligent agent.
[0014] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned multi-agent reinforcement learning method when executing the computer program.
[0015] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned multi-agent reinforcement learning method are implemented.
[0016] The multi-agent reinforcement learning method and device, electronic device, and storage medium provided by the embodiments of the present disclosure have the following beneficial effects:
[0017] First, the multi-agent reinforcement learning method provided in this embodiment can calculate the correlation between multi-agents based on the generated hypergraph structure, and obtain the message weights between multi-agents, thereby capturing the potential, more complex, and higher-order interactions and dependencies between multi-agents.
[0018] Second, the multi-agent reinforcement learning method provided in this embodiment can overcome the functional differences between multiple heterogeneous agents. Through the reinforcement learning method, multiple heterogeneous information from different agents are communicated and integrated to obtain the final integrated communication information. The corresponding action of the next step is executed based on the integrated communication information, thereby improving the efficiency of information exchange between multiple agents. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0020] Figure 1 A schematic diagram of a multi-agent reinforcement learning method according to an embodiment of the present disclosure;
[0021] Figure 2 A hyperedge generation flow chart provided for an embodiment of the present disclosure;
[0022] Figure 3 A diagram of the theory of mind network model architecture provided for an embodiment of the present disclosure;
[0023] Figure 4 A schematic diagram of a KNN algorithm provided in one embodiment of the present disclosure;
[0024] Figure 5 A schematic diagram of a comprehensive transportation network provided in an embodiment of the present disclosure;
[0025] Figure 6 A structural block diagram of a multi-agent reinforcement learning device provided by an embodiment of the present disclosure;
[0026] Figure 7 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0027] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.
[0028] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.
[0029] Please refer to Figure 1 , Figure 1 A schematic diagram of a multi-agent reinforcement learning method provided in an embodiment of the present disclosure, the method comprising:
[0030] S101: Generate multiple hyperedges based on the observation information of each agent in the multi-agent system. Each hyperedge connects multiple agents, and multiple hyperedges form a hypergraph structure. The hypergraph structure is the communication structure corresponding to the multi-agent system.
[0031] In this embodiment, agent is a very important concept in the field of artificial intelligence. Any independent entity that can think and interact with the environment can be abstracted as an agent. Multi-agent is a collection of multiple agents that can express the structure, function and behavior characteristics of the overall system through communication, cooperation, mutual understanding, coordination, scheduling, management and control between each agent. Collaboration between multiple agents can be divided into homogeneous agent collaboration and heterogeneous agent collaboration. The difference between the two is: (1) Functional difference: Heterogeneous agents refer to agents with different functions, capabilities or characteristics in collaborative tasks. They may have different sensors, actuators, or have different knowledge, experience and decision-making strategies. Homogeneous agents refer to agents with the same functions and capabilities, which perform similar tasks and behaviors in collaborative tasks. (2) Information exchange method: In heterogeneous agent collaboration, due to the functional differences between agents, information exchange may require more complex processing. Different types of agents may use different communication methods, transmission protocols or encoding rules. In homogeneous agent collaboration, because agents have similar functions and capabilities, information exchange can be more simplified and unified.
[0032] In current research, most of the research on multi-agent reinforcement learning focuses on homogeneous agent collaboration, but less attention is paid to heterogeneous agent collaboration. The reasons are: (1) Attention-based communication methods usually tend to assign attention weights to agents at a specific time point, but do not fully consider the dynamic nature of communication and the correlation between agent connections. In some multi-agent systems, especially in situations where flexibility is required to adapt to dynamic environments and changes in relationships between agents, this static attention allocation may not be sufficient to achieve optimal communication collaboration. (2) Attention mechanisms usually calculate attention weights based on the current state or features, but they may ignore the impact of communication history on future communications. The dynamic nature of communication includes the historical patterns of communication between agents and their changes over time. Failure to consider communication history may lead to ignoring the dynamic nature of communication. Therefore, based on the above two points, the understanding and fusion of messages between multiple heterogeneous agents in existing heterogeneous agent collaboration remains an unresolved problem.
[0033] In this embodiment, each agent can obtain observation information about itself and judgment of observation information of each other agent in the environment through interaction with the environment. The observation information can also be combined with the hidden state information of each agent. Hidden state information refers to the internal state of the agent during reinforcement learning and active reasoning. Hidden state information may also include the internal representation, memory or attention of the agent. In the process of reinforcement learning, the agent learns which actions to perform in different states to obtain the maximum reward or return by interacting with the environment. The hidden state information can help the agent better understand, adapt to and utilize the environment.
[0034] Based on the observation information of each agent in the multi-agent system, multiple hyperedges can be generated. Hyperedges can be regarded as a higher-level structure connecting multiple states, actions or rewards, which can represent complex interactions and dependencies between multiple states, actions or rewards. Each hyperedge can connect multiple agents, and multiple hyperedges can form a hypergraph structure. A hypergraph is a complex data structure used to describe the interaction between agents and the environment. Hypergraphs are different from traditional graphs. They can express richer and more complex interaction patterns. In particular, for multi-agent systems involving multiple interactions and dependencies, the application of hypergraphs is very important. Hypergraphs are composed of nodes and hyperedges, where nodes usually represent the states, actions or rewards of multiple agents, and hyperedges represent the relationship between them. This hypergraph structure is the communication structure corresponding to the multi-agent system.
[0035] S102: Calculate the message weights between multiple agents based on the observation information of each agent.
[0036] In this embodiment, the message weights between multiple agents can be calculated based on the observation information of each agent. In the multi-agent reinforcement learning process, the message weight refers to the degree of influence of the message on the decision and behavior of the agent. Messages with higher weight values can usually provide more information about the state of the environment and the effect of the action, thereby helping the agent to better understand the environment and make better decisions.
[0037] The specific calculation process of the message weight between multiple agents is as follows: Assuming there are i agents, according to the connection relationship of the hyperedges in the hypergraph structure, it is determined that the associated agents of the i-th agent include: the first agent, the second agent, and the third agent. The message weights between the i-th agent and the first agent, the second agent, and the third agent are calculated respectively, and at the same time, it is determined whether there is repeated calculation between every two agents to avoid duplicate data.
[0038] S103: Generate the corresponding correlation degree of the multi-agent system based on the communication structure and message weight.
[0039] In this embodiment, based on the communication structure and the calculated message weights, the correlation corresponding to the multi-agent system can be generated. The communication structure is a matrix with non-zero values. This embodiment provides a reference example. For example, assuming that there are 4 agents, numbered 0, 1, 2, and 3, 0 and 1 are connected, 1 and 2 are connected, and 0 and 3 are connected, so the communication structure is:
[0040] [[1, 1, 0, 1],
[0041] [1, 1, 1, 0],
[0042] [0, 1, 1, 0],
[0043] [1, 0, 0, 1]]
[0044] The message weight is:
[0045] [[0.3, 0.2, 0.1, 0.4],
[0046] [0.2, 0.2, 0.1, 0.5],
[0047] [0.4, 0.2, 0.2, 0.2],
[0048] [0.2, 0.4, 0.2, 0.2]]
[0049] The corresponding correlation degree of the multi-agent system generated according to the communication structure and message weight is:
[0050] [[0.3, 0.2, 0, 0.4],
[0051] [0.2, 0.2, 0.1, 0],
[0052] [0, 0.2, 0.2, 0],
[0053] [0.2, 0, 0, 0.2]]
[0054] S104: Based on the correlation, the communication information of each intelligent agent is integrated to obtain the integrated communication information corresponding to each intelligent agent. The integrated communication information is used to guide the action of the corresponding intelligent agent.
[0055] In this embodiment, the communication information of each agent is fused according to the calculated correlation degree, so as to obtain the fused communication information corresponding to each agent. The fused communication information is the basis for guiding the corresponding agent to perform the next action.
[0056] It can be concluded from the above that the multi-agent reinforcement learning method provided in this embodiment can fuse heterogeneous information between multi-agents according to the communication structure and message weight, more accurately capture the potential information interaction between multi-agents, and obtain the fused communication message corresponding to each agent, thereby guiding the agent to take the next action. This method effectively improves the reinforcement learning process of multi-agents for heterogeneous information and improves the efficiency of information interaction between multi-agents.
[0057] In one embodiment of the present disclosure, multiple hyperedges are generated based on observation information of each agent in a multi-agent system, including:
[0058] The multi-agents are classified according to their attributes to obtain corresponding multiple classification categories. The multi-agents of the same classification category are divided into the same hyperedge to obtain multiple first hyperedges.
[0059] The multi-agents are clustered based on their observation information, and the multi-agents belonging to the same classification are divided into the same hyperedge to obtain multiple second hyperedges.
[0060] The intention information of each agent is predicted based on the observation information and historical actions of the multi-agents. The multi-agents are clustered based on their intention information, and the multi-agents belonging to the same classification are divided into the same hyperedge to obtain multiple third hyperedges.
[0061] The first hyperedge, the second hyperedge and the third hyperedge constitute a plurality of hyperedges.
[0062] In this embodiment, Figure 2 A hyperedge generation flow chart provided in an embodiment of the present disclosure is as follows: Figure 2 As shown in Figure 1, multiple hyperedges are generated based on the observation information of each agent in the multi-agent system, which can be described in detail as follows:
[0063] The multiple agents are classified according to their attributes to obtain multiple different classification categories, and the agents belonging to the same category are divided into the same hyperedge, thereby obtaining multiple first hyperedges.
[0064] The multi-agents are clustered according to their observation information, and the multi-agents belonging to the same classification are divided into the same hyperedge, thereby obtaining multiple second hyperedges. The number of the second hyperedges is the same as the number of the multi-agents.
[0065] Based on the observation information and historical actions of multiple agents, the next intention of each agent is predicted to obtain the intention information of each agent. Based on the intention information, the agents are clustered and agents belonging to the same category are divided into the same hyperedge to obtain multiple third hyperedges. The number of the third hyperedges is also the same as the number of agents. Among them, the historical action refers to the action that the multi-agent has completed before.
[0066] A plurality of first hyperedges, a second hyperedge, and a third hyperedge together constitute a plurality of hyperedges, and the number of the plurality of hyperedges is the sum of the number of the first hyperedges, the second hyperedges, and the third hyperedges. The corresponding hypergraph structure can be obtained by weighting the first hyperedge, the second hyperedge, and the third hyperedge. This embodiment provides a reference example, for example, the weights corresponding to the first hyperedge, the second hyperedge, and the third hyperedge are: 1, 1, 3, respectively, which are converted into weight coefficients: (0.2, 0.2, 0.6).
[0067] From the above, it can be concluded that this embodiment can generate the first hyperedge, the second hyperedge and the third hyperedge respectively according to the observation information and historical actions of the multiple agents, and the three together constitute multiple hyperedges. By generating multiple hyperedges, the potential information interaction between the multiple agents can be analyzed and the relationship between the multiple agents can be clarified.
[0068] In one embodiment of the present disclosure, the intention information of each agent is predicted based on the observation information and historical actions of multiple agents, including:
[0069] The observation information and historical actions of multiple agents are input into the theory of mind network model to obtain the intention information of each agent.
[0070] In this embodiment, the intention information of each agent is predicted based on the observation information and historical actions of multiple agents, which can be described in detail as follows:
[0071] The observation information and historical actions of multiple agents are input into the theory of mind network model, and the intention information of each agent is output through detection and recognition by the network model. The theory of mind network model broadly refers to the ability of humans to represent the mental state of others, including their desires, beliefs, and intentions. The theory of mind network provides a powerful prior model for the future behavior learning of multiple agents. Using a small amount of behavioral observations, it can make richer predictions about the characteristics and mental states of multiple agents. Figure 3 The architecture diagram of the theory of mind network model provided by an embodiment of the present disclosure is as follows: Figure 3 As shown, the theory of mind network model collects the past behavior trajectory (observations and actions) of each agent, encodes its behavior trajectory to obtain the agent's behavioral characteristics, encodes the agent's current behavior trajectory into the agent's mind, and combines it with the agent's current observations to output the agent's intention information.
[0072] It can be concluded from the above that the theory of mind network model provided in this embodiment can automatically predict the future intentions and actions of the intelligent agent based on the intelligent agent's past historical behavior trajectory, current behavior trajectory and current observation information, thereby improving the intelligent agent's decision-making accuracy.
[0073] In one embodiment of the present disclosure, multiple agents are clustered based on their observation information, and agents belonging to the same category are divided into the same hyperedge to obtain multiple second hyperedges, including:
[0074] Input the observation information into the preset network model to generate corresponding feature points.
[0075] For each feature point, the nearest N feature points are selected for clustering according to the distance, and multiple second hyperedges are obtained. N is a hyperparameter.
[0076] In this embodiment, multiple agents are clustered based on their observation information, and agents belonging to the same category are divided into the same hyperedge to obtain multiple second hyperedges, which can be described in detail as follows:
[0077] Input the observation information of multiple agents into the preset network model to generate corresponding feature points, where each feature point corresponds to an agent. Each feature point selects N nearest feature points from the surrounding area according to the distance to cluster, thereby obtaining multiple second hyperedges, where N represents a hyperparameter that can be customized.
[0078] The observation information of multiple agents can be clustered by the KNN (K Nearest Neighbors) algorithm model. The KNN algorithm is a basic supervised learning algorithm, commonly used for classification and regression problems. The basic idea of the algorithm is to classify or predict new samples by finding the N training samples that are most similar to the new samples (i.e., the nearest neighbors). In the KNN algorithm, the training samples of the N nearest neighbors are selected by calculating the distance between the new sample and the training sample (usually using the Euclidean distance or Manhattan distance).
[0079] The application idea of the algorithm in this embodiment is: for any observation information of M agents, each corresponds to a point in the feature space, so the M agents correspond to M feature points in the feature space, and each agent selects the N neighbor feature points closest to itself to form a hyperedge, and the M agents generate a hyperedge respectively, that is, M hyperedges are generated. Among them, N is a hyperparameter that can be customized. The observation information of multiple agents is input into the KNN algorithm model to generate multiple corresponding feature points. Each feature point selects the N feature points closest to it for clustering, so each feature point obtains a corresponding hyperedge, thereby obtaining multiple second hyperedges.
[0080] This embodiment provides a reference example such as: Figure 4 A schematic diagram of a KNN algorithm provided in an embodiment of the present disclosure is shown in FIG. Figure 4As shown in the figure, the entire coordinate system represents a feature space, in which the square is the main body, that is, the agent, and each other figure represents another agent in the feature space. When N=3, the circle contains three other figures centered on the square, which are the three agents closest to the main body in the feature space. Connecting the agents in the circle will result in a hyperedge. Each agent executes the KNN algorithm in turn, resulting in multiple second hyperedges.
[0081] It can be concluded from the above that the hyperedge generation method provided in this embodiment can quickly form multiple hyperedges and improve the efficiency of multi-agent reinforcement learning.
[0082] In one embodiment of the present disclosure, the message weights between multiple agents are calculated based on the observation information of each agent, including:
[0083] pass Calculate the message weights between multiple agents.
[0084] in, represents the weight. Q and K are obtained by linear transformation of the observation information of each agent. k Indicates the dimension size of K.
[0085] In this embodiment, the message weights between multiple agents are calculated based on the observation information of each agent, which can be described in detail as follows:
[0086] Can be Calculate the message weights between multiple agents. represents the weight. Q and K are obtained by linear transformation of the observation information of each agent. k Represents the dimensional size of K. The method calculates the message weights between multiple agents based on the attention mechanism. The attention mechanism is composed of machine learning and deep learning models, which are used to focus the model's attention on information related to the current task. In a typical attention mechanism, the model calculates the attention weight distribution to determine the importance of different parts of the input for the current task. According to the weight distribution, the model performs a weighted combination of the representations of different parts to find the focus. It can be concluded from the above that this embodiment introduces the attention mechanism, so that multiple agents have better performance when processing long sequences and large data sets, and enhance their ability to understand the input data.
[0087] In one embodiment of the present disclosure, the communication structure is a communication matrix. Generating the correlation degree corresponding to the multi-agent system based on the communication structure and the message weight includes:
[0088] The target position in the communication matrix is determined based on each hyperedge in the communication structure.
[0089] The numerical value of the target position in the communication matrix is determined based on the message weight.
[0090] In this embodiment, the communication structure generated based on the communication structure and the message weight is essentially a communication matrix. The correlation degree corresponding to the multi-agent system is generated based on the communication structure and the message weight, which can be described in detail as follows:
[0091] According to each hyperedge in the communication structure, the target position in the communication matrix can be determined, where the target position can be the position of the xth and yth columns, which means that there is a hyperedge between the agent corresponding to the xth point position and the agent corresponding to the yth point position, that is, there is a connection between x and y. The value of the target position is used to represent the message weight corresponding to the hyperedge, so the corresponding value of the target position in the communication matrix can be determined according to the calculated message weight, and the calculation based on the communication structure and the message weight is performed to obtain a complete association matrix.
[0092] This embodiment provides a reference example. For example, assume that there are 4 agents, including 3 type A agents (numbered 0, 1, and 2 respectively) and 1 type B agent (numbered 3). The weights of the first hyperedge, the second hyperedge, and the third hyperedge are 1, 1, and 3 respectively, and the corresponding weight coefficients are (0.2, 0.2, and 0.6). The first hyperedge, the second hyperedge, and the third hyperedge are obtained as follows:
[0093] First hyperedge: [(0, 1, 2), (1, 0, 2), (2, 0, 1)]
[0094] Second hyperedge: [(0, 1, 2), (1, 3, 0), (2, 3, 0), (3, 1, 2)]
[0095] The third hyperedge: [(0, 2, 3), (1, 3, 2), (2, 0, 1), (3, 1, 2)]
[0096] Based on the observation information of the four agents, multiple hyperedges can be generated: [(0, 1, 2), (1, 0, 2), (2, 0, 1), (0, 1, 2), (1, 3, 0), (2, 3, 0), (3, 1, 2), (0, 2, 3), (1, 3, 2), (2, 0, 1), (3, 1, 2)], and multiple hyperedges form a hypergraph. The communication structure is:
[0097] [[0, 2, 5, 3],
[0098] [2, 0, 4, 4],
[0099] [5, 4, 0, 1],
[0100] [0, 4, 4, 0]]
[0101] Set the calculated message weight to:
[0102] [[0.0001, 0.0001, 0.0001, 0.0001],
[0103] [0.0001, 0.0001, 0.0001, 0.0001],
[0104] [0.0001, 0.0001, 0.0001, 0.0001],
[0105] [0.0001, 0.0001, 0.0001, 0.0001]]
[0106] According to the communication structure and message weight, the two matrices are multiplied to obtain:
[0107] [[0.0000, 0.0002, 0.0005, 0.0003],
[0108] [0.0002, 0.0000, 0.0004, 0.0004],
[0109] [0.0005, 0.0004, 0.0000, 0.0001],
[0110] [0.0000, 0.0004, 0.0004, 0.0000]]
[0111] By normalizing the matrix, the correlation degree corresponding to the multi-agent system is obtained as follows:
[0112] [[0.0000, 0.3333, 0.3334, 0.3333],
[0113] [0.3333, 0.0000, 0.3334, 0.3334],
[0114] [0.3334, 0.3334, 0.0000, 0.3332],
[0115] [0.0000, 0.5000, 0.5000, 0.0000]].
[0116] It can be concluded from the above that this embodiment can fuse heterogeneous information among multiple agents by generating correlation, thereby clarifying the next execution action and achieving coordination of heterogeneous agent collaboration.
[0117] In one embodiment of the present disclosure, the communication information of each agent is fused based on the correlation to obtain the fused communication information corresponding to each agent, including:
[0118] By O i =∑kj *o j Calculate the fused communication information corresponding to each agent.
[0119] Among them, O i represents the fusion communication information. i represents the number of agents. k j Represents the value of the jth target position in the i-th row or i-th column of the communication matrix. j Represents the communication information of the agent corresponding to the jth target position in the i-th row or i-th column of the communication matrix.
[0120] In this embodiment, the communication information of each agent is fused based on the correlation to obtain the fused communication information corresponding to each agent, which can be described in detail as follows:
[0121] By O i =∑k j *o j The fused communication information corresponding to each intelligent agent is calculated, and the fused communication information is the effective information obtained after each intelligent agent fuses multiple heterogeneous messages.
[0122] Among them, O i represents the fusion communication information. i represents the number of agents. k j represents the value of the jth target position in the i-th row or i-th column, O j Represents the communication information of the agent corresponding to the jth target position in the i-th row or i-th column.
[0123] This embodiment provides a reference example. For example, assuming there are 4 agents, the fused communication information of the agent corresponding to the third target position in the 4th row is: O4=k1*O1+k2*O2+k3*O3. The agent can be guided to perform the next action based on the fused communication information.
[0124] From the above, it can be concluded that the multi-agent reinforcement learning method provided in this embodiment can process heterogeneous information to make it easier to understand, and effectively integrate it, ultimately achieving the purpose of guiding multiple agents to perform corresponding actions.
[0125] In one embodiment of the present disclosure, a multi-agent reinforcement learning method is applied to a traffic light control system to achieve a traffic goal of continuous green wave passage.
[0126] In this embodiment, in the traditional traffic light system, a fixed schedule is usually used to control the change of the traffic lights. However, this method cannot adapt to the changes in real-time traffic flow, and it is difficult to coordinate the traffic lights of multiple intersections in a complex traffic network. The use of multi-agent reinforcement learning methods can improve this problem. Each intersection is modeled as an agent, and the traffic lights are autonomously controlled according to the current traffic conditions. Each agent determines the state and duration of the traffic light based on the surrounding traffic flow, traffic congestion and other information. Figure 5 A schematic diagram of a comprehensive transportation network provided by an embodiment of the present disclosure, specifically as follows Figure 5 As shown, traffic light control can be achieved through the following steps:
[0127] Step 1: Environment modeling: Model the intersections, roads, vehicles and other elements in the traffic network to provide the agent with perception and status information.
[0128] Step 2: State representation: Define the state space for each intersection agent, which can include indicators such as traffic flow, waiting time, and road congestion.
[0129] Step 3: Action definition: Each agent selects an executable action based on the state of the environment, such as changing the state of the traffic light, extending or shortening the green light time, etc.
[0130] Step 4: Reward design: Define a reward function to measure the overall performance of the system, such as reducing traffic congestion, improving traffic efficiency, etc.
[0131] Step 5: Reinforcement learning training. Using a heterogeneous multi-agent reinforcement learning algorithm, each intersection agent learns the optimal strategy through interaction with the environment. Each agent selects an action based on the current state and adjusts the strategy through reward signals to gradually optimize the traffic signal control strategy.
[0132] From the above, it can be concluded that in the traffic light control system based on the multi-agent reinforcement learning method, each agent can autonomously control the traffic light according to the current traffic conditions to minimize traffic congestion and waiting time, and achieve the effect of continuous green wave through traffic. This method can more effectively adapt to changes in different time periods and traffic conditions, thereby improving urban traffic efficiency and reducing traffic congestion.
[0133] Corresponding to the multi-agent reinforcement learning method of the above embodiment, Figure 6 This is a block diagram of a multi-agent reinforcement learning device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 6The multi-agent reinforcement learning device 20 includes: a hypergraph generation module 21, a message weight calculation module 22, a correlation generation module 23 and an information fusion module 24.
[0134] The hypergraph generation module 21 is used to generate multiple hyperedges based on the observation information of each agent in the multi-agent system. Each hyperedge connects multiple agents, and multiple hyperedges form a hypergraph structure. The hypergraph structure is a communication structure corresponding to the multi-agent system.
[0135] The message weight calculation module 22 is used to calculate the message weights between multiple agents based on the observation information of each agent.
[0136] The correlation generation module 23 is used to generate the correlation corresponding to the multi-agent system based on the communication structure and the message weight.
[0137] The information fusion module 24 is used to fuse the communication information of each agent based on the correlation degree to obtain the fused communication information corresponding to each agent. The fused communication information is used to guide the actions of the corresponding agent.
[0138] In one embodiment of the present disclosure, the hypergraph generation module 21 is specifically used for:
[0139] The multi-agents are classified according to their attributes to obtain corresponding multiple classification categories. The multi-agents of the same classification category are divided into the same hyperedge to obtain multiple first hyperedges.
[0140] Based on the observation information of multiple agents, multiple agents are clustered, and agents belonging to the same category are divided into the same hyperedge to obtain multiple second hyperedges.
[0141] The intention information of each agent is predicted based on the observation information and historical actions of the multi-agents. The multi-agents are clustered based on their intention information, and the agents belonging to the same category are divided into the same hyperedge to obtain multiple third hyperedges.
[0142] The first hyperedge, the second hyperedge and the third hyperedge constitute a plurality of hyperedges.
[0143] In one embodiment of the present disclosure, the hypergraph generation module 21 is specifically used for:
[0144] The observation information and historical actions of multiple agents are input into the theory of mind network model to obtain the intention information of each agent.
[0145] In one embodiment of the present disclosure, the hypergraph generation module 21 is specifically used for:
[0146] Input the observation information into the preset network model to generate corresponding feature points.
[0147] For each feature point, the nearest N feature points are selected for clustering according to the distance, and multiple second hyperedges are obtained. N is a hyperparameter.
[0148] In one embodiment of the present disclosure, the message weight calculation module 22 is specifically used to:
[0149] The correlation between multiple agents is calculated based on the observation information of each agent, including:
[0150] pass Calculate the message weights between multiple agents.
[0151] in, represents the weight. Q and K are obtained by linear transformation of the observation information of each agent. k Indicates the dimension size of K.
[0152] In one embodiment of the present disclosure, the communication structure is a communication matrix, and the correlation generation module 23 is specifically used for:
[0153] The target position in the communication matrix is determined based on each hyperedge in the communication structure.
[0154] The numerical value of the target position in the communication matrix is determined based on the message weight.
[0155] In one embodiment of the present disclosure, the information fusion module 24 is specifically used to:
[0156] By O i =∑k j *o j Calculate the fused communication information corresponding to each agent.
[0157] Among them, O i represents the fusion communication information. i represents the number of agents. k j represents the value of the jth target position in the i-th row or i-th column of the communication matrix, O j Represents the communication information of the agent corresponding to the jth target position in the i-th row or i-th column of the communication matrix.
[0158] See also Figure 7 , Figure 7 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 7The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules in the above-mentioned device embodiments, such as Figure 6 The functions of modules 21 to 24 are shown.
[0159] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0160] The input device 302 may include a touch panel, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.
[0161] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.
[0162] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the multi-agent reinforcement learning method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.
[0163] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, wherein the program instructions are executed by a processor to implement all or part of the processes in the above-mentioned embodiment method, and the computer program can also be completed by instructing related hardware through a computer program, wherein the computer program can be stored in a computer-readable storage medium, and wherein the computer program can implement the steps of each of the above-mentioned method embodiments when executed by a processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the contents contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signal and telecommunication signal.
[0164] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
[0165] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.
[0166] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0167] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.
[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.
[0169] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0170] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A multi-agent reinforcement learning method, characterized in that: include: Generate multiple hyperedges based on the observation information of each agent in the multi-agent system; Each hyperedge connects multiple agents, and the multiple hyperedges form a hypergraph structure; The hypergraph structure is a communication structure corresponding to the multi-agent system; Calculate the message weights between the multiple agents based on the observation information of each agent; Generate a correlation degree corresponding to the multi-agent system based on the communication structure and the message weight; Based on the correlation, the communication information of each intelligent agent is integrated to obtain the integrated communication information corresponding to each intelligent agent; The fused communication information is used to guide the actions of the corresponding intelligent agent; The generating of multiple hyperedges based on the observation information of each agent in the multi-agent system includes: Classifying the multi-agents according to the attributes to obtain a plurality of corresponding classification categories; dividing the multi-agents of the same classification category into the same hyperedge to obtain a plurality of first hyperedges; Clustering multiple agents based on their observation information, dividing multiple agents belonging to the same category into the same hyperedge, and obtaining multiple second hyperedges; Predict the intention information of each agent based on the observation information and historical actions of the multi-agents; cluster the multi-agents based on their intention information, and divide the multi-agents belonging to the same classification into the same hyperedge to obtain multiple third hyperedges; The first super edge, the second super edge and the third super edge constitute the plurality of super edges; The method of calculating the message weights between multiple agents based on the observation information of each agent includes: pass Calculate the message weights between multiple agents; in, represents weight; Q and K are obtained by linear transformation of the observation information of each agent; d k Indicates the dimension size of K; The communication structure is a communication matrix; the generating of the correlation degree corresponding to the multi-agent system based on the communication structure and the message weight includes: Determining a target location in the communication matrix based on each hyperedge in the communication structure; determining a value of a target position in the communication matrix based on the message weight; The step of fusing the communication information of each agent based on the correlation to obtain the fused communication information corresponding to each agent includes: By O i =∑k j *o j Calculate the fused communication information corresponding to each intelligent agent; Among them, O i represents the fusion communication information; i represents the number of agents; k j represents the value of the jth target position in the i-th row or i-th column of the communication matrix, O j Represents the communication information of the agent corresponding to the jth target position in the i-th row or i-th column of the communication matrix.
2. The multi-agent reinforcement learning method according to claim 1, characterized in that: The method of predicting the intention information of each agent based on the observation information and historical actions of multiple agents includes: The observation information and historical actions of multiple agents are input into the theory of mind network model to obtain the intention information of each agent.
3. The multi-agent reinforcement learning method according to claim 1, characterized in that: The multi-agents are clustered based on the observation information of the multi-agents, and the multi-agents belonging to the same classification are divided into the same hyperedge, so as to obtain multiple second hyperedges, including: Inputting the observation information into a preset network model to generate corresponding feature points; For each feature point, the nearest N feature points are selected for clustering according to the distance to obtain multiple second hyperedges; N is a hyperparameter.
4. A multi-agent reinforcement learning device, characterized in that: include: A hypergraph generation module, used to generate multiple hyperedges based on the observation information of each agent in the multi-agent system; Each hyperedge connects multiple agents, and the multiple hyperedges form a hypergraph structure; the hypergraph structure is a communication structure corresponding to the multi-agent system; The generating of multiple hyperedges based on the observation information of each agent in the multi-agent system includes: Classifying the multi-agents according to the attributes to obtain a plurality of corresponding classification categories; dividing the multi-agents of the same classification category into the same hyperedge to obtain a plurality of first hyperedges; Clustering multiple agents based on their observation information, dividing multiple agents belonging to the same category into the same hyperedge, and obtaining multiple second hyperedges; Predict the intention information of each agent based on the observation information and historical actions of the multi-agents; cluster the multi-agents based on their intention information, and divide the multi-agents belonging to the same classification into the same hyperedge to obtain multiple third hyperedges; The first super edge, the second super edge and the third super edge constitute the plurality of super edges; The message weight calculation module is used to calculate the message weights between multiple agents based on the observation information of each agent; The method of calculating the message weights between multiple agents based on the observation information of each agent includes: pass Calculate the message weights between multiple agents; in, represents weight; Q and K are obtained by linear transformation of the observation information of each agent; d k Indicates the dimension size of K; A correlation generation module, used to generate a correlation corresponding to the multi-agent system based on the communication structure and the message weight; The communication structure is a communication matrix; the generating of the correlation degree corresponding to the multi-agent system based on the communication structure and the message weight includes: Determining a target location in the communication matrix based on each hyperedge in the communication structure; determining a value of a target position in the communication matrix based on the message weight; An information fusion module, used to fuse the communication information of each intelligent agent based on the correlation degree to obtain the fused communication information corresponding to each intelligent agent; the fused communication information is used to guide the action of the corresponding intelligent agent; The step of fusing the communication information of each agent based on the correlation to obtain the fused communication information corresponding to each agent includes: By O i =∑k j *o j Calculate the fused communication information corresponding to each intelligent agent; Among them, O i represents the fusion communication information; i represents the number of agents; k j represents the value of the jth target position in the i-th row or i-th column of the communication matrix, O j Represents the communication information of the agent corresponding to the jth target position in the i-th row or i-th column of the communication matrix.
5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.
6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.
