Multi-agent reinforcement learning method and device with explicit communication mechanism

Abstract

The application discloses a kind of multi-agent reinforcement learning method and device with explicit communication mechanism, the method includes: between multi-agent, establish the display communication-sharing mechanism;Through the communication-sharing mechanism, the perception data of same team agent and different team agent, environmental information are obtained;Policy training is carried out using the deep reinforcement learning framework based on A3C, the perception data of same team agent and different team agent, environmental information, determine the current model parameter of each agent, each agent carries out action selection based on current model parameter.The application can better integrate the advantages of wireless communication technology and reinforcement learning technology, can better perceive enemy and me multi-agent and environmental information under complex environment, carry out reinforcement learning training using the deep reinforcement learning framework based on A3C, effectively reduce the difficulty of multi-agent game confrontation learning, provide a new effective way for solving future unmanned cooperative confrontation problem.

Description

Technical Field

[0001] This invention relates to the field of multi-agent game adversarial, and in particular to a multi-agent reinforcement learning method with an explicit communication mechanism. Background Technology

[0002] With the development of artificial intelligence technology, dynamic game theory with incomplete information in complex environments has become a pressing frontier issue, and multi-agent real-time policy adversarial technology is one of its core key aspects. Currently, multi-agent reinforcement learning technology, which uses sensors as the primary means of perception, has attracted considerable attention from researchers. However, its limited perception range in complex environments leads to low learning efficiency and an inability to obtain optimal cooperative game strategies. With the rapid development of wireless communication technologies such as 5G, multi-agent game adversarial strategies with communication capabilities represent the forefront of current research in cooperation and competition among multi-agents, offering new and effective approaches to solving future problems of cooperative adversarial games.

[0003] Existing reinforcement learning methods treat other agents as part of the environment and use their onboard sensors (such as cameras, lidar, etc.) to perceive the external environment (including information from agents in the same team and different teams) within a certain range. However, due to the limited detection range of traditional sensors, which can only perceive the environment within the agent's line of sight, the environmental information obtained by the agent itself is also relatively limited. This leads to low efficiency in multi-agent game adversarial learning and the resulting cooperative adversarial strategies are not optimal, thus having certain limitations. Summary of the Invention

[0004] To address the aforementioned shortcomings in the prior art, this invention provides a multi-agent reinforcement learning method and apparatus with an explicit communication mechanism.

[0005] This invention provides a multi-agent reinforcement learning method with an explicit communication mechanism, comprising:

[0006] Establish explicit communication-sharing mechanisms among multiple agents;

[0007] The communication-sharing mechanism is used to acquire perception data and environmental information from both team-mate and team-disagree agents.

[0008] The strategy is trained using a deep reinforcement learning framework based on A3C (Actor-Critic Algorithm), perceptual data from agents in the same team and agents in different teams, and environmental information. The current model parameters of each agent are determined, and each agent selects actions based on the current model parameters.

[0009] In some embodiments, the A3C-based deep reinforcement framework includes a global network, which includes an actor module and a critic module. The input of the actor module includes perception data and environmental information of teammate and out-of-team agents, and the output of the actor module is the behavior of teammate and out-of-team agents. The input of the critic module is the perception data and behavior of teammate and out-of-team agents, and the output of the actor module is the evaluation information of the perception data and behavior.

[0010] In some embodiments, policy training is performed using an A3C-based deep reinforcement learning framework, perception data from co-team agents and out-of-team agents, and environmental information to determine the current model parameters for each agent, including:

[0011] In multiple threads, each agent interacts and learns with its respective environment. The gradient is passed through the policy according to the loss function for training, and the obtained policy gradient is asynchronously transmitted back to the global network for updating within a certain period. The updated global network parameters are then copied to the agent policy parameters in each thread to obtain the current model parameters of each agent. The deep reinforcement learning framework based on A3C includes local networks, which include agents and corresponding environments. Each thread corresponds to one local network.

[0012] In some embodiments, each agent selects actions based on current model parameters, including:

[0013] Each agent selects actions based on the current model parameters through a decoding function.

[0014] In some embodiments, the communication-sharing mechanism includes communication method, sharing method, communication content, and communication frequency.

[0015] In some embodiments, the communication methods include one-to-one, one-to-many, many-to-many, and broadcast methods; the sharing methods include designated sharing, local sharing, and global sharing; and the communication content includes the speed, position, direction, and actions performed by agents in the same and different teams.

[0016] In some embodiments, after each agent selects an action based on the current model parameters, the method further includes:

[0017] Each intelligent agent interacts with the environment and receives its own reward.

[0018] Secondly, embodiments of the present invention provide a multi-agent reinforcement learning device with an explicit communication mechanism, comprising:

[0019] Establish a module to create an explicit communication-sharing mechanism among multiple agents;

[0020] The acquisition module is used to acquire perception data and environmental information of intelligent agents in the same team and those in different teams through the communication-sharing mechanism;

[0021] The training module is used to train policies using a deep reinforcement learning framework based on A3C, perception data from agents in the same team and agents in different teams, and environmental information. It determines the current model parameters of each agent and allows each agent to select actions based on the current model parameters.

[0022] In some embodiments, the A3C-based deep reinforcement framework includes a global network, which includes an actor module and a critic module. The input of the actor module includes perception data and environmental information of teammate and out-of-team agents, and the output of the actor module is the behavior of teammate and out-of-team agents. The input of the critic module is the perception data and behavior of teammate and out-of-team agents, and the output of the actor module is the evaluation information of the perception data and behavior.

[0023] In some embodiments, the training module is specifically used for:

[0024] In multiple threads, each agent interacts and learns with its respective environment. The gradient is passed through the policy according to the loss function for training, and the obtained policy gradient is asynchronously transmitted back to the global network for updating within a certain period. The updated global network parameters are then copied to the agent policy parameters in each thread to obtain the current model parameters of each agent. The deep reinforcement learning framework based on A3C includes local networks, which include agents and corresponding environments. Each thread corresponds to one local network.

[0025] This invention utilizes 5G and other wireless communication technologies to establish an explicit communication and sharing mechanism among multiple agents, breaking through the detection range of traditional sensors. It employs beyond-line-of-sight communication to perceive environmental information surrounding the agents and enables information sharing among agents within the same team, enhancing collaboration and effectively increasing the available environmental information and its utilization rate. This invention effectively integrates the advantages of wireless communication and reinforcement learning technologies, enabling better perception of multiple agents (both friendly and adversarial) and environmental information in complex environments. By utilizing a deep reinforcement learning framework based on A3C, it effectively reduces the difficulty of multi-agent adversarial learning, providing a new and effective approach to solving future unmanned cooperative adversarial problems. Attached Figure Description

[0026] Figure 1 A flowchart illustrating a multi-agent reinforcement learning method with an explicit communication mechanism, provided as an embodiment of the present invention;

[0027] Figure 2 A schematic diagram illustrating the framework of a multi-agent reinforcement learning method with an explicit communication mechanism, provided in an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of a multi-agent reinforcement learning device with an explicit communication mechanism, provided as an embodiment of the present invention. Detailed Implementation

[0029] Exemplary embodiments will be described more fully below with reference to the accompanying drawings; however, these exemplary embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and that those skilled in the art will fully understand the scope of the invention.

[0030] Where there is no conflict, the embodiments of the present invention and the features thereof can be combined with each other.

[0031] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated features, integrals, steps, operations, elements, and / or components is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded.

[0033] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be understood to have the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be understood to have an idealized or overly formal meaning unless expressly so defined herein.

[0034] To address the issues of insufficient cooperation and poor cooperation effects in current multi-agent game adversarial games, this application proposes a multi-agent reinforcement learning method and device with an explicit communication mechanism, taking advantage of the high transmission speed and wide range of wireless communication technology.

[0035] To enable those skilled in the art to better understand the technical solution, the following detailed description, in conjunction with the accompanying drawings, describes a multi-agent reinforcement learning method and apparatus with an explicit communication mechanism provided by the present invention.

[0036] Firstly, such as Figure 1 , Figure 2 As shown, this embodiment of the invention provides a multi-agent reinforcement learning method with an explicit communication mechanism, including:

[0037] Step S101: Establish an explicit communication-sharing mechanism among multiple agents;

[0038] Step S102: Obtain perception data and environmental information of the same team's intelligent agents and different teams' intelligent agents through the communication-sharing mechanism;

[0039] Step S103: Use the deep reinforcement learning framework based on A3C, the perception data of the same team agents and different team agents, and environmental information to train the policy, determine the current model parameters of each agent, and each agent selects actions based on the current model parameters.

[0040] Sensing data includes information such as location, speed, direction, and actions performed.

[0041] In the multi-agent interaction process within a team, each agent acquires and shares some of its observations. Furthermore, there are certain connections between agents, and they influence each other's decisions. Therefore, agents need to exchange internal information to better achieve their respective goals. Each agent extracts its implicit information (perceptual data) from its local observation state and communicates this implicit information according to a specific communication-sharing mechanism. Using the explicit communication-sharing method described in step S101, direct perception can be achieved for agents within the same team outside the detection range of traditional sensors. Indirect perception can also be achieved for agents in different teams outside the detection range of traditional sensors. By sharing the perception data of multiple agents within the same team regarding agents in different teams, indirect perception beyond line-of-sight can be achieved. This communication-sharing mechanism effectively increases the available environmental information and improves the efficiency of information utilization in complex environments.

[0042] Communication is a crucial feature in multi-agent interaction. This invention utilizes wireless communication technologies such as 5G to establish an explicit communication and sharing mechanism among multiple agents, breaking through the detection range of traditional sensors. By employing beyond-line-of-sight communication, it perceives environmental information surrounding the agents and enables information sharing among agents within the same team, enhancing collaboration and effectively increasing the available environmental information and its utilization rate. This invention effectively integrates the advantages of wireless communication and reinforcement learning technologies, enabling better perception of multiple agents (both friendly and adversarial) and environmental information in complex environments. Utilizing a deep reinforcement learning framework based on A3C for training effectively reduces the difficulty of multi-agent adversarial learning, providing a new and effective approach to solving future unmanned collaborative adversarial problems.

[0043] In some embodiments, such as Figure 2 As shown, the deep strong chemical framework based on A3C includes a global network, which comprises an actor module and a critic module. The input to the actor module includes perception data from both team agents and different teams, as well as environmental information. Figure 2 The state of the actors (in the context of the game), the output of the actor module is the behavior of the teammates and the opponents, and the input of the critic module is the perception data of the teammates and the opponents, and the behavior of the teammates and the opponents. Figure 2 The output of the actor module is an evaluation of the perceived data and the behavior.

[0044] In this embodiment of the invention, the beyond-line-of-sight environment information (including perception information of agents in the same team and different teams) constructed using a multi-agent wireless communication-sharing mechanism is used as input.

[0045] In some embodiments, policy training is performed using an A3C-based deep reinforcement learning framework, perception data from agents in the same team and from agents in different teams, and environmental information to determine the current model parameters of each agent (i.e., step S103), including:

[0046] In multiple threads, each agent interacts and learns with its respective environment. The gradient is passed through the policy according to the loss function for training, and the obtained policy gradient is asynchronously transmitted back to the global network for updating within a certain period. The updated global network parameters are then copied to the agent policy parameters in each thread to obtain the current model parameters of each agent. The deep reinforcement learning framework based on A3C includes local networks, which include agents and corresponding environments. Each thread corresponds to one local network.

[0047] In this embodiment of the invention, a policy training framework based on an improved actor-critic distributed reinforcement learning is used, and a multi-threaded agent interacts with the environment to pass gradients during training.

[0048] In some embodiments, each agent selects actions based on current model parameters, including:

[0049] Each agent selects actions based on the current model parameters through a decoding function.

[0050] In some embodiments, the communication-sharing mechanism includes communication method, sharing method, communication content, and communication frequency.

[0051] In some embodiments, the communication methods include one-to-one, one-to-many, many-to-many, and broadcast methods; the sharing methods include designated sharing, local sharing, and global sharing; and the communication content includes the speed, position, direction, and actions performed by agents in the same and different teams.

[0052] In this embodiment of the invention, a suitable communication method can be selected based on the amount of information and requirements. Designated sharing refers to sharing information with specific teammates, local sharing refers to sharing information with teammates within a certain range, and global sharing refers to sharing information with all teammates. The communication content includes the speed, position, direction, and actions performed by teammates and different teammates.

[0053] In some embodiments, after each agent selects an action based on the current model parameters, the method further includes:

[0054] Each intelligent agent interacts with the environment and receives its own reward.

[0055] Secondly, such as Figure 3 As shown, this embodiment of the invention provides a multi-agent reinforcement learning device with an explicit communication mechanism, comprising:

[0056] Establish module 11 to establish an explicit communication-sharing mechanism among multiple agents;

[0057] The acquisition module 12 is used to acquire the perception data and environmental information of the same team's intelligent agents and different teams' intelligent agents through the communication-sharing mechanism;

[0058] Training module 13 is used to train policies using a deep reinforcement learning framework based on A3C, perception data of agents in the same team and agents in different teams, and environmental information, to determine the current model parameters of each agent, and each agent selects actions based on the current model parameters.

[0059] In some embodiments, the A3C-based deep reinforcement framework includes a global network, which includes an actor module and a critic module. The input of the actor module includes perception data and environmental information of teammate and out-of-team agents, and the output of the actor module is the behavior of teammate and out-of-team agents. The input of the critic module is the perception data and behavior of teammate and out-of-team agents, and the output of the actor module is the evaluation information of the perception data and behavior.

[0060] In some embodiments, the training module 13 is specifically used for:

[0061] In multiple threads, each agent interacts and learns with its respective environment. The gradient is passed through the policy according to the loss function for training, and the obtained policy gradient is asynchronously transmitted back to the global network for updating within a certain period. The updated global network parameters are then copied to the agent policy parameters in each thread to obtain the current model parameters of each agent. The deep reinforcement learning framework based on A3C includes local networks, which include agents and corresponding environments. Each thread corresponds to one local network.

[0062] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some embodiments, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth by the appended claims.

Claims

1. A multi-agent reinforcement learning method with an explicit communication mechanism, characterized in that, include: Establish explicit communication-sharing mechanisms among multiple agents; The communication-sharing mechanism is used to acquire perception data and environmental information from both team-mate and team-disagree agents. The strategy is trained using a deep reinforcement learning framework based on A3C, perceptual data from agents in the same team and agents in different teams, and environmental information. The current model parameters of each agent are determined, and each agent selects actions based on the current model parameters. The A3C-based deep reinforcement learning framework includes a global network, which comprises an actor module and a critic module. The inputs of the actor module include perception data and environmental information of teammate and out-of-team agents, and the output of the actor module is the behavior of teammate and out-of-team agents. The inputs of the critic module are perception data and behavior of teammate and out-of-team agents, and the output of the actor module is evaluation information on the perception data and behavior. Policy training is performed using a deep reinforcement learning framework based on A3C, perception data from agents in the same team and agents in different teams, and environmental information. This determines the current model parameters for each agent, including: In multiple threads, each agent interacts and learns with its respective environment. The gradient is passed through the policy according to the loss function for training. The obtained policy gradient is asynchronously transmitted back to the global network for updating within a certain period. The updated global network parameters are then copied to the agent policy parameters in each thread to obtain the current model parameters of each agent. The deep reinforcement learning framework based on A3C includes local networks, which include agents and corresponding environments. Each thread corresponds to one local network. Each agent selects actions based on the current model parameters, including: Each agent selects actions based on the current model parameters through a decoding function.

2. The multi-agent reinforcement learning method with explicit communication mechanism according to claim 1, characterized in that, The communication-sharing mechanism includes communication method, sharing method, communication content, and communication frequency.

3. The multi-agent reinforcement learning method with explicit communication mechanism according to claim 2, characterized in that, The communication methods include one-to-one, one-to-many, many-to-many, and broadcast methods. The sharing methods include designated sharing, local sharing, and global sharing. The communication content includes the speed, position, direction, and actions performed by intelligent agents in the same and different teams.

4. The multi-agent reinforcement learning method with explicit communication mechanism according to claim 1, characterized in that, After each agent selects an action based on the current model parameters, the process also includes: Each intelligent agent interacts with the environment and receives its own reward.

5. A multi-agent reinforcement learning device with an explicit communication mechanism, characterized in that, include: Establish a module to create explicit communication-sharing mechanisms among multiple agents; The acquisition module is used to acquire perception data and environmental information of intelligent agents in the same team and those in different teams through the communication-sharing mechanism; The training module is used to train policies using a deep reinforcement learning framework based on A3C, perception data of agents in the same team and agents in different teams, and environmental information, to determine the current model parameters of each agent, and each agent selects actions based on the current model parameters. The A3C-based deep reinforcement learning framework includes a global network, which comprises an actor module and a critic module. The inputs of the actor module include perception data and environmental information of teammate and out-of-team agents, and the output of the actor module is the behavior of teammate and out-of-team agents. The inputs of the critic module are perception data and behavior of teammate and out-of-team agents, and the output of the actor module is evaluation information on the perception data and behavior. The training module is specifically used for: In multiple threads, each agent interacts and learns with its respective environment. The gradient is passed through the policy according to the loss function for training. The obtained policy gradient is asynchronously transmitted back to the global network for updating within a certain period. The updated global network parameters are then copied to the agent policy parameters in each thread to obtain the current model parameters of each agent. The deep reinforcement learning framework based on A3C includes local networks, which include agents and corresponding environments. Each thread corresponds to one local network. Each agent selects actions based on the current model parameters, including: Each agent selects actions based on the current model parameters through a decoding function.

Images