Multi-agent algorithm model based on distributed deep Q network
By adopting a distributed deep Q network algorithm model in multi-agent systems, information sharing and independent strategy learning between agents are realized, and the problems of low efficiency, insufficient real-time and poor adaptability of existing systems in complex dynamic environments are solved, which significantly improves the system's combat effectiveness and decision-making accuracy.
Patent Information
- Application Number
- CN202411712764.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-04-18
AI Technical Summary
Existing multi-agent systems have low efficiency in collaborative decision-making, insufficient real-time and poor adaptability in complex dynamic environments.
The multi-agent algorithm model based on distributed deep Q network is adopted, and through the global state information database, information sharing layer, decision-making layer and multi-agent action selection steps, information sharing and independent strategy learning between agents are realized, and the real-time and adaptability of collaborative decisions are improved.
It significantly improves the combat effectiveness and decision-making accuracy of multi-agent systems in high-dynamic environments such as air combat, enhances the real-time and environmental adaptability of the system, and has good robustness and scalability.
Smart Images
Figure CN119849537A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent air combat auxiliary systems and relates to a multi-agent algorithm model based on a distributed deep Q network. Background Art
[0002] A multi-agent system (MAS) is a distributed system composed of multiple agents that can work independently or collaboratively to complete complex tasks. MAS is widely used in robot group control, traffic management, distributed control systems, and collaborative operations. In MAS, each agent can be regarded as an individual that makes autonomous decisions, can perceive the environment, interact with other agents, and take corresponding actions to achieve its goals. With the improvement of computer capabilities and the development of distributed system architecture, the application of MAS has become more extensive and in-depth. The core of MAS technology lies in the collaboration and distributed decision-making between agents. How to achieve effective cooperation between agents without relying on central control has become an important research direction of MAS. Existing MAS technologies include distributed control, decision-making methods based on game theory, and collaborative strategies based on reinforcement learning. These technologies provide technical support for decision optimization in multi-agent systems.
[0003] At the same time, the distributed deep Q network (D-DQN) is a distributed reinforcement learning algorithm for multi-agent systems. By deploying independent Q networks on different agents, each agent can independently learn the optimal strategy based on local and global information. The core of D-DQN technology lies in the global information sharing mechanism and distributed strategy learning: in the global information sharing mechanism, each agent shares global state information to fully utilize information and avoid the local optimal problem of decision-making; in distributed strategy learning, each agent independently learns strategies and achieves the approximation of the global optimal strategy in a distributed manner. The application of D-DQN technology in distributed systems provides a new solution for multi-agent collaborative decision-making. Especially in tasks with high real-time requirements, D-DQN can effectively balance computing resources and decision-making accuracy, and has broad application prospects.
[0004] The existing multi-agent system, reinforcement learning, and distributed learning technology provide a solid theoretical foundation and technical support for the realization of the agent algorithm model. By combining the distributed deep Q network technology, it is theoretically possible to achieve real-time and accurate decision optimization in multi-agent collaborative combat, effectively improving the overall combat capability of the air combat system. However, the current multi-agent system still has many problems in use, as follows: First, the adaptability to dynamic environments. The adaptive ability of multi-agent systems in dynamic and complex environments still needs to be improved. Agents should be able to quickly identify environmental changes and make corresponding adjustments; second, standardized communication protocols. The current lack of a unified standardized communication protocol leads to poor interoperability between different agent systems, and there is an urgent need to develop a general communication framework; then there is the collaborative decision-making algorithm. The existing decision-making algorithm has low efficiency and effectiveness when dealing with large-scale agents. It is necessary to improve the collaborative decision-making algorithm to support the cooperation of more agents; finally, in terms of real-time performance, with the increase in the number of agents, the amount of data required for real-time processing increases dramatically, resulting in an increased computational burden and difficulty in meeting the real-time requirements. Therefore, a multi-agent algorithm model based on a distributed deep Q network is urgently needed to solve the above problems. Summary of the invention
[0005] A multi-agent algorithm model based on distributed deep Q-network is proposed to solve the technical problems of low efficiency, insufficient real-time performance and poor adaptability of existing multi-agent systems in collaborative decision-making in complex dynamic environments by constructing a distributed deep Q-network architecture.
[0006] In order to achieve the above technical objectives, the specific technical solutions adopted by the present invention are:
[0007] A multi-agent algorithm model based on a distributed deep Q network includes the following steps:
[0008] S1, running global status information base;
[0009] Set in the global state information library. When the model starts running, the global state information is initialized through the global state information library, and the global state information is updated after each round of running.
[0010] S2, operation information sharing layer;
[0011] A global state information sharing module is set up to integrate the battlefield situation, aircraft status and target information into global state information and share it with all intelligent agents. Each intelligent agent generates its own action probability distribution after receiving the global state information.
[0012] S3, operation decision-making layer;
[0013] The action probability distribution of each intelligent agent transmitted by the information sharing layer is received and stacked to form an action probability distribution array. The intelligent algorithm makes decisions based on the array information to generate a multi-agent probability distribution update strategy.
[0014] S4, perform multi-agent action selection;
[0015] The agents in the environment select actions based on the probability distribution update strategy provided by the decision layer, and the actions of all agents are integrated into an action set.
[0016] S5. Update global status information;
[0017] After multiple agents make corresponding actions according to the action set, the state of the environment changes, the global state information is updated accordingly, and the updated global state information is synchronized to the global state information library.
[0018] Furthermore, a global loss function calculation module is set up. After step S5, the global state information obtained through communication between independently deployed intelligent agents is used as input, and the reward value of the current action is calculated through the loss function. This reward value will be used as input information for the next round of information sharing layer to update the action probability distribution of multiple agents in the next round.
[0019] The present invention aims to realize information sharing, independent strategy learning and collaborative decision-making among multiple intelligent agents, improve the combat effectiveness and decision-making accuracy of multi-agent systems in high dynamic environments such as air combat, optimize the resource utilization efficiency of intelligent agents through a distributed update mechanism, meet real-time requirements, and enhance the system's adaptability to environmental changes.
[0020] The principle is as follows:
[0021] Reinforcement learning is a machine learning method that learns the optimal strategy by interacting with the environment. In reinforcement learning, the agent continuously tries different actions through interaction with the environment and adjusts its strategy based on the rewards obtained to maximize the cumulative rewards. Reinforcement learning technology has significant advantages in solving complex problems, especially in dynamic and uncertain environments. Deep Q network is a typical application of reinforcement learning. It combines deep learning and Q learning, approximates the Q value function through deep neural networks, and realizes the processing of high-dimensional state space.
[0022] Distributed learning and decision-making technology refers to the use of distributed architecture to achieve independent learning and decision-making optimization of each agent in a multi-agent system. This technology can achieve global optimal decision-making through information sharing and collaboration mechanisms while ensuring the independence of each agent. Distributed learning technologies include distributed reinforcement learning, distributed supervised learning, and distributed decision-making based on game theory.
[0023] In distributed learning, how to effectively share information among multiple agents to prevent information islands, and how to coordinate the learning process of each agent to avoid conflicts and competition are the research focuses in this field. Existing distributed reinforcement learning methods, such as distributed algorithms based on the Actor-Critic framework and multi-agent learning methods based on experience sharing, provide theoretical support and technical means for multi-agent collaborative decision-making.
[0024] With the development of deep learning technology, deep reinforcement learning is increasingly used in complex systems. In distributed systems, DRL introduces deep neural networks to enable agents to process high-dimensional perceptual information and optimize in complex decision spaces.
[0025] Distributed Deep Q Network (D-DQN) is a distributed reinforcement learning algorithm for multi-agent systems. By deploying independent Q networks on different agents, each agent can independently learn the optimal strategy based on local and global information.
[0026] The existing multi-agent system, reinforcement learning, and distributed learning technologies provide a solid theoretical basis and technical support for the implementation of the present invention. By combining the distributed deep Q network technology, the present invention can achieve real-time and accurate decision optimization in multi-agent collaborative combat, and effectively improve the overall combat capability of the air combat system. The specific construction content is as follows:
[0027] Deep Q networks (DQNs) are deployed independently on multiple intelligent agents. Each intelligent agent is equipped with an independent deep neural network for decision-making learning and optimization. Each functional agent obtains the global situation information of the battlefield through the global information sharing mechanism to achieve comprehensive perception of the environment.
[0028] Design a global information sharing mechanism to ensure that each agent can obtain the perception information and local status of other agents in real time.
[0029] Each agent learns strategies independently and updates its strategies through a distributed deep Q-network structure using local perception information and global shared information. Each agent independently adjusts its strategies based on its own experience and feedback from the environment to achieve the global optimal strategy for the entire system.
[0030] Since each intelligent agent has a different position in the entire air combat system, its weight is also different. Designing a global loss function enhances the accuracy of system decision-making and improves the efficiency of model convergence.
[0031] In practical applications, algorithms are continuously iterated and optimized to adapt them to different combat scenarios and missions.
[0032] By adopting the above technical solution, the present invention can also bring the following beneficial effects:
[0033] 1. The present invention mentions a multi-agent algorithm model based on a distributed deep Q network, which significantly improves the collaborative combat capability of various units in air combat, enhances the real-time and environmental adaptability of the system, and also has good robustness and scalability. Even if some agents fail or are disturbed, the system can continue to operate and maintain high combat effectiveness. The present invention is also optimized specifically for air combat environments with high dynamics and uncertainties, so that the multi-agent system can effectively respond to various emergencies under complex battlefield conditions and ensure the smooth execution of combat missions.
[0034] 2. The present invention mentions a multi-agent algorithm model based on a distributed deep Q network, which adopts a hierarchical structure to design the algorithm model. Each layer of the algorithm model has a clear division of labor, and only a predetermined communication interface is retained between the layers. The specific implementation inside each layer is invisible to the outside. This modular design mode is conducive to improving the efficiency of the multi-agent decision-making system. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0036] Figure 1 The present invention mentions a multi-agent algorithm model structure diagram based on a distributed deep Q network. DETAILED DESCRIPTION
[0037] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0038] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.
[0039] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present invention, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.
[0040] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. The drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0041] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.
[0042] Example 1
[0043] In one embodiment of the present invention, see Figure 1 The present invention mentions a multi-agent algorithm model based on a distributed deep Q network, and the specific steps include:
[0044] 1. The global state information library initializes the entire multi-agent environment. Assume that the multi-agent environment includes two transport planes, four fighter planes, four helicopters, and ten drones. Each aircraft is an independent agent, and deploys an independent distributed deep Q network according to its positioning in the air combat system, for example: A1, A2; B1, B2, B3, B4; C1, C2, C3, C4; D1, D2...D10; (where A represents transport planes, B represents fighter planes, C represents helicopters, and D represents drones). Assuming that the initial time is t0, the global state information library initializes the environment to state St0 after the agent is configured.
[0045] 2. After the environment is initialized, each agent communicates through the interface in the information sharing layer. After the communication is completed, the probability distribution of the action is generated according to the reward value rt (no reward value in the first round). For example, suppose fighter B1 has three actions: climb, hover, dive, roll, jump, sharp turn and bomb. After the information sharing layer calculates, the agent B1 forms the action probability distribution (0.05, 0.04, 0.03, 0.05, 0.7, 0.1, 0.03) data represents the probability of each action. From the above data, it can be seen that the probability of the agent B1 taking a roll (70% probability) at the next moment is relatively high.
[0046] 3. After each agent generates the action probability distribution, it sends the data to the decision layer. After stacking the data, the decision layer applies the multi-agent decision algorithm to select the action of each agent and output the agent action set. For example, AT0[at1,at2,…at20], atN represents the action to be taken by the Nth agent at the next moment.
[0047] 4. After receiving the action set AT0, the agents in the environment find the corresponding actions through the index and execute them.
[0048] 5. After the agent performs the corresponding action, the global environment state changes. At this time, the global state information is recorded as St1, and the latest global state information is synchronized to the global state information library.
[0049] 6. The global loss function calculation module generates a reward value rt0 by comparing the previous state (St0) and the current state (St1). This reward value will be input into the information sharing layer to help the agent make better decisions in the next round.
[0050] In summary, by constructing a distributed deep Q-network architecture, the present invention can realize information sharing, independent strategy learning and collaborative decision-making among multiple agents, improve the combat effectiveness and decision-making accuracy of multi-agent systems in high-dynamic environments such as air combat, and optimize the resource utilization efficiency of the agents through a distributed update mechanism to meet real-time requirements and enhance the system's adaptability to environmental changes.
[0051] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims
1. A multi-agent algorithm model based on a distributed deep Q network, characterized in that: The following steps are involved: S1, running global status information base; Set in the global state information library. When the model starts running, the global state information is initialized through the global state information library, and the global state information is updated after each round of running. S2, operation information sharing layer; A global state information sharing module is set up to integrate battlefield situation, aircraft state and target information into global state information and share it with all agents. Each agent generates its own action probability distribution after receiving the global state information. S3, operation decision-making layer; Receive the action probability distribution of each agent transmitted by the information sharing layer, and stack them to form an action probability distribution array. The intelligent algorithm makes decisions based on the array information to generate a multi-agent probability distribution update strategy; S4, perform multi-agent action selection; The agents in the environment update their strategies based on the probability distribution provided by the decision layer to select actions, and the actions of all agents are integrated into an action set; S5. Update global status information; After multiple agents make corresponding actions according to the action set, the state of the environment changes, the global state information is updated accordingly, and the updated global state information is synchronized to the global state information library.
2. A multi-agent algorithm model based on a distributed deep Q network as claimed in claim 1, characterized in that: A global loss function calculation module is set up. After step S5, the global state information obtained through communication between independently deployed intelligent agents is used as input, and the reward value of the current action is calculated through the loss function. This reward value will be used as the input information of the next round of information sharing layer to update the action probability distribution of multiple agents in the next round.