Unmanned aerial vehicle game confrontation island seizing method based on multi-agent reinforcement learning algorithm

Abstract

The invention belongs to the technical field of deep reinforcement learning. The invention provides an unmanned aerial vehicle game confrontation island seizing method based on a multi-agent reinforcement learning algorithm. According to the embodiment of the invention, a Rainbow algorithm module is introduced into the MADDPG algorithm, such as priority experience playback, multi-step TD updating and the like, and the convergence speed at the initial stage of training is accelerated by using behavior cloning. And the sample utilization efficiency and the strategy stability are remarkably improved, so that the intelligent agent can obtain excellent performance in a more complex task environment. The reward function combining the reward return value and the battle power preservation rate is designed, the intelligent agent can be effectively guided to complete the island seizing task target, the intelligent agent is encouraged to optimize resource utilization, and battle damage in the task process is reduced. The reward mechanism can dynamically evaluate the task completion effect, the strategy efficiency and the resource consumption balance, and the strategy learning ability of the intelligent agent is improved. The introduction of the war power preserving rate provides a brand-new optimization dimension for reinforcement learning tasks.

Description

Technical Field

[0001] The disclosed embodiments relate to the technical field of deep reinforcement learning, and in particular to a method for drone game confrontation and island capture based on a multi-agent reinforcement learning algorithm. Background Art

[0002] With the rapid development of drone technology and its widespread application in the military field, the collaborative combat capability of drones in complex game-based island-grabbing environments has become a research focus. Drones can not only perform tasks individually, but also collaborate in clusters to take advantage of group intelligence. In complex battlefield environments, especially when resources are limited, drone swarms need to make optimal strategies through game decisions to achieve mission objectives. The combination of game theory and deep reinforcement learning technology provides a new solution for drone swarm confrontation.

[0003] Against this backdrop, MADDPG (Multi-Agent Deep Deterministic Policy Gradient), as a multi-agent reinforcement learning algorithm, has shown great potential in drone game confrontation. MADDPG can effectively handle the coordination and competition issues of multiple drones, and when faced with highly uncertain and complex environments, its reinforcement learning framework can enable the red and blue drones to learn their own confrontation strategies through repeated interactions with the environment. This adversarial learning method can not only improve the strategy level of individual drones, but also optimize group collaboration, allowing multi-agents to play a greater role in highly dynamic and complex tasks, especially in island-grabbing missions, where the red and blue drone groups need to collaborate in confrontation and formulate offensive or defensive strategies.

[0004] However, the traditional MADDPG algorithm suffers from common problems such as low convergence efficiency, unstable strategy, and difficulty in learning sparse reward tasks in some practical applications, especially in complex adversarial game scenarios in drone adversarial tasks.

[0005] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0006] It should be noted that this section is intended to provide background or context for the technical solutions of the present disclosure stated in the claims. The description herein is not admitted to be prior art by virtue of being included in this section. Summary of the invention

[0007] The purpose of the embodiments of the present disclosure is to provide a drone game confrontation island capture method based on a multi-agent reinforcement learning algorithm, thereby overcoming one or more problems caused by the limitations and defects of related technologies at least to a certain extent.

[0008] According to an embodiment of the present disclosure, a method for drone game confrontation and island capture based on a multi-agent reinforcement learning algorithm is provided, the method comprising: Step S1, construct a multi-agent simulation environment: Based on the UAV game confrontation platform MaCA, the motion model, perception range and dynamic environment parameters of the intelligent agent are set to simulate the collaboration and confrontation scenarios of the intelligent agent in two-dimensional or three-dimensional space; Step S2, agent modeling and initialization: Assign initial position, speed and task role to each agent, and build independent strategy network and value network based on MADDPG algorithm; The policy network generates action instructions based on the state space of the agent, which includes position, speed, state of neighboring agents, and target area information; The value network evaluates the expected return of action instructions and optimizes the strategy based on global information; Step S3, design an average reward mechanism: The average reward value is dynamically calculated according to the agent type and mission objectives. The agent types include reconnaissance aircraft and attack aircraft, and the reward values ​​include successful attack reward, effective detection reward, survival reward, and destruction penalty. Step S4, strategy optimization and training: The MADDPG algorithm is used for multi-agent strategy learning, and the priority experience replay mechanism of the Rainbow algorithm is introduced to dynamically adjust the sample sampling weight according to the TD-error; Combined with the multi-step TD update mechanism, the agent's ability to perceive long-term rewards; Initialize the policy network through behavioral cloning technology to solve the problem of convergence difficulties in the early stage of training; Step S5, combat power preservation rate evaluation: After the confrontation task is completed, the combat power preservation rate is calculated based on the ratio of the number of surviving agents to the initial total number, and the strategy performance is comprehensively evaluated in combination with the reward return value; Step S6, iterative optimization and deployment: The network parameters are updated through multiple rounds of training to generate the final strategy model and apply it to the actual drone game against island capture missions.

[0009] Furthermore, in step S3, the sum of the rewards of all agents in each round is divided by the number of agents to obtain the average reward value of each round. The average reward value is calculated as:

[0010] in, is the number of agents, For the The reward of each agent in this round.

[0011] Furthermore, the introduction of the Rainbow algorithm in step S4 specifically includes: Use the soft update mechanism of the target network and the main network to reduce the strategy fluctuation during training; Improve strategy diversity and avoid local optimal solutions through distributed Q-value estimation.

[0012] Furthermore, the differentiated reward mechanism is as follows: The reward function of the attack aircraft includes the reward for destroying the enemy target, the reward for occupying the target area, and the reward for its own survival; The reward function of the reconnaissance aircraft includes the detection reward for enemy targets, the information collection reward for the target area, and the survival reward for avoiding attacks.

[0013] Furthermore, the calculation formula for combat power preservation rate is: Combat Power Preservation Rate = ×100% Among them, m is the total number of initial friendly agents, and n is the number of surviving friendly agents; And combined with the enemy's combat power loss rate, dynamically adjust the strategy optimization direction.

[0014] Furthermore, behavioral cloning technology specifically includes: Generate initial training data using expert strategies and pre-train the strategy network through supervised learning; In the reinforcement learning process, the weight of cloned data is gradually reduced, and the transition to autonomous strategy learning is made.

[0015] Furthermore, the step size of the multi-step TD update mechanism ranges from 3 to 5 steps, which is used to balance the weights of immediate rewards and long-term returns.

[0016] Furthermore, electronic interference, multi-band radar detection and passive collaborative positioning functions are simulated in the simulation environment to enhance the authenticity of the confrontation scenario.

[0017] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects: In the embodiment of the present disclosure, through the above-mentioned drone game confrontation island capture method based on multi-agent reinforcement learning algorithm, on the one hand, the Rainbow algorithm module is introduced into the MADDPG algorithm, such as priority experience replay, multi-step TD update, etc., and behavior cloning is used to accelerate the convergence speed in the early stage of training. The combination of these technologies significantly improves the sample utilization efficiency and the stability of the strategy, enabling the agent to achieve excellent performance in a more complex task environment. On the other hand, a reward function combining the reward return value and the combat power preservation rate is designed, which can not only effectively guide the agent to complete the island capture task goal, but also encourage the agent to optimize resource utilization and reduce combat damage during the task. Such a reward mechanism can dynamically evaluate the task completion effect, strategy efficiency and resource consumption balance, and improve the strategy learning ability of the agent. The introduction of combat power preservation rate provides a new optimization dimension for reinforcement learning tasks. This indicator evaluates the survivability of the agent in the task in real time. By combining with the reward function, the agent can minimize resource consumption while completing the task and maximize the benefits of the island capture task. On the other hand, special optimization is carried out for the complex confrontation game task of drone island capture. By simulating confrontation and collaboration scenarios, a multi-agent strategy framework that can balance collaboration and competition is designed, enabling the agents to demonstrate excellent learning and decision-making capabilities in dynamic environments, making up for the shortcomings of existing technologies in similar scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0019] Figure 1 A specific flow chart showing a method for drone game confrontation and island capture based on a multi-agent reinforcement learning algorithm in an exemplary embodiment of the present disclosure is shown; Figure 2 A diagram showing a red and blue model constructed on the MaCA platform in an exemplary embodiment of the present disclosure is shown; Figure 3 A MADDPG algorithm structure diagram of the Rainbow algorithm module introduced in the exemplary embodiment of the present disclosure is shown; Figure 4 A schematic diagram showing a game of drones between red and blue parties to seize an island in an exemplary embodiment of the present disclosure; Figure 5 A comparison diagram of the average reward value of the MADDPG algorithm before and after the introduction of the Rainbow algorithm module in an exemplary embodiment of the present disclosure is shown; Figure 6A comparison chart of the combat power preservation rate of the MADDPG algorithm before and after the introduction of the Rainbow algorithm module in the exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0020] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0021] In addition, the accompanying drawings are only schematic illustrations of the embodiments of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0022] This example implementation provides a drone game confrontation island capture method based on a multi-agent reinforcement learning algorithm. Figure 1 As shown in , the drone game confrontation island capture method based on the multi-agent reinforcement learning algorithm may include: Step S1, construct a multi-agent simulation environment: Based on the UAV game confrontation platform MaCA, the motion model, perception range and dynamic environment parameters of the intelligent agent are set to simulate the collaboration and confrontation scenarios of the intelligent agent in two-dimensional or three-dimensional space; Step S2, agent modeling and initialization: Assign initial position, speed and task role to each agent, and build independent strategy network and value network based on MADDPG algorithm; The policy network generates action instructions based on the state space of the agent, which includes position, speed, state of neighboring agents, and target area information; The value network evaluates the expected return of action instructions and optimizes the strategy based on global information; Step S3, design an average reward mechanism: The average reward value is dynamically calculated according to the agent type and mission objectives. The agent types include reconnaissance aircraft and attack aircraft, and the reward values ​​include successful attack reward, effective detection reward, survival reward, and destruction penalty. Step S4, strategy optimization and training: The MADDPG algorithm is used for multi-agent strategy learning, and the priority experience replay mechanism of the Rainbow algorithm is introduced to dynamically adjust the sample sampling weight according to the TD-error; Combined with the multi-step TD update mechanism, the agent's ability to perceive long-term rewards; Initialize the policy network through behavior cloning technology to solve the problem of convergence difficulties in the early stage of training; Step S5, combat power preservation rate evaluation: After the confrontation task is completed, the combat power preservation rate is calculated based on the ratio of the number of surviving agents to the initial total number, and the strategy performance is comprehensively evaluated in combination with the reward return value; Step S6, iterative optimization and deployment: The network parameters are updated through multiple rounds of training to generate the final strategy model and apply it to the actual drone game against island capture missions.

[0023] Through the above-mentioned drone game confrontation island capture method based on multi-agent reinforcement learning algorithm, on the one hand, the Rainbow algorithm module is introduced into the MADDPG algorithm, such as priority experience replay, multi-step TD update, etc., and behavior cloning is used to accelerate the convergence speed in the early stage of training. The combination of these technologies significantly improves the sample utilization efficiency and the stability of the strategy, enabling the agent to achieve excellent performance in a more complex task environment. On the other hand, a reward function combining the reward return value and the combat power preservation rate is designed, which can not only effectively guide the agent to complete the island capture task goal, but also encourage the agent to optimize resource utilization and reduce combat losses during the task. Such a reward mechanism can dynamically evaluate the task completion effect, strategy efficiency and resource consumption balance, and improve the strategy learning ability of the agent. The introduction of combat power preservation rate provides a new optimization dimension for reinforcement learning tasks. This indicator evaluates the survivability of the agent in the task in real time. By combining with the reward function, the agent can minimize resource consumption while completing the task and maximize the benefits of the island capture task. On the other hand, special optimization is carried out for the complex confrontation game task of drone island capture. By simulating confrontation and collaboration scenarios, a multi-agent strategy framework that can balance collaboration and competition is designed, enabling the agents to demonstrate excellent learning and decision-making capabilities in dynamic environments, making up for the shortcomings of existing technologies in similar scenarios.

[0024] Next, we will refer to Figures 1 to 6 The various steps of the above-mentioned drone game against island capture method based on multi-agent reinforcement learning algorithm in this example implementation are described in more detail.

[0025] In one embodiment, a simulation environment including multiple agents is constructed, and the existing drone game confrontation platform MaCA is used to set the motion model, perception range and other environmental factors of the agents.

[0026] Modeling of intelligent agents (i.e. drones). Initialize each intelligent agent, including its initial position, speed, and strategy. Each intelligent agent uses the MADDPG algorithm to build an independent policy network and value network in order to learn different strategies in a multi-agent environment and design the state space, action space, and reward mechanism of the intelligent agent.

[0027] Set up island-taking confrontation scenarios to simulate competition and cooperation between drones. For example, in a drone game to take over an island, some drones are set as attackers, whose goal is to capture the target area, while others are defenders, whose goal is to prevent the island from being captured by the enemy. By designing a reward mechanism function, the agent is encouraged to improve its strategy in the confrontation.

[0028] Strategy optimization,During the training process of the agent, the MADDPG algorithm is used for strategy optimization, and Rainbow is used to improve the MADDPG algorithm.,Each agent updates its strategy based on its own observations and environmental feedback.

[0029] Combining behavior cloning with reinforcement learning algorithms solves the problem that traditional reinforcement learning algorithms are difficult to converge in the early stages of multi-agent reinforcement learning. Through multiple rounds of training, a model of the drone in the game of island capture was obtained.

[0030] Results evaluation and optimization: By comparing the experimental results of different strategies, the performance of the drone in the island capture mission is evaluated, including the reward return value and combat power preservation rate.

[0031] In one embodiment, this application proposes a MADDPG drone game confrontation island capture method that introduces the Rainbow algorithm, aiming to solve the problems of low convergence efficiency, unstable strategies, and difficulty in learning sparse reward tasks that are common in existing methods in complex confrontation game scenarios. By organically combining the Rainbow algorithm module with the MADDPG framework, this application significantly improves the collaboration efficiency and confrontation capabilities among multiple agents, especially in complex tasks such as drone island capture confrontation games, showing excellent performance.

[0032] In the specific implementation, this application introduces a number of innovative technical modules. The sample utilization efficiency is optimized through the priority experience replay mechanism, and the sampling weight is dynamically adjusted according to the TD-error to ensure that the agent can quickly focus on key samples. Behavior cloning is also introduced, which can effectively solve the problem of difficulty in convergence in the early stages of training. This not only improves the intelligence of operators in multi-agent game confrontation, but also the simulation experimental results conducted in the experimental platform show that the MADDPG algorithm has significant improvements in training effect and convergence speed, especially when using prior knowledge, which can make the training converge faster.

[0033] To meet the special task requirements of UAV game confrontation for seizing islands, this application has customized and optimized the reward function and strategy design. This includes attack result rewards, detection rewards, and being destroyed rewards, thereby ensuring that the generated strategy is not only effective but also practically executable. In addition, in adversarial scenarios, this application adopts a joint training mechanism to dynamically adjust the agent's strategy, enabling it to maintain a competitive advantage when facing the opponent's constantly changing strategies.

[0034] In view of the task characteristics of sparse rewards and delayed feedback, this application introduces a multi-step TD update mechanism. By integrating the cumulative effect of multi-step rewards into the target calculation, it enhances the agent's perception ability of long-term rewards and significantly improves the performance of the strategy in complex tasks.

[0035] The evaluation metrics of this application include reward return values and combat power preservation rates. Reward return values can be used to evaluate the effectiveness of the current strategy in long-term tasks. A strategy that continuously obtains high reward values indicates that the agent can take correct and efficient actions in the task, while low reward values indicate that the strategy needs further optimization. For game confrontation tasks, reward return values can measure whether the agent's performance in the confrontation is better than that of the opponent. For example, a higher reward value indicates that the agent has an advantage in the game and can more effectively suppress the opponent's strategy. It can also reflect the agent's cooperation ability from the side.

[0036] This application integrates the modules of the Rainbow algorithm and the multi-agent reinforcement learning framework of MADDPG, and uses behavior cloning to accelerate the convergence speed, forming a multi-agent reinforcement learning method with high efficiency, high robustness, and adversarial optimization ability. Through the synergistic effect of these innovative technologies, this application not only effectively improves the learning efficiency of the multi-agent system in complex adversarial tasks, but also significantly enhances the stability and adaptability of the strategy. Especially in the UAV game confrontation for seizing islands task, this method can quickly converge in various complex environments to form a dominant strategy, which is an important technical means to solve complex multi-agent task scenarios.

[0037] In a specific embodiment, to make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will elaborate on each embodiment of this application with reference to the accompanying drawings. However, those of ordinary skill in the art can understand that in each embodiment of this application, many technical details are presented to help readers better understand this application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in this application can still be implemented. The following division of each embodiment is for convenience of description and should not constitute any limitation to the specific implementation manner of this application. Each embodiment can be combined and cross-referenced with each other without conflict.

[0038] In order to solve the problem of coordinated combat of drones in a complex game-based island-grabbing environment, the embodiment of this application introduces the MADDPG algorithm drone game-based island-grabbing method of the Rainbow algorithm module. The following is a detailed description of the implementation details of this embodiment. The following content is only for the convenience of understanding the implementation details provided, and is not necessary for the implementation of this solution.

[0039] like Figures 1 to 6 As shown, this embodiment provides a drone game confrontation island capture method based on a multi-agent reinforcement learning algorithm, comprising the following steps: I. Conduct multi-agent environment modeling. To build a simulation environment containing multiple agents, using the existing drone game confrontation platform MaCA, you first need to set up a suitable motion model and perception range. The two-dimensional space set up in the environment allows drones to fly freely in it. The adaptability of the agent is enhanced through reasonable parameter settings. The motion model can include acceleration control, speed control, and a simplified dynamics model to simulate the position and speed changes of the agent in the two-dimensional space. The cooperation and confrontation strategies of the agent are also reflected in the environment, such as target allocation, cooperation rewards, and behavior modeling of hostile agents. By real-time monitoring of the state and task progress of the agent, strategy analysis and optimization are facilitated. These settings ensure that the agent can perform tasks efficiently in a dynamic and complex simulation environment.

[0040] II. Model the intelligent agent in the MaCA simulation platform. First, each intelligent agent needs to be initialized. This includes setting the initial position and speed of the intelligent agent, and building an independent strategy network and value network for it. Figure 2 shown.

[0041] The design of the state space of the agent includes several key factors. The state space contains not only the position and speed of the drone, but also the status of neighboring drones and relevant information about the target area. For example, the agent needs to understand its relative position to other agents and the status of the target island. This data helps the agent make effective decisions, adjust flight strategies, and cooperate or compete.

[0042] The action space defines the behaviors that the drone can take. Each agent can choose the direction of movement, speed control, and even attack or evasion behavior. The direction of movement and speed control determine the flight trajectory of the drone in three-dimensional space, while attack and evasion behaviors mainly make decisions on hostile targets or defense strategies. The design of these action spaces ensures that the agents can perform a variety of operations to adapt to different mission requirements.

[0043] III. Introduction of the island capture scenario. The random actions of the attack aircraft are generated mainly based on the relative position between the attack aircraft and the target island. First, the distance between the attack aircraft and the center of the island is calculated to determine whether it is close to the island. If the distance is greater than the radius of the island, it means that the attack aircraft has not yet reached the vicinity of the island. At this time, the system will calculate a direction angle pointing to the island and adjust it according to the current direction of the attack aircraft to ensure that the attack aircraft moves towards the island. This adjustment takes into account the current flight trajectory of the attack aircraft, allowing it to approach the target area smoothly. If the attack aircraft is close to the island, the system will randomly generate a steering angle to allow the attack aircraft to maneuver flexibly. After approaching the island, the system will create an array to represent the probability distribution of different attack targets. This array generates random numbers and normalizes them to a sum of 1 to ensure that the distribution weights of each attack target are reasonable and dynamic.

[0044] The logic for generating random actions for reconnaissance aircraft is similar to that for attack aircraft, but there are some differences. The goal of the reconnaissance aircraft is to conduct reconnaissance as close to the target island as possible, rather than to attack. When the reconnaissance aircraft is far away from the island, the system also calculates the direction to the island and adjusts the orientation of the reconnaissance aircraft to ensure that the reconnaissance aircraft flies towards the target island. Once the reconnaissance aircraft approaches the island, the system no longer instructs the reconnaissance aircraft to perform attack actions, but instead switches to reconnaissance or detour missions. The actions of the reconnaissance aircraft can include tasks such as circling the island and long-range reconnaissance to collect more environmental information. Therefore, the action strategy of the reconnaissance aircraft focuses more on the exploration and information collection of the target area, rather than direct confrontation with the enemy.

[0045] In this way, the random action generation method of attack aircraft and reconnaissance aircraft can be dynamically adjusted in the game environment according to different task requirements, which can not only reflect the role differences between the two in the game, but also increase the diversity and challenge in the simulation environment.

[0046] IV. Reward function design. The reward mechanism is the core of agent learning. It evaluates and provides feedback based on the agent's behavior. For example, behaviors such as defeating enemy drones, detecting enemy drones, staying alive, and avoiding attacks will receive positive rewards, while being destroyed by enemy drones will receive penalties. Through these reward and penalty mechanisms, the agent can continuously optimize its behavior strategy and achieve better performance in the task.

[0047] The average reward value is an important indicator for evaluating the overall strategy effect in a multi-agent system. It measures the performance of the entire system in each round by dividing the total reward obtained by the agent in each round by the number of participating agents. Through this indicator, the optimization effect of the algorithm in terms of task goal achievement, resource utilization, game confrontation, etc. can be comprehensively evaluated. The reward value settings are shown in Table 1 below.

[0048] Table 1 Reward value settings

[0049] Specifically, the average reward value is calculated by dividing the sum of the rewards of all agents in each round by the number of agents to obtain the average reward value for each round. This value reflects the comprehensive performance of the drone group in a round and can help determine whether the algorithm can effectively coordinate the behavior of the drone group and achieve better overall results.

[0050]

[0051] in, is the number of agents, It is The average reward per round is calculated by summing up the rewards of all agents and dividing by the number of agents.

[0052] In a multi-agent system, the average reward value directly reflects the collaborative effect of the agents. A higher average reward value usually means good collaboration or confrontation, which helps achieve the mission objectives. For example, in a drone confrontation mission, a higher average reward value indicates that the agents successfully collaborate or confront, achieving goals such as occupying islands, avoiding attacks, and higher combat power preservation rates. In a sparse reward environment, the frequency of agents obtaining effective rewards is low, and the learning process is limited. At this time, the average reward value can measure the learning ability of the agent under limited feedback, reflecting whether it can gradually overcome the challenge of sparse rewards through experience accumulation and strategy optimization.

[0053] By using By using methods such as return, the agent is better able to infer the optimal strategy from the limited reward signal, thereby improving its long-term performance. As training progresses, the gradual increase in the average reward value indicates that the agent gradually learns how to optimize its behavior in sparse reward scenarios.

[0054]

[0055] in, is based on of Step back helps the agent to better optimize its strategy and effectively adapt to sparse reward scenarios.

[0056] V. Design of the MADDPG algorithm with the Rainbow algorithm module. The MADDPG algorithm is the core of this method, allowing each agent to use global information during training and only use its own observations during execution. The algorithm learns the agent's strategy through independent policy networks and value networks. In this implementation, the MADDPG class maintains the agent's policy network and value network.

[0057] In the initialization process of the MADDPG class, network creation is a key step. The Actor method and Critic method are used to create the policy network and value network respectively.

[0058] The Actor network is used to generate actions. Each agent has its own Actor network, because different agents may differ in observation space and action space. The output of the Actor network is an action, which is usually generated by inputting the current state and using a deep learning algorithm to generate a policy. The target network has the same structure as the main network, but is used to generate target Q values ​​during training, thereby providing training stability.

[0059] The Critic network is used to evaluate the value of a given state and action. It outputs a Q value, which represents the expected reward of taking a specific action in a specific state. The Critic network guides the training of the Actor network by evaluating the value of the current policy. By minimizing the critic loss, the performance of the policy can be improved.

[0060] The creation and initial settings of the target networks (actor_target_network and critic_target_network) are the same as the main network. The target network is a replica used to stabilize the learning process and reduce the instability of training by delaying updates. The target network does not directly participate in parameter updates in each training step, but gradually approaches the parameters of the main network through soft updates after each training. This strategy helps to smooth the learning process and avoid drastic fluctuations. The parameters of the actor and critic networks are optimized using the Adam optimizer.

[0061] However, MADDPG has the problems of high complexity, instability, and difficulty in handling sparse rewards during training. Therefore, in order to further improve the performance of the MADDPG algorithm, the core technology of the Rainbow algorithm can be introduced. The Rainbow algorithm combines multiple reinforcement learning technologies, which can greatly improve the stability of learning, sample efficiency, and strategy diversity. The experience replay mechanism in the Rainbow algorithm is based on the size of the TD error, and prioritizes experiences with larger errors for replay. After the introduction of priority experience replay in MADDPG, the sample efficiency can be improved, allowing the agent to learn from key experiences more quickly, especially in high-dimensional state space, which can effectively accelerate convergence. The Rainbow algorithm updates the Q value by considering rewards for multiple time steps, rather than relying solely on the immediate reward at the current moment. This method can make the learning process more stable and reduce the negative impact of reward delays. And by adding behavioral cloning technology, the problem of difficulty in convergence in the early stage of training is solved. The introduction of multi-step learning in MADDPG enables the agent to learn effective strategies more efficiently when facing long-term reward problems. The structure diagram is as follows Figure 3 shown.

[0062] VI. System training obtains the overall system parameters and environmental parameters, and randomly samples the experience buffer for learning. A total of 8,000 rounds of training are performed, and the performance of the agent is evaluated every 300 rounds of training, and the evaluation results and charts are saved.

[0063] VII. Evaluation of training results. Figure 4 As shown in the figure, it is a schematic diagram of the game between the red and blue drones to seize the island. Figure 5 and Figure 6 The figure shows the comparison of the average reward value and combat power preservation rate of the MADDPG algorithm and the MADDPG algorithm with the Rainbow algorithm module. After the training is completed, the reward return value of the training process is obtained. Then the trained model is used for testing and the combat power preservation rate is recorded. Assume that the team has a total of A drone was destroyed after a game of island seizure. The combat power preservation rate is: Combat Power Preservation Rate = ×100% In summary, this application proposes a drone game confrontation island capture method based on a multi-agent reinforcement learning algorithm, aiming to enhance the collaboration and confrontation capabilities of multiple agents in a complex game environment. By designing an independent strategy network and value network for each agent, and combining the reward mechanism, state space and action space settings, the agent behavior optimization in tasks such as island capture, attack, and detection is achieved. At the same time, the Rainbow algorithm is introduced to optimize MADDPG, which enhances training stability and strategy exploration capabilities. This method can be widely used in drone group combat, intelligent transportation and other fields, and has strong practical application potential.

[0064] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0065] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

[0066] Through the above-mentioned drone game confrontation island capture method based on multi-agent reinforcement learning algorithm, on the one hand, the Rainbow algorithm module is introduced into the MADDPG algorithm, such as priority experience replay, multi-step TD update, etc., and behavior cloning is used to accelerate the convergence speed in the early stage of training. The combination of these technologies significantly improves the sample utilization efficiency and the stability of the strategy, enabling the agent to achieve excellent performance in a more complex task environment. On the other hand, a reward function combining the reward return value and the combat power preservation rate is designed, which can not only effectively guide the agent to complete the island capture task goal, but also encourage the agent to optimize resource utilization and reduce combat losses during the task. Such a reward mechanism can dynamically evaluate the task completion effect, strategy efficiency and resource consumption balance, and improve the strategy learning ability of the agent. The introduction of combat power preservation rate provides a new optimization dimension for reinforcement learning tasks. This indicator evaluates the survivability of the agent in the task in real time. By combining with the reward function, the agent can minimize resource consumption while completing the task and maximize the benefits of the island capture task. On the other hand, special optimization is carried out for the complex confrontation game task of drone island capture. By simulating confrontation and collaboration scenarios, a multi-agent strategy framework that can balance collaboration and competition is designed, enabling the agents to demonstrate excellent learning and decision-making capabilities in dynamic environments, making up for the shortcomings of existing technologies in similar scenarios.

[0067] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0068] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A drone game confrontation island capture method based on a multi-agent reinforcement learning algorithm, characterized in that: The method includes: Step S1, construct a multi-agent simulation environment: Based on the UAV game confrontation platform MaCA, the motion model, perception range and dynamic environment parameters of the intelligent agent are set to simulate the collaboration and confrontation scenarios of the intelligent agent in two-dimensional or three-dimensional space; Step S2, agent modeling and initialization: Assign initial position, speed and task role to each agent, and build independent strategy network and value network based on MADDPG algorithm; The policy network generates action instructions based on the state space of the agent, which includes position, speed, state of neighboring agents, and target area information; The value network evaluates the expected return of action instructions and optimizes the strategy based on global information; Step S3, design an average reward mechanism: The average reward value is dynamically calculated according to the agent type and mission objectives. The agent types include reconnaissance aircraft and attack aircraft, and the reward values ​​include successful attack reward, effective detection reward, survival reward, and destruction penalty. Step S4, strategy optimization and training: The MADDPG algorithm is used for multi-agent strategy learning, and the priority experience replay mechanism of the Rainbow algorithm is introduced to dynamically adjust the sample sampling weight according to the TD-error; Combined with the multi-step TD update mechanism, the agent's ability to perceive long-term rewards; Initialize the policy network through behavioral cloning technology to solve the problem of convergence difficulties in the early stage of training; Step S5, combat power preservation rate evaluation: After the confrontation task is completed, the combat power preservation rate is calculated based on the ratio of the number of surviving agents to the initial total number, and the strategy performance is comprehensively evaluated in combination with the reward return value; Step S6, iterative optimization and deployment: The network parameters are updated through multiple rounds of training to generate the final strategy model and apply it to the actual drone game against island capture missions.

2. The method for island capture by drone game based on multi-agent reinforcement learning algorithm according to claim 1 is characterized in that: In step S3, the sum of the rewards of all agents in each round is divided by the number of agents to obtain the average reward value of each round The average reward value is calculated as: in, is the number of agents, For the The reward of each agent in this round.

3. The method for capturing islands by drone game based on multi-agent reinforcement learning algorithm according to claim 1 is characterized in that: The introduction of the Rainbow algorithm in step S4 specifically includes: Use the soft update mechanism of the target network and the main network to reduce the strategy fluctuation during training; Improve strategy diversity and avoid local optimal solutions through distributed Q-value estimation.

4. The method for island capture by drone game based on multi-agent reinforcement learning algorithm according to claim 1 is characterized in that: The differentiated reward mechanism is as follows: The reward function of the attack aircraft includes the reward for destroying the enemy target, the reward for occupying the target area, and the reward for its own survival; The reward function of the reconnaissance aircraft includes the detection reward for enemy targets, the information collection reward for the target area, and the survival reward for avoiding attacks.

5. The method for island capture by drone game based on multi-agent reinforcement learning algorithm according to claim 1 is characterized in that: The calculation formula of combat power preservation rate is: Combat Power Preservation Rate = ×100% Among them, m is the total number of initial friendly agents, and n is the number of surviving friendly agents; And combined with the enemy's combat power loss rate, dynamically adjust the strategy optimization direction.

6. The method for capturing islands by drone game based on multi-agent reinforcement learning algorithm according to claim 1 is characterized in that: Behavioral cloning technology specifically includes: Generate initial training data using expert strategies and pre-train the strategy network through supervised learning; In the reinforcement learning process, the weight of cloned data is gradually reduced to transition to autonomous strategy learning.

7. The method for island capture by drone game based on multi-agent reinforcement learning algorithm according to claim 1 is characterized in that: The multi-step TD update mechanism has a step size range of 3-5 steps, which is used to balance the weights of immediate rewards and long-term returns.

8. The drone game confrontation island capture method based on multi-agent reinforcement learning algorithm according to claim 1 is characterized in that: Electronic interference, multi-band radar detection and passive collaborative positioning functions are also simulated in the simulation environment to enhance the realism of the confrontation scenario.

Images