A hybrid intelligent-based cooperative confrontation control method for a drone cluster

Abstract

The application discloses a kind of unmanned aerial vehicle cluster cooperative confrontation control method based on hybrid intelligence, comprising: step 1, initialize unmanned aerial vehicle cluster system, configure the initial state and action space of unmanned aerial vehicle;Step 2, establish the task decision rule base based on scene, set corresponding rule trigger and decision rule;Step 3, obtain the environment state and the observation information of each unmanned aerial vehicle at the current time;Step 4, analyze current state, evaluate current scene according to rule trigger, generate task decision;Step 5, generate maneuver control action based on the observation information of each unmanned aerial vehicle and task decision, each unmanned aerial vehicle independently generates action;Step 6, record the decision execution process of unmanned aerial vehicle cluster;Step 7, repeat step 3-step 6 until the number of samples in experience replay pool meets the training requirement;Step 8, update network parameters;Step 9, when completion rate reaches preset threshold, training is completed, save model parameters, otherwise repeat step 3-step 8.

Description

Technical Field

[0001] This invention relates to unmanned aerial vehicle (UAV) swarm control, and more particularly to a UAV swarm cooperative countermeasure control method based on hybrid intelligence. Background Technology

[0002] With the widespread application of drone technology in industries such as industry, agriculture, and surveying, drone swarm systems are attracting increasing attention. In complex environments, drone swarms need to cope with various interferences from the external environment, which places higher demands on the intelligence level of the swarm control system. Traditional drone swarm control methods mainly rely on manual remote operation and preset programs, which often fall short when facing complex and ever-changing environments.

[0003] Currently, control methods based on prior knowledge remain the primary means of controlling UAV swarms. These methods analyze environmental conditions and formulate corresponding strategies, including path planning, altitude adjustment, speed control, target selection, and mission decision-making. While this approach ensures system safety and reliability, its adaptability to the environment is limited, making it difficult to guide the swarm to achieve optimal collaborative states.

[0004] Reinforcement learning, as an important branch of artificial intelligence, has shown great potential in the field of drone control in recent years. However, applying reinforcement learning directly to drone swarm control faces many challenges: First, there is the safety issue, as unpredictable behavior may occur during the learning process; second, there is the convergence issue, as the complex environmental states and action spaces make it difficult for the learning process to converge effectively; and finally, there is the coordination issue, as the state space of multi-drone systems increases dramatically, making it difficult to achieve effective coordinated control.

[0005] In drone swarm collaboration scenarios, several challenges remain: how to leverage reinforcement learning to optimize the overall swarm state, making it more conducive to executing higher-level task strategies; how to improve control adaptability while ensuring system safety; and how to achieve collaborative cooperation among multiple drones. Therefore, developing a hybrid control method that integrates prior knowledge and intelligent learning has significant practical implications. Summary of the Invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide a method for collaborative control of unmanned aerial vehicle (UAV) swarms based on hybrid intelligence, which addresses the shortcomings of the existing technology.

[0007] To address the aforementioned technical problems, this invention discloses a method for cooperative adversarial control of unmanned aerial vehicle (UAV) swarms based on hybrid intelligence, comprising:

[0008] Step 1: Initialize the UAV swarm system in the simulation environment through the simulation platform module, and configure the initial state and action space of the UAVs;

[0009] Step 2: The expert strategy module establishes a scenario-based task decision rule base and sets corresponding rule triggers and decision rules;

[0010] Step 3: Obtain the current environmental state and observation information of each UAV through the simulation platform;

[0011] Step 4: The expert strategy module analyzes the current state, evaluates the current scenario based on rule triggers, and generates task decisions;

[0012] Step 5: The UAV decision network generates maneuver control actions based on the observation information of each UAV (i.e., the relevant information that can be obtained within the UAV's field of vision) and the task decisions generated by expert strategies. Each UAV independently generates actions using a distributed Actor network to ensure the real-time performance of the system.

[0013] Step 6: Execute the task decision in Step 4 and the control actions selected by each UAV in Step 5 in the simulation platform. The simulation platform records the decision execution process of the UAV cluster, including state transition, observation update, and reward calculation, and stores this data in the experience replay pool for subsequent training.

[0014] Step 7: Store the transferred samples in the experience replay pool. When the number of samples in the experience replay pool meets the training requirements, collect sample data for training. Otherwise, repeat steps 3-6.

[0015] Step 8: The network optimizer updates the network parameters using the MAPPO algorithm;

[0016] Step 9: Evaluate the task completion rate. When the completion rate reaches the preset threshold, training is complete, and the model parameters are saved. Otherwise, repeat steps 3-8.

[0017] The simulation platform module builds and updates the mission environment to simulate the actual mission scenarios of the drone swarm.

[0018] The expert strategy module includes a preset task decision rule library. Based on different target characteristics and task requirements, it sets corresponding strategy rules to determine the overall task objectives and allocation scheme.

[0019] The UAV decision network is trained using a multi-agent reinforcement learning algorithm and generates maneuver control actions based on task decisions generated from local observation information and expert strategies.

[0020] The experience replay pool stores transfer samples generated by the drone swarm during mission execution;

[0021] The network optimizer updates the parameters of the UAV decision network based on data samples collected from the experience replay pool.

[0022] The expert strategy module includes:

[0023] Rule triggers and decision rule base;

[0024] The rule trigger assesses scenario characteristics based on the current environmental state and activates the corresponding task decision rule when the triggering condition is met.

[0025] The decision rule base contains decision trees for different scenarios and generates task decision instructions based on the evaluation results of rule triggers.

[0026] The UAV decision-making network adopts a centralized training and distributed execution framework, including:

[0027] Centralized Critic networks and distributed Actor networks;

[0028] The centralized Critic network evaluates the state value based on the global state, local observation information from each UAV, and mission decisions.

[0029] The distributed Actor network independently generates the maneuver decision probability distribution for each UAV based on its local observation information and mission decisions.

[0030] The centralized Critic network includes:

[0031] Main network and target network;

[0032] The main network evaluates the value function in the current state;

[0033] The target network has the same structure as the main network, calculates the target value, and copies network parameters from the main network at intervals of training.

[0034] The network optimizer updates network parameters using the MAPPO algorithm, including:

[0035] Strategy optimization loss function and value optimization loss function;

[0036] The strategy optimizes the loss function, which is used to optimize the Actor network parameters;

[0037] The value optimization loss function is used to optimize the parameters of the Critic network.

[0038] In the simulation environment of Step 1, each UAV in the UAV swarm acquires state information about the current environment through its sensor system, including position, velocity, and target information. This information is processed to form local observation data input to the UAV decision-making network. The observation information from each UAV is then integrated for task decision-making by the expert strategy. The relevant information acquired by each UAV locally through sensors generally includes all relevant information from all UAVs.

[0039] The design of the rule triggers in step 2 includes: T1 = f1(D), a target allocation trigger T1 based on the target distance matrix D; T2 = f2(D,e), an interference decision trigger T2 based on the target distance matrix D and the energy state vector e of the friendly UAV cluster. When the triggering condition is met, the corresponding task decision rule is activated and the corresponding task decision instruction is generated.

[0040] Step 8, updating network parameters using the MAPPO algorithm, specifically includes:

[0041] Calculate the importance sampling ratio and the advantage function based on the value assessment of the centralized Critic network;

[0042] For the Actor network, the parameters are updated by optimizing the loss function through minimizing the policy;

[0043] For the Critic network, the parameters are updated by optimizing the loss function to minimize the value.

[0044] The key feature is that the expert strategy module in step 4 includes a preset task decision rule library. Based on the target characteristics and task requirements, corresponding strategy rules are set to determine the overall task objectives and allocation schemes. The rule library contains decision trees for different scenarios, which can quickly match applicable task strategies. These strategies will serve as behavioral constraints for the MAPPO algorithm to ensure the reliability of the decision.

[0045] This method, based on a prior policy that determines the task objective, optimizes the collaborative state of the swarm through multi-agent reinforcement learning, thereby achieving precise control of the UAV swarm. This invention enhances the system's environmental adaptability through reinforcement learning while ensuring the reliable execution of the basic policy.

[0046] Beneficial effects:

[0047] This invention applies reinforcement learning to cluster control, enhancing the system's environmental adaptability while ensuring the safety of the underlying system. Scenario-based prior policies guide task objectives, while reinforcement learning optimizes the cluster's collaborative state, significantly improving overall system performance. Through a framework design combining centralized training and distributed execution, fast and accurate real-time control is achieved. Attached Figure Description

[0048] Figure 1 This is an architecture diagram of the drone swarm collaborative control system of the present invention.

[0049] Figure 2 This is a flowchart of the UAV swarm hybrid intelligence method of the present invention.

[0050] Figure 3 This is a schematic diagram of a collaborative task scenario for the simulation platform of the present invention. Detailed Implementation

[0051] This invention proposes a hybrid intelligent control method that integrates prior policies and reinforcement learning. Based on the task objectives determined by the prior policies, this method optimizes the collaborative state of the swarm through multi-agent reinforcement learning, achieving precise control of the UAV swarm. This invention enhances the system's environmental adaptability through reinforcement learning while ensuring the reliable execution of the basic policies.

[0052] The core technology of this invention lies in making high-level task decisions through a priori strategy module, while using reinforcement learning to optimize the collaborative state of the cluster, thereby improving the overall performance of the cluster.

[0053] To address the aforementioned technical problems, this invention proposes a hybrid intelligent control method that integrates prior policies and reinforcement learning. Based on the task objectives determined by the prior policies, this method optimizes the collaborative state of the swarm through multi-agent reinforcement learning, achieving precise control of the UAV swarm. This invention enhances the system's environmental adaptability through reinforcement learning while ensuring the reliable execution of the basic policies.

[0054] A method for cooperative adversarial control of unmanned aerial vehicle (UAV) swarms based on hybrid intelligence, comprising:

[0055] Step 1: Initialize the UAV swarm system in the simulation environment through the simulation platform module, including setting the initial parameters such as the position, speed, and heading of each UAV, configuring the environmental elements and constraints related to the mission, and configuring the initial state and action space of the UAVs.

[0056] Step 2: The expert strategy module establishes a scenario-based task decision rule base, setting corresponding rule triggers and decision rules. Based on different environmental scenarios, it sets corresponding behavior triggers and writes behavior triggering conditions and decision rules. A rule trigger should include: a triggering scenario and several behavior triggers. Behavior triggers then include more specific triggering conditions and the specific behavior decision after triggering.

[0057] Step 3: Obtain the current environmental status and observation information of each UAV through the simulation platform, including the position, speed, altitude, sensor data, target information and other environmental parameters of each UAV;

[0058] Step 4: The expert strategy module analyzes the current state, evaluates the current scenario based on rule triggers, and generates task decisions. The expert strategy module makes task-level decisions based on the current state, analyzes the environmental scenario and matches the corresponding strategy rules, determines the overall behavioral goal of the cluster, and outputs specific task decision instructions. This decision will serve as the behavioral constraint for the MAPPO algorithm.

[0059] Step 5: The UAV decision network generates maneuver control actions based on the observation information of each UAV and the task decisions generated by expert strategies. Each UAV independently generates actions using a distributed Actor network to ensure the real-time performance of the system. The UAV decision network includes modules such as state encoding, action generation, and value estimation.

[0060] Step 6: Execute the task decision in Step 4 and the control actions selected by each UAV in Step 5 in the simulation platform. The simulation platform records the decision execution process of the UAV cluster, including state transition, observation update, and reward calculation, and stores this data in the experience replay pool for subsequent training.

[0061] Step 7: Store the transferred samples in the experience replay pool. When the number of samples in the experience replay pool meets the training requirements, collect sample data for training. Otherwise, repeat steps 3-6.

[0062] Step 8: The network optimizer uses the MAPPO (Multi-Agent Proximal Policy Optimization) algorithm to update the network parameters. By minimizing the policy loss function, it optimizes the control policy of the UAV, so as to improve the collaborative performance of the cluster while ensuring the execution of expert policies.

[0063] Step 9: Evaluate the task completion rate. When the completion rate reaches the preset threshold, training is complete, and the model parameters are saved. Otherwise, repeat steps 3-8.

[0064] Example:

[0065] According to the appendix Figure 1 As shown, this embodiment provides a hybrid intelligent control method that integrates prior policies and reinforcement learning. The method mainly consists of core modules such as a simulation platform, an expert policy module, an UAV decision network, a collaborative optimization module, an experience replay pool, and a network optimizer.

[0066] The simulation platform module is used to build and update the mission environment. Within this simulation environment, each UAV in the swarm acquires state information about the current environment through its sensor system, including position, velocity, and target information. This information is processed to form local observation data suitable for input into the decision-making network. Furthermore, the observation information from each UAV is integrated for mission decision-making using expert strategies.

[0067] The expert strategy module contains a pre-defined task decision rule base. Based on different target characteristics and task requirements, corresponding strategy rules are set to determine the overall task objectives and allocation schemes. The rule base includes decision trees for different scenarios, enabling rapid matching of applicable task strategies. These strategies will serve as behavioral constraints for the reinforcement learning module, ensuring the reliability of decisions.

[0068] The UAV decision-making network is trained using the MAPPO algorithm, a multi-agent reinforcement learning method. During the training phase, a centralized evaluation network guides the optimization of each UAV's strategy; during the execution phase, a distributed Actor network makes independent decisions to ensure the system's real-time performance.

[0069] like Figure 2 As shown:

[0070] Step 1: Initialize the UAV swarm system in the simulation environment. This embodiment uses a simulation platform developed based on QT to configure the initial state of 4 UAVs, including their position coordinates (x, y, y). i ,y i , z i) Velocity components The parameters are as follows: the specific task is for our four UAVs to conduct jamming countermeasures against the enemy's four UAVs. The UAV's maneuver control action space includes basic maneuvers such as level flight, acceleration, deceleration, turning, climbing, and diving. The mission decision space includes higher-level decision actions such as target selection and jamming decisions, i = 1, 2, 3, 4;

[0071] Step 2: Establish a scenario-based expert strategy library: Set corresponding rule triggers and decision rules according to different environmental scenarios and task types. The design of the rule triggers includes: T1 = f1(D), a target allocation trigger based on the target distance matrix D; T2 = f2(D,e), an interference decision trigger based on the target distance matrix D and the energy state vector e of the friendly UAV cluster. When the triggering condition is met, the corresponding task decision rule is activated, and the corresponding task decision instruction is generated.

[0072] Step 3: Obtain the environmental state S at time t using the simulation platform. t and observation information from various drones

[0073] Step 4: The expert strategy module analyzes the current state S. t Based on the rule triggers defined in step 2, the current scenario is evaluated. When the conditions of rule triggers T1 and T2 are met, the expert strategy module generates the corresponding task decision set a. task ={a target ,a act}, where a target For the allocation of targets to the friendly drone swarm, a act To make interference decisions for our own drones;

[0074] Step 5: The decision-making network of each UAV is based on observation information. Task decision generated by expert strategy a task Generate motor control actions

[0075] The system employs a centralized training-distributed learning framework to achieve multi-agent collaborative training. During the training phase, a centralized Critic network is used. Integrate global state, observation information from all UAVs, and high-level mission decisions to assess state value. in, For Critic Network The parameters; to achieve stable training, a structure is also maintained. Same target network Used to calculate target value For target network The parameters, from each training iteration at intervals... The network parameters are replicated, and during the execution phase, each drone uses a distributed Actor network. Action probability distribution is generated independently based on local observations and task decisions. Selecting the optimal maneuver decision for each UAV Where, θ i For Actor Network Parameters;

[0076] Step 6: Execute task decisions and control actions to obtain the new state S. t+1 and new observation information from various UAVs Simultaneously calculate the reward R at time t. t The rewards include penalties for deviating from the mission area, penalties for drone interference, penalties for mission failure, rewards for successful interference, and rewards for mission completion. The specific reward functions are as follows:

[0077]

[0078] Step 7: Transfer the sample Store the data in the experience replay pool P, i = 1, 2, 3, 4. When |P| ≥ N, collect a batch of N sample data from the replay pool for training, where N is the batch size of the samples required for training, and |P| represents the number of samples in the experience replay pool P; otherwise, repeat steps 3-6 to continue collecting data.

[0079] Step 8: Update network parameters using the MAPPO algorithm. During the centralized training phase, the policy optimization loss function for the Actor network of each drone i is:

[0080]

[0081] Where, θ i For Actor Network The parameters are: E represents the expected value, and clip(·) is the clipping function.

[0082] in, Importance sampling ratio:

[0083]

[0084] in, This represents the Actor network before its last update.

[0085] in, The advantage function is calculated based on the value assessment of the centralized Critic network:

[0086]

[0087] Where ε = 0.1 is the clipping parameter, H(·) is the entropy function, λ = 0.01 is the entropy regularization coefficient, and γ = 0.95 is the decay factor.

[0088] The value optimization loss function for a centralized Critic network is:

[0089]

[0090] in, For Critic Network The parameters.

[0091] By minimizing the total loss function, the system optimizes the cluster's collaborative performance while ensuring the effectiveness of task execution.

[0092] Step 9: Evaluate the completion rate η of the most recent 50 tasks; when η≥90%, training is complete, save the model parameters, and each drone only needs to deploy the Actor network to execute; otherwise, repeat steps 3-8.

[0093] This embodiment achieves reliable task decision-making and flexible maneuver control by integrating expert strategies and reinforcement learning. Expert strategies ensure the reliability of critical task decisions, while multi-agent reinforcement learning algorithms optimize the collaborative control performance of the cluster. The centralized training and distributed execution framework guarantees both training effectiveness and meets the requirements of real-time execution.

[0094] This invention provides a method for cooperative countermeasure control of unmanned aerial vehicle (UAV) swarms based on hybrid intelligence. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for cooperative adversarial control of unmanned aerial vehicle (UAV) swarms based on hybrid intelligence, characterized in that, include: Step 1: Initialize the UAV swarm system in the simulation environment through the simulation platform module, and configure the initial state and action space of the UAVs; Step 2: The expert strategy module establishes a scenario-based task decision rule base and sets corresponding rule triggers and decision rules; Step 3: Obtain the current environmental state and observation information of each UAV through the simulation platform; Step 4: The expert strategy module analyzes the current state, evaluates the current scenario based on rule triggers, and generates task decisions; Step 5: The UAV decision network generates maneuver control actions based on the observation information of each UAV and the task decisions generated by the expert strategy module. Each UAV independently generates actions using a distributed Actor network. Step 6: Execute the task decision in Step 4 and the control actions selected by each UAV in Step 5 in the simulation platform. The simulation platform records the decision execution process of the UAV cluster, including state transition, observation update, and reward calculation, and stores this data in the experience replay pool for subsequent training. Step 7: Store the transfer samples generated by the drone cluster during the mission execution into the experience replay pool. When the number of samples in the experience replay pool meets the training requirements, collect sample data for training. Otherwise, repeat steps 3-6. Step 8: The network optimizer updates the network parameters using the MAPPO algorithm; Step 9: Evaluate the completion rate of the most recent tasks. When the completion rate reaches the preset threshold, training is complete and the model parameters are saved. Otherwise, repeat steps 3-8. In step 4, the expert strategy module contains a preset task decision rule library. Based on the target characteristics and task requirements, it sets corresponding strategy rules to determine the overall task objectives and allocation schemes. The rule library contains decision trees for different scenarios, which can quickly match applicable task strategies. This strategy will serve as the behavioral constraint for the MAPPO algorithm.

2. The method for cooperative countermeasure control of unmanned aerial vehicle (UAV) swarms based on hybrid intelligence according to claim 1, characterized in that, The simulation platform module builds and updates the mission environment to simulate the actual mission scenarios of the drone swarm. The expert strategy module includes a preset task decision rule library. Based on different target characteristics and task requirements, it sets corresponding strategy rules to determine the overall task objectives and allocation scheme. The UAV decision network is trained using a multi-agent reinforcement learning algorithm and generates maneuver control actions based on task decisions generated from local observation information and expert strategies. The experience replay pool stores transfer samples generated by the drone swarm during mission execution; The network optimizer updates the parameters of the UAV decision network based on data samples collected from the experience replay pool.

3. The method for cooperative countermeasure control of unmanned aerial vehicle (UAV) swarms based on hybrid intelligence according to claim 2, characterized in that, The expert strategy module includes: Rule triggers and decision rule base; The rule trigger assesses scenario characteristics based on the current environmental state and activates the corresponding task decision rule when the triggering condition is met. The decision rule base contains decision trees for different scenarios and generates task decision instructions based on the evaluation results of rule triggers.

4. The method for cooperative countermeasure control of unmanned aerial vehicle (UAV) swarms based on hybrid intelligence according to claim 2, characterized in that, The UAV decision-making network adopts a centralized training and distributed execution framework, including: Centralized Critic networks and distributed Actor networks; The centralized Critic network evaluates the state value based on the global state, local observation information from each UAV, and mission decisions. The distributed Actor network independently generates the maneuver decision probability distribution for each UAV based on its local observation information and mission decisions.

5. The method for cooperative countermeasure control of unmanned aerial vehicle (UAV) swarms based on hybrid intelligence according to claim 4, characterized in that, The centralized Critic network includes: Main network and target network; The main network evaluates the value function in the current state; The target network has the same structure as the main network, calculates the target value, and copies network parameters from the main network at intervals of training.

6. The method for cooperative countermeasure control of unmanned aerial vehicle (UAV) swarms based on hybrid intelligence according to claim 2, characterized in that, The network optimizer updates network parameters using the MAPPO algorithm, including: Strategy optimization loss function and value optimization loss function; The strategy optimizes the loss function, which is used to optimize the Actor network parameters; The value optimization loss function is used to optimize the parameters of the Critic network.

7. The method for cooperative counter-control of unmanned aerial vehicle (UAV) swarms based on hybrid intelligence according to claim 1, characterized in that, In the simulation environment of step 1, each UAV in the UAV swarm acquires the current environmental state information, including position information, speed information, and target information, through the sensor system. After processing, this information forms local observation data that is input into the UAV decision network. The observation information of each UAV is integrated for the task decision of the expert strategy.

8. The method for cooperative countermeasure control of unmanned aerial vehicle (UAV) swarms based on hybrid intelligence according to claim 1, characterized in that, The design of the rule trigger in step 2 includes: Target assignment trigger based on target distance matrix D Interference decision trigger based on target distance matrix D and energy state vector e of friendly UAV swarm When the triggering condition is met, the corresponding task decision rule is activated, and the corresponding task decision instruction is generated.

9. The method for cooperative countermeasure control of unmanned aerial vehicle (UAV) swarms based on hybrid intelligence according to claim 1, characterized in that, Step 8, updating the network parameters using the MAPPO algorithm, specifically includes: Calculate the importance sampling ratio and the advantage function based on the value assessment of the centralized Critic network; For the Actor network, the parameters are updated by optimizing the loss function through minimizing the policy; For the Critic network, the parameters are updated by optimizing the loss function to minimize the value.

Images