A method, device and equipment for cooperative confrontation control of unmanned aerial vehicles, and a storage medium

Abstract

The application provides a kind of unmanned aerial vehicle cooperative counter-control method, device, equipment and storage medium, by acquiring the situation information of multi-unmanned aerial vehicle system, the situation information is preprocessed by situation awareness module to generate standardization state feature;The hierarchical control strategy includes upper layer option decision and bottom layer multi-agent cooperative control, according to the hierarchical control strategy, guide unmanned aerial vehicle group to execute task, adopt priori knowledge rule to execute approach maneuver and surround strategy in initial stage of task, after surrounding is completed, realize upper layer option selection by Option-Critic algorithm, realize bottom layer multi-agent cooperative control by QMix+DQN algorithm;Get the feedback data of unmanned aerial vehicle executing task, process the feedback data by hierarchical reward mechanism to optimize decision strategy and update network parameter, and dynamically adjust option switching condition, to form closed loop optimization, solve the complex decision problem that unmanned aerial vehicle cannot effectively process high-dimensional state space and large-scale action space.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned aerial vehicles, and in particular to a method and device for cooperative confrontation control of unmanned aerial vehicles, and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, cooperative confrontation control of unmanned aerial vehicles has become a core technology of modern military intelligent systems. In a complex multi-agent cooperative combat environment, improving the autonomous decision-making ability and cooperative combat efficiency of unmanned aerial vehicle groups is crucial for the technical advantage of military equipment. However, existing unmanned aerial vehicle control methods often have limitations and cannot fully adapt to the needs of a dynamically changing combat environment.

[0003] Existing unmanned aerial vehicle control relies heavily on traditional manual operation or single rule control, lacks autonomous learning ability, and cannot effectively handle complex decision-making problems in high-dimensional state space and large-scale action space. Although traditional reinforcement learning methods have certain autonomous learning ability, in complex military scenarios with multiple tasks coupled, due to the high number of combat units, high situation dimension, and large action space, there are problems such as slow convergence speed and low sample efficiency, making it difficult to achieve efficient multi-agent cooperative control, limiting its effectiveness in actual military applications.

[0004] In view of the above, the present application is proposed. SUMMARY

[0005] The present application discloses a method and device for cooperative confrontation control of unmanned aerial vehicles, and a storage medium, aiming to solve the problem that unmanned aerial vehicles cannot effectively handle complex decision-making problems in high-dimensional state space and large-scale action space.

[0006] The first embodiment of the present application provides a method for cooperative confrontation control of unmanned aerial vehicles, comprising:

[0007] Obtaining situation information of a multi-unmanned aerial vehicle system, and preprocessing the situation information through a situation awareness module to generate standardized state features, wherein the situation information includes unmanned aerial vehicle position, velocity, attitude data, enemy target position, velocity, threat level data, and battlefield environment boundary, obstacle, missile threat data;

[0008] Constructing a hierarchical control strategy based on the standardized state features, wherein the hierarchical control strategy includes upper-level option decision-making and bottom-level multi-agent cooperative control, the upper-level option decision-making is responsible for selecting among three options of maneuvering, evading, and striking, and the bottom-level multi-agent cooperative control is responsible for executing specific cooperative actions;

[0009] According to the hierarchical control strategy, the UAV group is guided to perform a task, wherein, in an initial stage of the task, a priori knowledge rule is adopted to perform a closing maneuver and a surrounding strategy, after completion of the surrounding, an upper-layer option selection is realized through an Option-Critic algorithm, and a bottom-layer multi-agent collaborative control is realized through a QMix+DQN algorithm;

[0010] Feedback data of the UAV performing the task is acquired, the feedback data is processed through a hierarchical reward mechanism to optimize a decision strategy and update network parameters, and an option switching condition is dynamically adjusted to form a closed-loop optimization.

[0011] Preferably, situation information of the multi-UAV system is acquired, and the situation information is preprocessed through a situation awareness module to generate standardized state features, specifically:

[0012] The original situation data of each UAV i is normalized to generate a standardized feature vector , and the calculation formula is:

[0013]

[0014] wherein, is the original situation data of the UAV i, and are minimum and maximum values of the data, respectively;

[0015] The standardized feature vectors of all UAVs are combined to generate a global state feature , and the expression is:

[0016] [ ]

[0017] wherein, n is the number of UAVs of the own side, is an enemy target feature vector, is an environment feature vector.

[0018] Preferably, a hierarchical control strategy is constructed based on the standardized state features, specifically:

[0019] An upper-layer option decision strategy is constructed , which outputs a current option w to be performed according to the global state feature , wherein w∈{0,1,2} respectively represents a maneuver, evasion, and strike option;

[0020] A bottom-layer multi-agent collaborative control strategy is constructed , which outputs a specific action according to the local observation of the agent i and the upper-layer option instruction w;

[0021] Definition of option termination condition , for judging whether the current option should end and re-decision of upper layer, the calculation formula is:

[0022]

[0023] wherein, sigmoid activation function, termination condition weight matrix, bias vector.

[0024] Preferably, the priori knowledge rule is used to perform the approach maneuver and the surrounding strategy in the initial stage of the task, specifically:

[0025] In the initial stage of the task, the UAV group is controlled to move to the target area according to the predefined approach rule, and the approach rule is: maintaining the formation spacing, moving to the target point along the shortest path, and avoiding the known threat area

[0026] When the surrounding trigger condition is met, the surrounding strategy is started, and the surrounding trigger condition is: the distance between the UAV and the enemy target is less than or equal to the preset surrounding distance threshold, and the threat level of the enemy target is greater than or equal to the preset threat level threshold, both conditions need to be met simultaneously; wherein, the surrounding strategy distributes the UAV to the predetermined position around the target through geometric calculation to form a surrounding circle.

[0027] Preferably, the upper layer option selection is realized by the Option-Critic algorithm after the surrounding is completed, specifically:

[0028] Definition of option value function , represents the expected cumulative reward of selecting the option in state :

[0029]

[0030] wherein, is the cumulative reward from the current time to the termination of the option;

[0031] The upper layer strategy selects the option through the ε-greedy strategy, wherein the selection probability of the optimal option is the preset basic probability plus the average allocation of the exploration probability, and the other options obtain the average allocation of the exploration probability.

[0032] Preferably, the bottom layer multi-agent collaborative control is realized by the QMix+DQN algorithm, specifically:

[0033] Construct a local Q network for each agent i , the input is the local observation , and the output is the Q value of each action, Network regulation parameters

[0034] Building global mixed network Mixing the local Q value of each agent into a global Q value:

[0035]

[0036] wherein, is the observation-action history of agent n, is the action of agent n, is a monotonic mixing function;

[0037] The network parameters are updated using the TD error, and the loss function is:

[0038]

[0039] wherein, , is the immediate reward, is a discount factor for balancing immediate rewards and future rewards, is the next observation-action history, is the next action, is the target network parameter, is the target Q value, is the current global Q value.

[0040] Preferably, it further comprises: before the agent selects an action, filtering the unactionable action through a constraint checking mechanism, the constraint checking mechanism comprising a boundary constraint checking, a collision constraint checking and a fuel constraint checking, and the action that satisfies all constraint conditions at the same time is marked as feasible;

[0041] The boundary constraint checking verifies whether the position of the agent after performing the action is still within the preset valid combat area;

[0042] The collision constraint checking verifies whether the distance between the agent after performing the action and all other agents is greater than a preset safety distance threshold;

[0043] The fuel constraint checking verifies whether the remaining fuel of the agent after performing the action is still greater than a preset fuel reserve threshold.

[0044] The second embodiment of the present application provides a UAV cooperative confrontation control device, characterized by comprising:

[0045] A preprocessing unit is configured to acquire situation information of a multi-UAV system, and generate standardized state features by preprocessing the situation information through a situation awareness module, wherein the situation information includes UAV position, speed, attitude data, enemy target position, speed, threat level data, and battlefield environment boundary, obstacle, and missile threat data;

[0046] A hierarchical control strategy construction unit is configured to construct a hierarchical control strategy based on the standardized state features, wherein the hierarchical control strategy includes upper-layer option decision and bottom-layer multi-agent collaborative control, the upper-layer option decision is configured to select among three options of maneuvering, evading, and attacking, and the bottom-layer multi-agent collaborative control is configured to execute specific collaborative actions;

[0047] An execution unit is configured to guide the UAV group to execute a task according to the hierarchical control strategy, wherein priori knowledge rules are used to execute approach maneuvering and surrounding strategies in an initial stage of the task, the upper-layer option selection is realized through an Option-Critic algorithm after the surrounding is completed, and the bottom-layer multi-agent collaborative control is realized through a QMix+DQN algorithm.

[0048] A feedback unit is configured to acquire feedback data of the UAV executing the task, process the feedback data through a hierarchical reward mechanism to optimize a decision strategy and update network parameters, and dynamically adjust option switching conditions to form a closed-loop optimization.

[0049] The third embodiment of the present application provides a UAV cooperative confrontation control device, which comprises a memory and a processor, the memory stores a computer program, and the computer program can be executed by the processor to realize the UAV cooperative confrontation control method according to any one of the above.

[0050] The fourth embodiment of the present application provides a computer readable storage medium, characterized in that it stores a computer program, and the computer program can be executed by a processor of a device where the computer readable storage medium is located to realize the UAV cooperative confrontation control method according to any one of the above.

[0051] Based on the unmanned aerial vehicle cooperative confrontation control method, device, equipment and storage medium provided by the application, the situation information of a multi-unmanned aerial vehicle system is acquired, the situation information is preprocessed by a situation awareness module to generate standardized state features, a hierarchical control strategy is constructed based on the standardized state features, wherein the hierarchical control strategy includes upper-layer option decision and bottom-layer multi-agent cooperative control, the unmanned aerial vehicle group is guided to perform a task according to the hierarchical control strategy, wherein the priori knowledge rule is adopted to perform a close-in maneuver and a surrounding strategy in the initial stage of the task, the upper-layer option selection is realized by an Option-Critic algorithm after the surrounding is completed, the bottom-layer multi-agent cooperative control is realized by a QMix+DQN algorithm, the feedback data of the unmanned aerial vehicle performing the task is acquired, the feedback data is processed by a hierarchical reward mechanism to optimize the decision strategy and update the network parameters, and the option switching condition is dynamically adjusted to form a closed-loop optimization, so that the problem that the unmanned aerial vehicle cannot effectively process the complex decision of a high-dimensional state space and a large-scale action space is solved. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a flowchart of a kind of unmanned aerial vehicle cooperative confrontation control method provided by the first embodiment of the application;

[0053] Figure 2 is a module schematic diagram of a kind of unmanned aerial vehicle cooperative confrontation control device provided by the second embodiment of the application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0055] In order to better understand the technical solutions of the application, the embodiments of the application will be described in detail below with reference to the drawings.

[0056] It should be clear that the described embodiments are only part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0057] The terms used in the embodiments of the application are only for the purpose of describing the specific embodiments, and are not intended to limit the application. The singular forms "a", "said" and "the" used in the embodiments of the application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0058] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0059] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0060] The "first" and "second" mentioned in the embodiments are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or precedence of "first" and "second" can be interchanged where appropriate. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0061] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0062] The present invention discloses a UAV cooperative confrontation control method, device, equipment and storage medium, aiming to solve the complex decision-making problem that UAVs cannot effectively process high-dimensional state space and large-scale action space.

[0063] See also Figure 1 The first embodiment of the present invention provides a UAV cooperative confrontation control method, which can be executed by a UAV cooperative confrontation control device (hereinafter referred to as the control device), and in particular, is executed by one or more processors in the control device to implement at least the following steps:

[0064] S101, acquiring situation information of a multi-UAV system, and preprocessing the situation information to generate standardized state features using a situation awareness module, wherein the situation information includes UAV position, speed, and attitude data, enemy target position, speed, and threat level data, and battlefield environment boundary, obstacle, and missile threat data;

[0065] In this embodiment, the control device may be a server, which can establish communication with the drone cluster. The control device may be installed with a corresponding operating system and application software, and the functions required by this embodiment can be realized through the combination of the operating system and application software.

[0066] In a specific embodiment of the present invention, the situational awareness module first collects status data such as position coordinates, flight speed vectors, and attitude angles from the sensor system of each drone in real time, and at the same time obtains the real-time position, movement trajectory, and threat level of enemy targets assessed according to target type through radar detection and intelligence fusion. Battlefield environment information is obtained by combining pre-loaded maps and real-time reconnaissance, including the coordinates of the boundary of the combat area, the distribution of static obstacles, and the position and trajectory information of dynamic missile threats. In order to ensure the comparability of different types of data and the stability of network training, the Min-Max normalization method is used to standardize the original situation data of each drone, and the formula is used. Map each data item to a uniform range from 0 to 1, where and After the normalization process is completed, the standardized feature vectors of each drone, the enemy target feature vectors and the environment feature vectors are sequentially spliced ​​in accordance with the predetermined format to form the global state feature vectors. [ ], the global feature vector contains complete information about the current battlefield situation, and serves as a unified input interface for subsequent hierarchical control strategies, ensuring that upper-level option decisions and lower-level collaborative control are based on the same situation understanding.

[0067] S102: Constructing a hierarchical control strategy based on the standardized state characteristics, wherein the hierarchical control strategy includes an upper-level option decision-making layer and a lower-level multi-agent collaborative control layer. The upper-level option decision-making layer is responsible for selecting among the three options of maneuver, evasion, and attack, and the lower-level multi-agent collaborative control layer is responsible for executing specific collaborative actions.

[0068] It should be noted that the hierarchical control strategy adopts a two-layer architecture design, and the upper option decision strategy It receives global state features as input, calculates and outputs the most suitable combat options under the current battlefield situation through deep neural networks. The options are encoded as w∈{0,1,2}, corresponding to the three high-level tactical actions of maneuvering approach, threat avoidance and target strike. Under the constraints of receiving upper-level option instructions, the system generates specific flight control actions in combination with the local observation information of each agent, realizing the mapping from abstract tactical intentions to specific execution actions. In order to ensure the timeliness of option execution, the system designs a dynamic option termination mechanism, which is implemented by formula

[0069] The execution status of the current option is evaluated in real time, where the sigmoid activation function maps the linear combination result to a termination probability between 0 and 1, and the weight matrix and the bias vector Through training, the system learns the optimal execution duration of different options under various circumstances. When the termination probability exceeds a preset threshold, the system automatically terminates the current option and reactivates the upper-level decision-making process, ensuring that the control strategy can promptly adjust tactical behavior based on changes in the battlefield situation.

[0070] S103, guiding the drone swarm to perform the mission according to the hierarchical control strategy, wherein, in the initial stage of the mission, a priori knowledge rules are used to execute the approach maneuver and encirclement strategy. After the encirclement is completed, the upper layer option selection is realized by the Option-Critic algorithm, and the lower layer multi-agent collaborative control is realized by the QMix+DQN algorithm;

[0071] It should be noted that during the initial phase of the mission, a fixed strategy based on prior knowledge is employed to ensure the swarm can efficiently approach the target and establish a favorable position. During the approach maneuver phase, the swarm maintains a standard formation spacing based on predefined expert rules. A path planning algorithm calculates the shortest safe path from the current position to the target area. Simultaneously, a threat assessment module identifies and avoids known enemy defense zones and danger zones, ensuring the safety and concealment of the swarm during the approach. As the swarm approaches the target, the system monitors two key conditions in real time: whether the spatial distance between the drones and the enemy target is within a preset encirclement distance threshold, and whether the enemy target's threat level, calculated by the threat assessment algorithm, meets the minimum requirement for initiating an encirclement operation. Only when both the distance and threat level conditions are met will the encirclement strategy be switched. During the encirclement strategy execution, the system performs geometric calculations based on the target's current position and motion trends, assigning participating drones to tactical positions around the target at preset angular intervals, forming a polygonal encirclement. Each drone adjusts its flight trajectory based on its assigned encirclement position, ensuring effective control of the target from multiple directions.

[0072] Furthermore, in this embodiment, the upper-level option decision-making adopts the Option-Critic algorithm framework to evaluate the long-term benefits of different tactical options by constructing an option value function. The system establishes a value function for each possible option. , which is trained through reinforcement learning to estimate the current global state The expected cumulative reward for executing a specific option ω under in represents the total reward earned from the start of option execution until the termination condition is met, encompassing multiple dimensions such as task completion reward, time efficiency reward, and safety reward. During the option selection phase, the system employs an ε-greedy strategy to balance the use of the learned optimal strategy with the exploration of potentially superior strategies. Specifically, this strategy prioritizes the option with the highest value function evaluation under the current state while retaining the opportunity to randomly select other options with a small probability. This strategy controls the degree of exploration through a preset exploration rate parameter, ε. The optimal option receives a high probability of selection, consisting of a uniform distribution of the base utilization probability plus the exploration probability, while other non-optimal options receive only an average distribution of the exploration probability. This ensures that the system primarily executes the tactical option currently deemed optimal while maintaining its ability to explore and discover superior strategies, enabling continuous improvement and adaptation of upper-level decision-making within a dynamic combat environment.

[0073] In the specific implementation of the present invention, the underlying multi-agent collaborative control adopts the QMix+DQN algorithm framework to handle the collaborative decision-making problem of multiple drones. The system constructs an independent local Q network for each agent. The network takes the local observation information of the agent as input, including its own state, relative target position, relative relationship with teammates, etc., and outputs the value evaluation of the agent's execution of various optional actions under the current observation. In order to achieve effective multi-agent coordination, the system designed a global hybrid network , through the mixing function Combine the local Q value of each agent with the global state information to generate a global collaborative Q value:

[0074]

[0075] in Represents the historical observation-action sequence of agent n. The hybrid function design ensures monotonicity constraints and ensures the consistency between the individual optimal decision and the global optimal decision. The network training process adopts the temporal difference learning method to minimize the loss function To update the network parameters, the target Q value Combining the immediate reward r and the discounted future maximum Q value, the target network parameter Regular replication from the main network improves training stability, allowing each agent to achieve global collaborative optimization while learning its own optimal strategy.

[0076] S104, obtaining feedback data on the drone's mission execution, processing the feedback data through a hierarchical reward mechanism to optimize the decision-making strategy and update network parameters, and dynamically adjusting option switching conditions to form a closed-loop optimization.

[0077] It should be noted that the system has established a comprehensive closed-loop feedback optimization mechanism to continuously improve the performance of the control strategy. The feedback data collection module monitors each UAV's mission execution status in real time, including multi-dimensional information such as option completion status, action execution effectiveness, coordination, and overall mission progress. The hierarchical reward mechanism designs corresponding reward functions based on decision-making objectives at different levels. The upper-level rewards primarily evaluate the correctness of option selection and execution efficiency, with positive rewards awarded when options successfully achieve their intended goals and negative rewards applied when they fail or time out. The lower-level rewards focus on the execution quality of specific actions, including individual task completion, multi-agent coordination, and constraint satisfaction. Network parameter updates utilize a gradient descent method, calculating policy gradients and value function errors based on collected empirical data. Backpropagation is then used to simultaneously optimize the parameters of both the upper-level option decision network and the lower-level action execution network. The system also implements a dynamic adjustment mechanism for option switching conditions. Based on historical execution data, statistical analysis is performed to determine the optimal execution time of different options under various scenarios. The system then adaptively adjusts the weight parameters and threshold settings in the termination conditions, ensuring that option switching more closely matches actual operational requirements. The entire optimization process forms a complete closed loop from execution feedback to strategy improvement to parameter update, ensuring that the control system can continuously learn and improve during continuous mission execution, and gradually enhance the overall effectiveness of multi-UAV collaborative confrontation.

[0078] In one possible implementation of the present invention, the system performs a comprehensive constraint check before each agent selects an action to ensure the safety and feasibility of the selected action. This mechanism employs a three-tiered constraint verification system. First, a boundary constraint check is performed. By predicting the agent's position coordinates after executing a candidate action and comparing them with the preset effective combat zone boundaries, the drone is guaranteed not to fly outside its designated combat range or enter a no-fly zone. A collision constraint check calculates the expected distance between the agent and all other agents after executing the action, ensuring that the minimum separation between any two drones exceeds a preset safety distance threshold, thus avoiding the risk of mid-air collisions caused by coordinated maneuvers. A fuel constraint check estimates the fuel consumption of the candidate action and compares it with the current fuel reserve to ensure that the remaining fuel after executing the action still meets the preset fuel reserve requirement, ensuring that the drone has sufficient fuel to complete the subsequent mission or return safely. Only actions that pass all three constraint checks are marked as feasible and included in the agent's candidate action set. Actions that fail any constraint are directly filtered out. This pre-screening mechanism effectively reduces ineffective exploration during reinforcement learning.

[0079] In one possible implementation of the present invention, the hierarchical reward function design adopts a differentiated evaluation strategy to guide decision optimization at different levels. The upper-level reward mechanism mainly evaluates the overall effect of option execution. When the option successfully completes the expected goal, the system gives a basic success reward and adds a time reward based on the speed of completion to encourage efficient tactical choices; when the option fails to execute, the system imposes a failure penalty and adds an additional time penalty based on the length of execution time to avoid inefficient tactical decisions; when the option execution exceeds the preset maximum time limit, the system directly imposes a timeout penalty to encourage the upper-level strategy to learn the reasonable option switching timing. The bottom-level reward function adopts a multi-dimensional weighted combination method to linearly combine individual task completion rewards, collaborative cooperation rewards and constraint violation penalties according to preset weight coefficients to ensure that each intelligent agent takes into account the team collaboration effect while pursuing individual goals. Individual mission rewards are calculated differently based on the type of option currently being executed. Maneuver options primarily reward approaching the target and penalize distancing too far, while also encouraging actions with a positive approach velocity component. Evasion options prioritize distance from the threat source while controlling fuel consumption, balancing safety and economy. Strike options directly reward successful hits and factor in ammunition costs to improve strike efficiency. Coordination rewards evaluate teamwork effectiveness by calculating the similarity of actions and spatial proximity between agents. Action similarity uses the cosine similarity method to measure the angle between the action vectors of different agents, while positional proximity uses a Gaussian function to quantify the spatial relationship between agents. Closer agents contribute more to the reward through their collaborative actions. This design encourages adjacent agents to execute coordinated actions, fostering effective teamwork.

[0080] See also Figure 2 The second embodiment of the present invention provides a UAV cooperative countermeasure control device, which is characterized by comprising:

[0081] A preprocessing unit 201 is configured to obtain situation information of the multi-UAV system and preprocess the situation information to generate standardized state features through a situation awareness module, wherein the situation information includes the position, speed, and attitude data of the UAVs, the position, speed, and threat level data of the enemy targets, and the boundaries, obstacles, and missile threat data of the battlefield environment;

[0082] A hierarchical control strategy construction unit 202 is configured to construct a hierarchical control strategy based on the standardized state characteristics, wherein the hierarchical control strategy includes an upper-layer option decision-making and a lower-layer multi-agent collaborative control. The upper-layer option decision-making is responsible for selecting among the three options of maneuver, evasion, and attack, and the lower-layer multi-agent collaborative control is responsible for executing specific collaborative actions.

[0083] An execution unit 203 is configured to guide the drone swarm to perform a mission according to the hierarchical control strategy. In the initial phase of the mission, prior knowledge rules are used to execute approach maneuvers and encirclement strategies. After encirclement is completed, the upper-layer option selection is implemented using the Option-Critic algorithm, and the lower-layer multi-agent collaborative control is implemented using the QMix+DQN algorithm.

[0084] The feedback unit 204 is used to obtain feedback data on the drone's mission execution, process the feedback data through a hierarchical reward mechanism to optimize the decision-making strategy and update network parameters, and dynamically adjust the option switching conditions to form a closed-loop optimization.

[0085] A third embodiment of the present invention provides a UAV collaborative confrontation control device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a UAV collaborative confrontation control method as described in any one of the above items.

[0086] A fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, and the computer program can be executed by a processor of the device where the computer-readable storage medium is located to implement a drone collaborative confrontation control method as described in any one of the above items.

[0087] The present invention provides a method, apparatus, device and storage medium for cooperative countermeasure control of unmanned aerial vehicles (UAVs). The method obtains situation information of a multi-UAV system and pre-processes the situation information through a situation awareness module to generate standardized state features. A hierarchical control strategy is constructed based on the standardized state features, wherein the hierarchical control strategy includes upper-level option decision-making and lower-level multi-agent cooperative control. The UAV swarm is guided to perform a task according to the hierarchical control strategy. In the initial stage of the task, a priori knowledge rule is used to execute approach maneuvers and encirclement strategies. After the encirclement is completed, the upper-level option selection is realized through the Option-Critic algorithm, and the lower-level multi-agent cooperative control is realized through the QMix+DQN algorithm. Feedback data of the UAVs performing the task is obtained, and the feedback data is processed through a hierarchical reward mechanism to optimize the decision strategy and update the network parameters. The option switching conditions are dynamically adjusted to form a closed-loop optimization, thereby solving the complex decision-making problem that UAVs cannot effectively handle high-dimensional state spaces and large-scale action spaces.

[0088] For example, the computer programs described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in implementing a drone cooperative countermeasure control device. For example, the device described in the second embodiment of the present invention.

[0089] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the UAV cooperative countermeasure control method, and utilizes various interfaces and lines to connect the various parts of the UAV cooperative countermeasure control method.

[0090] The memory can be used to store the computer programs and / or modules. The processor implements various functions of a method for cooperative countermeasure control of unmanned aerial vehicles by running or executing the computer programs and / or modules stored in the memory and accessing data stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function or a text conversion function); the data storage area may store data generated based on the use of the mobile phone (such as audio data and text message data). Furthermore, the memory may include high-speed random access memory (RAM) and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0091] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the process steps in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0092] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0093] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A UAV cooperative confrontation control method, characterized in that: include: Acquire situational information of the multi-UAV system and pre-process the situational information to generate standardized state features using a situational awareness module, wherein the situational information includes the position, speed, and attitude data of the UAVs; the position, speed, and threat level data of enemy targets; and the boundaries, obstacles, and missile threat data of the battlefield environment; Based on the standardized state characteristics, a hierarchical control strategy is constructed, specifically: constructing an upper-level option decision strategy , the strategy is based on the global state characteristics Output the option w that should be executed currently, where w∈{0,1,2} represents the maneuver, avoidance, and attack options respectively; construct the underlying multi-agent collaborative control strategy , the strategy is based on the local observation of agent i And the upper option command w outputs specific actions ; Define option termination conditions , used to determine whether the current option should be ended and the upper-level decision should be made again. The calculation formula is: ,in, is the sigmoid activation function, is the termination condition weight matrix, is a bias vector, wherein the hierarchical control strategy includes upper-layer option decision-making and lower-layer multi-agent collaborative control. The upper-layer option decision-making is responsible for selecting among the three options of maneuver, avoidance, and attack, and the lower-layer multi-agent collaborative control is responsible for executing specific collaborative actions; Guide the drone swarm to perform the mission according to the hierarchical control strategy. In the initial stage of the mission, the approach maneuver and encirclement strategy are executed using a priori knowledge rules. After the encirclement is completed, the upper layer option selection is realized through the Option-Critic algorithm, and the lower layer multi-agent collaborative control is realized through the QMix+DQN algorithm. Obtain feedback data on the drone's mission execution, process the feedback data through a hierarchical reward mechanism to optimize the decision-making strategy and update network parameters, and dynamically adjust option switching conditions to form a closed-loop optimization.

2. The UAV cooperative confrontation control method according to claim 1, characterized in that: The acquiring of situation information of the multi-UAV system and the pre-processing of the situation information by the situation awareness module to generate standardized state features are specifically as follows: Normalize the original situation data of each UAV i to generate a standardized feature vector , the calculation formula is: in, is the original situation data of UAV i, and are the minimum and maximum values ​​of this type of data respectively; Combine the normalized feature vectors of all drones to generate the global state feature , the expression is: [ ] Among them, n is the number of our drones, is the enemy target feature vector, is the environmental feature vector.

3. The UAV cooperative countermeasure control method according to claim 1, characterized in that: The aforementioned prior knowledge rules are used to execute the approach maneuver and encirclement strategy at the initial stage of the mission, specifically: At the initial stage of the mission, the drone group is controlled to move towards the target area according to the predefined approach rules. The approach rules are: maintain the formation spacing, move to the target point along the shortest path, and avoid known threat areas. The encirclement strategy is activated when the encirclement trigger conditions are met. The encirclement trigger conditions are: the distance between the drone and the enemy target is less than or equal to the preset encirclement distance threshold, and the threat level of the enemy target is greater than or equal to the preset threat level threshold. Both conditions must be met at the same time; among them, the encirclement strategy uses geometric calculations to assign drones to predetermined positions around the target, forming an encirclement.

4. The UAV cooperative countermeasure control method according to claim 1, characterized in that: After the encirclement is completed, the upper layer option selection is realized by the Option-Critic algorithm, specifically: Defining the option value function , indicating that in the state Select option Expected cumulative reward: in, is the cumulative reward from the current moment to the termination of the option; The upper-level strategy selects options through the ε-greedy strategy, where the selection probability of the optimal option is the preset base probability plus the average distribution of the exploration probability, and other options obtain the average distribution of the exploration probability.

5. The UAV cooperative confrontation control method according to claim 1, characterized in that: The QMix+DQN algorithm is used to implement the underlying multi-agent collaborative control, specifically: Build a local Q network for each agent i , the input is a local observation , the output is the Q value of each action, Network adjustment parameters; Building a global hybrid network , mix the local Q values ​​of each agent into the global Q value: in, is the observation-action history of agent n, is the action of agent n, is a monotonic mixing function; Use TD error to update the network parameters, and the loss function is: in, , For instant rewards, is a discount factor used to balance immediate rewards and future rewards, For the next observation-action history, For the next action, are the target network parameters, is the target Q value, is the current global Q value.

6. The UAV cooperative confrontation control method according to claim 1, characterized in that: Also includes: Before the agent selects an action, it filters out infeasible actions through a constraint checking mechanism, which includes boundary constraint checking, collision constraint checking, and fuel constraint checking. Actions that satisfy all constraints are marked as feasible. The boundary constraint check verifies whether the position of the agent after performing the action is still within the preset valid combat area; The collision constraint check verifies whether the distance between the agent and all other agents after the agent performs the action is greater than a preset safety distance threshold; The fuel constraint check verifies whether the remaining fuel of the agent after performing the action is still greater than a preset fuel reserve threshold.

7. A UAV cooperative countermeasure control device, characterized in that: include: a preprocessing unit, configured to obtain situational information of the multi-UAV system and preprocess the situational information to generate standardized state features through a situational awareness module, wherein the situational information includes the position, speed, and attitude data of the UAVs, the position, speed, and threat level data of the enemy targets, and the boundaries, obstacles, and missile threat data of the battlefield environment; A hierarchical control strategy construction unit is used to construct a hierarchical control strategy based on the standardized state characteristics, specifically for: constructing an upper-level option decision strategy , the strategy is based on the global state characteristics Output the option w that should be executed currently, where w∈{0,1,2} represents the maneuver, avoidance, and attack options respectively; construct the underlying multi-agent collaborative control strategy , the strategy is based on the local observation of agent i And the upper option command w outputs specific actions ; Define option termination conditions , used to determine whether the current option should be ended and the upper-level decision should be made again. The calculation formula is: ,in, is the sigmoid activation function, is the termination condition weight matrix, is a bias vector, wherein the hierarchical control strategy includes upper-layer option decision-making and lower-layer multi-agent collaborative control. The upper-layer option decision-making is responsible for selecting among the three options of maneuver, avoidance, and attack, and the lower-layer multi-agent collaborative control is responsible for executing specific collaborative actions; An execution unit, configured to guide the drone swarm to perform a mission according to the hierarchical control strategy, wherein, in the initial stage of the mission, a priori knowledge rules are used to execute approach maneuvers and encirclement strategies, and after encirclement is completed, the upper layer option selection is implemented through the Option-Critic algorithm, and the lower layer multi-agent collaborative control is implemented through the QMix+DQN algorithm; The feedback unit is used to obtain feedback data on the drone's mission execution, process the feedback data through a hierarchical reward mechanism to optimize the decision-making strategy and update the network parameters, and dynamically adjust the option switching conditions to form a closed-loop optimization.

8. A UAV cooperative countermeasure control device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a UAV cooperative confrontation control method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that A computer program is stored, and the computer program can be executed by a processor of the device where the computer-readable storage medium is located to implement a drone cooperative confrontation control method as described in any one of claims 1 to 6.

Images