Unmanned aerial vehicle cooperative confrontation control method, device and equipment and storage medium

Abstract

The invention provides an unmanned aerial vehicle cooperative confrontation control method, device and equipment and a storage medium, and the method comprises the steps: obtaining the situation information of a multi-unmanned aerial vehicle system, and carrying out the preprocessing of the situation information through a situation awareness module, and generating a standardized state feature; the hierarchical control strategy comprises upper-layer option decision and bottom-layer multi-agent cooperative control, guiding an unmanned aerial vehicle group to execute a task according to the hierarchical control strategy, executing an approaching maneuvering and surrounding strategy by adopting a priori knowledge rule in an initial stage of the task, and realizing upper-layer option selection through an Option-Critic algorithm after surrounding is completed, bottom-layer multi-agent cooperative control is realized through a QMix + DQN algorithm; the feedback data of task execution of the unmanned aerial vehicle is obtained, the feedback data is processed through a hierarchical reward mechanism to optimize a decision strategy and update network parameters, and option switching conditions are dynamically adjusted to form closed-loop optimization, so that the problem that the unmanned aerial vehicle cannot effectively process complex decisions of a high-dimensional state space and a large-scale action space is solved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicles, and particularly to a method, device, equipment and storage medium for collaborative countermeasure control of unmanned aerial vehicles. Background Art

[0002] With the rapid development of artificial intelligence technology, the collaborative countermeasure control of unmanned aerial vehicles has become the core technology of modern military intelligent systems. In a complex multi-agent collaborative combat environment, improving the autonomous decision-making ability and collaborative combat efficiency of unmanned aerial vehicle swarms is crucial for the technical advantages of military equipment. However, existing unmanned aerial vehicle control methods often have limitations and cannot fully meet the requirements of dynamically changing combat environment.

[0003] The control of unmanned aerial vehicles in the prior art mostly relies on traditional manual operations or single-rule control, lacking autonomous learning ability and being unable to effectively handle complex decision-making problems in high-dimensional state spaces and large-scale action spaces. Although traditional reinforcement learning methods have certain autonomous learning abilities, in complex military scenarios with multi-task coupling, due to a large number of combat units, high-dimensional situation dimensions, and huge action spaces, there are problems such as slow convergence speed and low sample efficiency, making it difficult to achieve efficient multi-agent collaborative control and limiting their effectiveness in actual military applications.

[0004] In view of this, the present application is proposed. Summary of the Invention

[0005] The present invention discloses a method, device, equipment and storage medium for collaborative countermeasure control of unmanned aerial vehicles, aiming to solve the problem that unmanned aerial vehicles cannot effectively handle complex decision-making problems in high-dimensional state spaces and large-scale action spaces.

[0006] The first embodiment of the present invention provides a method for collaborative countermeasure control of unmanned aerial vehicles, including: Obtain the situation information of a multi-unmanned aerial vehicle system, and preprocess the situation information through a situation awareness module to generate standardized state features. Among them, the situation information includes the position, speed, and attitude data of the unmanned aerial vehicles, the position, speed, and threat level data of enemy targets, and the boundary, obstacles, and missile threat data of the battlefield environment; Construct a hierarchical control strategy based on the standardized state features. Among them, the hierarchical control strategy includes upper-layer option decision-making and lower-layer multi-agent collaborative control. The upper-layer option decision-making is responsible for selecting among three options: maneuvering, evading, and striking, and the lower-layer multi-agent collaborative control is responsible for executing specific collaborative actions; Guide the UAV swarm to execute tasks according to the hierarchical control strategy. Among them, in the initial stage of the task, the prior knowledge rule is used to execute the approach maneuver and the encirclement strategy. After the encirclement is completed, the upper-layer option selection is realized through the Option-Critic algorithm, and the lower-layer multi-agent collaborative control is realized through the QMix+DQN algorithm; Obtain the feedback data of the UAV executing the task, process the feedback data through the hierarchical reward mechanism to optimize the decision-making strategy and update the network parameters, and dynamically adjust the option switching conditions to form a closed-loop optimization. Preferably, obtain the situation information of the multi-UAV system, and preprocess the situation information through the situation awareness module to generate standardized state features. Specifically: Normalize the original situation data of each UAV i to generate a standardized feature vector , and the calculation formula is:

[0007] Among them, is the original situation data of UAV i, and are the minimum and maximum values of this type of data respectively; Combine the standardized feature vectors of all UAVs to generate the global state feature , and the expression is: Among them, n is the number of our UAVs, is the enemy target feature vector, is the environmental feature vector.

[0008] Preferably, construct a hierarchical control strategy based on the standardized state features. Specifically: Construct the upper-layer option decision-making strategy , and this strategy outputs the option w that should be executed currently according to the global state feature , where w∈{0,1,2} represents the maneuver, avoidance, and strike options respectively; Construct the lower-layer multi-agent collaborative control strategy , and this strategy outputs the specific action according to the local observation of agent i and the upper-layer option instruction w; Define the option termination condition , which is used to judge whether the current option should end and re-make the upper-layer decision. The calculation formula is:

[0009] Among them, ​​is the sigmoid activation function, is the termination condition weight matrix, is the bias vector.

[0010] Preferably, the approaching maneuver and encirclement strategy are executed using prior knowledge rules at the initial stage of the mission, specifically: At the initial stage of the mission, control the UAV swarm to move towards the target area according to the predefined approaching rules. The approaching rules are: maintain the formation spacing, move along the shortest path towards the target point, and avoid known threat areas When the encirclement trigger condition is met, start the encirclement strategy. The encirclement trigger condition is: the distance between the UAV and the enemy target is less than or equal to the preset encirclement distance threshold, and the threat level of the enemy target is greater than or equal to the preset threat level threshold. Both conditions need to be met simultaneously; among them, the encirclement strategy allocates the UAVs to the predetermined positions around the target through geometric calculations to form an encirclement.

[0011] Preferably, the upper-level option selection is realized through the Option-Critic algorithm after the encirclement is completed, specifically: Define the option value function , indicating the expected cumulative reward for selecting option in state :

[0012] Among them, is the cumulative reward from the current moment to the termination of the option; The upper-level strategy selects options through the ε-greedy strategy. Among them, the selection probability of the optimal option is the preset base probability plus the average distribution of the exploration probability, and the other options obtain the average distribution of the exploration probability.

[0013] Preferably, the bottom-layer multi-agent collaborative control is realized through the QMix+DQN algorithm, specifically: Construct a local Q network for each agent i, with the input being the local observation and the output being the Q values of each action, Network adjustment parameters; Construct a global mixing network

[0014] Among them, is the observation-action history of agent n, is the action of agent n, is the monotonic mixing function; Update the network parameters using the TD error, and the loss function is:

[0015] wherein, , is the immediate reward, is the discount factor, which is used to balance the immediate reward and future rewards, is the next observation-action history, is the next action, is the target network parameter, is the target Q value, is the current global Q value.

[0016] Preferably, it further includes: before the agent selects an action, filter out infeasible actions through a constraint check mechanism, and the constraint check mechanism includes boundary constraint check, collision constraint check and fuel constraint check, and the actions that satisfy all the constraint conditions are marked as feasible; The boundary constraint check verifies whether the position of the agent after executing the action is still within the preset effective combat area; The collision constraint check verifies whether the distance between the agent and all other agents after executing the action is greater than the preset safety distance threshold; The fuel constraint check verifies whether the remaining fuel of the agent after executing the action is still greater than the preset fuel reserve threshold.

[0017] The second embodiment of the present invention provides a UAV cooperative confrontation control device, which is characterized by including: A preprocessing unit, configured to obtain the situation information of the multi-UAV system, and preprocess the situation information through a situation awareness module to generate standardized state features, wherein the situation information includes UAV position, speed, attitude data, the position, speed, threat level data of enemy targets, and the boundary, obstacles, missile threat data of the battlefield environment; A hierarchical control strategy construction unit, configured to construct a hierarchical control strategy based on the standardized state features, wherein the hierarchical control strategy includes upper-layer option decision-making and lower-layer multi-agent collaborative control, and the upper-layer option decision-making is responsible for selecting among three options: maneuvering, evading, and striking, and the lower-layer multi-agent collaborative control is responsible for executing specific collaborative actions; An execution unit, configured to guide the UAV swarm to execute tasks according to the hierarchical control strategy, wherein in the initial stage of the task, the prior knowledge rules are used to execute the approach maneuver and encirclement strategy, and after the encirclement is completed, the upper-layer option selection is realized through the Option-Critic algorithm, and the lower-layer multi-agent collaborative control is realized through the QMix+DQN algorithm; A feedback unit for obtaining feedback data of the UAV during task execution, processing the feedback data through a hierarchical reward mechanism to optimize the decision-making strategy and update network parameters, and dynamically adjusting the option switching conditions to form a closed-loop optimization.

[0018] The third embodiment of the present invention provides a UAV cooperative confrontation control device, including a memory and a processor. A computer program is stored in the memory and can be executed by the processor to implement a UAV cooperative confrontation control method as described in any one of the above.

[0019] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program that can be executed by the processor of the device where the computer-readable storage medium is located to implement a UAV cooperative confrontation control method as described in any one of the above.

[0020] Based on a UAV cooperative confrontation control method, device, equipment, and storage medium provided by the present invention, by obtaining the situation information of a multi-UAV system, preprocessing the situation information through a situation awareness module to generate standardized state features; constructing a hierarchical control strategy based on the standardized state features, where the hierarchical control strategy includes upper-layer option decision-making and lower-layer multi-agent cooperative control, guiding the UAV swarm to execute tasks according to the hierarchical control strategy. Among them, in the initial stage of the task, prior knowledge rules are used to execute approach maneuvers and encirclement strategies. After the encirclement is completed, upper-layer option selection is realized through the Option-Critic algorithm, and lower-layer multi-agent cooperative control is realized through the QMix+DQN algorithm; obtaining feedback data of the UAV during task execution, processing the feedback data through a hierarchical reward mechanism to optimize the decision-making strategy and update network parameters, and dynamically adjusting the option switching conditions to form a closed-loop optimization, solving the problem of complex decision-making for UAVs unable to effectively handle high-dimensional state spaces and large-scale action spaces. Description of the Drawings

[0021] Figure 1 is a flowchart of a UAV cooperative confrontation control method provided by the first embodiment of the present invention; Figure 2 is a module diagram of a UAV cooperative confrontation control device provided by the second embodiment of the present invention. Detailed Embodiments

[0022] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0023] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0024] It should be clear that the described embodiments are only part of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0025] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "the" and "said" used in the embodiments of the present invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0026] It should be understood that the term " / and" used herein is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after.

[0027] Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detecting (stated condition or event)" may be interpreted as "when determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)".

[0028] The "first / second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It can be understood that the "first / second" can be interchanged in a specific order or sequence when allowed. It should be understood that the objects distinguished by "first / second" can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0029] The following will describe the specific embodiments of the present invention in detail with reference to the accompanying drawings.

[0030] The present invention discloses a method, device, equipment and storage medium for collaborative confrontation control of unmanned aerial vehicles, aiming to solve the complex decision-making problem that unmanned aerial vehicles cannot effectively handle high-dimensional state spaces and large-scale action spaces.

[0031] Please refer to Figure 1, the first embodiment of the present invention provides a method for cooperative confrontation control of unmanned aerial vehicles (UAVs), which can be executed by a UAV cooperative confrontation control device (hereinafter referred to as the control device), and specifically, by one or more processors in the control device to at least achieve the following steps S101, Obtain the situation information of the multi-UAV system, and preprocess the situation information through a situation awareness module to generate standardized state features. Among them, the situation information includes UAV position, speed, attitude data, the position, speed, and threat level data of enemy targets, and the boundaries, obstacles, and missile threat data of the battlefield environment; In this embodiment, the control device can be a server, which can establish communication with the UAV cluster. The control device can be installed with corresponding operating systems and application software, and the functions required in this embodiment can be realized through the combination of the operating system and application software.

[0032] In the specific implementation of the present invention, the situation awareness module first collects state data such as position coordinates, flight speed vectors, and attitude angles in real time from the sensor systems of each UAV. At the same time, it obtains the real-time position, movement trajectory, and threat level evaluated according to the target type of the enemy target through radar detection and intelligence fusion. The battlefield environment information is obtained through a combination of preloaded maps and real-time reconnaissance, including the boundary coordinates of the combat area, the distribution of static obstacles, and the position and trajectory information of dynamic missile threats. To ensure the comparability of different types of data and the stability of network training, the Min-Max normalization method is used to standardize the original situation data of each UAV. Through the formula Map each item of data to a unified range from 0 to 1, where and are the historical minimum and maximum values of this type of data or the preset physical limit values respectively. After the normalization process, the standardized feature vectors of each UAV, the enemy target feature vectors, and the environment feature vectors are concatenated in sequence according to a predetermined format to form a global state feature , this global feature vector contains the complete information of the current battlefield situation, and serves as the unified input interface for the subsequent hierarchical control strategy, ensuring that both the upper-layer option decision and the lower-layer cooperative control are based on the same situation understanding for decision-making.

[0033] S102, Construct a hierarchical control strategy based on the standardized state features. Among them, the hierarchical control strategy includes upper-layer option decision-making and lower-layer multi-agent cooperative control. The upper-layer option decision-making is responsible for selecting among three options: maneuvering, evading, and striking. The lower-layer multi-agent cooperative control is responsible for executing specific cooperative actions; It should be noted that the hierarchical control strategy adopts a two-layer architecture design, and the upper-layer option decision-making strategy ​Receives the global state features as input, calculates and outputs the most suitable combat option under the current battlefield situation through a deep neural network. The options are encoded as w ∈ {0, 1, 2}, corresponding to three high-level tactical actions of maneuvering approach, threat avoidance, and target strike respectively. The underlying multi-agent collaborative control strategy Then, under the constraint of receiving the upper-layer option instructions, combines the local observation information of each agent to generate specific flight control actions, realizing the mapping from abstract tactical intentions to specific execution actions. To ensure the timeliness of option execution, the system designs a dynamic option termination mechanism, through the formula

[0034] Evaluates the execution status of the current option in real time, where the sigmoid activation function maps the linear combination result to a termination probability between 0 and 1, and the weight matrix And the bias vector 0]Learn through training the optimal execution duration of different options in various situations. When the termination probability exceeds the preset threshold, the system automatically ends the execution of the current option and reactivates the upper-layer decision-making process, ensuring that the control strategy can adjust tactical behaviors in a timely manner according to the changes in the battlefield situation.

[0035] S103, Guide the UAV swarm to execute tasks according to the hierarchical control strategy. Among them, in the initial stage of the task, use prior knowledge rules to execute the approach maneuver and encirclement strategy. After the encirclement is completed, implement upper-layer option selection through the Option-Critic algorithm, and implement underlying multi-agent collaborative control through the QMix+DQN algorithm; It should be noted that in the initial stage of the task, a fixed strategy based on prior knowledge is adopted to ensure that the UAV swarm can approach the target efficiently and establish a favorable situation. In the approach maneuver stage, control the UAV swarm to maintain the standard formation spacing according to predefined expert rules, calculate the shortest safe path from the current position to the target area through the path planning algorithm, and at the same time use the threat assessment module to identify and avoid known enemy defense areas and dangerous zones to ensure the safety and concealment of the UAV swarm during the approach process. When the UAV swarm approaches the target, the system monitors the satisfaction of two key conditions in real time: one is whether the spatial distance between the UAV and the enemy target is reduced to within the preset encirclement distance threshold, and the other is whether the threat level of the enemy target calculated by the threat assessment algorithm reaches the minimum requirement for starting the encirclement operation. Only when both the distance condition and the threat level condition are met can the encirclement strategy switch be triggered. When the encirclement strategy is executed, the system performs geometric calculations based on the current position and movement trend of the target, distributes the UAVs participating in the encirclement to the tactical positions around the target at preset angular intervals to form a polygon encirclement. Each UAV adjusts its flight trajectory according to the assigned encirclement position to ensure effective control of the target from multiple directions.

[0036] Furthermore, in this embodiment, the upper-layer option decision-making adopts the Option-Critic algorithm framework, and evaluates the long-term benefits of different tactical choices by constructing an option value function. The system establishes a value function for each possible option , and this function estimates the expected cumulative reward for executing a specific option ω under the current global state through reinforcement learning training. The calculation formula in represents the sum of rewards obtained from the start of the option execution until the termination condition is met, including multiple dimensions such as task completion rewards, time efficiency rewards, and safety rewards. In the option selection stage, the system adopts an ε-greedy strategy to balance the relationship between exploiting the learned optimal strategy and exploring potentially better strategies. Specifically, it is implemented as mostly selecting the option with the highest value function evaluation in the current state, while reserving a small probability of randomly selecting other options. This strategy controls the exploration degree through a preset exploration rate parameter ε. The optimal option obtains a higher selection probability, which is composed of the basic exploitation probability plus the uniform distribution of the exploration probability, while other non-optimal options only obtain the average distribution of the exploration probability. This not only ensures that the system mainly executes the currently considered optimal tactical choice but also maintains the exploration ability to discover better strategies, enabling the upper-layer decision-making to continuously improve and adapt in a dynamic combat environment.

[0037] In the specific implementation of the present invention, the bottom-layer multi-agent collaborative control adopts the QMix+DQN algorithm framework to handle the collaborative decision-making problem of multiple unmanned aerial vehicles. The system constructs an independent local Q-network for each agent , which takes the local observation information of the agent as input, including its own state, relative target position, relative relationship with teammates, etc., and outputs the value evaluation of the agent for performing various optional actions under the current observation. To achieve effective multi-agent coordination, the system designs a global mixing network , and combines the local Q-values of each agent with the global state information through a mixing function to generate a global collaborative Q-value:

[0038] where represents the historical observation-action sequence of agent n. The design of the mixing function ensures monotonicity constraints to ensure the consistency between individual optimal decisions and global optimal decisions. The network training process adopts the temporal difference learning method, and updates the network parameters by minimizing the loss function , where the target Q-value combines the immediate reward r and the discounted future maximum Q-value. The target network parameters are periodically copied from the main network to improve training stability, enabling each agent to achieve global collaborative optimization while learning individual optimal strategies.

[0039] S104. Obtain the feedback data of the UAV's mission execution, process the feedback data through a hierarchical reward mechanism to optimize the decision-making strategy and update the network parameters, and dynamically adjust the option switching conditions to form a closed-loop optimization. It should be noted that the system has established a complete closed-loop feedback optimization mechanism to continuously improve the performance of the control strategy. The feedback data collection module monitors the mission execution of each UAV in real time, including multi-dimensional information such as option completion status, action execution effect, cooperation degree, and overall mission progress. The hierarchical reward mechanism designs corresponding reward functions according to different levels of decision-making goals. The upper-level reward mainly evaluates the correctness and execution efficiency of option selection. When the option successfully achieves the expected goal, a positive reward is given, and a negative reward is imposed when the execution fails or times out. The lower-level reward focuses on the execution quality of specific actions, including individual task completion, multi-agent cooperation effect, and satisfaction of constraint conditions. The network parameter update uses the gradient descent method. Based on the collected experience data, the policy gradient and value function error are calculated, and the parameters of the upper-level option decision network and the lower-level action execution network are optimized simultaneously through the backpropagation algorithm. The system also implements a dynamic adjustment mechanism for option switching conditions. According to the statistical analysis of historical execution data, the optimal execution duration of different options in various situations is calculated, and the weight parameters and threshold settings in the termination conditions are adaptively adjusted to make the option switching more in line with the actual combat requirements. The entire optimization process forms a complete closed-loop from execution feedback to strategy improvement and then to parameter update, ensuring that the control system can continuously learn and improve during continuous mission execution, and gradually enhance the overall effectiveness of multi-UAV cooperative confrontation.

[0040] In a possible implementation manner of the present invention, the system will execute a comprehensive constraint check mechanism before each agent makes an action selection to ensure the safety and feasibility of the selected action. This mechanism adopts a three-layer constraint verification system. First, boundary constraint checking is performed. By predicting the position coordinates of the agent after executing the candidate action and comparing them with the boundary of the preset effective combat area, it is ensured that the UAV will not fly out of the specified combat range or enter the no-fly zone. Collision constraint checking calculates the expected distance between the agent and all other agents after executing the action to ensure that the minimum distance between any two UAVs is greater than the preset safety distance threshold, avoiding the risk of mid-air collision caused by cooperative maneuvers. Fuel constraint checking estimates the fuel consumption of the candidate action and compares it with the current fuel stock to ensure that the remaining fuel after executing the action can still meet the preset fuel reserve requirements, ensuring that the UAV has enough fuel to complete subsequent tasks or return safely. Only the actions that pass all the above three constraint checks will be marked as actionable and included in the candidate action set of the agent. Actions that do not meet any of the constraint conditions will be directly filtered out. This pre-screening mechanism effectively reduces the ineffective exploration in the reinforcement learning process.

[0041] In a possible implementation of the present invention, the hierarchical reward function design adopts a differential evaluation strategy to guide the optimization of decision-making at different levels. The upper-layer reward mechanism mainly evaluates the overall effect of option execution. When the option successfully achieves the expected goal, the system gives a basic success reward and an additional time reward according to the speed of completion time, encouraging efficient tactical choices; when the option execution fails, the system imposes a failure penalty and an additional time penalty according to the length of execution time, avoiding inefficient tactical decisions; when the option execution exceeds the preset maximum time limit, the system directly gives a timeout penalty, prompting the upper-layer strategy to learn the reasonable option switching timing. The lower-layer reward function adopts a multi-dimensional weighted combination method, linearly combining the individual task completion reward, the cooperation reward, and the constraint violation penalty according to the preset weight coefficients, ensuring that each agent takes into account the team collaboration effect while pursuing individual goals. The individual task reward is calculated differentially according to the type of option currently being executed. For the maneuver option, behaviors close to the target are mainly rewarded and behaviors too far from the target are penalized, while actions with a positive approaching speed component are encouraged; for the avoidance option, behaviors far from the threat source are mainly rewarded and fuel consumption is controlled to balance safety and economy; for the strike option, the result of a successful hit is directly rewarded and the ammunition consumption cost is considered to improve the strike efficiency. The cooperation reward evaluates the team cooperation effect by calculating the similarity of actions and the proximity of spatial positions among agents. The action similarity measures the angle between the action vectors of different agents using the cosine similarity method, and the position proximity uses the Gaussian function to quantify the spatial relationship among agents. The closer the agents are, the greater the contribution of their cooperative behaviors to the reward. This design encourages adjacent agents to execute coordinated actions and form an effective team cooperation mode.

[0042] Please refer to Figure 2 , the second embodiment of the present invention provides a UAV cooperative confrontation control device, which is characterized by including: A preprocessing unit 201, configured to obtain the situation information of the multi-UAV system, and preprocess the situation information through a situation awareness module to generate standardized state features, where the situation information includes UAV position, speed, attitude data, the position, speed, and threat level data of enemy targets, and the boundary, obstacles, and missile threat data of the battlefield environment; A hierarchical control strategy construction unit 202, configured to construct a hierarchical control strategy based on the standardized state features, where the hierarchical control strategy includes upper-layer option decision-making and lower-layer multi-agent cooperative control. The upper-layer option decision-making is responsible for selecting among three options: maneuver, avoidance, and strike, and the lower-layer multi-agent cooperative control is responsible for executing specific cooperative actions; The execution unit 203 is configured to guide the UAV swarm to execute tasks according to the hierarchical control strategy. Specifically, in the initial stage of the task, the prior knowledge rules are adopted to execute the approaching maneuver and the encirclement strategy. After the encirclement is completed, the upper-layer option selection is realized through the Option-Critic algorithm, and the lower-layer multi-agent collaborative control is realized through the QMix+DQN algorithm; The feedback unit 204 is configured to obtain the feedback data of the UAV executing the task, process the feedback data through the hierarchical reward mechanism to optimize the decision-making strategy and update the network parameters, and dynamically adjust the option switching conditions to form a closed-loop optimization.

[0043] The third embodiment of the present invention provides a UAV cooperative confrontation control device, including a memory and a processor. The memory stores a computer program, and the computer program can be executed by the processor to implement a UAV cooperative confrontation control method as described in any one of the above.

[0044] The fourth embodiment of the present invention provides a computer-readable storage medium, which is characterized in that it stores a computer program, and the computer program can be executed by the processor of the device where the computer-readable storage medium is located to implement a UAV cooperative confrontation control method as described in any one of the above.

[0045] Based on the UAV cooperative confrontation control method, device, equipment and storage medium provided by the present invention, by obtaining the situation information of the multi-UAV system, the situation perception module preprocesses the situation information to generate standardized state features; a hierarchical control strategy is constructed based on the standardized state features, wherein the hierarchical control strategy includes upper-layer option decision-making and lower-layer multi-agent collaborative control, and the UAV swarm is guided to execute tasks according to the hierarchical control strategy. Specifically, in the initial stage of the task, the prior knowledge rules are adopted to execute the approaching maneuver and the encirclement strategy. After the encirclement is completed, the upper-layer option selection is realized through the Option-Critic algorithm, and the lower-layer multi-agent collaborative control is realized through the QMix+DQN algorithm; the feedback data of the UAV executing the task is obtained, and the feedback data is processed through the hierarchical reward mechanism to optimize the decision-making strategy and update the network parameters, and the option switching conditions are dynamically adjusted to form a closed-loop optimization, thereby solving the problem that UAVs cannot effectively handle complex decision-making problems in high-dimensional state spaces and large-scale action spaces.

[0046] Exemplarily, the computer program described in the third and fourth embodiments of the present invention may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in implementing a drone cooperative confrontation control device. For example, the device described in the second embodiment of the present invention.

[0047] The so-called processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor is the control center of the drone cooperative confrontation control method, and uses various interfaces and lines to connect all parts of the implementation of a drone cooperative confrontation control method.

[0048] The memory can be used to store the computer program and / or modules. The processor realizes various functions of a drone cooperative confrontation control method by running or executing the computer program and / or modules stored in the memory, and by calling the data stored in the memory. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, a text conversion function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0049] Among them, if the implemented module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-described embodiment methods of the present invention, it can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0050] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the attached drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines. Those of ordinary skill in the art can understand and implement it without creative effort.

[0051] As described above, the above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A collaborative countermeasure control method for unmanned aerial vehicles, characterized in that, Including: Obtain the situation information of the multi-UAV system, and preprocess the situation information through a situation awareness module to generate standardized state features. Among them, the situation information includes UAV position, speed, attitude data, the position, speed, and threat level data of enemy targets, and the boundary, obstacles, and missile threat data of the battlefield environment; Construct a hierarchical control strategy based on the standardized state features. Among them, the hierarchical control strategy includes upper-layer option decision-making and lower-layer multi-agent collaborative control. The upper-layer option decision-making is responsible for selecting among three options: maneuvering, evading, and striking. The lower-layer multi-agent collaborative control is responsible for executing specific collaborative actions; Guide the UAV swarm to execute tasks according to the hierarchical control strategy. Among them, in the initial stage of the task, prior knowledge rules are used to execute the approach maneuver and encirclement strategy. After the encirclement is completed, the upper-layer option selection is realized through the Option-Critic algorithm, and the lower-layer multi-agent collaborative control is realized through the QMix+DQN algorithm; Obtain the feedback data of the UAV executing the task, process the feedback data through a hierarchical reward mechanism to optimize the decision-making strategy and update the network parameters, and dynamically adjust the option switching conditions to form a closed-loop optimization.

2. The method for collaborative countermeasure control of an unmanned aerial vehicle according to claim 1, wherein The obtaining of the situation information of the multi-UAV system and the preprocessing of the situation information through a situation awareness module to generate standardized state features are specifically as follows: Normalize the original situation data of each drone i to generate a standardized feature vector , and the calculation formula is as follows: Among them, is the original situation data of UAV i, and are the minimum value and the maximum value of this type of data respectively; Combine the standardized feature vectors of all drones to generate the global state features , and the expression is: [ ] where n is the number of our UAVs, is the enemy target feature vector, is the environmental feature vector.

3. The method for collaborative confrontation control of an unmanned aerial vehicle according to claim 1, wherein The constructing of a hierarchical control strategy based on the standardized state features is specifically as follows: Construct the upper-layer option decision-making strategy , which outputs the option w to be executed currently according to the global state characteristics , where w ∈ {0, 1, 2} represents maneuvering, avoidance, and striking options respectively; Construct the underlying multi-agent collaborative control strategy , and this strategy outputs specific actions according to the local observation of agent i and the upper-layer option instruction w ; Define option termination conditions , which is used to determine whether the current option should end and a higher-level decision should be made again. The calculation formula is as follows: Among them, is the sigmoid activation function, is the termination condition weight matrix, is the bias vector.

4. A method for collaborative countermeasure control of an unmanned aerial vehicle according to claim 1, characterized in that The executing of the approach maneuver and encirclement strategy using prior knowledge rules in the initial stage of the task is specifically as follows: In the initial stage of the task, control the UAV swarm to move towards the target area according to the predefined approach rules. The approach rules are: maintain the formation spacing, move along the shortest path towards the target point, and avoid known threat areas When the encirclement trigger condition is met, start the encirclement strategy. The encirclement trigger condition is: the distance between the UAV and the enemy target is less than or equal to the preset encirclement distance threshold, and the threat level of the enemy target is greater than or equal to the preset threat level threshold. Both conditions need to be met simultaneously. Among them, the encirclement strategy distributes the UAVs to predetermined positions around the target through geometric calculations to form an encirclement.

5. A method for collaborative countermeasure control of unmanned aerial vehicles according to claim 1, characterized in that, The realizing of the upper-layer option selection through the Option-Critic algorithm after the encirclement is completed is specifically as follows: Define the option value function , which represents the expected cumulative reward of selecting option in state : wherein, is the cumulative reward from the current moment to the option termination; The upper-layer strategy selects options through the ε-greedy strategy. Among them, the selection probability of the optimal option is the preset base probability plus the average distribution of the exploration probability, and the other options obtain the average distribution of the exploration probability.

6. The method for collaborative countermeasure control of an unmanned aerial vehicle according to claim 1, wherein The realizing of the lower-layer multi-agent collaborative control through the QMix+DQN algorithm is specifically as follows: Construct a local Q-network for each agent i , with the local observation as the input , and the Q-values of each action as the output Network adjustment parameters; Construct a global hybrid network , and mix the local Q-values of each agent into a global Q-value: wherein, is the observation-action history of agent n, is the action of agent n, is a monotonic mixing function; Use the TD error to update the network parameters, and the loss function is: Among them, , is the immediate reward, is the discount factor, which is used to balance the immediate reward and the future reward, is the next observation-action history, is the next action, is the target network parameter, is the target Q value, is the current global Q value.

7. A method for cooperative countermeasure control of unmanned aerial vehicles according to claim 1, characterized in that, Also including: Before the agent selects an action, filter out infeasible actions through a constraint check mechanism. The constraint check mechanism includes boundary constraint check, collision constraint check, and fuel constraint check. Actions that satisfy all constraint conditions simultaneously are marked as feasible; The boundary constraint check verifies whether the position of the agent after executing the action is still within the preset effective combat area; The collision constraint check verifies whether the distance between the agent and all other agents after executing the action is greater than the preset safety distance threshold; The remaining fuel after the fuel constraint check and verification agent executes an action is checked to see if it is still greater than a preset fuel reserve threshold.

8. An unmanned aerial vehicle collaborative countermeasure control device, characterized in that, It includes: A preprocessing unit for obtaining the situation information of the multi-UAV system, and preprocessing the situation information through a situation awareness module to generate standardized state features. Among them, the situation information includes UAV position, speed, attitude data, the position, speed, and threat level data of enemy targets, and the boundary, obstacles, and missile threat data of the battlefield environment; A hierarchical control strategy construction unit for constructing a hierarchical control strategy based on the standardized state features. Among them, the hierarchical control strategy includes upper-layer option decision-making and lower-layer multi-agent collaborative control. The upper-layer option decision-making is responsible for selecting among the three options of maneuvering, evading, and striking, and the lower-layer multi-agent collaborative control is responsible for executing specific collaborative actions; An execution unit for guiding the UAV swarm to execute tasks according to the hierarchical control strategy. Among them, in the initial stage of the task, the prior knowledge rule is used to execute the approach maneuver and the encirclement strategy. After the encirclement is completed, the upper-layer option selection is realized through the Option-Critic algorithm, and the lower-layer multi-agent collaborative control is realized through the QMix+DQN algorithm; A feedback unit for obtaining the feedback data of the UAV executing tasks, processing the feedback data through a hierarchical reward mechanism to optimize the decision-making strategy and update the network parameters, and dynamically adjusting the option switching conditions to form a closed-loop optimization.

9. An unmanned aerial vehicle collaborative countermeasure control device, characterized in that, It includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement a UAV cooperative confrontation control method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored, and the computer program can be executed by the processor of the device where the computer-readable storage medium is located to implement a UAV cooperative confrontation control method according to any one of claims 1 to 7.

Images