Multi-unmanned aerial vehicle adaptive intelligent decision reinforcement learning method and system

Abstract

The invention provides a reinforcement learning method and system for adaptive intelligent decision making of multiple unmanned aerial vehicles. The method comprises the following steps: acquiring an unmanned aerial vehicle formation defense strategy for indicating a decision scheme of multiple unmanned aerial vehicles, wherein the decision scheme comprises a defense action and a flight direction; the defense actions comprise air patrol, calling of a friend machine and withdrawing of a local machine. And controlling the defense party unmanned aerial vehicle formation to execute the unmanned aerial vehicle formation defense strategy, obtaining the patrol unmanned aerial vehicle in the patrol state, and obtaining the detection result of the patrol unmanned aerial vehicle. And generating a decision scheme updating model according to the detection result, solving, and updating the decision scheme of the patrol unmanned aerial vehicle according to the model solution. And updating an unmanned aerial vehicle formation defense strategy according to the updated decision scheme, and controlling the defense party unmanned aerial vehicle formation to execute the updated unmanned aerial vehicle formation defense strategy. The decision-making scheme of the unmanned aerial vehicles can be updated through the detection results of the unmanned aerial vehicles, so that the scale of the unmanned aerial vehicle formation during task execution is scheduled in real time, and the defense effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a reinforcement learning method and system for adaptive intelligent decision-making of multiple unmanned aerial vehicles. Background Art

[0002] With the improvement of the intelligence and performance of drones, the use of multiple drone formations to perform air combat missions has become a hot topic in the field. The attacker can use drones to pose a serious threat to the defender's core assets with their wide coverage and powerful firepower. Therefore, anti-air operations have become an indispensable key element in military strategic planning, which covers two core strategies: offensive anti-air operations (OCA) and defensive anti-air operations (DCA). The main goal of OCA is to minimize the combat effectiveness of the attacker near its source by destroying, disrupting or suppressing the attacker's forces; while DCA focuses on intercepting air targets and ensuring the safety and stability of the defender's important assets by reducing their threat level.

[0003] In the complex DCA scenario, the defending drone must intercept the incoming attacking drones to the maximum extent possible, while adopting continuous and flexible maneuvering strategies to cope with the rapid changes in the battlefield environment.

[0004] However, the arrival of the attacking drones is often uncertain, and they will gradually approach the defender's predetermined target over time. In addition, when the number of attacking drones is unknown, if the defender uses a preset scale of drones, it may waste or be insufficient resources, making it difficult to respond to emergencies and having a poor defense effect. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a reinforcement learning method and system for adaptive intelligent decision-making of multiple UAVs, which solves the technical problem in the prior art that the defense effect is poor when the defending UAV uses a preset scale of UAVs for defense.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] The present invention solves the technical problem by providing a reinforcement learning method for adaptive intelligent decision-making of multiple UAVs. The reinforcement learning method is executed by a computer and comprises the following steps:

[0010] A reinforcement learning method for adaptive intelligent decision-making of multiple unmanned aerial vehicles, wherein the reinforcement learning method is executed by a computer and is characterized in that it includes the following steps:

[0011] Acquire the drone formation information and drone formation defense strategy of the defending party; the drone formation of the defending party includes multiple drones; the drone formation defense strategy is used to indicate the decision-making plan of the multiple drones, and the decision-making plan includes defense actions and flight directions; the defense actions include air patrol, calling friendly aircraft and the own aircraft retreating;

[0012] Controlling the defending drone formation to execute the drone formation defense strategy;

[0013] Acquire a patrol drone in a patrol state, and acquire a detection result of the patrol drone;

[0014] Generate a decision-making scheme update model according to the detection results; the decision-making scheme update model includes a state space, a decision scheme, a reward return and a benefit function; the state space is generated by the detection results;

[0015] Solving the decision-making scheme update model, and updating the decision-making scheme of the patrol drone according to the model solution;

[0016] The drone formation defense strategy is updated according to the updated decision plan, and the defending drone formation is controlled to execute the updated drone formation defense strategy.

[0017] Preferably, solving the decision solution update model includes:

[0018] Acquire drone state information of the attacking drone according to the state space;

[0019] Calculate reward return according to the drone status information;

[0020] The profit function is solved according to the reward return.

[0021] Preferably, the drone status information includes the number of first attacking drones defeated by the patrol drone and the number of second attacking drones entering the defense position of the defending drone formation. Calculating the reward return based on the drone status information includes:

[0022] Obtaining a first reward return according to the drone state information includes:

[0023] R target =m1×f1+m2×f2

[0024] in,

[0025] R target Indicates the first reward return;

[0026] m1 represents the number of drones of the first attacking party, and m2 represents the number of drones of the second attacking party;

[0027] f1 represents the bonus score for defeating the attacking drone, and f2 represents the bonus score for the attacking drone entering the defensive position;

[0028] Get the second reward, including:

[0029] R time =-0.01×m3

[0030] in,

[0031] R time represents the second reward return; m3 represents the number of patrol drones;

[0032] Calculate reward returns, including:

[0033] R=R target +R time

[0034] in,

[0035] R represents reward return.

[0036] Preferably, the profit function is:

[0037]

[0038] in,

[0039] R (t) represents the reward return of the tth cycle;

[0040] γ represents the discount factor, γ t Represents the discount factor raised to the power of t.

[0041] Preferably, after executing the acquisition of the patrol drone in the patrol state, the method further includes:

[0042] Control patrol drones to time and detect the timing duration;

[0043] If the patrol drone does not detect the attacking drone within the preset patrol period, the decision plan of the patrol drone is updated to set the defense action of the patrol drone to retreat;

[0044] If the patrol drone detects the attacking drone within the preset patrol cycle, the timing duration is updated to 0, and the patrol drone is controlled to perform timing.

[0045] Preferably, after obtaining the detection result of the patrol drone, the method further includes:

[0046] Obtaining the patrol threat value of the patrol drone;

[0047] If the patrol threat value is greater than or equal to a preset threat value threshold, updating the decision plan of the patrol drone to set the defense action of the patrol drone to call a friendly aircraft;

[0048] If the patrol threat value is less than a preset threat value threshold, the step of generating a decision solution and updating the model according to the detection result is performed.

[0049] Preferably, obtaining the patrol threat value of the patrol drone includes:

[0050] Acquire the target attacking drone detected by the patrol drone according to the detection result of the patrol drone;

[0051] Obtaining the action strategy of the target attacking drone; the action strategy includes an advance strategy, a retreat strategy, and other strategies; the advance strategy is the behavior of the target attacking drone advancing toward the defense position of the defending drone formation, the retreat strategy is the behavior of the target attacking drone retreating, and the other strategies are strategy behaviors other than the advance strategy and the retreat strategy;

[0052] Obtaining a strategy threat value according to the action strategy;

[0053] Get the location threat value, including:

[0054]

[0055] in,

[0056] W distance Indicates the position threat value, d max Indicates the maximum patrol distance of the drones in the defending drone formation. Indicates the distance between the target attacking drone i and the defense position;

[0057] The patrol threat value of the patrol drone is calculated according to the strategy threat value and the position threat value.

[0058] Preferably, the model solution includes probability distributions of multiple decision-making solutions;

[0059] The decision plan of the patrol drone is updated according to the model solution, including:

[0060] Obtaining a decision plan with the highest probability according to the model solution, and updating it as the decision plan of the patrol drone in the next cycle;

[0061] The drone formation defense strategy is updated according to the updated decision plan, including:

[0062] According to the update decision plan, the number of third drones whose defensive action is to call friendly drones is obtained, and the number of retreating drones whose defensive action is to retreat.

[0063] Acquire a number of standby drones corresponding to the third number of drones in the defending drone formation, and update the decision plan of the standby drones to set the defense action of the standby drones to air patrol;

[0064] Setting the standby drone as a patrol drone and canceling the setting of the retreating drone as a patrol drone;

[0065] Statistics are collected on the updated decision plans of multiple drones in the defending drone formation to generate the drone formation defense strategy for the next cycle.

[0066] The present invention solves the technical problem by providing a reinforcement learning system for multi-UAV adaptive intelligent decision-making, the system comprising:

[0067] The information acquisition module is configured to acquire the defender's drone formation information and the drone formation defense strategy; the defender's drone formation includes multiple drones; the drone formation defense strategy is used to indicate the decision-making plan of the multiple drones, the decision-making plan includes defense actions and flight directions; the defense actions include air patrol, calling friendly aircraft and the aircraft's own retreat;

[0068] An execution module, configured to control the defending drone formation to execute the drone formation defense strategy;

[0069] A detection module is configured to obtain a patrol drone in a patrol state and obtain a detection result of the patrol drone;

[0070] A model generation module is configured to generate a decision solution update model according to the detection result; the decision solution update model includes a state space, a decision solution, a reward return and a benefit function; the state space is generated by the detection result;

[0071] A decision solution updating module is configured to solve the decision solution updating model and update the decision solution of the patrol drone according to the model solution;

[0072] The defense strategy update module is configured to update the drone formation defense strategy according to the updated decision plan, and control the defending drone formation to execute the updated drone formation defense strategy.

[0073] The present invention solves its technical problem by providing a computer-readable storage medium, which stores a computer program for reinforcement learning of multi-UAV adaptive intelligent decision-making, wherein the computer program enables a computer to execute the reinforcement learning method for multi-UAV adaptive intelligent decision-making as described above.

[0074] (III) Beneficial effects

[0075] The present invention provides a reinforcement learning method and system for adaptive intelligent decision-making of multiple UAVs. Compared with the prior art, it has the following beneficial effects:

[0076] The present invention can obtain the drone formation information and drone formation defense strategy of the defender, wherein the drone formation defense strategy is used to indicate the decision-making plan of multiple drones, and the decision-making plan includes defense actions and flight directions; the defense actions include air patrol, calling friendly aircraft and the own aircraft retreat. The drone formation of the defender is controlled to execute the drone formation defense strategy, and the patrol drone in the patrol state is acquired, and the detection results of the patrol drone are acquired. According to the detection results, a decision-making plan update model is generated and solved, and the decision-making plan of the patrol drone is updated according to the model solution. The drone formation defense strategy is updated according to the updated decision-making plan, and the drone formation of the defender is controlled to execute the updated drone formation defense strategy. The decision-making plan of the drone can be updated through the detection results of the drone, so as to real-time schedule the scale of the drone formation when performing tasks, thereby improving the defense effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0078] Figure 1 A schematic diagram of a scenario of a reinforcement learning method for adaptive intelligent decision-making of multiple UAVs provided in an embodiment of the present invention;

[0079] Figure 2 A schematic diagram showing the attack range and detection range of a drone in some embodiments;

[0080] Figure 3 A schematic diagram of the state space in some embodiments is shown. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] The embodiments of the present application solve the problem of poor defense effect of the defending drone formation in the prior art by providing a reinforcement learning method and system for adaptive intelligent decision-making of multiple drones, and improve the quality of defense execution by the drone formation.

[0083] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0084] Multiple drone formations can perform air combat missions, and the attacker can use drone formations to attack the defender's defensive positions. For the defender, it can perform anti-air operations to prevent the attacker's drone formation from entering the defensive position.

[0085] The defender can implement two strategies: offensive counter-air operations (OCA) and defensive counter-air operations (DCA). Among them, the main goal of OCA is for the defender's drones to destroy, disrupt or suppress the attacker's forces and minimize their combat effectiveness near their source. DCA is for the defender's drones to intercept the attacker's drones and ensure the safety and stability of the defender's important assets by reducing their threat level.

[0086] As a key strategy to ensure airspace security, DCA covers two major aspects: active air defense and missile defense. It relies on a variety of asset types and system integration, including but not limited to fighter jets, surface-to-air missiles, anti-aircraft guns, electromagnetic warfare systems, and ballistic missile defense systems. These systems work together to destroy enemy forces or effectively weaken their offensive capabilities. When performing DCA missions, the fighter jets used may be traditional manned models or advanced unmanned aerial vehicles (UAVs). In particular, weaponized drones designed specifically for combat, namely unmanned combat aerial vehicles (UCAVs), can be remotely controlled and have the ability to perform tasks autonomously.

[0087] When using the DCA strategy, the defender usually fixes the size of the drones that perform the mission. For example, if the drone formation consists of 5 drones, the defender usually lets all 5 drones take off and perform the defense mission. In the embodiment of the present application, performing the defense mission means that the defender's drone patrols in the air to detect the attacker's drone and attack it.

[0088] However, the arrival of the attacking drones is often uncertain, and the number of attacking drones is usually unknown. The attacking drones may advance towards the defense position in batches and at multiple time periods. The defender's use of a preset scale of drones may result in a waste or shortage of resources, making it difficult to respond to emergencies and resulting in poor defense results.

[0089] To solve the above problems, an embodiment of the present invention provides a reinforcement learning method for adaptive intelligent decision-making of multiple UAVs, which is executed by a computer and includes the following steps:

[0090] S1. Obtain the drone formation information and drone formation defense strategy of the defending party. The drone formation of the defending party includes multiple drones; the drone formation defense strategy is used to indicate the decision-making plan of the multiple drones, and the decision-making plan includes defense actions and flight directions; the defense actions include air patrol, calling friendly aircraft and the own aircraft retreating;

[0091] S2. Control the defending drone formation to execute the drone formation defense strategy;

[0092] S3. Acquire a patrol drone in a patrol state, and acquire a detection result of the patrol drone;

[0093] S4, generating a decision-making scheme update model according to the detection result; the decision-making scheme update model includes a state space, a decision scheme, a reward return and a benefit function; the state space is generated by the detection result;

[0094] S5, solving the decision-making scheme update model, and updating the decision-making scheme of the patrol drone according to the model solution;

[0095] S6. Update the drone formation defense strategy according to the updated decision plan, and control the defending drone formation to execute the updated drone formation defense strategy.

[0096] Figure 1 This is a schematic diagram of a scenario of a reinforcement learning method for adaptive intelligent decision-making of multiple drones provided by an embodiment of the present invention. Each step is analyzed in detail below.

[0097] In step S1, the defender's drone formation information and drone formation defense strategy are obtained.

[0098] The defending drone formation may include multiple drones. The defending drone formation information includes information about each drone in the defending drone formation. All drones in the defending drone formation may be of the same model, that is, the speed, maximum fuel capacity and other parameters are the same. Each drone may be numbered in advance to distinguish them.

[0099] When the defending drone formation performs the defense mission, it can execute it according to the drone formation defense strategy. The drone formation defense strategy can indicate the action strategy of each drone, which is called a decision plan in the embodiment of the present application.

[0100] The decision plan can be represented by a two-dimensional vector (θ, z). θ represents the flight direction, that is, the maneuvering angle of the drone. The flight direction is a continuous value ranging from (0, 2π), which represents the maneuvering angle of the drone. z represents the defensive action, that is, the action that the drone should perform. The defensive actions include air patrol, calling friendly aircraft, and the drone's retreat. The drone can perform one of these defensive actions at the same time.

[0101] Air patrol refers to a drone flying in the air to patrol and detect the attacker's drone. In the embodiment of the present application, the attacker's drone is referred to as the attacker's drone. The drone has a detection function and can detect other drones in a certain area ahead. It can detect the attacker's drone and the friendly drone.

[0102] Calling friendly aircraft means that a drone detects attacking drones that exceed its own processing capacity, for example, the number of attacking drones is large, and thus calls the ground formation at the defensive position to send friendly aircraft for reinforcement.

[0103] Local retreat refers to the drone retreating back to the defensive position, including the need to retreat due to the drone's own factors such as insufficient battery and fuel to continue patrolling or the drone itself being damaged.

[0104] UAVs can be equipped with fire attack weapons and electronic jamming weapons, and UAVs can attack a certain enemy UAV. Affected by weapon factors, UAV weapons have a certain effective area, that is, the attack of UAVs is limited to a certain range. In the embodiment of this application, the weapon effective area is called the attack range.

[0105] The attack range of a drone can be considered as a part of the detection range, that is, the attack range is smaller than the detection range. Figure 2 A schematic diagram showing the attack range and detection range of a drone in some embodiments.

[0106] When the defending drone detects the attacking drone, that is, when a certain attacking drone enters the detection range of the defending drone, the defending drone will fly towards the attacking drone until the attacking drone enters the attacking range of the defending drone, and the defending drone will attack the attacking drone with weapons. At the same time, it is set in the embodiment of the present application that when the defending drone encounters an attacking drone within its attack range, it will definitely destroy the attacking drone.

[0107] When the defending drone detects the attacking drone and flies towards it, it will lock onto the attacking drone and chase it until it enters the attack range. At the same time, in the embodiment of the present application, if the defending drone detects that an attacking drone has entered the defense position of the defending party, it is determined that the attacking drone has completed the penetration mission, and the defense of the defending drone against the attacking drone has failed. At this time, the attacking drone is no longer considered, and the defending drone no longer chases the attacking drone. The defending drone resumes patrol to detect the next attacking drone.

[0108] It should be noted that in order to cope with the variability of the attacking drones, the number of drones performing patrol missions at the same time is set to be variable in the embodiment of the present application, that is, the scale of drones flying patrols can be scheduled. For the defending drone formation, only some of the drones may perform flight patrols, while the other drones remain in the defense position on standby.

[0109] At the same time, in the embodiment of the present application, a scheduling cycle is pre-set, for example, it can be 2 seconds. The defending drone formation can update the drone formation defense strategy according to the scheduling cycle. After determining a drone formation defense strategy, the drone formation can execute the strategy for a scheduling cycle, and then update the drone formation defense strategy to adjust the scale of drones flying patrols.

[0110] Before the defense mission begins, all drones in the defense drone formation are located in the defense position. At this time, an initial drone formation defense strategy can be set in advance, for example, the decision plan of each drone can be randomly set.

[0111] In step S2, the defending drone formation is controlled to execute the drone formation defense strategy.

[0112] At the beginning of the defense mission, the defending drone formation can execute the preset initial drone formation defense strategy. The execution time of the initial strategy is the preset scheduling period. After the execution time reaches the scheduling period, the drone formation defense strategy can be updated.

[0113] In step S3, a patrol drone in a patrol state is obtained, and a detection result of the patrol drone is obtained.

[0114] In order to update the drone formation defense strategy, the task execution status of each drone in the current scheduling cycle can be obtained first. In the embodiment of the present application, the drone that is currently flying and in a patrol state is called a patrol drone.

[0115] All patrol drones in the current cycle can be determined, and the detection results of each patrol drone can be obtained. The detection result refers to the detection of drones within its detection range by the drone, including the detection of attacking drones. It should be noted that the drone can perform detection in real time and can perform detection multiple times in a scheduling cycle. In the embodiment of the present application, the detection result obtained is set to be the detection situation of the drone at the last time point of the scheduling cycle.

[0116] In step S4, a decision solution update model is generated according to the detection result.

[0117] The decision-making scheme update model is used to update the decision-making scheme of the patrol drone. The decision-making scheme update model can include the state space, the current drone formation defense strategy, the reward return and the benefit function.

[0118] The decision solution update model can use Markov Game (MG) to formulate the DCA problem. MG is an extension of Markov Decision Process (MDP) to multi-agent problems.

[0119] The MG for air combat is defined as a tuple<N,S,A,R,T> , where N is a set of n drones; S and A are the joint state space and joint action space of multiple drones, respectively, where S = (S1, S2, ..., S n ), A=(A1,A2,…,A n ), S i and A i are the state space and action space of drone i respectively. The reward function of drone i is R i Indicates that in state S i Take action A i The reward obtained. The probability of the state transition function T entering the next state S' when taking action A in state S. In the embodiment of the present application, the next state is the next scheduling cycle. Since all drones in the air combat environment are homogeneous, all drones share parameters and share a strategy π w , represents the probability of the drone selecting an action step by step. The profit function aims to maximize the cumulative return.

[0120] In some embodiments, the state space is generated by the detection result. The detection result can represent the information of the drones detected by the patrol drone within its detection range. The detection result can be intuitively represented in the form of a matrix, and the matrix can be used as the state space.

[0121] For the state space S i , used to represent the discrete space describing the position of drone i, the position of friendly aircraft, and the position of enemy aircraft. In the embodiment of the present application, a double matrix is ​​constructed to represent the state space of drone i in the tth scheduling period The first matrix is ​​used to observe the position of friendly aircraft, and the second matrix is ​​used to observe the position of enemy aircraft.

[0122] It should be noted that, considering that the detection area of ​​the drone is fan-shaped, not all of a matrix is ​​the detection range of the drone. In the embodiment of the present application, the matrix elements outside the detection range are set to -1, and the matrix elements within the detection range are set to 1 or 0, where 0 indicates that the drone is not detected and 1 indicates that the drone is detected.

[0123] Figure 3 Schematic diagrams of state spaces in some embodiments are shown. Figure 3 As shown, two initial matrices are set in the initial state, representing the friendly aircraft matrix and the enemy aircraft matrix respectively. After determining the patrol range of the patrol drone, the initial matrix is ​​adjusted to determine the matrix elements corresponding to the detection area. Among them, the matrix elements outside the detection range, the drone not detected within the detection range, and the drone detected within the detection range are represented in three forms. After obtaining the detection results of the patrol drone, the matrix elements corresponding to the detection area in the matrix are assigned accordingly according to the detection results, so as to obtain the state space of the patrol drone in the current cycle.

[0124] The matrix of the state space can be expressed as:

[0125]

[0126]

[0127] in, The element in the kth row and jth column of the friendly aircraft matrix of drone i is -1 when the position is not within the detection range of the drone. If it is within the detection range and there is a friendly aircraft, the matrix element is 1, and if there is no friendly aircraft, the matrix element is 0.

[0128] The element in the kth row and jth column of the enemy aircraft matrix of drone i is -1 when the position is not within the detection range of the drone. If it is within the detection range and there is an enemy aircraft, the matrix element is 1, and if there is no enemy aircraft, the matrix element is 0.

[0129] The detection results can be input into a decision solution updating model, which can transform the detection results into a state space.

[0130] The action space is used to represent the flight action of the patrol drone in performing the mission, and the action space is the decision plan of the drone. In the embodiment of the present application, it is set that the decision plan calculated in the previous scheduling cycle is used as the action space in the current cycle.

[0131] The defensive actions in the decision scheme can be expressed as:

[0132]

[0133] The reward return is used to characterize the performance of the patrol drone's defense mission in the current scheduling cycle. In the embodiment of the present application, the reward return is set to include an event-triggered reward (also called a first reward return) and a time-continuous reward (also called a second reward return).

[0134] Event-triggered rewards: This type of reward is only given once when the corresponding event is triggered. This type of reward is a goal-oriented reward and punishment item, namely, capture and escape rewards. This reward indicates the reward value generated when our drone successfully captures an enemy aircraft and the enemy aircraft successfully enters our defense line. This type of reward value is relatively large. This reward mainly allows our drone to learn the mission purpose, that is, to minimize the enemy's entry into our defense line, and learn to capture enemy aircraft.

[0135] Time-continuous rewards: This type of reward is calculated and accumulated continuously during the confrontation process. Time step penalty, each drone on the field will deduct a small reward value at each time step. This reward, on the one hand, encourages drones to capture enemy aircraft as quickly as possible, and on the other hand, it can limit the number of drones on the field.

[0136] In the embodiment of the present application, the profit function is set as:

[0137]

[0138] in,

[0139] R (t) represents the reward return of the tth cycle;

[0140] γ represents the discount factor, which is a constant between 0 and 1. t Represents the discount factor raised to the power of t.

[0141] It should be noted that the process of drones performing air defense has high requirements for the real-time update of decision plans. For this reason, patrol drones can be set to update their own decision plans. An update unit can be pre-set in the drone, and a decision plan update model is stored in the update unit. After the patrol drone executes a decision plan for a scheduling cycle, it can directly input the detection results into the decision plan update model in the machine, so as to use the decision plan update model to update the decision plan for the next scheduling cycle.

[0142] In step S5, the decision-making scheme update model is solved, and the decision-making scheme of the patrol drone is updated according to the model solution.

[0143] The model solution includes the following steps:

[0144] S501, obtaining drone state information of the attacking drone according to the state space, wherein the drone state information refers to the attack result of the patrol drone on the detected attacking drone, including the attacking drone defeated by the patrol drone and the attacking drone that broke through the attacked drone and entered the defense position.

[0145] Specifically, the drone status information may include the number of attacking drones defeated by the patrol drone, which is referred to as the first number of attacking drones in the embodiment of the present application. The drone status information also includes the number of attacking drones that enter the defense position of the defending drone formation, which is referred to as the second number of attacking drones in the embodiment of the present application.

[0146] S502, calculating the reward return according to the drone status information. The steps include:

[0147] Obtaining a first reward return according to the drone state information includes:

[0148] R target =m1×f1+m2×f2

[0149] in,

[0150] R target Indicates the first reward return;

[0151] m1 represents the number of drones of the first attacking party, and m2 represents the number of drones of the second attacking party;

[0152] f1 represents the bonus score for defeating the attacking drone, and f2 represents the bonus score for the attacking drone entering the defensive position. f1 can be 1, and f2 can be -1.

[0153] Get the second reward, including:

[0154] R time =-0.01×m3

[0155] in,

[0156] R time represents the second reward return; m3 represents the number of patrol drones.

[0157] Calculate reward returns, including:

[0158] R=R target +R time

[0159] in,

[0160] R represents reward return.

[0161] S503: Solve the profit function according to the reward return. The reward return value can be substituted into the profit function to obtain a model solution that maximizes the profit function.

[0162] The model solution includes the probability distribution of multiple decision-making options, that is, the probability of the decision-making options of the patrol drone in the next cycle.

[0163] The decision-making plan of the patrol drone can be updated based on the model solution, including:

[0164] The decision plan with the highest probability is obtained according to the model solution, and the decision plan with the highest probability is updated as the decision plan of the patrol drone in the next cycle.

[0165] If the patrol drone updates the decision plan by itself, the updated decision plan can be returned to the ground command center.

[0166] In step S6, the drone formation defense strategy is updated according to the updated decision plan, and the defending drone formation is controlled to execute the updated drone formation defense strategy.

[0167] Wherein, updating the drone formation defense strategy includes the following steps:

[0168] S601. According to the update decision plan, obtain the number of third drones whose defensive action is to call friendly drones, and obtain the number of retreating drones whose defensive action is to retreat.

[0169] It should be noted that when a patrol drone calls a friendly drone, the ground command center needs to dispatch a new drone as a patrol drone to perform flight patrol. To this end, the number of drones calling friendly drones can be counted to dispatch the corresponding number of drones for support.

[0170] S602: Acquire the number of standby drones corresponding to the third number of drones in the defending drone formation, and update the decision plan of the standby drones to set the defense action of the standby drones to air patrol.

[0171] A backup drone can be selected from the drones currently staying in the defensive position to perform the flight patrol mission.

[0172] S603: Setting the standby drone as a patrol drone, and canceling the setting of the retreating drone as a patrol drone.

[0173] S604. Count the updated decision plans of multiple drones in the defending drone formation to generate the drone formation defense strategy for the next cycle.

[0174] In the next cycle, the defending drone formation can be controlled to execute the updated drone formation defense strategy.

[0175] In some embodiments, considering that the patrol drone may encounter some unexpected situations during the patrol process, such as insufficient power or fuel, it is necessary to retreat directly.

[0176] To this end, after obtaining the patrol drones in the patrol state, the situation of each patrol drone can be detected to determine whether it needs to be withdrawn.

[0177] In the embodiment of the present application, a patrol cycle is set. When a drone continuously patrols for a period of time that reaches the patrol cycle and no attacking drone is detected during this period, the drone is ordered to retreat.

[0178] To this end, the patrol drone can be controlled to start timing after taking off to perform patrol, and the timing duration can be detected.

[0179] If the patrol drone does not detect the attacking drone within a preset patrol period, the decision plan of the patrol drone is updated to set the patrol drone's defense action to retreat.

[0180] If the patrol drone detects an attacking drone within a preset patrol period, the timing duration is updated to 0, and the patrol drone is controlled to restart the timing and continue patrolling.

[0181] In some embodiments, if the attacking drone poses a greater threat to the defending position, the drone may be required to directly call a friendly aircraft.

[0182] In the embodiment of the present application, it is set as follows: after the patrol drone detects a scheduling cycle, the position of the current attacking drone can be first obtained according to the detection result. If the threat level is large, a friendly drone is directly called. If the threat level is small, the decision plan is updated through the decision plan update model.

[0183] Therefore, after obtaining the detection result of the patrol drone, the patrol threat value of the patrol drone can be obtained. In the embodiment of the present application, the patrol threat is set to include two types: strategic threat and position threat.

[0184] Obtaining patrol threat value includes the following steps:

[0185] The target attacking drone detected by the patrol drone is obtained according to the detection result of the patrol drone.

[0186] Obtaining the action strategy of the target attacking drone. The action strategy of the target attacking drone includes an advance strategy, a retreat strategy, and other strategies. The advance strategy is the behavior of the target attacking drone advancing toward the defense position of the defending drone formation, the retreat strategy is the behavior of the target attacking drone retreating, and the other strategies are strategy behaviors other than the advance strategy and the retreat strategy;

[0187] The strategy threat value is obtained according to the action strategy. The threat value of the advance strategy can be set to 50, the threat value of the retreat strategy can be set to 10, and the threat value of other strategies can be set to 30.

[0188] Get the location threat value, including:

[0189]

[0190] in,

[0191] W distance Indicates the position threat value, d max Indicates the maximum patrol distance of the drones in the defending drone formation. Indicates the distance between the target attacking drone i and the defense position.

[0192] The patrol threat value of the patrol drone is calculated according to the strategy threat value and the position threat value.

[0193] For each target attacking drone, the strategy threat value and position threat value are calculated to obtain the threat value of each target attacking drone.

[0194] The patrol drone may detect multiple target attack drones, and the threat values ​​of all target attack drones are summed up to obtain the patrol threat value of the patrol drone.

[0195] In some embodiments,

[0196] If the patrol threat value is greater than or equal to a preset threat value threshold, the decision plan of the patrol drone is updated to set the patrol drone's defense action to call a friendly drone. The threat value threshold can be set to 100.

[0197] If the patrol threat value is less than a preset threat value threshold, the step of generating a decision solution and updating the model according to the detection result is performed.

[0198] In some embodiments, when a patrol drone detects multiple attacking drones, it may attack the attacking drones in order of threat value from high to low.

[0199] The embodiment of the present invention further provides a reinforcement learning system for adaptive intelligent decision-making of multiple unmanned aerial vehicles, characterized in that the system includes:

[0200] an information acquisition module, configured to acquire information about a defending drone formation and a drone formation defense strategy; the defending drone formation includes a plurality of drones; the drone formation defense strategy is used to indicate a decision plan for the plurality of drones, the decision plan includes a defense action and a flight direction; the defense action includes air patrol, calling friendly aircraft, and the aircraft's own retreat;

[0201] An execution module, configured to control the defending drone formation to execute the drone formation defense strategy;

[0202] A detection module is configured to obtain a patrol drone in a patrol state and obtain a detection result of the patrol drone;

[0203] A model generation module is configured to generate a decision solution update model according to the detection result; the decision solution update model includes a state space, a decision solution, a reward return and a benefit function; the state space is generated by the detection result;

[0204] A decision solution updating module is configured to solve the decision solution updating model and update the decision solution of the patrol drone according to the model solution;

[0205] The defense strategy update module is configured to update the drone formation defense strategy according to the updated decision plan, and control the defending drone formation to execute the updated drone formation defense strategy.

[0206] It is understandable that the target allocation system provided in the embodiment of the present invention corresponds to the reinforcement learning method. The explanation, examples, beneficial effects and other aspects of the relevant contents can refer to the corresponding contents in the reinforcement learning method for adaptive intelligent decision-making of multiple drones, which will not be repeated here.

[0207] The embodiment of the invention also provides a computer-readable storage medium storing a computer program for reinforcement learning of multi-UAV adaptive intelligent decision-making, wherein the computer program enables a computer to execute the reinforcement learning method for multi-UAV adaptive intelligent decision-making as described above.

[0208] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0209] The embodiment of the present invention can obtain the drone formation information and drone formation defense strategy of the defender, wherein the drone formation defense strategy is used to indicate the decision plan of multiple drones, and the decision plan includes defense actions and flight directions; the defense actions include air patrol, calling friendly aircraft and the own aircraft retreat. The drone formation of the defender is controlled to execute the drone formation defense strategy, and the patrol drone in the patrol state is acquired, and the detection results of the patrol drone are acquired. The decision plan update model is generated and solved according to the detection results, and the decision plan of the patrol drone is updated according to the model solution. The drone formation defense strategy is updated according to the updated decision plan, and the drone formation of the defender is controlled to execute the updated drone formation defense strategy. The decision plan of the drone can be updated through the detection results of the drone, so as to real-time schedule the scale of the drone formation when performing the task, thereby improving the defense effect.

[0210] It should be noted that, through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform. The above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform the method of each embodiment or some parts of the embodiment. In the specification provided here, a large number of specific details are explained. However, it can be understood that the embodiments of the present invention can be practiced without these specific details. In some examples, known methods, structures and techniques are not shown in detail so as not to blur the understanding of this specification.

[0211] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0212] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reinforcement learning method for adaptive intelligent decision-making of multiple unmanned aerial vehicles, wherein the reinforcement learning method is executed by a computer and is characterized in that: The following steps are involved: Acquire the drone formation information and drone formation defense strategy of the defending party; the drone formation of the defending party includes multiple drones; the drone formation defense strategy is used to indicate the decision-making plan of the multiple drones, and the decision-making plan includes defense actions and flight directions; the defense actions include air patrol, calling friendly aircraft and the own aircraft retreating; Controlling the defending drone formation to execute the drone formation defense strategy; Acquire a patrol drone in a patrol state, and acquire a detection result of the patrol drone; Generate a decision-making scheme update model according to the detection results; the decision-making scheme update model includes a state space, a decision scheme, a reward return and a benefit function; the state space is generated by the detection results; Solving the decision-making scheme update model, and updating the decision-making scheme of the patrol drone according to the model solution; The drone formation defense strategy is updated according to the updated decision plan, and the defending drone formation is controlled to execute the updated drone formation defense strategy.

2. The reinforcement learning method according to claim 1, characterized in that: Solving the decision solution update model includes: Acquire drone state information of the attacking drone according to the state space; Calculate reward return according to the drone status information; The profit function is solved according to the reward return.

3. The reinforcement learning method according to claim 2, characterized in that: The drone status information includes the number of first attacking drones defeated by the patrol drone and the number of second attacking drones entering the defense position of the defending drone formation; The reward return is calculated according to the drone status information, including: Obtaining a first reward return according to the drone state information includes: R target =m1×f1+m2×f2 in, R target Indicates the first reward return; m1 represents the number of drones of the first attacking party, and m2 represents the number of drones of the second attacking party; f1 represents the bonus score for defeating the attacking drone, and f2 represents the bonus score for the attacking drone entering the defensive position; Get the second reward, including: R time =-0.01×m3 in, R time represents the second reward return; m3 represents the number of patrol drones; Calculate reward returns, including: R=R target +R time in, R represents reward return.

4. The reinforcement learning method according to claim 3, characterized in that: The profit function is: in, R (t) represents the reward return of the tth cycle; γ represents the discount factor, γ t Represents the discount factor raised to the power of t.

5. The reinforcement learning method according to claim 1, characterized in that: After executing the acquisition of the patrol drone in patrol state, it also includes: Control patrol drones to time and detect the timing duration; If the patrol drone does not detect the attacking drone within the preset patrol period, the decision plan of the patrol drone is updated to set the patrol drone's defense action to retreat; If the patrol drone detects the attacking drone within the preset patrol cycle, the timing duration is updated to 0, and the patrol drone is controlled to perform timing.

6. The reinforcement learning method according to claim 1, characterized in that: After obtaining the detection result of the patrol drone, the method further includes: Obtaining the patrol threat value of the patrol drone; If the patrol threat value is greater than or equal to a preset threat value threshold, updating the decision plan of the patrol drone to set the defense action of the patrol drone to call a friendly aircraft; If the patrol threat value is less than a preset threat value threshold, the step of generating a decision solution and updating the model according to the detection result is performed.

7. The reinforcement learning method according to claim 6, characterized in that: Obtaining the patrol threat value of the patrol drone, including: Acquire the target attacking drone detected by the patrol drone according to the detection result of the patrol drone; Obtaining the action strategy of the target attacking drone; the action strategy includes an advance strategy, a retreat strategy, and other strategies; the advance strategy is the behavior of the target attacking drone advancing toward the defense position of the defending drone formation, the retreat strategy is the behavior of the target attacking drone retreating, and the other strategies are strategy behaviors other than the advance strategy and the retreat strategy; Obtaining a strategy threat value according to the action strategy; Get the location threat value, including: in, W distance Indicates the position threat value, d max Indicates the maximum patrol distance of the drones in the defending drone formation. Indicates the distance between the target attacking drone i and the defense position; The patrol threat value of the patrol drone is calculated according to the strategy threat value and the position threat value.

8. The reinforcement learning method according to claim 1, characterized in that: The model solution includes probability distributions of multiple decision-making options; The decision plan of the patrol drone is updated according to the model solution, including: Obtaining a decision plan with the highest probability according to the model solution, and updating it as the decision plan of the patrol drone in the next cycle; The drone formation defense strategy is updated according to the updated decision plan, including: According to the update decision plan, the number of third drones whose defensive action is to call friendly drones is obtained, and the number of retreating drones whose defensive action is to retreat. Acquire a number of standby drones corresponding to the third number of drones in the defending drone formation, and update the decision plan of the standby drones to set the defense action of the standby drones to air patrol; Setting the standby drone as a patrol drone and canceling the setting of the retreating drone as a patrol drone; Statistics are collected on the updated decision plans of multiple drones in the defending drone formation to generate the drone formation defense strategy for the next cycle.

9. A reinforcement learning system for adaptive intelligent decision-making of multiple drones, characterized in that: The system comprises: The information acquisition module is configured to acquire the defender's drone formation information and the drone formation defense strategy; the defender's drone formation includes multiple drones; the drone formation defense strategy is used to indicate the decision-making plan of the multiple drones, the decision-making plan includes defense actions and flight directions; the defense actions include air patrol, calling friendly aircraft and the aircraft's own retreat; An execution module, configured to control the defending drone formation to execute the drone formation defense strategy; A detection module is configured to obtain a patrol drone in a patrol state and obtain a detection result of the patrol drone; A model generation module is configured to generate a decision solution update model according to the detection result; the decision solution update model includes a state space, a decision solution, a reward return and a benefit function; the state space is generated by the detection result; A decision solution updating module is configured to solve the decision solution updating model and update the decision solution of the patrol drone according to the model solution; The defense strategy update module is configured to update the drone formation defense strategy according to the updated decision plan, and control the defending drone formation to execute the updated drone formation defense strategy.

10. A computer-readable storage medium, characterized in that: It stores a computer program for reinforcement learning of multi-UAV adaptive intelligent decision-making, wherein the computer program enables a computer to execute the reinforcement learning method for multi-UAV adaptive intelligent decision-making as described in any one of claims 1 to 8.

Images