Reinforcement learning method and system for intelligent decision making of unmanned aerial vehicle, and storage medium

Abstract

The invention provides a reinforcement learning method and system for unmanned aerial vehicle intelligent decision, and a storage medium, and relates to the technical field of unmanned aerial vehicles. According to the invention, on one hand, the decision-making scheme of the unmanned aerial vehicle can be updated through the detection result of the unmanned aerial vehicle, thereby scheduling the task execution scale of the unmanned aerial vehicle formation in real time, and improving the defense effect. And on the other hand, a boundary grid reward obtained based on the differential game is designed, it is ensured that a reward mechanism can accurately reflect the target achievement situation of both game parties, and through continuous iterative optimization of reinforcement learning, the unmanned aerial vehicle can gradually learn an effective unmanned aerial vehicle formation defense strategy in a complex environment.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a reinforcement learning method, system and storage medium for intelligent decision-making of unmanned aerial vehicles. Background Art

[0002] With the increasing intelligence and performance of drones, the use of multiple drone formations to perform air missions has become a hot topic in the field. The attacker can use drones to pose a serious threat to the defender's core assets with their wide coverage and powerful firepower. Therefore, anti-air operations have become an indispensable key element in military strategic planning, covering two core strategies: offensive anti-air operations (OCA) and defensive anti-air operations (DCA). The main goal of OCA is to minimize the combat effectiveness of the attacker near its source by destroying, disrupting or suppressing the attacker's forces; while DCA focuses on intercepting air targets and ensuring the safety and stability of the defender's important assets by reducing their threat level.

[0003] As a cutting-edge technology in the field of artificial intelligence, multi-agent reinforcement learning has achieved remarkable results in complex game scenarios such as StarCraft and Texas Hold'em with its self-learning and strong exploration capabilities. In recent years, this method has also been widely used to solve the DCA problem. The reward function is an important part of the reinforcement learning algorithm. Due to the existence of the reward function, the algorithm can iterate the network parameters and finally train an algorithm model that meets the research problem. Generally speaking, rewards are artificially created based on experience. In simple research problems, reward functions are easy to construct and easy to understand. However, as the complexity of the problem increases, such as the DCA problem, rewards become more difficult to design, especially intermediate continuous rewards.

[0004] In view of this, it is necessary to provide a new reinforcement learning scheme for UAV intelligent decision-making to make up for the deficiency of multi-agent reinforcement learning in DCA problem that the reward function is difficult to design accurately. Summary of the invention

[0005] 1. Technical issues to be resolved In view of the shortcomings of the prior art, the present invention provides a reinforcement learning method, system and storage medium for unmanned aerial vehicle intelligent decision-making, which solves the technical problem that the reward function of multi-agent reinforcement learning in the DCA problem is difficult to accurately design.

[0006] (II) Technical solution To achieve the above objectives, the present invention is implemented through the following technical solutions: A reinforcement learning method for intelligent decision-making of unmanned aerial vehicles, comprising: Acquire the drone formation information and drone formation defense strategy of the defending party; the drone formation of the defending party includes multiple drones; the drone formation defense strategy is used to indicate the decision-making plan of the multiple drones, and the decision-making plan includes defense actions and flight directions; the defense actions include air patrol, calling friendly aircraft and the own aircraft retreating; Controlling the defending drone formation to execute the drone formation defense strategy; Acquire a patrol drone in a patrol state, and acquire a detection result of the patrol drone; A decision-making scheme update model is generated according to the detection results; the decision-making scheme update model includes a state space, a decision scheme, a reward return and a benefit function; the state space is generated by the detection results; the reward return at least includes a boundary fence reward obtained based on a differential game; Solving the decision-making scheme update model, and updating the decision-making scheme of the patrol drone according to the model solution; The drone formation defense strategy is updated according to the updated decision plan, and the defending drone formation is controlled to execute the updated drone formation defense strategy.

[0007] Preferably, solving the decision solution update model includes: Acquire drone state information of the attacking drone according to the state space; Calculate a reward return including at least the Jieshan reward according to the drone status information; The profit function is solved according to the reward return.

[0008] Preferably, the drone status information includes a speed ratio between the patrol drone and the attacking drone; The process of obtaining the boundary fence reward includes: Based on the speed ratio and the position of the defense position of the defense drone formation, differential game is used to construct the boundary fence of the patrol drone to construct the pursuit zone and escape zone of the patrol drone; Get Boundary Fence rewards, including: Among them, the subscript C CZ , C EZ Respectively represent all attacking drones located in the pursuit zone and escape zone of the patrol drone; It represents the reduction in the distance between the patrol drone and all attacking drones in the pursuit area in a single cycle; It represents the reduction in the distance between the patrol drone and all attacking drones in the escape zone in a single cycle; The drone status information also includes the number of first attacking drones defeated by the patrol drone and the number of second attacking drones that entered the defense position of the defending drone formation; Calculating a reward return including at least the Jieshan reward according to the drone state information also includes: Obtaining a first reward return according to the drone state information includes: in, R target Indicates the first reward return; m 1 indicates the number of drones of the first attacking party, m 2 represents the number of drones of the second attacking party; Defeated the attacking drone j Bonus score, if the attacking drone j If a patrol drone is defeated in its escape zone after it calls a friendly drone, the bonus score is upgraded to the preset multiple of the first basic bonus score, otherwise it remains at the first basic bonus score; Indicates the attacking drone j Bonus points for entering a defensive position, if the attacking drone j If you enter a defensive position in a patrol drone's pursuit zone after the patrol drone calls a friendly drone, the bonus score will be upgraded to the preset multiple of the second basic bonus score, otherwise it will remain at the second basic bonus score; Get the second reward, including: in, R time Indicates the second reward return; m 3 indicates the number of patrol drones; Calculate reward returns, including: in, R Indicates reward return.

[0009] Preferably, the profit function is: Among them, argmaxf( x ) means that the objective function f( x ) The variable with the maximum value x The value of θ Indicates the flight direction; R ( t ) indicates the t Reward return for each cycle; γ represents the discount factor, γ t The discount factor t Power.

[0010] Preferably, after executing the acquisition of the patrol drone in the patrol state, the method further includes: Control patrol drones to time and detect the timing duration; If the patrol drone does not detect the attacking drone within the preset patrol period, the decision plan of the patrol drone is updated to set the patrol drone's defense action to retreat; If the patrol drone detects the attacking drone within the preset patrol cycle, the timing duration is updated to 0, and the patrol drone is controlled to perform timing.

[0011] Preferably, after obtaining the detection result of the patrol drone, the method further includes: Obtaining the patrol threat value of the patrol drone; If the patrol threat value is greater than or equal to a preset threat value threshold, updating the decision plan of the patrol drone to set the defense action of the patrol drone to call a friendly aircraft; If the patrol threat value is less than a preset threat value threshold, the step of generating a decision solution and updating the model according to the detection result is performed.

[0012] Preferably, obtaining the patrol threat value of the patrol drone includes: Acquire the target attacking drone detected by the patrol drone according to the detection result of the patrol drone; Obtaining the action strategy of the target attacking drone; the action strategy includes an advance strategy, a retreat strategy, and other strategies; the advance strategy is the behavior of the target attacking drone advancing toward the defense position of the defending drone formation, the retreat strategy is the behavior of the target attacking drone retreating, and the other strategies are strategy behaviors other than the advance strategy and the retreat strategy; Obtaining a strategic threat value according to the action strategy; Get the location threat value, including: in, W distance Indicates the position threat value, d max Indicates the maximum patrol distance of the drones in the defending drone formation. Indicates the target attacking drone j * distance from defensive positions; The patrol threat value of the patrol drone is calculated according to the strategy threat value and the position threat value.

[0013] Preferably, the model solution includes probability distributions of multiple decision-making solutions; The decision plan of the patrol drone is updated according to the model solution, including: Obtaining a decision plan with the highest probability according to the model solution, and updating it as the decision plan of the patrol drone in the next cycle; The drone formation defense strategy is updated according to the updated decision plan, including: According to the updated decision plan, the number of third drones whose defensive action is to call friendly drones is obtained, and the number of retreating drones whose defensive action is to retreat. Acquire a number of standby drones corresponding to the third number of drones in the defending drone formation, and update the decision plan of the standby drones to set the defense action of the standby drones to air patrol; Setting the standby drone as a patrol drone and canceling the setting of the retreating drone as a patrol drone; Statistics are collected on the updated decision plans of multiple drones in the defending drone formation to generate the drone formation defense strategy for the next cycle.

[0014] A reinforcement learning system for intelligent decision-making of unmanned aerial vehicles, the system comprising: The information acquisition module is configured to acquire the defender's drone formation information and the drone formation defense strategy; the defender's drone formation includes multiple drones; the drone formation defense strategy is used to indicate the decision-making plan of the multiple drones, the decision-making plan includes defense actions and flight directions; the defense actions include air patrol, calling friendly aircraft and the aircraft's own retreat; A strategy execution module, configured to control the defending drone formation to execute the drone formation defense strategy; A result detection module is configured to obtain a patrol drone in a patrol state and obtain a detection result of the patrol drone; A model generation module is configured to generate a decision solution update model according to the detection result; the decision solution update model includes a state space, a decision solution, a reward return and a benefit function; the state space is generated by the detection result; the reward return at least includes a boundary fence reward obtained based on a differential game; A solution updating module is configured to solve the decision solution updating model and update the decision solution of the patrol drone according to the model solution; The strategy update module is configured to update the drone formation defense strategy according to the updated decision plan, and control the defending drone formation to execute the updated drone formation defense strategy.

[0015] A storage medium storing a computer program for reinforcement learning of drone intelligent decision-making, wherein the computer program enables a computer to execute the reinforcement learning method for drone intelligent decision-making as described above (III) Beneficial effects The present invention provides a reinforcement learning method, system and storage medium for intelligent decision-making of unmanned aerial vehicles. Compared with the prior art, it has the following beneficial effects: In the present invention, on the one hand, the decision-making plan of the drone can be updated through the detection results of the drone, so as to real-time dispatch the scale of the drone formation when executing the task, thereby improving the defense effect. On the other hand, the boundary fence reward obtained based on differential game is designed to ensure that the reward mechanism can accurately reflect the goal achievement of both parties in the game, and through the continuous iterative optimization of reinforcement learning, the drone can gradually learn the effective drone formation defense strategy in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A schematic diagram of a scenario of a reinforcement learning method for intelligent decision-making of a drone provided by an embodiment of the present invention; Figure 2 A schematic diagram of the attack range and detection range of a drone provided by an embodiment of the present invention; Figure 3 A schematic diagram of a state space provided by an embodiment of the present invention; Figure 4 A schematic diagram of a boundary gate structure provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The embodiments of the present application solve the technical problem that the reward function in the DCA problem of multi-agent reinforcement learning is difficult to accurately design by providing a reinforcement learning method, system and storage medium for drone intelligent decision-making.

[0020] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0021] Multiple drone formations can perform air missions, and the attacker can use drone formations to attack the defender's defensive positions. For the defender, it can perform anti-air operations to prevent the attacker's drone formation from entering the defensive position.

[0022] The defender can implement two strategies: offensive counter-air operations (OCA) and defensive counter-air operations (DCA). Among them, the main goal of OCA is for the defender's drones to destroy, disrupt or suppress the attacker's forces, and minimize their combat effectiveness near their source. DCA is for the defender's drones to intercept the attacker's drones, and by reducing their threat level, ensure the safety and stability of the defender's important assets.

[0023] As a key strategy to ensure airspace security, DCA covers two major aspects: active air defense and missile defense. It relies on a variety of asset types and system integration, including but not limited to fighter jets, surface-to-air missiles, anti-aircraft guns, electromagnetic warfare systems, and ballistic missile defense systems. These systems work together to destroy enemy forces or effectively weaken their offensive capabilities. When performing DCA missions, the fighter jets used may be traditional manned models or advanced unmanned aerial vehicles (UAVs). In particular, weaponized drones designed for combat, namely unmanned combat aerial vehicles (UCAVs), can be remotely controlled and have the ability to perform tasks autonomously.

[0024] When using the DCA strategy, the defender usually fixes the size of the drones that perform the mission. For example, if the drone formation consists of 5 drones, the defender usually lets all 5 drones take off and perform the defense mission. In the embodiment of the present application, performing the defense mission means that the defender's drone patrols in the air to detect the attacker's drone and attack it.

[0025] However, the arrival of the attacking drones is often uncertain, and the number of attacking drones is usually unknown. The attacking drones may advance towards the defense position in batches and at multiple time periods. The defender's use of a preset scale of drones may result in a waste or shortage of resources, making it difficult to respond to emergencies and resulting in poor defense results.

[0026] In particular, multi-agent reinforcement learning methods are also widely used to solve DCA problems. The embodiments of the present application also recognize that the reward function is an important part of the reinforcement learning algorithm. Due to the existence of the reward function, the algorithm can iterate the network parameters and finally train an algorithm model that meets the research problem. Different from simple research problems, the reward function is easy to construct and easy to understand. Since the DCA problem is more complex, its rewards are also more difficult to design, especially the intermediate continuous rewards.

[0027] Embodiment 1: To solve the above problems, an embodiment of the present invention provides a reinforcement learning method for intelligent decision-making of a drone, which is executed by a computer and includes the following steps: A reinforcement learning method for intelligent decision-making of unmanned aerial vehicles, characterized by comprising: S1. Obtaining the drone formation information and drone formation defense strategy of the defending party; the drone formation of the defending party includes multiple drones; the drone formation defense strategy is used to indicate the decision-making plan of the multiple drones, and the decision-making plan includes defense actions and flight directions; the defense actions include air patrol, calling friendly aircraft and the own aircraft retreating; S2. Control the defending drone formation to execute the drone formation defense strategy; S3. Acquire a patrol drone in a patrol state, and acquire a detection result of the patrol drone; S4, generating a decision-making scheme update model according to the detection results; the decision-making scheme update model includes a state space, a decision scheme, a reward return and a benefit function; the state space is generated by the detection results; the reward return at least includes a boundary fence reward obtained based on a differential game; S5, solving the decision-making scheme update model, and updating the decision-making scheme of the patrol drone according to the model solution; S6. Update the drone formation defense strategy according to the updated decision plan, and control the defending drone formation to execute the updated drone formation defense strategy.

[0028] Figure 1 This is a schematic diagram of a scenario of a reinforcement learning method for intelligent decision-making of a drone provided by an embodiment of the present invention. The following is a detailed analysis of each step: In step S1, the defender's drone formation information and drone formation defense strategy are obtained.

[0029] The defending drone formation may include multiple drones. The defending drone formation information includes information about each drone in the defending drone formation. All drones in the defending drone formation may be of the same model, that is, the speed, maximum fuel capacity and other parameters are the same. Each drone may be numbered in advance to distinguish them.

[0030] When the defending drone formation performs the defense mission, it can execute it according to the drone formation defense strategy. The drone formation defense strategy can indicate the action strategy of each drone, which is called a decision plan in the embodiment of the present application.

[0031] The decision solution can be made using a two-dimensional vector ( θ , z ) indicates. Among them, θ Indicates the flight direction, that is, the maneuvering angle of the drone. The flight direction is a continuous value ranging from (0,2 π ), which indicates the maneuvering angle of the drone. z Indicates the defensive action, that is, the action that the drone should perform. The defensive actions include air patrol, calling friendly aircraft and the drone retreating. The drone can perform one of these defensive actions at the same time.

[0032] Air patrol refers to a drone flying in the air to patrol and detect the attacker's drone. In the embodiment of the present application, the attacker's drone is referred to as the attacker's drone. The drone has a detection function and can detect other drones in a certain area ahead. It can detect the attacker's drone and the friendly drone.

[0033] Calling friendly aircraft means that a drone detects attacking drones that exceed its own processing capacity, for example, the number of attacking drones is large, and thus calls the ground formation at the defensive position to send friendly aircraft for reinforcement.

[0034] Local retreat refers to the drone retreating back to the defensive position, including the need to retreat due to the drone's own factors such as insufficient battery and fuel to continue patrolling or the drone itself being damaged.

[0035] UAVs can be equipped with fire attack weapons and electronic jamming weapons, and UAVs can attack a certain enemy UAV. Affected by weapon factors, UAV weapons have a certain effective area, that is, the attack of UAVs is limited to a certain range. In the embodiment of this application, the weapon effective area is called the attack range.

[0036] The attack range of a drone can be considered as a part of the detection range, that is, the attack range is smaller than the detection range. Figure 2 A schematic diagram showing the attack range and detection range of a drone in some embodiments.

[0037] When the defending drone detects the attacking drone, that is, when a certain attacking drone enters the detection range of the defending drone, the defending drone will fly towards the attacking drone until the attacking drone enters the attacking range of the defending drone, and the defending drone will attack the attacking drone with weapons. At the same time, it is set in the embodiment of the present application that when the defending drone encounters an attacking drone within its attack range, it will definitely destroy the attacking drone.

[0038] When the defending drone detects the attacking drone and flies towards it, it will lock onto the attacking drone and chase it until it enters the attack range. At the same time, in the embodiment of the present application, if the defending drone detects that an attacking drone has entered the defense position of the defending party, it is determined that the attacking drone has completed the penetration mission, and the defense of the defending drone against the attacking drone has failed. At this time, the attacking drone is no longer considered, and the defending drone no longer pursues the attacking drone. The defending drone resumes patrol to detect the next attacking drone.

[0039] It should be noted that in order to cope with the variability of the attacking drones, the number of drones performing patrol missions at the same time is set to be variable in the embodiment of the present application, that is, the scale of drones flying patrols can be scheduled. For the defending drone formation, only some of the drones may perform flight patrols, while the other drones remain in the defense position on standby.

[0040] At the same time, in the embodiment of the present application, a scheduling cycle is pre-set, for example, it can be 2 seconds. The defending drone formation can update the drone formation defense strategy according to the scheduling cycle. After determining a drone formation defense strategy, the drone formation can execute the strategy for a scheduling cycle, and then update the drone formation defense strategy to adjust the scale of drones flying patrols.

[0041] Before the defense mission begins, all drones in the defense drone formation are located in the defense position. At this time, an initial drone formation defense strategy can be set in advance, for example, the decision plan of each drone can be randomly set.

[0042] In step S2, the defending drone formation is controlled to execute the drone formation defense strategy.

[0043] At the beginning of the defense mission, the defending drone formation can execute the preset initial drone formation defense strategy. The execution time of the initial strategy is the preset scheduling period. After the execution time reaches the scheduling period, the drone formation defense strategy can be updated.

[0044] In step S3, a patrol drone in a patrol state is obtained, and a detection result of the patrol drone is obtained.

[0045] To update the drone formation defense strategy, the task execution status of each drone in the current scheduling cycle may be obtained first. In the embodiment of the present application, the drone currently flying and in a patrol state is referred to as a patrol drone.

[0046] All patrol drones in the current cycle can be determined, and the detection results of each patrol drone can be obtained. The detection result refers to the detection of drones within its detection range by the drone, including the detection of attacking drones. It should be noted that the drone can perform detection in real time and can detect multiple times in a scheduling cycle. In the embodiment of the present application, the detection result obtained is set to be the detection of the drone at the last time point of the scheduling cycle.

[0047] In step S4, a decision solution update model is generated according to the detection result.

[0048] The decision-making scheme update model is used to update the decision-making scheme of the patrol drone. The decision-making scheme update model can include the state space, the current drone formation defense strategy, the reward return and the benefit function.

[0049] The decision solution update model can use Markov game (MG) to formulate the DCA problem. MG is an extension of Markov decision process (MDP) to multi-agent problems.

[0050] The MG of the DCA problem is defined as a tuple < N,S,A,R,T >, where N yes n A collection of drones; S and A are the joint state space and joint action space of multiple UAVs, respectively, where S= ( S 1 ,S 2 ,…,S n ), A= ( A 1 ,A 2 ,…,A n ), S i and A i UAV i The state space and action space of drones i The reward function is R i Indicates in status S i Take action whenA i The reward obtained. State transfer function T In Status S Take action when A Enter the next state S’ In the embodiment of the present application, the next state is the next scheduling cycle. Since all drones in the environment are homogeneous, all drones share parameters and a strategy. π w , represents the probability of the drone selecting an action step by step. The profit function aims to maximize the cumulative return.

[0051] In some embodiments, the state space is generated by the detection result. The detection result can represent the information of the drones detected by the patrol drone within its detection range. The detection result can be intuitively represented in the form of a matrix, and the matrix can be used as the state space.

[0052] For the state space S i , used to describe drones i The discrete space of the position of the aircraft, the position of the friendly aircraft, and the position of the attacking drone. In the embodiment of the present application, a double matrix is ​​constructed to represent the drone i In the t The state space of the scheduling cycle , the first matrix is ​​used to observe the position of the friendly aircraft, and the second matrix is ​​used to observe the position of the attacking drone.

[0053] It should be noted that, considering that the detection area of ​​the drone is fan-shaped, not all of a matrix is ​​the detection range of the drone. In the embodiment of the present application, the matrix elements outside the detection range are set to -1, and the matrix elements within the detection range are set to 1 or 0, where 0 indicates that the drone is not detected and 1 indicates that the drone is detected.

[0054] Figure 3 Schematic diagrams of state spaces in some embodiments are shown. Figure 3 As shown in the figure, two initial matrices are set in the initial state, representing the friendly drone matrix and the attacking drone matrix respectively. After determining the patrol range of the patrol drone, the initial matrix is ​​adjusted to determine the matrix elements corresponding to the detection area. Among them, the matrix elements outside the detection range, the drone not detected within the detection range, and the drone detected within the detection range are represented in three forms. After obtaining the detection results of the patrol drone, the matrix elements corresponding to the detection area in the matrix are assigned accordingly according to the detection results, so as to obtain the state space of the patrol drone in the current cycle.

[0055] The matrix of the state space can be expressed as: in, Drones i Friendship Matrix k Row, No. j Column elements, when the position is not within the detection range of the drone, the matrix element is -1, if it is within the detection range, the matrix element is 1 when there is a friendly drone, and the matrix element is 0 when there is no friendly drone.

[0056] Drones i The attacking drone matrix k Row, No. j Column elements, when the position is not within the detection range of the drone, the matrix element is -1, if it is within the detection range, the matrix element is 1 when there is an attacking drone, and the matrix element is 0 when there is no attacking drone.

[0057] The detection results can be input into a decision solution updating model, which can transform the detection results into a state space.

[0058] The action space is used to represent the flight action of the patrol drone in performing the mission, and the action space is the decision plan of the drone. In the embodiment of the present application, it is set that the decision plan calculated in the previous scheduling cycle is used as the action space in the current cycle.

[0059] The defensive actions in the decision scheme can be expressed as: The reward return is used to characterize the performance of the patrol drone's defense mission in the current scheduling cycle. In the embodiment of the present application, in order for the drone to gradually learn effective strategies in complex environments, and to make up for the deficiency of the difficulty in accurately designing the reward function of reinforcement learning in the DCA problem, the reward return is set to at least include the boundary fence reward obtained based on the differential game, and the reward is shaped by using the differential game to ensure that the reward mechanism can accurately reflect the goal achievement of both parties in the game (i.e., the defending drone and the attacking drone).

[0060] It should be noted that the barrier is an important concept in qualitative differential game theory and is widely used in pursuit, military confrontation and other fields. The barrier divides the confrontation area into an escape zone and a pursuit zone, and satisfies the following conclusions: for the attacker's drone target in the escape zone, no matter what maneuver strategy the defender adopts, the attacker's drone target must have at least one strategy that can successfully reach the destination; for the attacker's drone target in the pursuit zone, no matter what penetration strategy the attacker's drone adopts, the defender must have a maneuver strategy that can successfully intercept the attacker's drone.

[0061] In addition, the reward returns can also be set to include event-triggered rewards (also called first reward returns) and time-continuing rewards (also called second reward returns).

[0062] Event-triggered rewards: This type of reward is only given once when the corresponding event is triggered. This type of reward is a goal-oriented reward and punishment item, namely, capture and escape rewards. This reward means that when our drone successfully captures the attacker's drone and the attacker's drone successfully enters our defense position, the reward value generated is relatively large. This reward mainly allows our drone to learn the mission purpose, that is, to minimize the number of enemies entering our defense position, and learn to capture the attacker's drone.

[0063] Furthermore, the above-mentioned corresponding events can also be associated with the concepts related to the escape zone and pursuit zone of the aforementioned boundary fence rewards to design the above-mentioned reward upgrade method for event-triggered rewards. It will not be introduced in detail here. For details, please refer to the construction content of the first reward return in the following text.

[0064] Time-continuous rewards: This type of reward is calculated and accumulated continuously during the confrontation. Time step penalty, each drone on the field will deduct a small reward value at each time step. This reward, on the one hand, encourages drones to capture the attacking drones as quickly as possible, and on the other hand, it can limit the number of drones on the field.

[0065] In the embodiment of the present application, the profit function is set as: Among them, argmaxf( x ) means that the objective function f( x ) The variable with the maximum value x The value of θ Indicates the flight direction; R ( t ) indicates the t Reward return for each cycle; γ represents the discount factor, γ t The discount factor t Power.

[0066] It should be noted that the process of drones performing air defense has high requirements for the real-time update of decision plans. For this reason, patrol drones can be set to update their own decision plans. An update unit can be pre-set in the drone, and a decision plan update model is stored in the update unit. After the patrol drone executes a decision plan for a scheduling cycle, it can directly input the detection results into the decision plan update model in the machine, so as to use the decision plan update model to update the decision plan for the next scheduling cycle.

[0067] In step S5, the decision-making scheme update model is solved, and the decision-making scheme of the patrol drone is updated according to the model solution.

[0068] The model solution includes the following steps: S501. Acquire drone state information of the attacking drone according to the state space.

[0069] Among them, drone status information refers to the attack results of patrol drones on detected attacking drones, including situations where patrol drones defeat attacking drones, and situations where attacked drones break through and enter defensive positions.

[0070] Specifically, the drone status information may include a speed ratio between the patrol drone and the attacking drone.

[0071] Furthermore, the drone status information may also include the number of attacking drones defeated by the patrol drone, which is referred to as the first number of attacking drones in the embodiment of the present application. The drone status information also includes the number of attacking drones that enter the defense position of the defending drone formation, which is referred to as the second number of attacking drones in the embodiment of the present application.

[0072] S502. Calculate a reward return including at least the Jieshan reward according to the drone status information.

[0073] Referring to the boundary fence construction theory of the existing literature (Rui Yan, Zongying Shi, and Yisheng Zhong, "Reach-Avoid Games With Two Defenders and One Attacker: An Analytical Approach," IEEE Transactions on Cybernetics, Volume: 49, Issue: 3, March 2019.), under the condition that the speed ratio of both parties and the position of the defense position of the defender's drone formation are known, the boundary fence of each patrol drone can be quickly solved to obtain the escape zone and pursuit zone. Based on this, a boundary fence is constructed for each patrol drone, and the action strategy of the patrol drone is guided according to the position of the attacker drone in the pursuit zone and the escape zone.

[0074] Figure 4 Schematic diagrams of boundary fence structures in some embodiments are shown, taking the patrol drone on the far left in the figure as an example: the dotted part is its boundary fence, the left side of the dotted line is its escape zone, and the right side of the dotted line is its pursuit zone.

[0075] On this basis, the calculation of the reward return including at least the Jieshan reward according to the drone state information includes the following steps: Based on the speed ratio and the position of the defense position of the defending drone formation, a boundary fence of the patrol drone is constructed using differential game to construct the pursuit zone and escape zone of the patrol drone: in, Patrol drone i of the pursuit area; Patrol drone i escape zone; Barrier It represents the relevant formula for constructing the boundary fence in the above-mentioned document "Reach-Avoid Games With Two Defenders and One Attacker: An Analytical Approach", which will not be explained in detail here.

[0076] When at least one attacking drone is attacked by a patrol drone i Observe and define two sets: Indicates that it is located Attacker drones within Indicates that it is located All attacking drones within.

[0077] Get Boundary Fence rewards, including: Subscript C CZ , C EZ Respectively represent all attacking drones located in the pursuit zone and escape zone of the patrol drone; Indicates the number of patrol drones and pursuit zones in a single cycle. The distance between all attacking drones is reduced; It represents the reduction in the distance between the patrol drone and all attacking drones in the escape zone in a single cycle, that is, the patrol drone i With Collection The amount by which the distance between all attacking drones in a .

[0078] Calculating a reward return including at least the Jieshan reward according to the drone state information also includes: Obtaining a first reward return according to the drone state information includes: in, R targetIndicates the first reward return; m 1 indicates the number of drones of the first attacking party, m 2 indicates the number of drones of the second attacking party.

[0079] Defeated the attacking drone j Bonus score, if the attacking drone j If a patrol drone is defeated in the escape zone after the patrol drone calls a friendly drone, the bonus score will be upgraded to the preset multiple of the first basic bonus score, otherwise it will remain at the first basic bonus score; that is, when the attacking drone j ∈ , patrol drone i After the output z is 1, the patrol drone i Defeat the attacking drone j ,but The reward will be upgraded, otherwise the original reward score will be maintained.

[0080] For example, the first basic reward score is 1, and the preset multiplier is a positive integer. k .

[0081] Indicates the attacking drone j Bonus points for entering a defensive position, if the attacking drone j If a patrol drone enters a defensive position after calling a friendly drone in the pursuit area, the bonus score will be upgraded to the preset multiple of the second basic bonus score, otherwise it will remain at the second basic bonus score; that is, when the attacking drone enters a defensive position after calling a friendly drone ... j ∈ , patrol drone i After the output z is 1, the patrol drone i Defeat the attacking drone j ,but The reward will be upgraded, otherwise the original reward score will be maintained.

[0082] For example, the second basic reward score is -1, and the preset multiplier is a positive integer. k .

[0083] It is not difficult to understand that in the embodiment of the present application, when the patrol drone i When there is an attacking drone in the escape zone, it is hoped that the patrol drone iCan call for support. However, in this case, the action only needs to be done once. For example, when the robot is learning to enter the door, it needs to press the doorbell first and then enter the door. Not pressing the doorbell or continuously pressing the doorbell is not the optimal result. In this regard, the embodiment of the present application is based on the escape zone and pursuit zone in the Jieshan reward, and designs the above-mentioned reward upgrade method for event-triggered rewards to achieve more rewards and penalties after the patrol drone performs a specific action, further ensuring that the provided reward mechanism can accurately reflect the actual interaction and goal achievement of the two parties in the game.

[0084] Get the second reward, including: in, R time Indicates the second reward return; m 3 indicates the number of patrol drones.

[0085] Calculate reward returns, including: in, R Indicates reward return.

[0086] S503: Solve the profit function according to the reward return. The reward return value can be substituted into the profit function to obtain a model solution that maximizes the profit function.

[0087] The model solution includes the probability distribution of multiple decision-making options, that is, the probability of the decision-making options of the patrol drone in the next cycle.

[0088] The decision-making plan of the patrol drone can be updated based on the model solution, including: The decision plan with the highest probability is obtained according to the model solution, and the decision plan with the highest probability is updated as the decision plan of the patrol drone in the next cycle.

[0089] If the patrol drone updates the decision plan by itself, the updated decision plan can be returned to the ground command center.

[0090] In step S6, the drone formation defense strategy is updated according to the updated decision plan, and the defending drone formation is controlled to execute the updated drone formation defense strategy.

[0091] Wherein, updating the drone formation defense strategy includes the following steps: S601. According to the updated decision plan, obtain the number of third drones whose defensive action is to call friendly drones, and obtain the number of retreating drones whose defensive action is to retreat.

[0092] It should be noted that when a patrol drone calls a friendly drone, the ground command center needs to dispatch a new drone as a patrol drone to perform flight patrol. To this end, the number of drones calling friendly drones can be counted to dispatch the corresponding number of drones for support.

[0093] S602: Acquire the number of standby drones corresponding to the third number of drones in the defending drone formation, and update the decision plan of the standby drones to set the defense action of the standby drones to air patrol.

[0094] A backup drone can be selected from the drones currently staying in the defensive position to perform the flight patrol mission.

[0095] S603: Setting the standby drone as a patrol drone, and canceling the setting of the retreating drone as a patrol drone.

[0096] S604. Count the updated decision plans of multiple drones in the defending drone formation to generate the drone formation defense strategy for the next cycle.

[0097] In the next cycle, the defending drone formation can be controlled to execute the updated drone formation defense strategy.

[0098] In some embodiments, considering that the patrol drone may encounter some unexpected situations during the patrol process, such as insufficient power or fuel, it is necessary to retreat directly.

[0099] To this end, after obtaining the patrol drones in the patrol state, the situation of each patrol drone can be detected to determine whether it needs to be withdrawn.

[0100] In the embodiment of the present application, a patrol cycle is set. When a drone continuously patrols for a period of time that reaches the patrol cycle and no attacking drone is detected during this period, the drone is ordered to retreat.

[0101] To this end, the patrol drone can be controlled to start timing after taking off to perform patrol, and the timing duration can be detected.

[0102] If the patrol drone does not detect the attacking drone within a preset patrol period, the decision plan of the patrol drone is updated to set the patrol drone's defense action to retreat.

[0103] If the patrol drone detects an attacking drone within a preset patrol period, the timing duration is updated to 0, and the patrol drone is controlled to restart the timing and continue patrolling.

[0104] In some embodiments, if the attacking drone poses a greater threat to the defending position, the drone may be required to directly call a friendly aircraft.

[0105] In the embodiment of the application, it is set as follows: after the patrol drone detects a scheduling cycle, the position of the current attacking drone can be obtained according to the detection results. If the threat level is large, a friendly drone is directly called. If the threat level is small, the decision plan is updated through the decision plan update model.

[0106] Therefore, after obtaining the detection result of the patrol drone, the patrol threat value of the patrol drone can be obtained. In the embodiment of the present application, the patrol threat is set to include two types: strategic threat and position threat.

[0107] Obtaining patrol threat value includes the following steps: The target attacking drone detected by the patrol drone is obtained according to the detection result of the patrol drone.

[0108] Obtaining the action strategy of the target attacking drone. The action strategy of the target attacking drone includes an advance strategy, a retreat strategy, and other strategies. The advance strategy is the behavior of the target attacking drone advancing toward the defense position of the defending drone formation, the retreat strategy is the behavior of the target attacking drone retreating, and the other strategies are strategy behaviors other than the advance strategy and the retreat strategy; The strategy threat value is obtained according to the action strategy. The threat value of the advance strategy can be set to 50, the threat value of the retreat strategy can be set to 10, and the threat value of other strategies can be set to 30.

[0109] Get the location threat value, including: in, W distance Indicates the position threat value, d max Indicates the maximum patrol distance of the drones in the defending drone formation. Indicates the target attacking drone j * distance from defensive positions; The patrol threat value of the patrol drone is calculated according to the strategy threat value and the position threat value.

[0110] For each target attacking drone, the strategy threat value and position threat value are calculated to obtain the threat value of each target attacking drone.

[0111] The patrol drone may detect multiple target attack drones, and the threat values ​​of all target attack drones are summed up to obtain the patrol threat value of the patrol drone.

[0112] In some embodiments, If the patrol threat value is greater than or equal to a preset threat value threshold, the decision plan of the patrol drone is updated to set the patrol drone's defense action to call a friendly drone. The threat value threshold can be set to 100.

[0113] If the patrol threat value is less than a preset threat value threshold, the step of generating a decision solution and updating the model according to the detection result is performed.

[0114] In some embodiments, when a patrol drone detects multiple attacking drones, it may attack the attacking drones in order of threat value from high to low.

[0115] Embodiment 2: The embodiment of the present invention further provides a reinforcement learning system for intelligent decision-making of unmanned aerial vehicles, the system comprising: The information acquisition module is configured to acquire the defender's drone formation information and the drone formation defense strategy; the defender's drone formation includes multiple drones; the drone formation defense strategy is used to indicate the decision-making plan of the multiple drones, the decision-making plan includes defense actions and flight directions; the defense actions include air patrol, calling friendly aircraft and the aircraft's own retreat; A strategy execution module, configured to control the defending drone formation to execute the drone formation defense strategy; A result detection module is configured to obtain a patrol drone in a patrol state and obtain a detection result of the patrol drone; A model generation module is configured to generate a decision solution update model according to the detection result; the decision solution update model includes a state space, a decision solution, a reward return and a benefit function; the state space is generated by the detection result; the reward return at least includes a boundary fence reward obtained based on a differential game; A solution updating module is configured to solve the decision solution updating model and update the decision solution of the patrol drone according to the model solution; The strategy update module is configured to update the drone formation defense strategy according to the updated decision plan, and control the defending drone formation to execute the updated drone formation defense strategy.

[0116] Embodiment 3: An embodiment of the present invention also provides a storage medium that stores a computer program for reinforcement learning of drone intelligent decision-making, wherein the computer program enables a computer to execute the reinforcement learning method for drone intelligent decision-making as described in Example 1.

[0117] It is understandable that the reinforcement learning system and storage medium for intelligent decision-making of drones provided in the embodiments of the present invention correspond to the reinforcement learning method. The explanations, examples, beneficial effects, etc. of the relevant contents can refer to the corresponding contents in the reinforcement learning method for intelligent decision-making of drones, and will not be repeated here.

[0118] In summary, compared with the prior art, the present invention has the following beneficial effects: 1. The embodiment of the present invention combines differential countermeasures with reinforcement learning to overcome the difficulty of differential games in solving multi-machine collaborative boundary barriers. Specifically, the boundary barrier reward obtained based on differential games is designed to ensure that the reward mechanism can accurately reflect the goal achievement of both parties in the game, and through continuous iterative optimization of reinforcement learning, drones can gradually learn effective drone formation defense strategies in complex environments.

[0119] 2. The embodiment of the present invention can obtain the drone formation information and drone formation defense strategy of the defender, wherein the drone formation defense strategy is used to indicate the decision plan of multiple drones, and the decision plan includes defense actions and flight directions; defense actions include air patrol, calling friendly aircraft and the own aircraft retreat. The defender’s drone formation is controlled to execute the drone formation defense strategy, and the patrol drone in patrol state is acquired, and the detection results of the patrol drone are acquired. The decision plan update model is generated and solved according to the detection results, and the decision plan of the patrol drone is updated according to the model solution. The drone formation defense strategy is updated according to the updated decision plan, and the defender’s drone formation is controlled to execute the updated drone formation defense strategy. The decision plan of the drone can be updated through the detection results of the drone, so as to real-time schedule the scale of the drone formation when performing tasks, thereby improving the defense effect.

[0120] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0121] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reinforcement learning method for intelligent decision-making of unmanned aerial vehicles, characterized in that: include: Obtain the defender’s drone formation information and drone formation defense strategy; The defending drone formation includes multiple drones; the drone formation defense strategy is used to indicate the decision-making plan of the multiple drones, and the decision-making plan includes defense actions and flight directions; the defense actions include air patrol, calling friendly aircraft and the own aircraft retreating; Controlling the defending drone formation to execute the drone formation defense strategy; Acquire a patrol drone in a patrol state, and acquire a detection result of the patrol drone; A decision-making scheme update model is generated according to the detection results; the decision-making scheme update model includes a state space, a decision scheme, a reward return and a benefit function; the state space is generated by the detection results; the reward return at least includes a boundary fence reward obtained based on a differential game; Solving the decision-making scheme update model, and updating the decision-making scheme of the patrol drone according to the model solution; The drone formation defense strategy is updated according to the updated decision plan, and the defending drone formation is controlled to execute the updated drone formation defense strategy.

2. The reinforcement learning method according to claim 1, characterized in that: The step of solving the decision solution update model comprises: Acquire drone state information of the attacking drone according to the state space; Calculate a reward return including at least the Jieshan reward according to the drone status information; The profit function is solved according to the reward return.

3. The reinforcement learning method according to claim 2, characterized in that: The drone status information includes a speed ratio between the patrol drone and the attacking drone; The process of obtaining the boundary fence reward includes: Based on the speed ratio and the position of the defense position of the defense drone formation, differential game is used to construct the boundary fence of the patrol drone to construct the pursuit zone and escape zone of the patrol drone; Get Boundary Fence rewards, including: Among them, the subscript C CZ , C EZ They represent all the attacking drones in the pursuit zone and escape zone of the patrol drone respectively; It represents the reduction in the distance between the patrol drone and all attacking drones in the pursuit area in a single cycle; It represents the reduction in the distance between the patrol drone and all attacking drones in the escape zone in a single cycle; The drone status information also includes the number of first attacking drones defeated by the patrol drone and the number of second attacking drones that entered the defense position of the defending drone formation; Calculating a reward return including at least the Jieshan reward according to the drone state information also includes: Obtaining a first reward return according to the drone state information includes: in, R target Indicates the first reward return; m 1 indicates the number of drones of the first attacking party, m 2 represents the number of drones of the second attacking party; Defeated the attacking drone j Bonus score, if the attacking drone j If a patrol drone is defeated in its escape zone after it calls a friendly drone, the bonus score is upgraded to the preset multiple of the first basic bonus score, otherwise it remains at the first basic bonus score; Indicates the attacking drone j Bonus points for entering a defensive position, if the attacking drone j If you enter a defensive position in a patrol drone's pursuit zone after the patrol drone calls a friendly drone, the bonus score will be upgraded to the preset multiple of the second basic bonus score, otherwise it will remain at the second basic bonus score; Get the second reward, including: in, R time Indicates the second reward return; m 3 indicates the number of patrol drones; Calculate reward returns, including: in, R Indicates reward return.

4. The reinforcement learning method according to claim 3, characterized in that: The profit function is: Among them, argmaxf( x ) means that the objective function f( x ) The variable with the maximum value x The value of θ Indicates the flight direction; R ( t ) indicates the t Reward return for each cycle; γ represents the discount factor, γ t The discount factor t Power.

5. The reinforcement learning method according to claim 1, wherein: After executing the acquisition of the patrol drone in patrol state, it also includes: Control patrol drones to time and detect the timing duration; If the patrol drone does not detect the attacking drone within the preset patrol period, the decision plan of the patrol drone is updated to set the patrol drone's defense action to retreat; If the patrol drone detects the attacking drone within the preset patrol cycle, the timing duration is updated to 0, and the patrol drone is controlled to perform timing.

6. The reinforcement learning method according to claim 1, characterized in that: After obtaining the detection result of the patrol drone, the method further includes: Obtaining the patrol threat value of the patrol drone; If the patrol threat value is greater than or equal to a preset threat value threshold, updating the decision plan of the patrol drone to set the defense action of the patrol drone to call a friendly aircraft; If the patrol threat value is less than a preset threat value threshold, the step of generating a decision solution and updating the model according to the detection result is performed.

7. The reinforcement learning method according to claim 6, characterized in that: Obtaining the patrol threat value of the patrol drone, including: Acquire the target attacking drone detected by the patrol drone according to the detection result of the patrol drone; Obtaining the action strategy of the target attacking drone; the action strategy includes an advance strategy, a retreat strategy, and other strategies; the advance strategy is the behavior of the target attacking drone advancing toward the defense position of the defending drone formation, the retreat strategy is the behavior of the target attacking drone retreating, and the other strategies are strategy behaviors other than the advance strategy and the retreat strategy; Obtaining a strategy threat value according to the action strategy; Get the location threat value, including: in, W distance Indicates the position threat value, d max Indicates the maximum patrol distance of the drones in the defending drone formation. Indicates the target attacking drone j * distance from defensive positions; The patrol threat value of the patrol drone is calculated according to the strategy threat value and the position threat value.

8. The reinforcement learning method according to claim 1, characterized in that: The model solution includes probability distributions of multiple decision-making options; The decision plan of the patrol drone is updated according to the model solution, including: Obtaining a decision plan with the highest probability according to the model solution, and updating it as the decision plan of the patrol drone in the next cycle; The drone formation defense strategy is updated according to the updated decision plan, including: According to the updated decision plan, the number of third drones whose defensive action is to call friendly drones is obtained, and the number of retreating drones whose defensive action is to retreat. Acquire a number of standby drones corresponding to the third number of drones in the defending drone formation, and update the decision plan of the standby drones to set the defense action of the standby drones to air patrol; Setting the standby drone as a patrol drone and canceling the setting of the retreating drone as a patrol drone; Statistics are collected on the updated decision plans of multiple drones in the defending drone formation to generate the drone formation defense strategy for the next cycle.

9. A reinforcement learning system for intelligent decision-making of unmanned aerial vehicles, characterized in that: The system comprises: The information acquisition module is configured to acquire the defender's drone formation information and the drone formation defense strategy; the defender's drone formation includes multiple drones; the drone formation defense strategy is used to indicate the decision-making plan of the multiple drones, the decision-making plan includes defense actions and flight directions; the defense actions include air patrol, calling friendly aircraft and the aircraft's own retreat; A strategy execution module, configured to control the defending drone formation to execute the drone formation defense strategy; A result detection module is configured to obtain a patrol drone in a patrol state and obtain a detection result of the patrol drone; A model generation module is configured to generate a decision solution update model according to the detection result; the decision solution update model includes a state space, a decision solution, a reward return and a benefit function; the state space is generated by the detection result; the reward return at least includes a boundary fence reward obtained based on a differential game; A solution updating module is configured to solve the decision solution updating model and update the decision solution of the patrol drone according to the model solution; The strategy update module is configured to update the drone formation defense strategy according to the updated decision plan, and control the defending drone formation to execute the updated drone formation defense strategy.

10. A storage medium, characterized in that: It stores a computer program for reinforcement learning of drone intelligent decision-making, wherein the computer program enables a computer to execute the reinforcement learning method for drone intelligent decision-making as described in any one of claims 1 to 8.

Images