Unmanned aerial vehicle cluster collaborative strike control method based on reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle cluster cooperative strike control method based on reinforcement learning, and the method comprises the steps: firstly building a two-dimensional confrontation environment which comprises an unmanned aerial vehicle cluster, a maneuvering target and a no-fly zone constraint; and then designing an unmanned aerial vehicle-target combat geometrical relationship based on a line-of-sight distance and a line-of-sight angle change rate and an unmanned aerial vehicle cluster space-time collaborative saturation strike mechanism, describing a state space, an action space and a reward function according to a Markov decision process, and designing a distributed collaborative strike strategy training method based on reinforcement learning. And finally, inputting the local observation of the unmanned aerial vehicle into the strategy network and outputting an acceleration instruction. According to the invention, through designing a heuristic reward based on zero-control miss distance and fusing a composite reward function of collaborative strike efficiency, energy consumption and no-fly zone avoidance, battlefield situation observation is directly mapped into an acceleration control instruction of an unmanned aerial vehicle by adopting a reinforcement learning method; and efficient interception and space-time collaborative saturation strike of various maneuvering type targets are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of UAV cluster collaborative control, and specifically relates to a UAV cluster collaborative strike control method based on reinforcement learning. Background Art

[0002] As the combat applications of drone swarms become more complex, the effectiveness of individual drones in executing missions is limited and they are unable to cope with dynamic targets. Drone swarms take advantage of information collaboration and can carry out saturation strikes on targets through multi-dimensional collaboration in time and space, thereby improving strike effectiveness.

[0003] In complex dynamic mission scenarios, there are many confrontational elements, and the dynamic maneuvering of the target leads to many constraints on cluster coordinated strike missions, and higher coordination efficiency requirements. The traditional proportional guidance method cannot achieve multi-machine time synchronization and multi-angle encirclement strikes. The biased proportional guidance method can achieve time coordination or angle coordination to a certain extent, but it fails to fully consider the restrictions of the no-fly zone constraints on the UAV flight trajectory, resulting in mission failure. In addition, the above method is based on the small angle assumption and is difficult to deal with targets with strong excitation capabilities. Summary of the Invention

[0004] The present invention aims to overcome the shortcomings of existing technologies and provides a reinforcement learning-based method for controlling coordinated strikes by drone swarms. While considering the maneuverability of targets, this method also accounts for potential threats such as no-fly zones in adversarial environments. By designing a heuristic reward based on zero-miss distance and integrating a composite reward function that considers coordinated strike effectiveness, energy consumption, and no-fly zone avoidance, the method employs reinforcement learning to directly map battlefield situation observations into acceleration control commands for drones. This method achieves efficient interception and spatiotemporal coordinated saturation strikes against multiple maneuverable targets.

[0005] The purpose of the present invention is to achieve the following technical solution: a UAV cluster coordinated strike control method based on reinforcement learning, the specific steps are as follows:

[0006] S1. Establish a two-dimensional confrontation environment including drone clusters, maneuverable targets, and no-fly zone constraints; the specific method is: set a drone cluster coordinated strike scenario including a strike cluster composed of N drones and a target to be struck with maneuverability, and the strike scenario is l env ×l env The flight environment contains M no-fly zones.

[0007] Establish a two-degree-of-freedom kinematic model for the UAV. The specific form is as follows:

[0008]

[0009] Where V Uis the velocity vector of UAV U, (x, y) represents the position of UAV U, γ is the heading angle of UAV U, a long and a lat are the longitudinal and lateral accelerations of the UAV;

[0010] The no-fly zone distribution is modeled using piecewise potential field functions;

[0011] S2. Design the UAV-target engagement geometry based on the line-of-sight distance and line-of-sight angle change rate. The specific method is as follows: define the line-of-sight distance between the UAV U and the target T as r, the line-of-sight angle as λ, and the relative velocity component along the line-of-sight direction as Relative velocity component perpendicular to the line of sight and They represent the rate of change of sight distance and sight angle respectively, specifically:

[0012]

[0013] Where V U 、V T are the velocity vectors of the UAV and the target, γ U , γ T are the heading angles of the UAV and the target respectively;

[0014] Define the line of sight distance r in the relative velocity component V λ The projection in the direction is the zero control miss distance r ZEM , the specific form is:

[0015]

[0016] S3. Design a spatiotemporal coordinated saturation strike mechanism for drone swarms, including a time synchronization mechanism and a multi-angle coordinated mechanism. The details are as follows: define the strike time of the i-th drone on the target as t i , when the distance between the i-th UAV and the target T is less than the strike distance threshold R hit When , it is determined that the UAV has successfully hit the target;

[0017] The design time synchronization constraints are:

[0018]

[0019] Where ΔT is the strike time limit;

[0020] Define the attack time of the i-th UAV at t i The heading angle is γ i (t i ), design the angular dispersion metric function γ min To measure the attack angle distribution of the drone, the specific form is:

[0021] γ min =min 1≤i＜j≤N (min(|γ i -γ j |,2π-|γ i -γ j |))

[0022] The time-space coordinated saturation strike mission requires the UAV to attack the target within a specified time window and approach the target from multiple angles, making it difficult for the target to evade;

[0023] S4. Based on the Markov decision process, describe the state space, action space and reward function of the UAV swarm coordinated attack problem;

[0024] S5. Design a distributed coordinated strike strategy training method based on reinforcement learning;

[0025] S6, the mission execution phase, inputs the UAV’s local observation into the strategy network, and outputs acceleration instructions to control the UAV to complete the coordinated strike.

[0026] The beneficial effects of the present invention are as follows: the present invention discloses a method for controlling coordinated strikes of drone swarms based on reinforcement learning. First, a two-dimensional confrontation environment including drone swarms, maneuvering targets, and no-fly zone constraints is established. Then, a drone-target engagement geometry relationship based on line-of-sight distance and line-of-sight angle change rate is designed. Secondly, a spatiotemporal coordinated saturation strike mechanism for drone swarms is designed. Further, based on the Markov decision process, the state space, action space, and reward function of the drone swarm coordinated strike problem are described. On this basis, a distributed coordinated strike strategy training method based on reinforcement learning is designed. Finally, in the task execution link, the local observations of the drones are input into the strategy network, and acceleration commands are output to control the drones to complete the coordinated strike. The method of the present invention takes into account the maneuverability characteristics of the target while also taking into account possible threat factors such as no-fly zones in the confrontation environment. By designing a heuristic reward based on zero-control miss amount, integrating a composite reward function of coordinated strike effectiveness, energy consumption, and no-fly zone avoidance, and using reinforcement learning methods to directly map battlefield situation observations into acceleration control commands for drones, efficient interception and spatiotemporal coordinated saturation strikes against multiple types of maneuverable targets are achieved.

[0027] Compared with the existing proportional guidance method and biased proportional guidance method, the present invention comprehensively considers strike accuracy, cluster coordination effectiveness, energy consumption and no-fly zone avoidance, and can achieve spatiotemporal coordinated saturation strikes on maneuvering targets in complex confrontation scenarios, maintain high strike accuracy, and the UAV flight trajectory is safe and sustainable, showing good adaptability and robustness in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of an implementation method of a UAV cluster coordinated strike control method based on reinforcement learning according to the present invention;

[0029] Figure 2 Schematic diagram of the geometric relationship between drone and target engagement in an embodiment of the present invention

[0030] Figure 3 Schematic diagram of zero-control miss distance calculation in an embodiment of the present invention

[0031] Figure 4 This is a schematic diagram of the present invention using a reinforcement learning method to optimize the UAV coordinated strike strategy;

[0032] Figure 5 This is the UAV cluster coordinated strike control effect 1 in an embodiment of the present invention;

[0033] Figure 6 This is the UAV cluster coordinated strike control effect 2 in the embodiment of the present invention;

[0034] Figure 7 This is the UAV cluster coordinated strike control effect 3 in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0036] like Figure 1 As shown, the present invention is a UAV cluster coordinated attack control method based on reinforcement learning, and the specific steps are as follows:

[0037] S1. Establish a two-dimensional confrontation environment including drone clusters, maneuverable targets, and no-fly zone constraints. The specific method is: set a drone cluster coordinated strike scenario including a strike cluster composed of N drones and a maneuverable target to be struck T, and the strike scenario is l env ×l env The flight environment contains M no-fly zones O;

[0038] A two-degree-of-freedom kinematic model of a UAV with controllable speed and angular velocity is established. The specific form is as follows:

[0039]

[0040] Where V U is the velocity vector of UAV U, (x, y) represents the position of UAV U, γ is the heading angle of UAV U, a long and a lat are the longitudinal and lateral accelerations of the UAV, and the symbol · is used to find the first derivative;

[0041] Use differential equations to describe the kinematic characteristics of targets with different maneuverability, including but not limited to uniform linear motion, constant curvature circular motion, and random maneuvers;

[0042] The segmented potential field function is used to model the distribution of no-fly zones. The potential field size of the j-th no-fly zone for the UAV at the (x, y) position is:

[0043]

[0044] Among them, (x j ,y j ) represents the geometric center of the i-th no-fly zone, r j Indicates the radius of the no-fly zone; r safe A safe distance is set to balance the attenuation of the potential field with distance;

[0045] S2. Design the UAV-target engagement geometry based on the line of sight distance and line of sight angle change rate. The UAV-target engagement geometry is as follows: Figure 2 The specific method is as follows: define the line of sight distance r and line of sight angle λ between the UAV U and the target T in the inertial coordinate system (X, Y) as the key parameters of the engagement geometry, and further define the relative velocity component along the line of sight direction and the relative velocity component perpendicular to the line of sight in and They represent the rate of change of sight distance and sight angle respectively, and the specific forms are:

[0046]

[0047] Where V U 、V T are the velocity vectors of the UAV U and the target T, γ U , γ T are the heading angles of the UAV and the target respectively;

[0048] Define the line of sight distance r in the relative velocity component V λ The projection in the direction is the zero control miss distance r ZEM ,like Figure 3 As shown, the specific form is:

[0049]

[0050] S3. Design a spatiotemporal coordinated saturation strike mechanism for drone swarms, including a time synchronization mechanism and a multi-angle coordination mechanism. The spatiotemporal coordinated saturation strike mission requires controlling the drone swarm to target the same target, complete a synchronous attack within a specified time window, and achieve multi-angle encirclement strikes through distributed decision-making.

[0051] The specific method is as follows: Let the size of the drone group be N, and define the time when the i-th drone strikes the target as t i , when the distance between the i-th UAV and the target T is less than the strike distance threshold R hit When , it is determined that the UAV has successfully hit the target, the specific form is:

[0052] ‖(x i (t i ),y i (t i ))-(x T (t i ),y T (t i ))‖≤R hit

[0053] Among them, (x i (t i ),y i (t i )) is t i The position of the i-th drone at the moment;

[0054] The design time synchronization constraints are:

[0055]

[0056] Where ΔT is the strike time limit; this constraint can ensure that all drones complete the strike action within the specified time window and achieve overall time synchronization.

[0057] Define the attack time of the i-th UAV at t i The heading angle is γ i (t i ), design the angular dispersion metric function γ min To measure the attack angle distribution of the drone, the specific form is:

[0058] γ min =min 1≤i＜j≤N (min(|γ i -γ j |,2π-|γ i -γ j |))

[0059] γ i , γ j are the i-th and j-th UAVs at the attack time t i The heading angle;

[0060] The time-space coordinated saturation strike mission requires the drone to attack the target within a specified time window and approach the target from multiple angles, making it difficult for the target to evade.

[0061] S4. Based on the Markov decision process, describe the state space, action space, and reward function of the UAV swarm coordinated attack problem; the details are as follows:

[0062] S41, the state space represents all possible environmental states, including the UAV's own state information (UAV position, speed, heading angle, engagement geometry with the target, etc.), target position and motion state information, and no-fly zone distribution information;

[0063] The drone's own status information is:

[0064]

[0065] x i 、y i 、v i , γ i are the horizontal coordinate, vertical coordinate, speed and heading angle of the i-th UAV respectively;

[0066] The target position and motion status information are:

[0067]

[0068] x T 、y T 、v T , γ T are the horizontal coordinate, vertical coordinate, speed and heading angle of the target T respectively;

[0069] The distribution information of the no-fly zones is as follows:

[0070]

[0071] The state space of drone i is defined by combining the states of drone, target, and no-fly zone:

[0072] s=[s uav ,s target ,s NFZ ]

[0073] When performing the above calculations, the original information in the state space is first converted to the combat environment size l env 、UAV maximum speed v max Normalize the prior knowledge to avoid the problem of gradient disappearance and gradient saturation caused by excessive input values ​​during network training.

[0074] S42. The action space is the set of all control actions that the agent can perform. For aircraft guidance law design, a typical approach is to separate the aircraft's guidance loop and control loop through a hierarchical design. The outer loop outputs guidance commands based on the kinematic relationship between the aircraft and the target. This paper defines the action space a as the longitudinal and lateral acceleration commands:

[0075] a={(a long ,a lat )||a long |,|a lat |≤a max}

[0076] where a max is the maximum acceleration that the drone can exert;

[0077] S43. Design a heuristic reward function based on zero-control miss distance, and integrate zero-control miss distance, synergy, energy consumption and no-fly zone avoidance to construct a composite reward function r total , expressed as:

[0078] r total =r finish +μ1r zem +μ2r time +μ3r angle +μ4r energy +μ5r avoid

[0079] where μ i ,i∈{1,2,...,5} represents the weight factor, which is used to balance the importance of each part of the reward function;

[0080]

[0081] r finish It represents the strike completion reward, which is a discrete reward given when the relative position of the UAV and the target meets the strike distance constraint;

[0082]

[0083] r zem It represents the heuristic reward based on the zero-control miss amount, guiding the UAV to minimize the strike distance;

[0084]

[0085] r time Represents the time synchronization reward, which is based on discrete rewards when the strike time window of the drone cluster meets the time limit requirement;

[0086]

[0087] r angle represents the angle coordination reward, guiding the drone to maximize the angle difference during the attack process;

[0088]

[0089] r energy It represents energy consumption reward, guiding the UAV to reduce energy consumption and improve the sustainability of the UAV;

[0090]

[0091] r avoid Represents a no-fly zone avoidance reward, guiding the drone cluster to adjust its trajectory during maneuvers to avoid entering the no-fly zone and improve the survivability of the drones.

[0092] Through the above design, the zero-control miss amount, synergy efficiency and other branch task reward functions are integrated into the total reward function r total , comprehensively quantify the performance of UAV clusters in terms of strike accuracy, time coordination, space coordination, energy consumption and no-fly zone avoidance, and provide a quantifiable evaluation and optimization basis for the coordinated strike missions of UAV clusters in dynamic confrontation environments.

[0093] S5. Design a distributed coordinated attack strategy training method based on reinforcement learning; the process is as follows Figure 4 As shown, the details are as follows:

[0094] S51, initialize the parameters θ of the strategy network π and the parameters of the evaluation network V and the corresponding experience replay buffer D;

[0095] S52, initialize the training environment, obtain the environment state set at time t as the initial joint observation s t The state space and action space are sets. The state space represents the set of all possible states of the environment, and the action space represents the set of possible actions of the agent. Observation is the information about the environment perceived by the agent at time t, which is a subset of the state space. Action is an action taken by the agent at time t, which is a subset of the action space. Substituting the state at time t into the reward function yields the reward r for the current state. t ;

[0096] S53, sample action a according to the policy network θ i ~π(s t |θ), perform the joint action {a1,a2,...,a N}, get reward r t and the next joint observation s t+1 , store the experience into the experience buffer D;

[0097] S54. Randomly sample small batches of data from buffer D to calculate the generalized advantage estimate:

[0098]

[0099] Where V(s t ) is the state value function, V is the evaluation network, V(s t ) is the current state s t The value estimate of is generally called the state value function; γ is the discount rate;

[0100] S55. Update the policy network parameters θ using stochastic policy gradient ascent. During the update process, use the clipping objective function to limit the policy update range. The specific implementation method is as follows:

[0101]

[0102] Among them, r t (θ i ) is the current policy θ i The ratio between the new policy and the old policy; clip means clipping, ε is the clipping threshold, which controls the amplitude of policy update; represents the expectation at time t;

[0103] The evaluation network is updated by minimizing the mean square error. The specific implementation method is as follows:

[0104]

[0105] Indicates state s t The value function under is the parameter of the evaluation network; R t is the cumulative reward from the current moment t to the future, estimated by time difference:

[0106]

[0107] S56. Repeat steps S53-S55 until the training conditions are met and freeze the strategy network parameters θ.

[0108] S6, task execution phase, the UAV local observation s t Input policy network π θ , output acceleration instructions to control the coordinated flight of the drone cluster, and repeat the above steps until the coordinated strike mission is completed.

[0109] In this embodiment, the UAV cluster attack scenario includes a strike cluster U consisting of 4 UAVs and a moving target T with three different levels of maneuverability, namely, uniform linear motion (see Figure 5), constant curvature circular motion (see Figure 6 ), random maneuver (see Figure 7 ), the strike scene is a 5000×5000 square area, and the flight environment includes 7 no-fly zones O of different locations and sizes.

[0110] In this embodiment, the drone cluster needs to complete a synchronous attack within a specified time window and achieve multi-angle encirclement and strike through distributed decision-making. At the same time, the drones must avoid no-fly zones during flight to avoid crashing.

[0111] In this embodiment, the specific steps in steps S5-S6 are used to complete the UAV cluster coordinated strike strategy training and execution. Figure 5 、 6 , 7 respectively completed the coordinated saturation strike of drone clusters against various target maneuvers. During the process, the drones were able to effectively avoid the no-fly zone. The method can effectively coordinate the strike time and strike angle of the drone cluster, so that the drone cluster can approach the target from multiple angles at the terminal moment, thereby enhancing the strike effect on the target.

[0112] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A UAV swarm coordinated attack control method based on reinforcement learning, characterized in that: The specific steps are as follows: S1. Establish a two-dimensional confrontation environment including drone clusters, maneuverable targets, and no-fly zone constraints; the specific method is: set a drone cluster coordinated strike scenario including a strike cluster composed of N drones and a target to be struck with maneuverability, and the strike scenario is l env ×l env The flight environment contains M no-fly zones. Establish a two-degree-of-freedom kinematic model for the UAV. The specific form is as follows: Where V U is the velocity vector of UAV U, (x, y) represents the position of UAV U, γ is the heading angle of UAV U, a long and a lat are the longitudinal and lateral accelerations of the UAV; Use differential equations to describe the kinematic characteristics of targets with different maneuverability; The segmented potential field function is used to model the distribution of no-fly zones. The potential field size of the j-th no-fly zone for the UAV at the (x, y) position is: Among them, (x j ,y j ) represents the geometric center of the i-th no-fly zone, r j represents the radius of the no-fly zone, r safe For safe distance; S2. Design the UAV-target engagement geometry based on the line-of-sight distance and line-of-sight angle change rate. The specific method is as follows: define the line-of-sight distance between the UAV U and the target T as r, the line-of-sight angle as λ, and the relative velocity component along the line-of-sight direction as Relative velocity component perpendicular to the line of sight and They represent the rate of change of sight distance and sight angle respectively, specifically: Where V U 、V T are the velocity vectors of the UAV and the target, γ U , γ T are the heading angles of the UAV and the target respectively; Define the line of sight distance r in the relative velocity component V λ The projection in the direction is the zero control miss distance r ZEM , the specific form is: S3. Design a spatiotemporal coordinated saturation attack mechanism for drone swarms. The specific mechanism is as follows: define the attack time of the i-th drone on the target as t i , when the distance between the i-th UAV and the target T is less than the strike distance threshold R hit When , it is determined that the UAV has successfully hit the target; The design time synchronization constraints are: Where ΔT is the strike time limit; Define the attack time of the i-th UAV at t i The heading angle is γ i (t i ), design the angular dispersion metric function γ min To measure the attack angle distribution of the drone, the specific form is: c min =min 1≤i＜j≤N (min(|γ i -c j |,2π-|γ i -c j |)) The time-space coordinated saturation strike mission requires the UAV to attack the target within a specified time window and approach the target from multiple angles, making it difficult for the target to evade; S4. Based on the Markov decision process, describe the state space, action space and reward function of the UAV swarm coordinated attack problem; S5. Design a distributed coordinated strike strategy training method based on reinforcement learning; S6, the mission execution phase, inputs the UAV’s local observation into the strategy network, and outputs acceleration instructions to control the UAV to complete the coordinated strike.

2. The method for controlling the coordinated attack of a drone cluster based on reinforcement learning according to claim 1 is characterized in that: The step S4 is specifically as follows: S41, the state space includes the UAV's own state information, target position and motion state information, and no-fly zone distribution information; The drone's own status information is: x i 、y i 、v i , γ i are the horizontal coordinate, vertical coordinate, speed and heading angle of the i-th UAV respectively; The target position and motion status information are: x T 、y T 、v T , γ T are the horizontal coordinate, vertical coordinate, speed and heading angle of the target T respectively; The distribution information of the no-fly zones is as follows: The state space of drone i is defined by combining the states of drone, target, and no-fly zone: s=[s uav ,s target ,s NFZ ]; S42. Define the action space as longitudinal and lateral acceleration instructions: a={(a long ,a lat )∣|a long |,|a lat |≤a max } where a max is the maximum acceleration that the drone can exert; S43. Design a heuristic reward function based on the zero-control miss distance, expressed as: r total =r finish +μ1r zem +μ2r time +μ3r angle +μ4r energy +μ5r avoid where μ i ,i∈{1,2,...,5} represents the weight factor, which is used to balance the importance of each part of the reward function; r finish It represents the strike completion reward, which is a discrete reward given when the relative position of the UAV and the target meets the strike distance constraint; r zem represents the heuristic reward based on the zero-control miss amount; r time Represents the time synchronization reward, which is based on discrete rewards when the strike time window of the drone cluster meets the time limit requirement; r angle represents the angle synergy reward; r energy Indicates energy consumption reward; r avoid Indicates a no-fly zone avoidance reward.

3. The UAV swarm coordinated attack control method based on reinforcement learning according to claim 1 is characterized in that: The step S5 is specifically as follows: S51, initialize the parameters θ of the strategy network π and the parameters of the evaluation network V and the corresponding experience replay buffer D; S52, initialize the training environment, obtain the environment state set at time t as the initial joint observation s t ; S53, sample action a according to the policy network θ i ~π(s t |θ), perform the joint action {a1,a2,...,a N }, get reward r t and the next joint observation s t+1 , store the experience into the experience buffer D; S54. Randomly sample small batches of data from buffer D to calculate the generalized advantage estimate: Where V(s t ) is the state value function, γ is the discount rate; S55. Update the policy network parameters θ using stochastic policy gradient ascent. During the update process, use the clipping objective function to limit the policy update range. The specific implementation method is as follows: Among them, r t (θ i ) is the current policy θ i The ratio between the old strategy and the old strategy; clip means clipping, and ε is the clipping threshold; represents the expectation at time t; The evaluation network is updated by minimizing the mean square error. The specific implementation method is as follows: Indicates state s t The value function under R t is the cumulative reward from the current time t to the future, estimated by time difference; S56. Repeat steps S53-S55 until the training conditions are met and freeze the strategy network parameters θ.

Images