A reinforcement learning method for controlling unmanned aerial vehicle swarm based on damage maximization strategy

Abstract

The application discloses a reinforcement learning unmanned aerial vehicle cluster control method based on a damage maximization strategy, links guidance and damage in unmanned aerial vehicle cluster control, realizes cluster cooperative control and guarantees maximization of unmanned aerial vehicle damage efficiency. A damage-motion coupling model of the unmanned aerial vehicle is established to associate cooperative control with damage control; and a reward function for realizing damage maximization is designed based on the model, which is embedded into a MATD3 or other reinforcement learning framework, so that the algorithm can gradually learn to realize damage maximization in training with cooperative control as the target. The algorithm has clear logic, simple structure and is easy to realize.

Description

Technical Field

[0001] This invention belongs to the field of information processing technology, specifically relating to a reinforcement learning-based unmanned aerial vehicle (UAV) swarm control method based on a damage maximization strategy. Background Technology

[0002] With the increasing complexity and intelligence of modern military confrontation, multi-UAV swarm saturation attacks, with their multi-directional, multi-wave, and high-density coordinated strike capabilities, can effectively overcome the limitations of single platforms in terms of penetration, payload, and survivability, significantly improving the success rate of penetration against high-value, heavily protected targets, thus becoming a disruptive combat style. However, the realization of this combat style relies on the efficient collaborative decision-making and precise control of individual UAVs in complex and dynamic battlefield environments. It is essentially a highly complex multi-agent system control problem, requiring the fulfillment of requirements such as strong real-time performance, distributed perception, autonomous decision-making, collaborative planning, spatiotemporal synchronization, and strong anti-jamming capabilities. At the same time, ensuring system stability and robustness in scenarios with limited communication and uncertain environments makes the design of multi-UAV swarm strike control algorithms extremely challenging.

[0003] Existing multi-UAV swarm collaborative control methods are mainly divided into two categories: model-based and model-free. Early model-based control methods focused on optimizing time convergence performance. For example, Bhat and Bernstein proposed a finite-time stable design method based on homogeneity theory to solve the finite-time convergence problem where the system state depends on the initial state, providing a theoretical basis for cooperative attacks. However, the upper bound of the convergence time is heavily dependent on the initial state. Polyakov designed a nonlinear feedback controller to make the upper bound of the convergence time independent of the initial state, improving the predictability and robustness of cooperative time performance. However, it suffers from problems such as the inability to preset the upper bound of the convergence time, computational complexity, and difficulty in online adjustment according to operational needs. Song et al. proposed a predetermined time stable control strategy based on a time-varying scaling function, which can preset the exact convergence time (hit time) and improve the accuracy of time control. However, it relies on time-varying gain, and the gain tends to infinity when approaching the predetermined time. It is sensitive to actuator saturation, noise, and model uncertainty, and is difficult to implement in engineering. Kumar et al. combined sliding mode control and bias proportional guidance (BPN) to design a cooperative guidance law to solve the coupling problem of precise terminal attitude control and time coordination. However, it suffers from high computational complexity and limited scalability to multiple constraints. Furthermore, the aforementioned model-based methods still face significant challenges in handling highly nonlinear, strongly coupled, multi-constraint cooperative guidance problems, as well as in terms of adaptability and robustness under complex and uncertain environments (such as unknown target maneuvers and strong interference).

[0004] To overcome the shortcomings of model-based methods, multi-agent reinforcement learning cooperative control methods in model-free control have become a research hotspot. For example, Chen et al. proposed the MADDPG cooperative guidance algorithm based on the centralized training distributed execution (CTDE) framework, which learns the optimal policy through interaction between agents and the environment, solving the problems of traditional methods relying on accurate dynamic models, difficulty in handling complex high-dimensional state and action spaces, and difficulty in adaptive optimization under unknown perturbations. Xiaoqi Qiu et al. proposed the MATD3 cooperative guidance algorithm based on policy decomposition, which effectively solves the problems of time consistency and attack angle constraints.

[0005] However, both model-based control methods and model-free control methods based on multi-agent reinforcement learning share a common bottleneck: the technical gap between guidance and damage remains unbridged. The actual damage effect of a multi-UAV swarm on a target requires not only precise guidance and control but also ensuring the spatiotemporal matching between the warhead and the target. Taking fragmentation warheads as an example, their damage effectiveness function is coupled with the projectile-target distance and azimuth angle, exhibiting strong nonlinear characteristics. Existing multi-agent control methods can only achieve precise control of the detonation point distance, failing to guarantee the maximization of actual damage effectiveness. A new multi-UAV swarm control scheme is urgently needed to solve this problem. Based on the above description, this invention proposes a multi-UAV detonation point control algorithm based on multi-agent reinforcement learning for collaborative damage maximization. Summary of the Invention

[0006] The purpose of this invention is to propose a reinforcement learning-based UAV swarm control method based on a damage maximization strategy, so as to achieve a breakthrough between guidance and damage using reinforcement learning.

[0007] The technical solution for implementing this invention is: a reinforcement learning-based UAV swarm control method based on a damage maximization strategy, comprising the following steps:

[0008] Step 1: Establish a damage model and kinematic model for the UAV swarm, and then establish a damage-kinematic coupling model. Based on the established UAV swarm coupling model, determine the environmental state of the UAV swarm. Proceed to step 2.

[0009] Step 2: Based on the environmental conditions of the drone swarm Calculate the optimal detonation time and detonation point for the drone swarm, and calculate the minimum arrival time for each drone in its current state. Based on the kinematic function in the damage-kinematic coupling model, determine whether the UAV has reached the predetermined target. If the predetermined target has been reached, proceed to step 6; otherwise, proceed to step 3.

[0010] Step 3: Construct a structured reward function, including a time-coordinated reward function. Angle-constrained reward function and the hit location-guided reward function Calculate the reward value based on the structured reward function. Establish an experience pool mechanism to store data such as the drone's current actions, environmental status, and rewards in the experience pool. Proceed to step 4.

[0011] Step 4: Use MATD3 as a reinforcement learning framework for the damage-kinematic coupling model and the structured reward function, utilizing the experience pool. The damage-kinematic coupling model is trained by... The network and policy network are continuously updated to obtain the optimal policy, thereby obtaining the minimum exact time of arrival. And obtain the trained damage-kinematic coupling model, then proceed to step 5.

[0012] Step 5: Determine whether the predetermined target has been achieved based on the damage performance function. If the predetermined target has been achieved, proceed to step 6; otherwise, return to step 4.

[0013] Step 6: Output the policy network and value The network constructs a Lyapunov function to prove the convergence of the trained damage-kinematic coupling model, ensuring that the UAV arrives at the target simultaneously at the optimal detonation time.

[0014] Compared with the prior art, the significant advantages of this invention are:

[0015] (1) This invention combines the damage effectiveness function of the fragmentation warhead with the kinematic relationship of the projectile target to construct a detonation point control model for maximizing the damage of multiple UAVs in a coordinated manner, thereby guiding the optimal detonation point control for their clustered saturation attacks.

[0016] (2) The present invention can learn the optimal control strategy scheme that maximizes damage.

[0017] (3) This invention verifies that the proposed algorithm can guarantee that the UAVs will eventually reach the target simultaneously.

[0018] (4) The present invention has the advantages of high damage efficiency, strong correction capability and good robustness, and can be applied to various micro UAVs and other actual equipment and combat scenarios. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of target engagement by the UAV of the present invention.

[0020] Figure 2 This is a schematic diagram of the damage range of the present invention.

[0021] Figure 3 This is a flowchart of a reinforcement learning-based UAV swarm control method based on a damage maximization strategy.

[0022] Figure 4 This is a diagram of the algorithm framework of this invention. Detailed Implementation

[0023] This invention links guidance and damage, and proposes a reinforcement learning-based UAV swarm control method based on a damage maximization strategy, which can maximize the damage efficiency of UAV swarms.

[0024] Combination Figures 1-4 The present invention discloses a reinforcement learning-based UAV swarm control method based on a damage maximization strategy, comprising the following steps:

[0025] Step 1: Establish a damage model and kinematic model for the UAV swarm, and then establish a damage-kinematic coupling model. Based on the above damage-kinematic coupling model, determine the environmental state of the UAV swarm. , The specific model is constructed as follows:

[0026] For micro-UAVs, with limited payload capacity, fragmentation warheads with omnidirectional dispersion are often used to increase the probability of successfully destroying targets. For this type of warhead, the degree of damage to the target is positively correlated with the kinetic energy of the fragments and the damage coefficient of the fragments. Therefore, a damage efficiency function is introduced. The degree of damage is quantified and specifically expressed as follows:

[0027] (1),

[0028] in Damage efficiency coefficient This expresses the relationship between the velocity of the fragment and time.

[0029] Due to drag, the speed of the fragment gradually decreases during its flight. Let the drag coefficient of the fragment be... According to the law of resistance, the differential equation of motion for the fragment is:

[0030] (2),

[0031] in For the quality of the fragments, For the density of air, The area of ​​the fragment determines its velocity. With time The relationship can be represented as:

[0032] (3),

[0033] In the formula, This indicates the dynamic initial velocity of the fragments in the main dispersion direction during warhead detonation, such as... Figure 1 As shown, its initial velocity under static blasting conditions is... , No. The flight speed of the drone , the direction and angle of fragment scattering Related, then Represented as:

[0034] (4),

[0035] The damage efficiency coefficient in equation (1) is related to the fragment scattering characteristics. Due to the scattering characteristics of the fragments, more fragments with high kinetic energy are concentrated on the centerline of the damage area, and the damage efficiency coefficient can be expressed as: (5),

[0036] in, The standard deviation of damage efficiency This represents the damage efficiency gain coefficient. Indicates the first The line-of-sight angle between the drone and the target. This indicates that the maximum value between the two functions is taken. From the introduced damage efficiency function, it can be seen that the greater the velocity of the fragments upon impact with the target, and the closer the target is to the center line of the missile fragment distribution range, the better the damage effect.

[0037] like Figure 2 As shown, with the location of the target as the origin of the coordinate system, then the... The coordinates of the location of the drone are , The line-of-sight angle between the drone and the target is The line-of-sight distance between the drone and the target is The speed of the drone is The angle between the drone's velocity direction and the horizontal direction is... The magnitude of the drone's acceleration is The kinematic model of the drone relative to the ground is as follows:

[0038] (6),

[0039] in, express The first derivative, express The first derivative.

[0040] No. The trajectory of a drone can be represented as:

[0041] (7),

[0042] in and For the tangential and normal components of the UAV's acceleration, express The first derivative, express The first derivative.

[0043] Considering the regulation of drones:

[0044] (8),

[0045] in and The control input is obtained from the reinforcement learning of the damage maximization strategy.

[0046] By combining equations (5)-(8), a damage-kinematic coupling model can be established to describe the potential damage effectiveness of a UAV swarm under dynamic flight conditions, considering the environmental state. That is, the line-of-sight angle between the drone and the target is The line-of-sight distance between the drone and the target is The speed of the drone is The angle between the drone's velocity direction and the horizontal direction is... The magnitude of the drone's acceleration is .

[0047] Step 2: Based on the environmental conditions of the drone swarm Calculate the optimal detonation time and detonation point for the drone swarm, and calculate the minimum arrival time T1 for each drone in its current state. Based on the kinematic function in the damage-kinematic coupling model, determine whether the drone has reached the predetermined target. If it does, proceed to step 7; otherwise, proceed to step 3, as follows:

[0048] Optimal initiation time It is an important parameter that changes dynamically during the flight of a drone swarm, and the optimal detonation time is... The essence of solving this problem is to perform an optimization search within the feasible solution space formed by the constraints.

[0049] For the Calculate the theoretical time for a drone to reach its target location:

[0050] (9),

[0051] in For the first The initial position vector of the drone. The location of the target. Let be the current flight speed of the drone. Then define the damage effectiveness response function with respect to time and position:

[0052] (10)

[0053] (11),

[0054] in For fragment damage function, For drones from location The azimuth of the target. For time The distance between the drone and the target at all times. Indicates the first A drone on the coordinate axis velocity in direction and Indicates the first A drone on the coordinate axis velocity in the direction, This indicates the total number of drones.

[0055] Next, the function needs to be globally optimized to obtain the optimal detonation time and location, thus solving the problem in an optimized manner:

[0056] (12),

[0057] right Global optimization is performed to obtain the optimal detonation time and location, thus solving the problem in a way that optimizes the objective.

[0058] (13)

[0059] in, This indicates the time value representing the start time of the drone. The drone that took the longest time was The time value of the location.

[0060] Step 2.1: Substitute the environmental parameters at this time into formula (10), compare the calculated value with the optimal detonation time, and determine whether the UAV has reached the predetermined target. If it is within the constraints, the predetermined target has been reached, and proceed to step 6.

[0061] Step 2.2: Substitute the environmental parameters at this time into formula (10), compare the calculated value with the optimal detonation time, and if it is not within the constraints, proceed to step 3.

[0062] Step 3: Construct a structured reward function, including a time-coordinated reward function. Angle-constrained reward function and the hit location-guided reward function Calculate the reward value based on the structured reward function. Establish an experience pool mechanism to store the drone's current actions, environmental status, and reward data in the experience pool. In detail, as follows:

[0063] A time-coordinated reward function is designed based on the principle of ensuring that all drones arrive at the target simultaneously at the optimal detonation time. :

[0064] (14)

[0065] (15)

[0066] For the first The estimated arrival time of the drone. , It's about adjusting parameters. Refers to the optimal detonation time. Indicates the number of drones, The time when the last drone reaches the target. For the first The difference between the expected arrival time and the optimal detonation time of the drone.

[0067] Exponential terms in a function When the maximum time deviation approaches 0, it approaches 1, meaning that the drone can continuously provide rewards when the optimal detonation time is reached. When the deviation of each drone approaches 0, the value becomes 0, and no further penalty is imposed. Ultimately, this achieves time-based coordination among multiple drones, enabling them to reach the target simultaneously at the optimal detonation time.

[0068] According to the damage efficiency function, the closer the target is to the center line of the fragment's damage range, the greater the damage effectiveness. Therefore, the attack angle of the UAV needs to be adjusted. Based on this, an angle-constrained reward function is designed to guide each UAV to strike the target at a certain angle:

[0069] (16)

[0070] in , To adjust the parameters, the exponential term is in The reward is maximized when the angle approaches zero. When there is a large angular deviation, the angle-constrained reward function... The cosine term dominates, while the exponential term dominates when the angle deviation is small, thus achieving precise convergence.

[0071] To maximize damage, the drone's position needs to be guided, and a position-constrained reward function needs to be designed accordingly. :

[0072] (17)

[0073] in Indicates the current location of the drone. Indicates the first Damage value of a drone, This represents the theoretical maximum damage value. To determine the optimal detonation time and corresponding detonation location, This represents the spatial decay coefficient. The closer the drone's position is to the theoretically optimal detonation location, the greater the reward.

[0074] Thus, through the design of the above reward function, the innovation of this invention lies in the integration of the damage model and the reinforcement learning algorithm. This hierarchical reward design based on the fused damage model enables the reinforcement learning agent to autonomously train a multi-UAV cooperative attack strategy that meets the requirements of precise time coordination, strict terminal angle constraints, and ultimately maximizes the damage effect through trial and error learning in a complex continuous state-action space and interaction with the environment.

[0075] Step 4: Use MATD3 as a reinforcement learning framework for the damage-kinematic coupling model and the structured reward function, utilizing the experience pool. The damage-kinematic coupling model is trained by... The network and policy network are continuously updated to obtain the optimal policy, thereby obtaining a trained damage-kinematic coupling model and the minimum accurate arrival time. The details are as follows:

[0076] like Figure 3 As shown, a damage fusion algorithm based on MATD3 is used for training. An experience pool mechanism is added during the training process, and the data generated by each interaction step during training is put into the experience pool. In the middle, it will be updated immediately. The network then randomly samples data for learning; the entire process can utilize a set of state data. To indicate. At time The current environmental status is At this time, each drone uses an environmental observation function. Obtain the observations at the current time, and then according to the policy function. Get the current action The obtained action is used as a control variable to apply to the state transition function in the environment, thus obtaining the environment state at the next moment. At the same time, each drone receives rewards from the environment. The reward value is used to update the policy function. This allows the environment to gradually reach the desired state.

[0077] (18)

[0078] in, The asterisk represents the optimal state, which is the control input obtained from reinforcement learning of the damage maximization strategy.

[0079] The value function can be estimated by the total expected reward that the agent receives at that moment:

[0080] (19)

[0081] in, This represents the expected value after the drone performs the current action; For intelligent agents from time to time The total reward obtained by the end of this training round can be expressed as:

[0082] (20)

[0083] in, Indicates the first The reward value for each drone. This represents the total time from the start to the end of training. Indicates the step size. This represents the coefficient, and its range of values ​​is... .

[0084] The above strategies It is a probability function that represents the probability of an event occurring in the observed value. Take action below The probability of this restricts it to a discrete and finite action space, making it unsuitable for problems with continuous action spaces.

[0085] This problem is solved using a deterministic policy gradient algorithm, that is, by employing a parameterized policy. To approximate the optimal strategy During retraining, the strategy is continuously optimized by updating the parameters.

[0086] (twenty one),

[0087] in yes learning rate, This represents the gradient function.

[0088] According to the policy gradient theorem, we can obtain:

[0089] (twenty two),

[0090] Using parameterization network to replace That is, to continuously adjust during the training process. The value makes Once the maximum value is obtained, this optimization strategy can be applied.

[0091] The MATD3 algorithm also uses dual Network mechanism, equipping each drone with two network and and the two corresponding objectives network and .use Network parameters for policy networks Loss function during update Defined as:

[0092] (twenty three),

[0093] in, Indicates the status of the drone. Indicates the actions of the drone. Indicates the first The measurement network of drones, This indicates the number of layers in the network; here, the value is 64.

[0094] Used for updating loss function Defined as:

[0095] (twenty four),

[0096] The reward value function Defined as:

[0097] ,

[0098] in, This represents the reward value for the drone. This represents a coefficient with a value of 0.96. This indicates the evaluation network's calculation of the future after the drone performs its current action. Indicates the first The measurement network of drones, Indicates the first The actions of the drone Indicates the first The strategy network for the drone determines the next action based on the current observations. Indicates the first Observations from drones Indicates the first A network for determining the π of a drone; and The loss function can be updated using the gradient descent method shown in equation (12).

[0099] (25)

[0100] in It is the learning rate of the Q network. For gradient operators, The loss function is represented by the inertial update method shown in equation (13) for the target network parameters.

[0101] (26)

[0102] in It is the inertial update factor. At this point, combined with the design of the reward function in the previous section, the algorithm is continuously trained and iterated. Each training iteration substitutes the environmental parameters into formula (10), and the calculated value after substitution is compared with the optimal detonation time. If it is within the constraints, it is considered that the optimal strategy has been iterated. Thus, the specific training process of the entire algorithm is completed.

[0103] Step 5: Determine whether the predetermined target has been achieved based on the damage effectiveness function. If it is met, proceed to Step 6; otherwise, return to Step 4, as detailed below:

[0104] Step 5.1: Substitute the environmental parameters at this time into formula (10), compare the calculated value with the optimal detonation time, and if it is within the constraints, proceed to step 6.

[0105] Step 5.2: Substitute the environmental parameters at this time into formula (10), compare the calculated value with the optimal detonation time, and if it is not within the constraints, proceed to step 3.

[0106] Step 6: Output the policy network and value network, construct the Lyapunov function, and perform convergence proof on the trained damage-kinematic coupling model to ensure that the UAV arrives at the target simultaneously at the optimal detonation time, as follows:

[0107] The goal that needs to be achieved by multiple unmanned aerial vehicle (UAV) systems is:

[0108] (27)

[0109] in It is the first The estimated time for each drone to reach its target. This is the optimal detonation time calculated above. The extreme value of the difference between the two is 0, which indicates that all drones will detonate at the optimal time. Arrive at the target location on time.

[0110] Assuming there is a deviation between the estimated arrival time and the optimal detonation time of a single UAV during flight, this individual deviation... for:

[0111] (28)

[0112] Maximum time deviation of drone swarm for:

[0113] (29)

[0114] Among them, the first The estimated time for the drone to reach the target for:

[0115] (30)

[0116] in For the current moment, the first The remaining distance between the drone and the target point Let be the reference speed of the drone. Then, differentiating with respect to the time deviation, we get:

[0117] (31),

[0118] in The acceleration control variable for the UAV can be derived from the aforementioned algorithm:

[0119] (32),

[0120] Represents reward value and time The function, It is a symbolic function;

[0121] The following constructs a Lyapunov function. Based on the stability theorem of the function, it can be proved that the time deviation will converge to 0;

[0122] (33),

[0123] Taking the derivative of this function and substituting the reciprocal of the time deviation, we get:

[0124]

[0125]

[0126] (34),

[0127]

[0128]

[0129]

[0130] because ,so ,

[0131] Right now:

[0132] (35),

[0133] By Lyapunov's stability theorem, because the Lyapunov function... For any time deviation When not equal to 0, the function All are greater than 0, and only if all deviations When equal to 0, the function And because of the function Since the derivative of the function is always less than 0, the function satisfies the stability theorem, meaning the system is globally asymptotically stable at the equilibrium point. ,Right now This means that the system will converge at the optimal detonation time, and the drone will be able to reach the target at the optimal detonation time.

Claims

1. A reinforcement learning-based UAV swarm control method based on a damage maximization strategy, characterized in that, The steps are as follows: Step 1: Establish a damage model and kinematic model for the UAV swarm, and then establish a damage-kinematic coupling model. Based on the above damage-kinematic coupling model, determine the environmental state of the UAV swarm. , Proceed to step 2; Step 2: Based on the environmental conditions of the drone swarm Calculate the optimal detonation time and detonation point for the drone swarm, and calculate the minimum arrival time for each drone in its current state. Based on the kinematic function in the damage-kinematic coupling model, determine whether the UAV has reached the predetermined target. If the predetermined target has been reached, proceed to step 6; otherwise, proceed to step 3. Step 3: Construct a structured reward function, including a time-coordinated reward function. Angle-constrained reward function and the hit location-guided reward function Calculate the reward value based on the structured reward function. Establish an experience pool mechanism to store the drone's current actions, environmental status, and reward data in the experience pool. Proceed to step 4; Step 4: Use MATD3 as a reinforcement learning framework for the damage-kinematic coupling model and the structured reward function, utilizing the experience pool. The damage-kinematic coupling model is trained by... The network and policy network are continuously updated to obtain the optimal policy, thereby obtaining the minimum exact time of arrival. And obtain the trained damage-kinematic coupling model, then proceed to step 5; Step 5: Determine whether the predetermined target has been achieved based on the damage performance function. If the predetermined target has been achieved, proceed to step 6; otherwise, return to step 4. Step 6: Output the policy network and value The network constructs a Lyapunov function to prove the convergence of the trained damage-kinematic coupling model, ensuring that the UAV arrives at the target simultaneously at the optimal detonation time.

2. The reinforcement learning-based UAV swarm control method based on a damage maximization strategy according to claim 1, characterized in that, In step 1, a damage model and a kinematic model of the UAV swarm are established, followed by a damage-kinematic coupling model. Based on the damage-kinematic coupling model, the environmental state of the UAV swarm is determined. The details are as follows: Introducing a damage efficiency coefficient to quantify the degree of damage, the fragmentation damage function of a drone swarm. Specifically, it is expressed as follows: （1）， in, This represents the damage efficiency coefficient. Indicates the velocity of the fragment; Due to the influence of drag, let the flight drag coefficient of the fragment be... According to the law of resistance, the differential equation of motion for the fragment is: （2）， in, For the quality of the fragments, For the density of air, The area of ​​the fragment. Indicates time; The speed of the fragment With time The relationship is represented as: （3）， In the formula, This represents the dynamic initial velocity of the fragments in the main dispersion direction during warhead detonation, which is the same as the initial velocity of the fragments under static detonation conditions. , No. The flight speed of the drone , the direction and angle of fragment scattering Related, then Represented as: （4）， Considering the scattering characteristics of the fragments, the damage efficiency coefficient Represented as: （5）， in, The standard deviation of damage efficiency This represents the damage efficiency gain coefficient. Indicates the first The line-of-sight angle between the drone and the target. This indicates taking the maximum value between two functions; The damage efficiency function introduced shows that the greater the velocity of the fragments when they collide with the target, and the closer the target is to the center line of the missile fragment distribution range, the better the damage effect. Taking the location of the target as the origin of the coordinate system, then the... The drone's location coordinates are The line-of-sight angle between the drone and the target is The line-of-sight distance between the drone and the target is The speed of the drone is The angle between the drone's velocity direction and the horizontal direction is... The magnitude of the drone's acceleration is The kinematic model of the drone relative to the ground is as follows: （6）， in, express The first derivative, express The first derivative; No. The trajectory of the drone is represented as follows: （7）， in, and The first The acceleration of the drone in the tangential and normal directions. express The first derivative, express The first derivative; Considering the regulation of drones: （8）， in, and These are the control inputs obtained from reinforcement learning of the damage maximization strategy; By combining equations (5)-(8), a damage-kinematic coupling model can be established to describe the potential damage effectiveness of a UAV swarm under dynamic flight conditions. Environmental conditions of drones Including the line-of-sight angle between the drone and the target Line-of-sight distance between the drone and the target The speed of drones The angle between the drone's velocity direction and the horizontal direction And the acceleration of drones .

3. The reinforcement learning-based UAV swarm control method based on a damage maximization strategy according to claim 2, characterized in that, In step 2, based on the environmental conditions of the drone swarm Calculate the optimal detonation time and detonation point for the drone swarm, and calculate the minimum arrival time for each drone in its current state. Based on the kinematic functions in the damage-kinematic coupling model, it is determined whether the UAV has achieved the predetermined target, as follows: For the A drone is deployed to calculate its theoretical time to reach the target location. : （9）， in, For the first The initial position vector of the drone. The location of the target. This represents the current flight speed of the drone. Define the response function of damage effectiveness with respect to time and location. : （10）， （11）， in, For the fragmentation damage function of the drone swarm, For drones from location The azimuth angle of the target. for The distance between the drone and the target at any given time; Indicates the first A drone on the coordinate axis velocity in direction and Indicates the first A drone on the coordinate axis velocity in the direction, Indicates the total number of drones; right Global optimization is performed to obtain the optimal detonation time and location, thus solving the problem in a way that optimizes the objective. （12）， in, This indicates the optimal detonation time. express The solution that maximizes the function value in the space of all feasible solutions to the function; The constraints are: （13）， in, This indicates the time value representing the start time of the drone. The drone that took the longest time was The time value of the location; Substitute the environmental parameters at this time into formula (10), compare the calculated value with the optimal detonation time, and determine whether the UAV has reached the predetermined target. If it is within the constraints, that is, the predetermined target has been reached, and proceed to step 6; if it is not within the constraints, that is, the predetermined target has not been reached, and proceed to step 3.

4. The reinforcement learning-based UAV swarm control method based on a damage maximization strategy according to claim 3, characterized in that, In step 3, a structured reward function is constructed, including a time-coordinated reward function. Angle-constrained reward function and the hit location-guided reward function Calculate the reward value based on the structured reward function. Establish an experience pool mechanism to store the drone's current actions, environmental status, and reward data in the experience pool. In detail, as follows: A time-coordinated reward function is designed based on the principle of ensuring that all drones arrive at the target simultaneously at the optimal detonation time. : （14）， （15）， in, For the first The estimated arrival time of the drone. , All are adjustable parameters. Refers to the optimal detonation time. Indicates the number of drones, The time when the last drone reaches the target. For the first The difference between the expected arrival time and the optimal detonation time of the drone; Exponential terms in a function The value approaches 1 when the maximum time deviation is close to 0, meaning the drone can continuously provide rewards when the optimal detonation time is reached; exponential term When the deviation of each drone approaches 0, it is zero and no longer penalized, ultimately achieving time-coordination of multiple drones and enabling them to reach the target simultaneously at the optimal detonation time; Design an angle-constrained reward function to guide each drone to strike the target at a specific angle. : （16）， in , All are adjustment parameters, with the exponential term in The reward is maximized when the angle approaches zero. When there is a large angular deviation, the angle-constrained reward function... The cosine term dominates when the angle deviation is small, while the exponential term dominates, achieving precise convergence. To maximize damage, the drone's position needs to be guided, and a position-constrained reward function needs to be designed accordingly. : （17）， in, Indicates the current location of the drone. Indicates the first Damage value of a drone, theoretical maximum damage value , To determine the optimal detonation time and corresponding detonation location, This is the spatial attenuation coefficient; The closer the drone's location is to the theoretically optimal detonation position, the greater the reward.

5. The reinforcement learning-based UAV swarm control method based on a damage maximization strategy according to claim 4, characterized in that, In step 4, MATD3 is used as a reinforcement learning framework for the damage-kinematic coupling model and the structured reward function, utilizing the experience pool. The damage-kinematic coupling model is trained by... The network and policy network are continuously updated to obtain the optimal policy, thereby obtaining the minimum exact time of arrival. And obtain the trained damage-kinematic coupling model, as follows: The damage fusion algorithm based on MATD3 was used for training. An experience pooling mechanism was incorporated into the training process, where data generated at each interaction step was added to the experience pool. In the middle, it will be updated immediately. The network then randomly samples data for learning, and the entire process uses a set of state data. To indicate; At any moment The current environmental status is At this time, each drone uses an environmental observation function. Obtain the observations at the current time, and then according to the policy function. Get the current action The obtained action is used as a control variable to apply to the state transition function in the environment, thus obtaining the environment state at the next moment. At the same time, each drone receives rewards from the environment. The reward value is used to update the policy function. This allows the environment to gradually reach the desired state. （18）， in, The asterisk represents the best state, which is the control input obtained from reinforcement learning of the damage maximization strategy. value function Estimate using the agent's expected total reward at the current moment: （19）， in, This represents the expected value after the drone performs the current action; Represented as an intelligent agent from time... The total reward obtained by the end of this training round is expressed as: （20）， in, Indicates the first The reward value for each drone. This represents the total time from the start to the end of training. Indicates the step size. This represents the coefficient, and its range of values ​​is... ; The above strategies It is a probability function that represents the probability of an event occurring in the observed value. Take action below The probability of this restricts it to a discrete and finite action space, making it unsuitable for problems with continuous action spaces. This problem is solved using a deterministic policy gradient algorithm, that is, by employing a parameterized policy. To approximate the optimal strategy During training, the strategy is continuously optimized by updating the parameters. （21）， in, yes learning rate, Represents the gradient function; According to the policy gradient theorem, we get: （22）， Updating the strategy requires the use of a value network, which can be directly obtained in practical applications. It is very difficult, so parameterization is adopted. network to replace During training, it is necessary to continuously adjust the determined coefficients to measure the network. The value makes Once the maximum value is obtained, this optimization strategy can be applied again. Using double in the MATD3 algorithm Network mechanism, equipping each drone with two network and and the two corresponding objectives network and ,use Network parameters for policy networks Loss function during update Defined as: （23）， in, Indicates the status of the drone. Indicates the actions of the drone. Indicates the first The measurement network of drones, This indicates the number of layers in the network, which is 64 here. Used for updating loss function Defined as: （24）， Reward value function Defined as: in, This represents the reward value for the drone. This represents a coefficient with a value of 0.

96. This indicates the evaluation network's calculation of the future after the drone performs its current action. Indicates the first The actions of the drone Indicates the first The strategy network for the drone determines the next action based on the current observations. Indicates the first Observations from drones Indicates the first A network for determining the π of a drone; and Based on the loss function described above, the update is performed using the gradient descent method shown in equation (12): （25）， in yes The learning rate of the network, For gradient operators, Represents the loss function; The target network parameters are updated using the inertial update method shown in equation (13): （26）， in, It is the inertial update factor; Combined with the design of the reward function in step 3, the algorithm is continuously trained and iterated. Each training session substitutes the environmental parameters into equation (10), and the calculated value after substitution is compared with the optimal detonation time. If it is within the constraints, it is considered that the optimal strategy has been iterated to obtain the trained damage-kinematic coupling model.

6. The reinforcement learning-based UAV swarm control method based on a damage maximization strategy according to claim 5, characterized in that, Determine whether the predetermined target has been achieved based on the damage effectiveness function. If the predetermined target has been achieved, proceed to step 6; otherwise, return to step 4. Substitute the environmental parameters at this time into formula (10), compare the calculated value with the optimal detonation time, and determine whether the UAV has reached the predetermined target. If it is within the constraints, that is, the predetermined target has been reached, proceed to step 6. If the conditions are not met, i.e. the predetermined goal is not achieved, proceed to step 3.

7. The reinforcement learning-based UAV swarm control method based on a damage maximization strategy according to claim 6, characterized in that, In step 6, the policy network and value are output. The network constructs a Lyapunov function to prove the convergence of the trained damage-kinematic coupling model, ensuring that the UAV arrives at the target simultaneously at the optimal detonation time, as detailed below: The goal of drone swarms is to achieve the following: （27）， in, It is the first The estimated time for the drone to reach the target. The optimal detonation time; the limit of the difference between the two is 0, which means that all drones will be detonated at the optimal time. Arrive at the target location on time; Assuming there is a deviation between the estimated arrival time and the optimal detonation time of a single UAV during flight, this individual deviation... for: （28）， Maximum time deviation of drone swarm for: （29）， Among them, the first The estimated time for the drone to reach the target for: （30）， in, For the current moment The line-of-sight distance between the drone and the target. This is the reference speed for the drone; Taking the derivative with respect to the time deviation, we get: （31）， in The acceleration control quantity for the UAV is expressed as: （32）， Represents reward value and time The function, It is a symbolic function; Construct a Lyapunov function According to the stability theorem of the function, the time deviation will converge to 0. （33）， Differentiate the function Substituting the reciprocal of the time deviation, we get: （34）， because , , All are adjustable parameters; so , Right now: （35）， By Lyapunov's stability theorem, the Lyapunov function For any time deviation When not equal to 0, the function All are greater than 0, and only if all deviations When equal to 0, the function And because of the function Since the derivative of the function is always less than 0, the function satisfies the stability theorem, meaning the system is globally asymptotically stable at the equilibrium point. ,Right now This indicates that the system will converge at the optimal detonation time, and the UAV can reach the target simultaneously at the optimal detonation time.

Images