An entropy-regularized driving method for cooperative non-cooperative target capture of aircraft cluster

Abstract

The application discloses an aircraft cluster cooperative non-cooperative target capturing method driven by entropy regularization, applies the entropy regularization thought to reinforcement learning, optimizes the updating method of the parameters of an evaluation network and a policy network, so that a UAV can obtain a more optimal maneuvering strategy, and the UAV performs actions according to the maneuvering strategy, and realizes the interception of an attacking aircraft. The aircraft cluster cooperative non-cooperative target capturing method driven by entropy regularization can not rely on a deterministic strategy in the interception process of multiple UAVs, but adopts a random strategy, avoids the training from falling into a local optimal point, and the actions of the aircraft are randomized as much as possible while the aircraft completes the task, so that the exploratory and robustness can be practically improved.

Description

Technical Field

[0001] This invention relates to an entropy regularization-driven method for cooperative non-cooperative target acquisition in aircraft swarms, belonging to the field of guidance technology. Background Technology

[0002] Traditional UAV cooperative interception methods mostly employ conventional analytical guidance and control techniques, which require extensive analytical solution calculations, making them complex to implement and lacking in flexibility.

[0003] Existing technologies also include collaborative interception methods based on learning algorithms. However, these methods require a large number of learning samples, resulting in high training costs and difficulties in obtaining samples, making it difficult to guarantee sample coverage. This leads to long training times and poor interception stability after training.

[0004] Furthermore, existing collaborative interception methods based on learning algorithms directly output deterministic actions through their policy networks. This prevents UAVs from effectively exploring the action space, easily leading to convergence to local optima. Moreover, these methods are less flexible and prone to interception failures.

[0005] Therefore, it is necessary to further study multi-target cooperative interception methods to solve the above problems. Summary of the Invention

[0006] To overcome the above problems, the inventors conducted in-depth research and provided an entropy regularization-driven method for cooperative non-cooperative target acquisition in aircraft swarms, comprising the following steps:

[0007] S1. Each drone in the drone swarm acquires its own flight information and observes the flight information of the attacking aircraft, storing the flight information in the experience pool;

[0008] S2. Data is sampled from the experience pool through the policy network to generate a maneuvering policy. The UAV performs maneuvering actions according to the maneuvering policy, and the flight information of the UAV and the attack aircraft is obtained during the maneuvering action. The flight information and maneuvering action values ​​are stored in the experience pool.

[0009] S3. Evaluate the value of the maneuver actions corresponding to the maneuver strategy through the evaluation network and store the value in the experience pool;

[0010] S4. Sample data from the experience pool and optimize the centralized evaluation network;

[0011] S5. Sample data from the experience pool and update the parameters of the policy network;

[0012] S6. Repeat steps S2 to S5 to obtain the strategy model. Set the strategy model on the UAV. The UAV flies according to the maneuver strategy output by the strategy model to intercept the attacking aircraft.

[0013] In a preferred embodiment, a separate policy network is set up for each UAV, multiple UAVs share a centralized evaluation network, and multiple policy networks update parameters simultaneously.

[0014] In a preferred embodiment, the maneuver actions corresponding to the maneuver strategies generated by multiple policy networks are stored in the same experience pool.

[0015] In a preferred embodiment, the input to the centralized evaluation network is global information, which includes the flight information of each UAV itself and the flight information of the attacking aircraft observed by each UAV.

[0016] In a preferred embodiment, the input in the policy network is local information, which includes the flight information of the corresponding UAV itself and the flight information of the attacking aircraft observed by the UAV.

[0017] In a preferred embodiment, a total value function for the UAV is set in the centralized evaluation network, and the parameters of the centralized evaluation network are updated based on maximizing the total value.

[0018] In a preferred embodiment, an entropy term is included in the total value function, along with a hyperparameter α used to adjust the importance of the entropy term relative to the reward.

[0019] In a preferred embodiment, the total value function is set as follows:

[0020]

[0021] Among them, V π (s) represents the total value of the maneuver strategy, t represents the time number, and s t Let a represent the state of the UAV at time t. t ρ represents the action taken by the drone at time t. π The expression represents the strategy π(a) t |s t The state-action distribution of the interaction trajectory caused by ) This represents the expectation of the distribution, where γ represents the discount factor, and r(s) represents the expected value of the distribution. t ,a t ) represents the reward, π(·∣s t This indicates the maneuvering strategy adopted by the drone. Let s0 represent the entropy term of the maneuver strategy π, and let s0 represent the expectation given the state s.

[0022] In a preferred embodiment, the hyperparameter α is updated during the policy network update process.

[0023] In a preferred embodiment, the parameter update of the policy network is expressed as:

[0024]

[0025] in, Describes the optimal joint strategy. This represents the maneuvering strategy output by the policy network for the i-th drone.

[0026] The beneficial effects of this invention include:

[0027] (1) During the interception of multiple drones, it can adopt a stochastic strategy instead of a deterministic strategy, thus avoiding the training from getting stuck in a local optimum.

[0028] (2) While completing the mission, the aircraft should make its actions as random as possible, which can effectively improve its exploratory nature and robustness;

[0029] (3) It can solve the maneuver strategy of each unmanned aerial vehicle in the cluster in the combat scenario at one time, saving the calculation of analytical solution in the traditional interception process;

[0030] (4) Integrating priority experience revisit technology into reinforcement learning improves the efficiency of data utilization in the reinforcement learning exploration and optimization process, which is beneficial to the training of UAV policy networks and increases the training speed. Attached Figure Description

[0031] Figure 1 This diagram illustrates a flow chart of an entropy regularization-driven cooperative non-cooperative target acquisition method for aircraft swarms according to a preferred embodiment of the present invention.

[0032] Figure 2 The position curves during the drone's interception of the attacking aircraft in Example 1 are shown;

[0033] Figure 3 The normal overload curve of the UAV in Example 1 is shown;

[0034] Figure 4 The heading angle curve of the UAV in Example 1 is shown. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Through these descriptions, the features and advantages of the present invention will become clearer and more apparent.

[0036] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. Although various aspects of embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated otherwise.

[0037] The present invention provides an entropy regularization-driven method for cooperative non-cooperative target acquisition in aircraft swarms, comprising the following steps:

[0038] S1. Each drone in the drone swarm acquires its own flight information and observes the flight information of the attacking aircraft, storing the flight information in the experience pool;

[0039] S2. Data is sampled from the experience pool through the policy network to generate a maneuvering policy. The UAV performs maneuvering actions according to the maneuvering policy, and the flight information of the UAV and the attack aircraft is obtained during the maneuvering action. The flight information and maneuvering action values ​​are stored in the experience pool.

[0040] S3. Evaluate the value of the maneuver actions corresponding to the maneuver strategy through the evaluation network and store the value in the experience pool;

[0041] S4. Sample data from the experience pool and optimize the evaluation network;

[0042] S5. Sample data from the experience pool and update the parameters of the policy network;

[0043] S6. Repeat steps S2 to S5 to obtain the strategy model. Set the strategy model on the UAV. The UAV flies according to the maneuver strategy output by the strategy model to intercept the attacking aircraft.

[0044] Preferably, a simulation environment is set up, and steps S1 to S5 are performed in the simulation environment. In the simulation environment, the motion model of the UAV is set as follows:

[0045]

[0046] Where x and y are the planar coordinates of the UAV, v is the speed of the UAV, and γ is the angle between the direction of the UAV's speed and the x-axis, i.e., the trajectory deflection angle.

[0047] Furthermore, the flight information includes position, speed, and line-of-sight angular velocity.

[0048] Preferably, after acquiring flight information, the relative position and relative speed between the UAV and the attacking aircraft, as well as the line-of-sight angular rate of the UAV, are also obtained and stored in the experience pool along with the flight information as a data entry.

[0049] In this invention, by adding relative position, relative velocity, and line-of-sight angular rate information as prediction conditions for the policy network, the accuracy of the output maneuver strategy of the policy network is improved.

[0050] Furthermore, unlike existing methods that require a large amount of additional and complex battlefield environment information to perform penetration missions, in this invention, defensive maneuver strategies can be obtained using only a portion of the perceptible combat environment information, thus reducing the difficulty of information perception.

[0051] Policy networks and evaluation networks are commonly used networks in reinforcement learning. Their specific structures are not particularly limited in this invention, and those skilled in the art can freely choose according to actual needs.

[0052] Unlike traditional reinforcement learning, this invention sets up a separate policy network for each UAV, and multiple policy networks update parameters simultaneously.

[0053] Traditional reinforcement learning uses only one policy network, and updates are performed on this single policy network. The conventional approach involves setting up a policy network and an evaluation network for a single drone, learning evaluation and rewards based on the drone's maneuvers to determine the final policy network, and then distributing this final policy network across different drones. In this invention, a separate policy network is set up for each drone, and multiple policy networks simultaneously perform step S2. This distributes the data sampling process across multiple parallel policy networks, thereby achieving efficient data sampling and utilization.

[0054] Furthermore, in order to enable multiple policy networks to execute in parallel, the maneuver actions corresponding to the maneuver policies generated by multiple policy networks are stored in the same experience pool, and the same centralized evaluation network is used to evaluate the multiple policy networks. Based on the evaluation values, the parameters of the multiple policy networks are updated simultaneously.

[0055] In a preferred embodiment, the input to the centralized evaluation network is global information, which includes the flight information of each UAV itself and the flight information of the attacking aircraft observed by each UAV.

[0056] In the policy network, the input is local information, which includes the flight information of the corresponding UAV itself and the flight information of the attacking aircraft observed by the UAV.

[0057] According to a preferred embodiment of the present invention, in the centralized evaluation network, a total value function of the UAV is set, and the parameters of the centralized evaluation network are updated based on maximizing the total value.

[0058] In this invention, by setting a total value function, greater emphasis is placed on the comprehensive effect of different UAV maneuvering strategies.

[0059] More preferably, the total value function includes an entropy term and a hyperparameter α used to adjust the importance of the entropy term relative to the reward.

[0060] The entropy term is used to characterize the uncertainty in the policy network output. By setting the entropy term, the randomness of the policy network output is increased. This enables the multi-drone interception process to adopt a random strategy instead of a deterministic one, thus avoiding training getting stuck in local optima.

[0061] Furthermore, in this invention, by using a total value function with hyperparameter α, not only can the expected reward be maximized, but also the entropy can be maximized, so that the aircraft's actions are as random as possible while completing the mission, which can practically improve exploratoryness and robustness.

[0062] In a preferred embodiment, the total value function is set as follows:

[0063]

[0064] Among them, V π (s) represents the total value of the maneuver strategy, t represents the time number, and s t Let a represent the state of the UAV at time t. t ρ represents the action taken by the drone at time t. π The expression represents the strategy π(a) t |s t The state-action distribution of the interaction trajectory caused by ) This represents the expectation of the distribution, where γ represents the discount factor, and r(s) represents the expected value of the distribution. t ,a t ) represents the reward, π(·∣s t This indicates the maneuvering strategy adopted by the drone. Let s0 represent the entropy term of the maneuver strategy π, and let s0 represent the expectation given the state s.

[0065] In a preferred embodiment, to maximize the total value, an improved Bellman backup operator is set based on the above total value function, and the total value is maximized by minimizing the Bellman residual.

[0066] The improved Bellman backup operator Set to:

[0067]

[0068] in,

[0069] Q(s t+1 ,a t+1 Let represent the action value function at time t+1.

[0070] Furthermore, the parameters of the lumped evaluation network are obtained by minimizing the Bellman residuals, and are expressed as:

[0071]

[0072] Where θ represents the parameters of the centralized evaluation network, J Q (θ) represents the objective optimization function of the value network, and N represents the number of sampling rounds during the interaction between the UAV swarm and the environment. Q represents the expected value operation. θ (s t ,a t V represents the action value function with parameter θ. θ (s t+1 ) indicates that the network parameters are The state-value function of the drone swarm.

[0073] Preferably, the parameters of the centralized evaluation network are optimized using the stochastic gradient descent method, as follows:

[0074]

[0075] in, This represents gradient descent with parameters, where r represents the reward value, and Q... θ (s t+1 ,a t+1 ) represents the action value function.

[0076] In traditional value functions, the optimization objective is based on the state value function. In this invention, the objective is based on the gradient descent method. Based on the action value function, it can better meet the game-theoretic interception tasks of multiple unmanned aerial vehicles in complex scenarios.

[0077] According to the present invention, the parameter update of the policy network is expressed as:

[0078]

[0079] in, Describes the optimal joint strategy. Let π represent the maneuvering strategy output by the policy network for the i-th UAV, π represent the maneuvering strategy adopted by the UAV, and Π represent the set of policies.

[0080] Based on the above updates, the parameters of the policy network are trained by minimizing the following formula:

[0081]

[0082] Where φ represents the set of parameters of the policy network, φ = {φ1, ..., φ} i ,...,φ N},φ i Let π represent the policy network parameters of the i-th drone. φThis represents the set of maneuver policies output by all policy networks. The policy network parameter is φ i At that time, the maneuvering strategy output by the policy network, This indicates the operation of seeking the expected value.

[0083] Furthermore, the parameters of the policy network are updated using stochastic gradient descent, as follows:

[0084]

[0085] in, This indicates the policy network parameter φ i Gradient descent calculation, This indicates that the logarithmic gradient descent method is used to update the policy neural network parameters φ of the i-th UAV. Let θ represent the action value function of the i-th drone in a neural network with parameter θ.

[0086] In a preferred embodiment, the hyperparameter α is updated during the policy network update process, and more preferably, during the entropy term regularization process, as expressed as:

[0087]

[0088] in, This represents the objective function for updating parameter α in a neural network. Let π represent the entropy term of the maneuver strategy.

[0089] In this invention, entropy regularization is combined with the learning reinforcement process, and a combined update method is adopted to achieve efficient and stable parameter update effect during training. At the same time, the aircraft's actions are made as random as possible while completing the mission, which can effectively improve exploratoryness and robustness.

[0090] In a preferred embodiment, a soft update method is used when updating the policy network and the evaluation network.

[0091] Soft update is a method that uses a weighted average of new and old parameter values ​​to gradually update parameters.

[0092] In a preferred embodiment, the soft update is represented as:

[0093]

[0094] in, This represents the updated value of the neural network parameter θ. i This represents the value of the neural network parameter θ before it is updated. This represents the updated value of the neural network parameter β. i τ represents the value of the neural network parameter β before the update, and τ represents the hyperparameter used to adjust the soft update.

[0095] In traditional reinforcement learning training, parameters are typically updated by uniformly sampling data from an experience pool. This invention improves upon this traditional method, specifically...

[0096] A priority experience replay area is set up in the experience pool to store data on the interaction between the UAV and the environment. Furthermore, in S2 to S5, when sampling data from the experience pool, the sampling probability of data in the priority experience replay area is different from that of data in other areas. The sampling probability of data in the priority experience replay area is higher than that of data in other areas. That is, the data stored in the priority experience replay area is weighted by importance weight and sampled using non-uniform probability.

[0097] More preferably, the sampled data includes:

[0098] Experience samples: Experience samples consist of multiple elements, including the drone's current state, actions, rewards, and next state;

[0099] Importance sampling weights: These weights are used to adjust the gradient updates for empirical samples;

[0100] The sampling probability is related to the priority of the empirical samples: samples with higher priority are more likely to be selected, thus increasing the chances of these samples being used for training.

[0101] In S6, the UAV obtains its own flight information and the flight information of the attacking aircraft observed by the UAV, inputs the above information into the UAV's strategy model, and the strategy model outputs the maneuver strategy.

[0102] Example

[0103] Example 1

[0104] The simulation experiment, which involves multi-UAV cooperative interception, includes the following steps:

[0105] S1. Each drone in the drone swarm acquires its own flight information and observes the flight information of the attacking aircraft, storing the flight information in the experience pool;

[0106] S2. Data is sampled from the experience pool through the policy network to generate a maneuvering policy. The UAV performs maneuvering actions according to the maneuvering policy, and the flight information of the UAV and the attack aircraft is obtained during the maneuvering action. The flight information and maneuvering action values ​​are stored in the experience pool.

[0107] S3. Evaluate the value of the maneuver actions corresponding to the maneuver strategy through the evaluation network and store the value in the experience pool;

[0108] S4. Sample data from the experience pool and optimize the centralized evaluation network;

[0109] S5. Sample data from the experience pool and update the parameters of the policy network;

[0110] S6. Repeat steps S2 to S5 to obtain the strategy model. Set the strategy model on the UAV, and the UAV will fly according to the maneuver strategy output by the strategy model to intercept the attacking aircraft.

[0111] In S1, the coordinates of the protected target are set to (-121.3m, -92.7m), and two attack aircraft are set to attack the target with initial positions of (56.5m, -525.3m) and (-232.1m, 553.6m) and initial speeds of 15.5m / s and 16.4m / s, respectively.

[0112] Five drones were deployed to intercept the attacking aircraft. The initial positions of the drones were (-16.5.5m, -34.7m), (-99.9m, 48.2m), (-37.7m, -74.3m), (-72.2m, -14.9m), and (-73.8m, -112.1m), and their initial velocities were 15.1m / s, 16.1m / s, 17.4m / s, 17.8m / s, and 17.9m / s, respectively.

[0113] A separate policy network is set up for each UAV, and multiple policy networks update their parameters simultaneously. The maneuver actions corresponding to the maneuver strategies generated by multiple policy networks are stored in the same experience pool, and the same centralized evaluation network is used to evaluate the multiple policy networks.

[0114] In a centralized evaluation network, the input is global information, while in a policy network, the input is local information.

[0115] In the centralized evaluation network, the total value function of the UAV is set as follows:

[0116]

[0117] To maximize the total value, based on the aforementioned total value function, an improved Bellman backup operator is set up. This operator maximizes the total value by minimizing the Bellman residual. Set to:

[0118]

[0119] The parameters of the lumped evaluation network are obtained by minimizing the Bellman residuals, and are expressed as follows:

[0120]

[0121] The parameters of the centralized evaluation network are optimized using the stochastic gradient descent method, and are expressed as follows:

[0122]

[0123] The parameter update of the policy network is represented as:

[0124]

[0125] The parameters of the policy network are trained by minimizing the following formula:

[0126]

[0127] The parameters of the policy network are updated using stochastic gradient descent, as follows:

[0128]

[0129] The hyperparameter α is updated during the policy network update process and is expressed as:

[0130]

[0131] A priority experience replay area is set in the experience pool. In S2 to S5, when sampling data from the experience pool, the sampling probability of data in the priority experience replay area is different from that of data in other areas.

[0132] Figure 2 The position curve during the drone interception process is shown. As can be seen from the figure, the drone's trajectory is smooth, and it intelligently assigned tasks to the attacking swarm, thus achieving the interception of the attacking aircraft.

[0133] Figure 3 The normal overload curve of the UAV is shown. Figure 4 The heading angle curve of the UAV is shown. It can be seen that both the normal overload curve and the heading angle curve are smooth and meet the overload constraint requirements, thus ensuring the flight stability of the UAV.

[0134] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front," and "rear," etc., indicate the orientation or positional relationship based on the orientation or positional relationship in the working state of this invention, and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. Furthermore, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0135] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0136] The present invention has been described above with reference to preferred embodiments; however, these embodiments are merely exemplary and illustrative. Various substitutions and modifications can be made to the present invention based on these embodiments, all of which fall within the scope of protection of the present invention.

Claims

1. An entropy regularization-driven method for cooperative non-cooperative target acquisition in aircraft swarms, characterized in that, Includes the following steps: S1. Each drone in the drone swarm acquires its own flight information and observes the flight information of the attacking aircraft, storing the flight information in the experience pool; S2. Data is sampled from the experience pool through the policy network to generate a maneuvering policy. The UAV performs maneuvering actions according to the maneuvering policy, and the flight information of the UAV and the attack aircraft is obtained during the maneuvering action. The flight information and maneuvering action values ​​are stored in the experience pool. S3. Evaluate the value of the maneuver actions corresponding to the maneuver strategy through the evaluation network and store the value in the experience pool; S4. Sample data from the experience pool and optimize the centralized evaluation network; S5. Sample data from the experience pool and update the parameters of the policy network; S6. Repeat steps S2 to S5 to obtain the strategy model. Set the strategy model on the UAV. The UAV flies according to the maneuver strategy output by the strategy model to intercept the attacking aircraft. In the centralized evaluation network, a total value function for the UAV is set, and the parameters of the centralized evaluation network are updated based on maximizing the total value. The total value function includes an entropy term and hyperparameters to adjust the importance of the entropy term relative to the reward. The total value function is set as follows: in, This represents the total value of the maneuver strategy. t Indicates the time sequence number. express t The status of the drone at all times. express t Real-time actions taken by drones Indicates by strategy The resulting state-action distribution of the interaction trajectory This indicates the expectation of the distribution. Indicates the discount factor. Indicates a reward. This indicates the maneuvering strategy adopted by the drone. Indicates maneuver strategy The entropy term, This represents the expectation given the state s.

2. The entropy regularization-driven method for cooperative non-cooperative target acquisition in aircraft swarms according to claim 1, characterized in that, A separate policy network is set up for each drone, and multiple drones share a centralized evaluation network. The parameters of multiple policy networks are updated simultaneously.

3. The entropy regularization-driven method for cooperative non-cooperative target acquisition in aircraft swarms according to claim 1, characterized in that, The maneuvers corresponding to the maneuver policies generated by multiple policy networks are stored in the same experience pool.

4. The entropy regularization-driven method for cooperative non-cooperative target acquisition in aircraft swarms according to claim 1, characterized in that, In the centralized evaluation network, the input is global information, which includes the flight information of each UAV itself and the flight information of the attacking aircraft observed by each UAV.

5. The entropy regularization-driven method for cooperative non-cooperative target acquisition in aircraft swarms according to claim 1, characterized in that, In the policy network, the input is local information, which includes the flight information of the corresponding UAV itself and the flight information of the attacking aircraft observed by the UAV.

6. The entropy regularization-driven method for cooperative non-cooperative target acquisition in aircraft swarms according to claim 1, characterized in that, hyperparameters It is updated during the policy network update process.

7. The entropy regularization-driven method for cooperative non-cooperative target acquisition in aircraft swarms according to claim 1, characterized in that, The parameter update of the policy network is represented as: in, Describes the optimal joint strategy. Indicates the first The maneuvering strategy output by the strategy network of each drone. This indicates the maneuvering strategy employed by the drone. Represents the set of strategies.

Images