Cooperative route planning method based on centralized improved PPO algorithm

Abstract

The invention belongs to the technical field of aircraft intelligent body flight path planning, and particularly relates to a collaborative flight path planning method based on a centralized improved PPO algorithm. Comprising the steps of 1, forming a local formation by an aggressor and a jammer / decoy aircraft according to a collaborative attack task, and establishing a collaborative route planning model based on the local formation; and 2, a strategy training skill is introduced, a lower-layer collaborative flight path planning model based on a centralized BRPPO beta loop near-end strategy optimization algorithm is established, an aircraft intelligent body iteratively loops in a flight path planning environment, an optimal strategy is solved, and the optimal strategy is collaborative flight path planning. According to the invention, a cooperative track planning method of offline training and online planning is provided, so that the time of online optimization is saved, and the solving speed and the real-time performance of track planning are improved; according to the method, the neural network based on learning is adopted to solve the end-to-end nonlinear approximate optimal track, the precision and optimality of track planning are improved, and implementation of multi-aircraft fine track and tactical cooperation is facilitated.

Description

Technical Field

[0001] The invention belongs to the technical field of aircraft intelligent body trajectory planning, and in particular relates to a collaborative trajectory planning method based on a centralized improved PPO algorithm. Background Art

[0002] In recent years, with the tremendous progress in sensors, functional payloads, control and communication technologies, multi-UAV collaborative trajectory planning has become a research hotspot. Trajectory planning is essentially a decision optimization problem, the purpose of which is to find the optimal solution to the trajectory optimization objective under constraints.

[0003] At present, the academic community mostly uses static intelligent optimization algorithms to solve the problem of multi-UAV trajectory planning, such as genetic algorithms, annealing algorithms, particle swarm algorithms, ant colony algorithms, etc. There are also methods based on agent modeling. Some scholars have adopted agent-based hierarchical behavior modeling and combined modularization to realize the dynamic self-organization of UAV cluster route reconnaissance. This method has strong interpretability and stability, but poor flexibility. Some scholars have completed the optimal trajectory planning of multiple UAVs based on tasks through genetic algorithms and mixed integer linear programming algorithms. Some scholars have improved the speed and accuracy of UAV trajectory planning by improving the standard ant colony algorithm.

[0004] In recent years, intelligent optimization algorithms have been able to solve the optimal or suboptimal solutions under low-dimensional space constraints in parallel by improving the initialization population, updating strategy optimization individual iteration methods, and mixing multiple algorithms, and have good convergence. However, its essence is still random search, which requires an explicit objective function, and can only be solved online each time, and cannot be generalized to dynamically changing unknown environments. Especially when calculating large-scale problems, the amount of calculation increases exponentially and the convergence speed drops sharply. Therefore, it has certain limitations for the rapid planning of unmanned swarm combat in complex scenarios in the future. Summary of the invention

[0005] The present invention aims at the above-mentioned technical problems. The method of the present invention forms a local formation of attack aircraft and jammers / decoy aircraft for coordinated strike tasks, describes the coordinated trajectory planning as a sequential decision-making problem, and establishes a corresponding MDP model; based on the principle of deep reinforcement learning and an improved proximal policy optimization algorithm; establishes a lower-level coordinated trajectory planning model of the centralized BRPPO algorithm, including designing the state space, action space, and reward function of the UAV, and introducing five strategy training techniques such as domain randomization, maximum entropy, parameter sharing, data standardization, and adaptive parameters; completes offline training and online planning simulation experiments of coordinated trajectory planning of attack aircraft and jammers and coordinated trajectory planning of attack aircraft and decoy aircraft, as well as method robustness research and ablation experiment comparison, verifying the advanced nature and potential value of the proposed BRPPO method.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A collaborative trajectory planning method based on a centralized improved PPO algorithm includes:

[0008] Step 1: For the coordinated strike mission, the attack aircraft and jammer / decoy aircraft are organized into a local formation, and a coordinated trajectory planning model based on the local formation is established;

[0009] Step 2: Based on step 1, the strategy training technique is introduced to establish a lower-level collaborative trajectory planning model based on the centralized BRPPO beta loop proximal strategy optimization algorithm. The aircraft agent iterates in the trajectory planning environment to solve the optimal strategy, which is the collaborative trajectory planning.

[0010] Preferably, the step 1 comprises:

[0011] Step 1.1: Establish the MDP model of the attack aircraft and jammer / decoy aircraft;

[0012] Step 1.2: Based on step 1, a collaborative trajectory planning strategy network model based on local formation is established.

[0013] Preferably, the step 1.1 comprises:

[0014] Step 1.1.1: Arrange the attack aircraft and jammer / decoy aircraft into a local two-aircraft formation;

[0015] Step 1.1.2: State design, using coordinates in two-dimensional space to represent the positions of the attack aircraft, jammer / decoy aircraft and ground-to-air missile, and the concatenated vectors of all positions are input into the state space;

[0016] Step 1.1.3: Action space, define the actions of the attack aircraft and jammer / decoy aircraft as heading angular velocity and longitudinal acceleration;

[0017] Step 1.1.4: Establish reward functions for guiding the learning of attack aircraft and jammers, as well as reward functions for coordinated strikes between attack aircraft and decoy aircraft;

[0018] Step 1.1.5: Establish the MDP model and represent it with a five-tuple (S, A, R, P, γ), where S represents the state, A represents the action, R represents the reward function, P represents the state transfer function, and γ represents the discount factor, and calculate the cumulative discounted reward.

[0019] Preferably, the step 1.2 is specifically as follows:

[0020] The position splicing vector of step 1.1 is used as the input of the collaborative trajectory planning strategy network model, and the heading angular velocity and longitudinal acceleration decision quantities of the attack aircraft and jammer / decoy aircraft are output.

[0021] Preferably, the BRPPO algorithm process of step 2 is specifically as follows:

[0022]

[0023]

[0024] Among them, the aircraft agent perceives the initial state of the environment s∈S, S represents the state, selects action a∈A, A represents the action, the aircraft agent receives an immediate reward r∈R, R represents the reward function, the next state s′∈S, P represents the state transfer function, d represents whether the task is completed, minibatch represents the batch, K represents the maximum number of updates, and M represents the maximum number of cycles.

[0025] Preferably, the optimal strategy of step 2 is Specifically:

[0026]

[0027] Preferably, the advantage function of step 2 is:

[0028]

[0029] Among them, γ represents the discount factor, λ is a constant, and r t represents the reward at time t, s t+l+1 represents the state at time t+l+1, V(s t+l+1 ) means s t+l+1 The status value of .

[0030] Preferably, in step 2:

[0031] Beta distribution is used to sample random actions using its boundedness to reduce training variance, as shown below:

[0032]

[0033] Among them, α and β represent the parameters of Γ distribution, x represents the input of Γ distribution, and a bounded action is obtained through this formula.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) A collaborative trajectory planning method with offline training and online planning is proposed, which saves the time of online optimization and improves the solution speed and real-time performance of trajectory planning;

[0036] (2) A learning-based neural network is used to solve the end-to-end nonlinear approximate optimal trajectory, which improves the accuracy and optimality of trajectory planning and is conducive to the implementation of multi-aircraft fine trajectory and tactical coordination. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0038] In the attached picture:

[0039] Figure 1 The strategy network of the present invention;

[0040] Figure 2 A collaborative trajectory planning environment for the present invention;

[0041] Figure 3 Reward the learning curve for this invention;

[0042] Figure 4 The coordinated attack trajectory planning process of the attack aircraft and the jammer of the present invention is shown as follows: (a) before the jamming (light blue is the radar detection range before the jamming), (b) after the jamming (the blue dotted line indicates that the ground-to-air missile detection capability disappears);

[0043] Figure 5 For the average reward comparison of the present invention;

[0044] Figure 6 The attack trajectory planning process of the attack aircraft and the decoy aircraft in the present invention is as follows: (a) the decoy aircraft first enters the detection envelope, and (b) the attack aircraft takes the opportunity to attack and knock down the ground-to-ground missile and the target;

[0045] Figure 7 Monte Carlo simulation for coordinated attack trajectory planning of attack aircraft and decoy aircraft of the present invention: (a) the decoy aircraft first enters the ground-guided detection envelope, (b) the attack aircraft takes the opportunity to attack and complete the mission;

[0046] Figure 8 is the average reward comparison of different training techniques of the present invention;

[0047] Fig. 9 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0048] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0049] Example:

[0050] See attached Figure 1-9 As shown, a collaborative trajectory planning method based on a centralized improved PPO algorithm is characterized by comprising:

[0051] Step 1: For the coordinated strike mission, the attack aircraft and jammer / decoy aircraft are organized into a local formation, and a coordinated trajectory planning model based on the local formation is established. Including:

[0052] Step 1.1: Establish the MDP (Markov decision process) model of the attack aircraft and the jammer / decoy aircraft.

[0053] Step 1.1.1: Arrange the attack aircraft and jammer / decoy aircraft into a local two-aircraft formation;

[0054] Step 1.1.2: State space, using coordinates in two-dimensional space (x f ,y f )、(x j ,y j )、(x d ,y d )、(x s ,y s ) represent the position status of the attack aircraft, jammer, decoy aircraft and ground-based missile respectively. When the ground-based missile is not found (x s ,y s ) coordinates are (0,0), so the state space inputs the concatenated vector of all position states. If it is an attack aircraft or jammer scenario, it is expressed as (x f ,y f ,x j ,y j ,x s ,y s ), if it is an attack aircraft or decoy aircraft scenario, it is represented as (x f ,y f ,x d ,y d ,x s ,y s ).

[0055] Step 1.1.3: Action space. The actions of the attack aircraft and the jammer / decoy aircraft are the heading angular velocity (lateral acceleration) and longitudinal acceleration. The angular velocity is controlled to ensure that the UAV track is smooth.

[0056] The action space is represented as and

[0057] Step 1.1.4: Establish reward functions to guide the learning of attack aircraft and jammers, as well as reward functions for coordinated strikes between attack aircraft and decoy aircraft.

[0058] If the jammer can suppress the ground-guided radar outside the range of the ground-guided radar, and the attack aircraft can destroy the ground-guided radar and the target outside the final attack range of the ground-guided radar, both aircraft will receive positive rewards at the same time; if the jammer or the attack aircraft enters the attack range of the ground-guided radar or goes out of bounds, they will both receive negative rewards; if the distance between the attack aircraft and the jammer is less than the minimum anti-collision distance, they will also receive a smaller negative reward to ensure safety; for other intermediate states, a continuous dense reward related to the relative distance is obtained based on empirical knowledge, thereby guiding the agent to learn. Therefore, the reward function is as follows:

[0059]

[0060] Among them, d fs ,d js Respectively represent the relative distances between the attack aircraft, jammer and ground-to-ground missile, d s Indicates the attack distance of the ground-to-ground missile, d f Indicates the attack distance of the attack aircraft, d j Indicates the jammer interference distance, d fj represents the relative distance between the attack aircraft and the jammer, d collide Indicates the minimum collision avoidance distance.

[0061] For the coordinated strike mission between the attack aircraft and the decoy aircraft, the decoy aircraft is responsible for luring the ground-to-air missile attack, and the attack aircraft uses the gap between the ground-to-air missile attacks to shoot down the ground-to-air missile and the target. Therefore, the reward function is modified as follows:

[0062]

[0063] Among them, d fs ,d ds Respectively represent the distance between the attack aircraft, decoy aircraft and ground-to-ground missile, d s Indicates the attack distance of the ground-to-ground missile, d f Represents the attack distance of the attack aircraft, k is an empirical parameter, and is taken as 0.8 here.

[0064] Step 1.1.5: Establish the MDP model and represent it with a five-tuple (S, A, R, P, γ), where S represents the state, A represents the action, R represents the reward function, P represents the state transfer function, and γ represents the discount factor, and calculate the cumulative discounted reward.

[0065] The aircraft agent (attack aircraft, decoy aircraft) perceives the initial state of the environment s∈S and selects an action a∈A. After the action takes effect on the environment, the aircraft agent receives an immediate reward r∈R and transfers to the next state s′∈S according to the state transition function. Then, it continues to select a new action in this state and continues the state transition until the terminal state. P is unknown, and it is assumed that the state transition function is stationary, that is, it does not change over time, and γ is a constant.

[0066] Step 1.2, based on step 1, establish a collaborative trajectory planning strategy network model based on local formation. Specifically: the position splicing vector of step 1.1 is used as the input of the collaborative trajectory planning strategy network model, and the heading angular velocity and longitudinal acceleration decision quantities of the attack aircraft, jammer / decoy aircraft are output. Among them, the state s and action a in the MDP, the collaborative trajectory planning strategy network model is a mapping from s to a, and the aircraft agent selects action a (such as turning left or moving forward) from the s state through the strategy network mapping and then transfers to the next state s'.

[0067] Step 2: Based on step 1, introduce the strategy training technique and establish the lower-level collaborative trajectory planning model based on the centralized BRPPO (Beta Cycle Proximal Policy Optimization) algorithm. The aircraft agent iterates in the trajectory planning environment to solve the optimal strategy and realize collaborative trajectory planning. The specific process of the BRPPO algorithm is as follows:

[0068]

[0069]

[0070] Among them, the aircraft agent perceives the initial state of the environment s∈S, S represents the state, selects action a∈A, A represents the action, the aircraft agent receives an immediate reward r∈R, R represents the reward function, the next state s′∈S, P represents the state transfer function, d represents whether the task is completed, minibatch represents the batch, K represents the maximum number of updates, and M represents the maximum number of cycles.

[0071] (1) The specific process of using domain randomization for position state is as follows:

[0072] In order to improve the robustness of the aircraft agent's strategy and adapt to environmental uncertainty, domain randomization is used to increase disturbances to the input position state during training, as shown in the following formula. That is, each time the position state is reset, random initialization is performed, and the aircraft agent perceives a variety of initial states, and then learns the corresponding strategies, thereby abstracting more robust strategy features, preventing overfitting to a monotonous strategy, and thus having stronger adaptability to unknown environments.

[0073]

[0074] Here, δ represents the disturbance.

[0075] (2) Optimal Strategy Specifically:

[0076]

[0077] Among them, γ t represents the discount factor, r t represents the reward at time t.

[0078] (3) The advantage function is:

[0079]

[0080] Among them, γ represents the discount factor, λ is a constant, and r t represents the reward at time t, s t+l+1 represents the state at time t+l+1, V(s t+l+1 ) means s t+l+1 The status value of .

[0081] The process of advantage function standardization is:

[0082]

[0083] Among them, A′ represents the advantage value after standardization, and A represents the advantage value before standardization, that is, μ A represents the average value of A, σ A is the standard deviation of A.

[0084] (4) Adaptive adjustment of parameter size is as follows:

[0085] a. Adaptive learning rate

[0086] The learning rate controls the algorithm update step size. In the early stage of simulation training, the learning rate is relatively large, which is convenient for rapid learning. During training, learning rate annealing is adopted to gradually reduce the learning rate to prevent skipping the optimal solution and facilitate convergence to the local optimum. The formula is:

[0087]

[0088] Among them, η is the learning rate, η 0 is the initial learning rate, i episode is the current round, i max_episode is the maximum number of rounds.

[0089] b. Adaptive CLIP value

[0090] The principle of adaptive CLIP value is consistent with adaptive learning rate. A larger CLIP value is adopted in the early stage of training to speed up the strategy update. During training, CLIP value annealing is adopted to gradually reduce it to ensure that the optimal solution is not missed. The formula is:

[0091]

[0092] Where ζ is the current value of CLIP, ζ 0 is the initial value of CLIP.

[0093] (5) The strategy loss function is:

[0094]

[0095] (6) The value function is:

[0096] The value function loss is also standardized and compared in the ablation experiment. The formula is:

[0097]

[0098] Among them, V′ represents the standardized value, V i represents the state value before normalization, μ Ri is the average return, is the standard deviation of returns and T is a constant.

[0099] The value function loss is:

[0100]

[0101] (7) The policy entropy loss function is as follows:

[0102]

[0103] (8) The loss function is shown as follows:

[0104]

[0105] in, is the strategy loss, is the value function loss, is the entropy loss function, λ critic is the value loss coefficient, λ entropy is the entropy coefficient.

[0106] (9) The back propagation update strategy parameters are as follows:

[0107]

[0108] Among them, θ′ represents the updated policy parameter, θ represents the updated policy parameter, represents the gradient of the loss function.

[0109] (10) The reward function R(τ) is:

[0110]

[0111] In step 2:

[0112] Beta distribution is used to obtain random actions by taking advantage of its boundedness, which prevents actions from going out of bounds and reduces training variance, as shown in the following formula:

[0113]

[0114] Among them, α and β represent the parameters of Γ distribution, x represents the input of Γ distribution, that is, the output of the policy network actor, and a bounded action is obtained through this formula.

[0115] Simulation analysis:

[0116] 1. Simulation settings:

[0117] The collaborative trajectory planning environment of the present invention is assumed to be a square area of ​​100km×100km. The starting position of the attack aircraft is a square area of ​​20km-50km from the origin, and the attack distance is 30km. The starting position of the jammer is 10km-40km, and the attack distance is 50km. The coordinates of the ground-guided missile are (80,80), and the detection attack distance is 40km. Assuming that the jammer implements effective electronic interference, the detection range of the ground-guided missile is reduced to 20km. Figure 2 As shown. In the simulation, the above distances are normalized to adapt to the range of activation function values, facilitate neural network training, and prevent gradient elimination. The simulation platform is PyCharm Community Edition, the programming language is Python3.6, and the deep learning framework is PyTorch. The collaborative trajectory planning simulations in three different scenarios are completed respectively, compared with the unimproved algorithm, and robustness tests and ablation experiments are completed to test the performance of the proposed BRPPO algorithm.

[0118] The hyperparameter settings are shown in Table 2. The neural network uses orthogonal initialization and the optimization uses the Adam optimizer that combines Adagrad and momentum gradient descent.

[0119] Table 2 Hyperparameter settings

[0120]

[0121] 2. Collaborative trajectory planning of attack aircraft and jammers

[0122] In Experiment 1, our forces consisted of 1 attack aircraft and 1 jammer, and the enemy forces consisted of 1 ground-to-air missile and 1 target. Therefore, the attack aircraft had to compress the detection range of the ground-to-air missile radar from 40km to 20km under the interference of the jammer before it could safely destroy the ground-to-air missile and complete the mission.

[0123] Therefore, the BRPPO model was trained for 300 rounds to learn the policy network model, and the trained policy model was used for online reasoning testing to verify the online intelligent trajectory planning capability of BRPPO.

[0124] 2.1 Offline Training

[0125] First, the policy network and value network of BRPPO are trained offline, the cumulative rewards of the time steps in the training are recorded, and the performance is compared with that of RPPO and standard PPO algorithms to obtain the cumulative average reward learning curves of the three algorithms, as shown in Figure 3 shown.

[0126] As can be seen from the above figure, the learning curve variance of the RPPO model is large, the final performance of the standard PPO model convergence is poor, and the proposed BRPPO model has a higher average reward and a smaller average reward variance, so the training is more stable and robust, and therefore has better performance.

[0127] 2.2 Online Planning

[0128] Next, we test the online planning capability of the BRPPO model. Load the trained BRPPO model and perform 10 Monte Carlo simulation tests under different initial situations to verify the online planning capability of the model. The coordinated attack trajectory planning results of the attack aircraft and the jammer are obtained, as shown in the figure below. Figure 4 shown.

[0129] Figure 4 In (a), the attack aircraft and the jammer launch a coordinated attack. To ensure safety, the jammer needs to be located at the rear to carry out the jamming. After the jammer successfully jams, the detection range of the ground-to-air radar is reduced, and the attack aircraft takes advantage of the situation to attack. Figure 4 In (b), after the jammer jams the ground-guided radar, the attack distance of the attack aircraft is greater than the detection distance of the ground-guided radar, so the attack aircraft launches an effective attack to complete the mission, and the ground-guided radar and the target turn black. In 10 simulations, the number of collisions between the two aircraft is 0, which reflects the strong collaborative trajectory planning capability of the proposed BRPPO method.

[0130] 3. Collaborative trajectory planning of attack aircraft and decoy aircraft

[0131] In order to further test the intelligent collaborative trajectory planning capability of the BRPPO model, a coordinated attack mission scenario between an attack aircraft and a decoy aircraft was designed. The ground-to-air missile attack range is 20km, and the attack aircraft attack range is also 20km. It is impossible to directly and safely destroy the ground-to-air missile and the target, while the decoy aircraft is a low-cost, consumable drone that can be sacrificed. Therefore, the attack aircraft must coordinate with the decoy aircraft to attack, sacrifice the decoy aircraft, and use the time gap between the ground-to-air missile attacking the decoy aircraft to destroy the ground-to-air missile and the target, and the attack aircraft successfully completes the mission.

[0132] 3.1 Offline Training

[0133] Similarly, in the coordinated trajectory planning of the attack aircraft and the decoy aircraft, the BRPPO model is retrained, the cumulative rewards during training are recorded, the average reward learning curve is obtained, and it is compared with the learning curves of the RPPO and PPO algorithms. The results are as follows: Figure 5 shown.

[0134] from Figure 5 It can be seen that the RPPO and PPO models have large variance, poor training stability, and insufficient convergence ability, while the proposed BRPPO model achieves a higher final reward, a relatively smooth learning curve, a steady increase in performance, and better convergence performance.

[0135] 3.2 Online Planning

[0136] Next, load the trained BRPPO model for online planning. Initialize the environment and run 10 Monte Carlo tests to observe the online planning capabilities of the converged agent model. The results are as follows: Figure 6 shown.

[0137] Figure 6 In (a), the decoy aircraft flies directly toward the ground-based missile, attracting the ground-based missile to detect, track and attack, while the attack aircraft waits for an opportunity. Figure 6 In (b), the attack aircraft used the ground-based missile to track the time interval of the attack decoy aircraft, flew straight to the ground-based missile, and successfully shot down the ground-based missile and the target. The decoy aircraft was sacrificed, but the entire mission was completed. This shows that the BRPPO model has a strong tactical collaborative trajectory planning capability.

[0138] 4. Robustness testing

[0139] The robustness test of the BRPPO model of coordinated trajectory planning convergence of the attack aircraft and the decoy aircraft was conducted. A certain amount of randomness was added to the simulation. The starting positions of the attack aircraft, the decoy aircraft and the ground-guided missile were randomly initialized in a certain area around them. Wind uncertainty disturbance was added. Monte Carlo tests were performed 100 times to verify the robustness of the proposed method to the uncertainty in the unknown environment. The results are as follows: Figure 7 shown.

[0140] Figure 7 In (a), the initial position of the attack aircraft is a square area of ​​20km to 40km, and the initial position of the decoy aircraft is a square area of ​​80km to 100km. As can be seen from the figure, the decoy aircraft takes the lead in launching the attack, and the attack aircraft waits for an opportunity. When the decoy aircraft enters the detection range of the ground-guided missile radar first, the attack aircraft also quickly flies to the ground-guided missile, successfully destroys the ground-guided missile and the target, and finally completes the mission, as shown in 7(b). After 100 tests, it was found that the collaborative success rate was 94%. The number of failures is due to the model's insufficient ability to learn diverse perturbations. It shows that the trained BRPPO model can adapt to diverse initial situations, has good robustness, and has certain practical application value.

[0141] 5. Ablation Experiment

[0142] Finally, we compared the effects of different training techniques on the performance of the BRPPO algorithm, that is, we compared the BRPPO using all training techniques with the algorithms using advantage function normalization, value function normalization, layer initialization, adaptive learning rate, and adaptive clip techniques, and completed the ablation experiment. The research results are as follows: Figure 8 shown.

[0143] from Figure 8 It can be seen that the complete BRPPO algorithm has a faster learning speed and a higher final reward, which shows the value of the proposed training techniques. Among them, the initial performance of BRPPO without advantage function standardization deteriorates, so advantage function standardization can greatly improve the convergence speed of the model in the early stage. The final reward of BRPPO without value function standardization is the smallest, so value function standardization can improve the final reward performance of the BRPPO algorithm, while other techniques have relatively little effect on the performance of the BRPPO algorithm.

[0144] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A collaborative trajectory planning method based on a centralized improved PPO algorithm, characterized in that: include: Step 1: For the coordinated strike mission, the attack aircraft and jammer / decoy aircraft are organized into a local formation, and a coordinated trajectory planning model based on the local formation is established; Step 2: Based on step 1, the strategy training technique is introduced to establish a lower-level collaborative trajectory planning model based on the centralized BRPPO beta loop proximal strategy optimization algorithm. The aircraft agent iterates in the trajectory planning environment to solve the optimal strategy, which is the collaborative trajectory planning.

2. The collaborative trajectory planning method based on the centralized improved PPO algorithm according to claim 1 is characterized in that: The step 1 comprises: Step 1.1: Establish the MDP model of the attack aircraft and jammer / decoy aircraft; Step 1.2: Based on step 1, a collaborative trajectory planning strategy network model based on local formation is established.

3. The collaborative trajectory planning method based on the centralized improved PPO algorithm according to claim 2 is characterized in that: The step 1.1 comprises: Step 1.1.1: Arrange the attack aircraft and jammer / decoy aircraft into a local two-aircraft formation; Step 1.1.2: State design, using coordinates in two-dimensional space to represent the positions of the attack aircraft, jammer / decoy aircraft and ground-to-air missile, and the concatenated vectors of all positions are input into the state space; Step 1.1.3: Action space, define the actions of the attack aircraft and jammer / decoy aircraft as heading angular velocity and longitudinal acceleration; Step 1.1.4: Establish reward functions for guiding the learning of attack aircraft and jammers, as well as reward functions for coordinated strikes between attack aircraft and decoy aircraft; Step 1.1.5: Establish the MDP model and represent it with a five-tuple (S, A, R, P, γ), where S represents the state, A represents the action, R represents the reward function, P represents the state transfer function, and γ represents the discount factor, and calculate the cumulative discounted reward.

4. The collaborative trajectory planning method based on the centralized improved PPO algorithm according to claim 3 is characterized in that: The step 1.2 is specifically as follows: The position splicing vector of step 1.1 is used as the input of the collaborative trajectory planning strategy network model, and the heading angular velocity and longitudinal acceleration decision quantities of the attack aircraft and jammer / decoy aircraft are output.

5. The collaborative trajectory planning method based on the centralized improved PPO algorithm according to claim 4 is characterized in that: The BRPPO algorithm process of step 2 is specifically as follows: Among them, the aircraft agent perceives the initial state of the environment s∈S, S represents the state, selects action a∈A, A represents the action, the aircraft agent receives an immediate reward r∈R, R represents the reward function, the next state s′∈S, P represents the state transfer function, d represents whether the task is completed, minibatch represents the batch, K represents the maximum number of updates, and M represents the maximum number of cycles.

6. The collaborative trajectory planning method based on the centralized improved PPO algorithm according to claim 5 is characterized in that: The optimal strategy for step 2 Specifically:

7. The collaborative trajectory planning method based on the centralized improved PPO algorithm according to claim 6 is characterized in that: The advantage function of step 2 is: Among them, γ represents the discount factor, λ is a constant, and r t represents the reward at time t, s t+l+1 represents the state at time t+l+1, V(s t+l+1 ) means s t+l+1 The status value of .

8. The collaborative trajectory planning method based on the centralized improved PPO algorithm according to claim 7 is characterized in that: In step 2: Beta distribution is used to sample random actions using its boundedness to reduce training variance, as shown below: Among them, α and β represent the parameters of Γ distribution, x represents the input of Γ distribution, and a bounded action is obtained through this formula.

Images