Multi-aircraft cooperative guidance method based on MAPPO algorithm

Abstract

The invention relates to a multi-aircraft cooperative guidance method based on an MAPPO algorithm, and the method comprises the steps: building a multi-aircraft cooperative guidance model which comprises an aircraft dynamics model, a process constraint model, a terminal constraint model, and a heeling angle amplitude constraint model; when the remaining flight range is larger than or equal to 500 km, a cooperative guidance law based on improved prediction correction is designed, and an aircraft attack angle and heeling angle guidance instruction is solved, and when the remaining flight range is smaller than 500 km, the cooperative guidance law based on the MAPPO algorithm is switched to, and the aircraft attack angle and heeling angle guidance instruction is solved. The method can accurately arrive at the middle-end shift change point in a relatively large time difference, and smoothly realizes a time cooperation task. The method has high robustness for state disturbance, and has a larger time cooperation range and a better time cooperation guidance capability.

Description

Technical Field

[0001] The invention belongs to the field of aircraft guidance control and relates to a multi-aircraft collaborative guidance method based on a MAPPO algorithm. Background Art

[0002] In recent years, the rapid development of hypersonic vehicles has attracted widespread attention from all walks of life. Thanks to its advantages such as long range, high flight speed, diverse and unpredictable trajectory, it has great development potential in military applications. Developed countries such as the United States, Russia, Germany, and the United Kingdom have implemented a number of research plans and achieved a series of research results. However, in the face of increasingly sophisticated anti-missile defense systems, traditional single-vehicle operations can no longer meet the combat mission requirements in complex battlefield environments, and multi-vehicle coordinated operations will inevitably become a typical combat style in future wars.

[0003] Multi-vehicle collaboration refers to the formation of multiple homogeneous or heterogeneous aircraft, which conduct coordinated flight in time and space dimensions according to mission requirements. At present, research on multi-vehicle collaboration is mostly focused on drone cluster collaboration, while research on hypersonic aircraft collaboration is relatively insufficient. Therefore, integrating the advantages of hypersonic aircraft and conducting research on multi-vehicle collaborative guidance is of great significance to promoting the leapfrog development of hypersonic aircraft. Summary of the invention

[0004] The technical problem solved by the present invention is to overcome the deficiencies of the prior art and propose a multi-aircraft cooperative guidance method based on the MAPPO algorithm.

[0005] The solution of the present invention is:

[0006] A multi-aircraft cooperative guidance method based on MAPPO algorithm, comprising:

[0007] Establishing a multi-aircraft collaborative guidance model, wherein the multi-aircraft collaborative guidance model includes a dynamic model of each aircraft, a process constraint model, a terminal constraint model, and a roll angle amplitude constraint model;

[0008] When the remaining flight range is greater than or equal to 500km, a collaborative guidance law based on improved prediction and correction is designed to solve the aircraft's angle of attack and roll angle guidance instructions:

[0009] When the remaining range is less than 500 km, switch to the collaborative guidance law based on the MAPPO algorithm to solve the aircraft's angle of attack and roll angle guidance instructions.

[0010] Preferably, in multiple aircraft, the dynamics model of each aircraft is as follows:

[0011]

[0012] Where r is the distance from the center of the earth to the aircraft, θ and φ represent the longitude and latitude of the aircraft respectively, V is the speed of the aircraft, γ and ψ represent the track angle and heading angle of the aircraft respectively, σ is the roll angle of the aircraft, m is the mass of the aircraft, g is the acceleration of gravity, L and D represent the lift and drag of the aircraft respectively.

[0013] Preferably, the process constraint model of each aircraft includes a heat flow rate constraint model, an overload constraint model, a dynamic pressure constraint model, and a quasi-equilibrium gliding condition constraint model;

[0014] The heat flow rate constraint model is as follows:

[0015]

[0016] In the formula, represents the heat flux rate at the stagnation point of the aircraft; k Q is the heat flow rate model coefficient; is the maximum allowable heat flux rate at the aircraft stagnation point, ρ is the atmospheric density at the aircraft altitude, and V is the aircraft speed;

[0017] The overload constraint model is as follows:

[0018]

[0019] Where n is the aircraft overload; n max is the maximum permissible overload; g0 is the gravitational acceleration at sea level, m is the mass of the aircraft, g is the gravitational acceleration, L and D are the lift and drag of the aircraft respectively;

[0020] The dynamic pressure constraint model is as follows:

[0021]

[0022] Where q is the dynamic pressure of the aircraft; q max is the maximum allowable dynamic pressure;

[0023] The quasi-balanced gliding condition constraint model is as follows:

[0024]

[0025] r is the distance from the center of the earth to the aircraft, and σ is the tilt angle of the aircraft.

[0026] Preferably, a collaborative guidance law based on improved prediction correction is designed to solve the guidance instructions of the aircraft angle of attack and roll angle, and the method is as follows:

[0027] Design a preset angle of attack profile for each aircraft;

[0028] Solve for the roll angle amplitude of each aircraft;

[0029] Solve for the sign of the roll angle for each vehicle.

[0030] Preferably, the preset attack angle α profile is in the following specific form:

[0031]

[0032] Where: α max is the maximum angle of attack; α LDmax is the angle of attack corresponding to the maximum lift-to-drag ratio; V0 is the initial speed of the aircraft; V aoa1 and V aoa2 are the speed of the aircraft at the segmentation points, It is the ideal speed for the mid- and end-shift handover points.

[0033] Preferably, the method for solving the roll angle amplitude of each aircraft is as follows:

[0034] The remaining range error and time error are predicted, and the overall error is obtained by weighting them. The secant method is used to obtain the roll angle amplitude that makes the overall error zero.

[0035] Preferably, the method for solving the sign of the roll angle of each aircraft is as follows:

[0036] A heading angle error corridor is designed for each aircraft, and the corridor width is adjusted based on the remaining flight time error. When the aircraft heading angle error exceeds the boundary of the aircraft heading angle error corridor, the aircraft roll angle is reversed, that is, the sign changes, thereby determining the sign of the aircraft roll angle.

[0037] Preferably, the collaborative guidance law based on the MAPPO algorithm is as follows:

[0038] (1) Designing the preset angle of attack profile of each aircraft;

[0039] (2) Design the observations, terminal rewards, and process rewards for each spacecraft;

[0040] (3) Use a fully connected neural network to establish the action network and evaluation network of all aircraft, and train the action network. The process is as follows:

[0041] S1. The aircraft action network outputs the aircraft roll angle command (including amplitude and sign) in real time according to the observed value, receives the process reward of environmental feedback, and stores the data of interaction with the environment in the experience pool until the aircraft arrives at the mid-terminal handover point and obtains the terminal reward; after all aircraft arrive at the terminal position, a learning cycle ends and the experience pool completes data collection;

[0042] S2. Using the data collected from the experience pool, the aircraft action network is trained in a centralized training method: During the training process, data is randomly extracted from the aircraft's experience pool data, the parameters of the evaluation network are updated, and the parameters of the action network are updated according to the value function generated by the evaluation network;

[0043] S3. After multiple update iterations, the data in the experience pool is cleared;

[0044] S4, return to step S1, and restart a new round of data collection and algorithm training process until the MAPPO algorithm converges and the action network parameter training is completed;

[0045] (4) Each aircraft uses a distributed execution method to output roll angle commands in real time based on the trained action network until the aircraft arrives at the mid-terminal handover point.

[0046] Preferably, in step (1), the specific form of the preset attack angle α profile is as follows:

[0047]

[0048] Where: α max is the maximum angle of attack; α LDmax is the angle of attack corresponding to the maximum lift-to-drag ratio; V0 is the initial speed of the aircraft; V aoa1 and V aoa2 are the speeds of the aircraft at the segmentation points respectively; It is the ideal speed for the mid- and end-shift handover points.

[0049] Preferably, in step (2), the observation quantity of each aircraft is The first six items represent the state of the aircraft; Δr = rr f is the difference between the current aircraft altitude and the terminal altitude; ΔV = VV f Indicates the difference between the current aircraft speed and the terminal speed; Δθ=θ-θ f Indicates the difference between the current aircraft longitude and the terminal longitude; Indicates the difference between the current aircraft latitude and the terminal latitude; S is the remaining range of the aircraft; is the rate of change of height.

[0050] Preferably, in step (2), the terminal reward for each aircraft is as follows:

[0051]

[0052] rr goal The reward for reaching the terminal position, rr dev Indicates the penalty value when the longitude and latitude of the aircraft deviate too far from the mid-terminal handover point when it arrives at the terminal position.

[0053] Preferably, in step (2), the process reward function of each aircraft includes a distance reward, a roll angle constraint reward, and a time constraint reward:

[0054] Distance Reward rr goto as follows:

[0055]

[0056] Where S represents the remaining range of the aircraft; R0 is the radius of the earth; represents normalization processing; k togo >0 is a constant;

[0057] Roll angle constraint reward rr diff as follows:

[0058]

[0059] where σ pre_cor represents the roll angle generated by the prediction correction guidance; σ mappo is the roll angle generated by the MAPPO algorithm; k comp >0 is a constant;

[0060] Time Constraint Reward rr time as follows:

[0061] rr time =-k time |t togo -t co |

[0062] The time constraint reward enables multiple aircraft to meet the coordinated time constraints.

[0063] Preferably, in the cooperative guidance law based on the MAPPO algorithm, t togo The remaining flight time of the aircraft is calculated by analytical expression. The analytical expression is as follows:

[0064]

[0065] Where: R0 is the radius of the earth; g0 is the gravitational acceleration at sea level; V is the current speed of the aircraft; V f is the speed at the middle and end handover points; σ mappo is the roll angle generated by the cooperative guidance law based on the MAPPO algorithm, L and D are the lift and drag of the aircraft, respectively.

[0066] The beneficial effects of the present invention compared with the prior art are:

[0067] Aiming at the time cooperative guidance problem of multiple hypersonic aircraft, the present invention designs a time cooperative guidance law based on improved prediction correction and a cooperative guidance law based on MAPPO algorithm, and forms a composite guidance method with the remaining range of the aircraft as the switching logic; the composite guidance law proposed by the present invention can accurately reach the mid-term handover point in a large time difference and smoothly realize the time cooperative task. The composite guidance law of the present invention has strong robustness to state disturbance, a larger time cooperative range and better time cooperative guidance capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is the longitude and latitude variation curve;

[0069] Figure 2 is the curve of height changing with speed;

[0070] Figure 3 is the curve of the roll angle changing with time;

[0071] Figure 4 is the curve of heading angle changing with time;

[0072] Figure 5 is the curve of track angle changing with time. DETAILED DESCRIPTION

[0073] The present invention will be further described below in conjunction with the accompanying drawings.

[0074] The present invention designs a phased time cooperative guidance law based on the remaining range. When the remaining range is large, an improved predictive correction guidance is adopted, and the remaining range error, the remaining cooperative flight time error and the terminal height error are considered at the same time. The roll angle instruction is obtained in combination with the heading angle error corridor to realize the re-entry guidance of the aircraft under time constraints. When the remaining range is small, in order to improve the guidance accuracy and time coordination capability, switch to the multi-agent reinforcement learning method, by designing the observation space and action space, and considering the distance between the aircraft and the mid-terminal handover point, and the time error between aircraft to design the reward function, adjust the reward weights and algorithm parameters to train the aircraft, and finally realize the time cooperative guidance of the aircraft. The specific method is as follows (each aircraft in the multi-aircraft is taken as an example below):

[0075] (1) Hypersonic vehicle modeling

[0076] (a) Dynamic modeling of each aircraft:

[0077]

[0078] Where r is the distance from the center of the earth to the aircraft, θ and φ represent the longitude and latitude of the aircraft respectively, V is the speed of the aircraft, γ and ψ represent the track angle and heading angle of the aircraft respectively, σ is the roll angle of the aircraft, m is the mass of the aircraft, g is the acceleration of gravity, L and D represent the lift and drag of the aircraft respectively.

[0079]

[0080] Where, ρ = ρ0e -βH is the atmospheric density at the height of the aircraft, ρ0 is the atmospheric density at sea level; β = 1 / H MCP , H MCP =7.11km is the base altitude; S ref is the characteristic area of ​​the aircraft; C L and C D are lift coefficient and drag coefficient, respectively, which are related to the aircraft attack angle α and speed V. The present invention selects the American general aviation aircraft CAV-H as the research object. The lift coefficient C L and the drag coefficient C D The expression is:

[0081]

[0082] In the formula, C L0 , C L1 , C L2 and C L3 is the fitting coefficient of lift expression; C D0 , C D1 , C D2 and C D3 is the drag expression fitting coefficient lift coefficient C L and the drag coefficient C D The relevant parameters are as follows:

[0083]

[0084] (b) Process Constraint Modeling

[0085] When a glider is flying at high speed, the high-speed airflow will generate a large amount of heat energy in the aircraft stationary area. Considering the safety of the aircraft, the heat flow rate constraint should be met:

[0086]

[0087] In the formula, represents the heat flux rate at the stagnation point of the aircraft; k Q is the heat flow rate model coefficient; is the maximum allowable heat flux rate at the aircraft stagnation point, ρ is the atmospheric density at the aircraft altitude, and V is the aircraft speed;

[0088] Overload refers to the multiple of the combined force of aerodynamic force and thrust acting on the aircraft relative to the aircraft's gravity. Due to the limited structural bearing capacity of the aircraft, the aircraft overload cannot exceed the maximum overload limit for safety reasons. Since the hypersonic aircraft in the present invention has no thrust effect during the gliding phase, its overload constraint is

[0089]

[0090] Where n is the aircraft overload; n max is the maximum allowable overload; g0 is the gravitational acceleration at sea level.

[0091] Also from the perspective of the structural bearing capacity of the aircraft, the aircraft should also meet the dynamic pressure constraint. During high-speed flight, the air flows in the opposite direction of the aircraft body at high speed. At this time, the air itself has kinetic energy. The pressure presented by this kinetic energy is called dynamic pressure:

[0092]

[0093] Where q is the dynamic pressure of the aircraft; q max is the maximum dynamic pressure.

[0094] The upper limit of the aircraft fuselage and wing structural strength and the bearing capacity of the aerodynamic control surface hinge determine the maximum allowable overload and dynamic pressure during the reentry flight. Heat flux constraint, overload constraint and dynamic pressure constraint are "hard constraints" that must be met during the flight of the aircraft. The quasi-balanced gliding condition is a soft constraint to ensure the stability of the aircraft's altitude and track angle and avoid large mutations. The mathematical form of this constraint is as follows:

[0095]

[0096] Substituting into the expression of the track angle of the dynamic model, the motion equation describing the quasi-balanced gliding can be obtained:

[0097]

[0098] It can be seen from the above formula that when the aircraft satisfies the above formula, the combined force of its gravity and aerodynamic lift provides the required centripetal force. At this time, the change in flight altitude is very small, and the rate of change of the track angle γ is close to zero.

[0099] (c) Terminal constraint modeling: The terminal constraints of a hypersonic vehicle include the altitude, speed, and longitude and latitude constraints at the mid-terminal handover point:

[0100]

[0101] In the formula, h f 、V f ,θ f ,φ fThey are the actual state quantities (altitude, speed, longitude and latitude) of the aircraft when it arrives at the mid-term and final handover points; It is the ideal state for the mid-term and end-term handover points; h , ε V , ε θ , ε φ are the maximum terminal errors allowed for each state variable (altitude, speed, longitude and latitude).

[0102] (d) Modeling of bank angle amplitude constraint: The aircraft needs to meet the above-mentioned hard constraints such as heat flux, overload and dynamic pressure during flight. However, repeated calculation of the aircraft's heat flux, overload and dynamic pressure will bring a large amount of calculation. Therefore, the hard constraints of heat flux, overload and dynamic pressure can be converted into constraints on bank angle amplitude, that is:

[0103]

[0104] The constraint form of the roll angle amplitude is:

[0105]

[0106] They are respectively used to convert the aircraft thermal flow, overload and dynamic pressure process constraints into the constraints of the roll angle amplitude.

[0107] In addition, in order to further suppress the oscillation of the reentry flight trajectory, especially to ensure that the first trough after reentry meets the constraint of the heat flux rate, the feedback control of the height change rate is added on the basis of the above-mentioned bank angle constraint. The purpose is to automatically adjust the amplitude of the bank angle according to the change of the aircraft altitude, so as to achieve the projection of the lift in the negative gravity direction Lcosσ can better meet the quasi-balanced gliding conditions, specifically:

[0108]

[0109] Where: σ(V) represents the roll angle of the aircraft as the speed changes, Indicates the current aircraft altitude change rate; is the reference height change rate; k is the feedback coefficient.

[0110] (2) Cooperative guidance law based on improved prediction and correction

[0111] When the remaining flight range is greater than 500km, the collaborative guidance law based on improved prediction correction is used to obtain the flight attack angle and roll angle of each aircraft.

[0112] (a) Angle of attack profile design: The control variables of a hypersonic vehicle during flight mainly include the angle of attack and the roll angle. However, considering that it is difficult to design both the angle of attack and the roll angle at the same time, the calculation amount is large. Therefore, the present invention adopts a preset angle of attack-velocity profile. The specific form of the angle of attack α profile is as follows:

[0113]

[0114] Where: α max is the maximum angle of attack; α LDmax is the angle of attack corresponding to the maximum lift-to-drag ratio; V0 is the initial speed of the aircraft; V aoa1 and V aoa2 are the vehicle speeds at the segmentation points respectively. It is the ideal speed for the mid- and end-shift handover points.

[0115] (b) Design of the roll angle amplitude

[0116] Transforming the dynamic equation in (1) into the form of energy E integral and ignoring the state variables that are irrelevant to the calculation of the remaining range S, we can obtain:

[0117]

[0118] Integrate the above equation and set the integration termination condition to E≤E f , E f is the terminal energy of the aircraft when it arrives at the mid-terminal handover point, that is, the optimal remaining range S(|σ opt |), σ opt represents the roll angle that makes the overall error zero. In addition, the actual remaining range of the aircraft S f The current latitude and longitude of the aircraft can be And the longitude and latitude of the mid-term and final handover points θ f , Direct access to:

[0119]

[0120] Therefore, the remaining range error is expressed as

[0121] ΔS(|σ opt |)=S(|σ opt |)-S f .

[0122] Remaining flight time t togo It can be obtained by integration:

[0123]

[0124] In the interval between two prediction and correction algorithm updates of the remaining flight time, the aircraft roll angle profile remains unchanged, so the aircraft's real-time remaining flight time is equal to the remaining flight time of the last prediction and correction algorithm update minus the flight time. On this basis, the collaborative remaining flight time t is defined as co is the average value of the remaining flight time corresponding to the current state of each aircraft t co :

[0125]

[0126] N is the total number of aircraft, t togo,i is the remaining flight time of the i-th aircraft.

[0127] The coordinated time error of each aircraft can be expressed as:

[0128] Δt=t togo -t co

[0129] In summary, the overall error Δe(|σ opt |) is represented as:

[0130] Δe(|σ opt |)=k S ΔS(|σ opt |)+k t Δt

[0131] Where: k S , k t are the weights corresponding to the remaining range error and the coordinated remaining flight time error respectively. Then, the secant method can be used to obtain the roll angle amplitude that makes the overall error zero:

[0132]

[0133] Where: k is the number of iterations required for the secant method iterative process to converge.

[0134] (c) Heel angle symbol design

[0135] The sign of the roll angle is mainly determined by the heading angle error corridor:

[0136]

[0137] Where: Δψ corridor is the width of the heading angle error corridor; Δψ out and Δψ in are the outer and inner boundary widths of the heading error corridor, Δψ out >Δψ in >0;V a and V bis a parameter to be set. When the aircraft heading angle error exceeds the boundary of the aircraft heading angle error corridor, the aircraft roll angle is reversed, that is, the sign changes. Therefore, the sign of the aircraft roll angle can be determined using the heading angle error corridor:

[0138]

[0139] Where: T hor is the lateral guidance period, t is the current moment, and the heading angle error Δψ is defined as the deviation between the aircraft heading angle ψ and the aircraft line of sight angle Φ:

[0140] Δψ=ψ-Φ

[0141] The aircraft line of sight angle Φ can be obtained through the spherical triangle:

[0142]

[0143] In addition, according to the current remaining flight time t of the aircraft togo and the coordinated remaining flight time t co Dynamically adjust the heading angle error corridor width based on the error:

[0144]

[0145] Where: T corridor The time error threshold for adjusting the heading angle error corridor width; Δψ ε In addition, in order to ensure that the lateral corridor has sufficient time to adjust while ensuring that the aircraft's roll angle reversal times are not too many and that the aircraft can successfully reach the mid-term handover point, the heading angle error corridor width should be limited to a certain range:

[0146]

[0147] Where: Δψ min(1,2,3) , Δψ max(1,2,3) They are respectively the minimum and maximum values ​​of the heading angle error corridor width corresponding to the aircraft speed at different stages.

[0148] (3) Cooperative guidance law based on MAPPO algorithm

[0149] When the remaining flight range is less than 500 km, the coordinated guidance law based on the MAPPO algorithm is adopted. In this algorithm, the observation quantity of each aircraft is designed to be The first six items represent the state of the aircraft; Δr = rr f is the difference between the current aircraft altitude and the terminal altitude; ΔV = VV f Indicates the difference between the current aircraft speed and the terminal speed; Δθ=θ-θ fIndicates the difference between the current aircraft longitude and the terminal longitude; Indicates the difference between the current aircraft latitude and the terminal latitude; S is the remaining range of the aircraft; is the altitude change rate. In the MAPPO algorithm, the action amount of each aircraft is a=σ, which is the roll angle of the aircraft.

[0150] Take each aircraft as an example:

[0151] (a) Terminal reward function

[0152]

[0153] The terminal reward function is divided into two parts, which represent the reward value obtained by the aircraft when it reaches the terminal position, rr goal The reward for reaching the terminal position is to determine whether the aircraft can meet the terminal constraints when arriving at the mid-terminal handover point. If so, a large reward value is given to encourage the aircraft to complete the time coordination task while meeting the terminal accuracy; dev Indicates the penalty value for the aircraft's longitude and latitude being too far away from the mid-terminal handover point when it reaches the terminal position. The terminal reward is set so that the aircraft can meet the specified accuracy when it reaches the mid-terminal handover point.

[0154] (b) Process reward function

[0155] The process reward function is mainly used to guide the aircraft to reach the mid-term and final handover points and meet the coordination time constraints:

[0156]

[0157] rr goto is the process reward, which is mainly used to guide the aircraft to fly to the mid-term handover point. S represents the remaining range of the aircraft; R0 is the radius of the earth; represents normalization processing; k togo >0 is a constant. The closer the aircraft is to the longitude and latitude of the mid-term and final handover points, the greater the reward for this part.

[0158]

[0159] rr diff represents the roll angle constraint reward, which uses the roll angle generated by the improved predictive correction guidance to constrain the roll angle generated by the MAPPO algorithm, where σ pre_cor represents the roll angle generated by the prediction correction guidance; σ mappo is the roll angle generated by the cooperative guidance law based on the MAPPO algorithm; k comp>0 is a constant. The purpose of this part of the reward is to speed up the algorithm training by imitating the predicted correction guidance roll angle, while leaving some room for improvement, that is, only when the difference between the two parts of the roll angle is greater than Δδ>0, will the penalty value constrain the roll angle generated by the MAPPO algorithm.

[0160] rr time =-k time |t togo -t co |

[0161] rr time represents the time constraint reward, which is mainly to make multiple aircraft meet the coordinated time constraints, where t togo The remaining flight time of the aircraft is calculated by analytical expression. The analytical expression is as follows:

[0162]

[0163] Where: R0 is the radius of the earth; g0 is the gravitational acceleration at sea level; V is the current speed of the aircraft; V f is the speed at the mid-terminal handover point. After obtaining the remaining flight time of all aircraft, average them to get t co , k time >0 is a constant. The purpose of this part of the reward is to influence the roll angle generated by the MAPPO algorithm through the penalty term, so that the remaining flight time of all aircraft is close to the average coordination time, thereby satisfying the coordination time constraint of multiple aircraft. Finally, the process reward obtained by each aircraft is obtained by adding the rewards of these three parts with weights.

[0164] (c) Training of MAPPO algorithm:

[0165] S1. Use a fully connected neural network to build an action network for all aircraft;

[0166] S2, the aircraft action network outputs the aircraft roll angle command (including amplitude and sign) in real time according to the observed value, receives rewards from the environment feedback, and stores the data of interaction with the environment in the experience pool until the aircraft arrives at the mid-terminal handover point and obtains the terminal reward. After all aircraft arrive at the terminal position, a learning cycle ends and the experience pool completes data collection;

[0167] S3. Using the data collected from the experience pool to train the aircraft action network in a centralized training method: During the training process, data is randomly extracted from the aircraft's experience pool data, the parameters of the evaluation network are updated, and the parameters of the action network are updated according to the value function generated by the evaluation network;

[0168] S4. After multiple update iterations, the data in the experience pool is cleared;

[0169] S5. Return to step S2 and restart a new round of data collection and algorithm training process until the MAPPO algorithm converges and the action network parameters are trained.

[0170] Finally, each aircraft adopts a distributed execution method and outputs roll angle commands in real time based on the trained action network until the aircraft arrives at the mid-term handover point.

[0171] Simulation Verification

[0172] The present invention takes the general aviation aircraft CAV-H as the research object, and the relevant parameters are set as follows: Earth radius R0 = 6378km; Sea level atmospheric density ρ0 = 1.2258kg / m 3 ; Gravitational acceleration at sea level g0 = 9.8 m / s 2 ; Aircraft mass m = 907.2 kg; Reference area S ref =0.4839m 2 ; Heat flow model coefficient k Q =5×10 -5 ; Maximum allowable heat flow rate Maximum allowable overload n max =3; Maximum allowable dynamic pressure q max =100kPa. In addition, the maximum angle of attack of the aircraft is α max =20°; Maximum lift-to-drag ratio corresponds to angle of attack α LDmax =20°; angle of attack profile velocity node V aoa1 =5000m / s, V aoa2 =2000m / s; lateral guidance period T hor = 0.1s. In the process of updating the roll angle amplitude by longitudinal prediction and correction, in order to ensure the terminal accuracy and reduce the amount of calculation for online correction of the roll angle, a prediction correction period T is designed to be shortened as the remaining range decreases. preco :

[0173]

[0174] In addition, the outer boundary of the aircraft heading angle error corridor Δψ out =21°; inner boundary Δψ in =3°; Node V a =5000m / s, V b =2500m / s. Dynamic adjustment threshold of heading corridor T corridor =0.8; heading angle corridor boundary width adjustment value When the remaining range of each aircraft in the composite guidance law reaches 400km, it switches to the cooperative guidance law based on the MAPPO algorithm. The number of aircraft N = 8 is selected, and the initial positions are shown in the following table:

[0175]

[0176] The MAPPO algorithm uses a distributed reinforcement learning method for training. The structure of the action network and the evaluation network uses a 2-layer 128-node fully connected network. The learning rate of the action network and the evaluation network is set to lr=5e-4, and the network is trained using the Adam optimizer, the discount factor is set to 0.99, and the GAE is set to 0.95. Since the MAPPO algorithm is an on-policy algorithm, the data stored in the experience pool is discarded after the update. Therefore, the batch size of the experience pool is set to 2000 to store all the trajectory data of the aircraft, the size of the mini batch is set to 600, and the number of reuses of the experience data is set to 6. The simulation results are shown in Figure 1 to Figure 5 From the simulation results, it can be seen that the method of the present invention can enable multiple aircraft to arrive at the same target point at the same time, and compared with the traditional method, the longitude and latitude errors and the time coordination errors are significantly reduced.

[0177] The core idea of ​​the present invention is to integrate the advantages of improved predictive correction guidance and collaborative guidance law based on MAPPO algorithm. When the remaining range is large, the improved predictive correction guidance can solve the collaborative flight trajectory with the smallest combined error of the remaining range error and the remaining flight time error through iterative optimization, and enable the aircraft to stably and reliably arrive near the mid-terminal handover point after a long-term and long-distance flight under the constraint of the heading angle corridor; when the remaining range of the aircraft reaches a certain distance, in order to achieve higher spatial and temporal collaborative guidance accuracy of the aircraft cluster, it is necessary to further increase the update frequency of the guidance command, while the ballistic integration process of the improved predictive correction guidance has a large amount of calculation and a long guidance cycle. The collaborative guidance law based on the MAPPO algorithm has the advantages of faster intelligent algorithm calculation speed, shorter guidance cycle, and higher degree of freedom of output roll angle command, which can further improve the collaborative guidance accuracy of the aircraft cluster.

[0178] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A multi-aircraft cooperative guidance method based on MAPPO algorithm, characterized in that: include: Establishing a multi-aircraft collaborative guidance model, wherein the multi-aircraft collaborative guidance model includes a dynamic model of each aircraft, a process constraint model, a terminal constraint model, and a roll angle amplitude constraint model; When the remaining flight range is greater than or equal to 500km, a collaborative guidance law based on improved prediction correction is designed to solve the aircraft's angle of attack and roll angle guidance instructions: When the remaining range is less than 500 km, switch to the collaborative guidance law based on the MAPPO algorithm to solve the aircraft's angle of attack and roll angle guidance instructions.

2. The multi-aircraft cooperative guidance method based on the MAPPO algorithm according to claim 1 is characterized in that: In multiple aircraft, the dynamic model of each aircraft is as follows: Where r is the distance from the center of the earth to the aircraft, θ and φ represent the longitude and latitude of the aircraft respectively, V is the speed of the aircraft, γ and ψ represent the track angle and heading angle of the aircraft respectively, σ is the roll angle of the aircraft, m is the mass of the aircraft, g is the acceleration of gravity, L and D represent the lift and drag of the aircraft respectively.

3. The multi-aircraft cooperative guidance method based on the MAPPO algorithm according to claim 1 is characterized in that: The process constraint model of each aircraft includes heat flow rate constraint model, overload constraint model, dynamic pressure constraint model, and quasi-equilibrium gliding condition constraint model; The heat flow rate constraint model is as follows: In the formula, represents the heat flux rate at the stagnation point of the aircraft; k Q is the heat flow rate model coefficient; is the maximum allowable heat flux rate at the aircraft stagnation point, ρ is the atmospheric density at the aircraft altitude, and V is the aircraft speed; The overload constraint model is as follows: Where n is the aircraft overload; n max is the maximum permissible overload; g0 is the gravitational acceleration at sea level, m is the mass of the aircraft, g is the gravitational acceleration, L and D are the lift and drag of the aircraft respectively; The dynamic pressure constraint model is as follows: Where q is the dynamic pressure of the aircraft; q max is the maximum allowable dynamic pressure; The quasi-balanced gliding condition constraint model is as follows: r is the distance from the center of the earth to the aircraft, and σ is the tilt angle of the aircraft.

4. The multi-aircraft cooperative guidance method based on the MAPPO algorithm according to claim 1 is characterized in that: Design a collaborative guidance law based on improved prediction correction to solve the guidance instructions of the aircraft angle of attack and roll angle. The method is as follows: Design a preset angle of attack profile for each aircraft; Solve for the roll angle amplitude of each aircraft; Solve for the sign of the roll angle for each vehicle.

5. The multi-aircraft cooperative guidance method based on MAPPO algorithm according to claim 4 is characterized in that: The specific form of the preset attack angle α profile is as follows: Where: α max is the maximum angle of attack; α LDmax is the angle of attack corresponding to the maximum lift-to-drag ratio; V0 is the initial speed of the aircraft; V aoa1 and V aoa2 are the speed of the aircraft at the segmentation points, It is the ideal speed for the mid- and end-shift handover points.

6. The multi-aircraft cooperative guidance method based on MAPPO algorithm according to claim 4 is characterized in that: The method to solve the roll angle amplitude of each aircraft is as follows: The remaining range error and time error are predicted, and the overall error is obtained by weighting them. The secant method is used to obtain the roll angle amplitude that makes the overall error zero.

7. The multi-aircraft cooperative guidance method based on MAPPO algorithm according to claim 4 is characterized in that: The method to solve for the sign of the roll angle for each aircraft is as follows: A heading angle error corridor is designed for each aircraft, and the corridor width is adjusted based on the remaining flight time error. When the aircraft heading angle error exceeds the boundary of the aircraft heading angle error corridor, the aircraft roll angle is reversed, that is, the sign changes, thereby determining the sign of the aircraft roll angle.

8. The multi-aircraft cooperative guidance method based on MAPPO algorithm according to claim 1 is characterized in that: The cooperative guidance law based on the MAPPO algorithm is as follows: (1) Designing the preset angle of attack profile of each aircraft; (2) Design the observations, terminal rewards, and process rewards for each spacecraft; (3) Use a fully connected neural network to establish the action network and evaluation network of all aircraft, and train the action network. The process is as follows: S1. The aircraft action network outputs the aircraft roll angle command (including amplitude and sign) in real time according to the observed value, receives the process reward of environmental feedback, and stores the data of interaction with the environment in the experience pool until the aircraft arrives at the mid-terminal handover point and obtains the terminal reward; after all aircraft arrive at the terminal position, a learning cycle ends and the experience pool completes data collection; S2. Using the data collected from the experience pool, the aircraft action network is trained in a centralized training method: During the training process, data is randomly extracted from the aircraft's experience pool data, the parameters of the evaluation network are updated, and the parameters of the action network are updated according to the value function generated by the evaluation network; S3. After multiple update iterations, the data in the experience pool is cleared; S4, return to step S1, and restart a new round of data collection and algorithm training process until the MAPPO algorithm converges and the action network parameter training is completed; (4) Each aircraft uses a distributed execution method to output roll angle commands in real time based on the trained action network until the aircraft arrives at the mid-terminal handover point.

9. The multi-aircraft cooperative guidance method based on MAPPO algorithm according to claim 8, characterized in that: In step (1), the specific form of the preset attack angle α profile is as follows: Where: α max is the maximum angle of attack; α LDmax is the angle of attack corresponding to the maximum lift-to-drag ratio; V0 is the initial speed of the aircraft; V aoa1 and V aoa2 are the speeds of the aircraft at the segmentation points respectively; It is the ideal speed for the mid- and end-shift handover points.

10. The multi-aircraft cooperative guidance method based on MAPPO algorithm according to claim 8, characterized in that: In step (2), the observations of each aircraft are The first six items represent the state of the aircraft; Δr = rr f is the difference between the current aircraft altitude and the terminal altitude; ΔV = VV f Indicates the difference between the current aircraft speed and the terminal speed; Δθ=θ-θ f Indicates the difference between the current aircraft longitude and the terminal longitude; Indicates the difference between the current aircraft latitude and the terminal latitude; S is the remaining range of the aircraft; is the rate of change of height.

11. The multi-aircraft cooperative guidance method based on MAPPO algorithm according to claim 8, characterized in that: In step (2), the terminal reward for each aircraft is as follows: rr goal The reward for reaching the terminal position, rr dev Indicates the penalty value when the longitude and latitude of the aircraft deviate too far from the mid-terminal handover point when it arrives at the terminal position.

12. The multi-aircraft cooperative guidance method based on MAPPO algorithm according to claim 8, characterized in that: In step (2), the process reward function of each aircraft includes distance reward, roll angle constraint reward, and time constraint reward: Distance Reward rr goto as follows: Where S represents the remaining range of the aircraft; R0 is the radius of the earth; represents normalization processing; k togo >0 is a constant; Roll angle constraint reward rr diff as follows: where σ pre_cor represents the roll angle generated by the prediction correction guidance; σ mappo is the roll angle generated by the MAPPO algorithm; k comp >0 is a constant; Time Constraint Reward rr time as follows: rr time =-k time |t togo -t co | The time constraint reward enables multiple aircraft to meet the coordinated time constraints.

13. The multi-aircraft cooperative guidance method based on MAPPO algorithm according to claim 12, characterized in that: In the cooperative guidance law based on the MAPPO algorithm, t togo The remaining flight time of the aircraft is calculated by analytical expression. The analytical expression is as follows: Where: R0 is the radius of the earth; g0 is the gravitational acceleration at sea level; V is the current speed of the aircraft; V f is the speed at the middle and end handover points; σ mappo is the roll angle generated by the cooperative guidance law based on the MAPPO algorithm, L and D are the lift and drag of the aircraft, respectively.

Images