On-line resolving method for throwing opportunity of child-mother aircraft based on reinforcement learning

Abstract

The invention provides a reinforcement learning-based on-line resolving method for the throwing opportunity of a child-mother aircraft, and the method comprises the steps: firstly predicting the trajectory of a mother aircraft based on an analytical solution / quasi-analytical solution, evaluating the flight capabilities of a child aircraft in different initial states based on an optimization method, building a flight capability data set of the child aircraft in different initial states, and carrying out the calculation of the throwing opportunity of the child aircraft; then, a Markov decision process model of the child-mother aircraft is established; and finally, on-line rapid solution of the scattering opportunity is realized based on a reinforcement learning algorithm fused with neural network optimization. The method solves the problem that the target arriving effect of the child aircrafts is seriously affected due to the fact that the actual scattering position of the child aircraft cluster deviates from the expected scattering position because the actual trajectory deviates from the expected trajectory when the mother aircraft is influenced by guidance precision, control precision, external interference and the like in the flight process. And the problem that in order to complete an expected flight task in various flight environments, the scattering opportunity needs to be quickly calculated on line is solved.

Description

Technical Field

[0001] The present invention relates to an online calculation method for the timing of mother-and-child aircraft scattering based on reinforcement learning, which is used to realize the online rapid calculation of the timing of the mother-and-child aircraft cluster scattering towards a typical arrival target, and belongs to the field of aircraft overall design and control technology. Background Art

[0002] A mother-and-child aircraft is a type of aircraft that is equipped with a certain number of the same or different types of child aircraft. The mother aircraft can drop a cluster of child aircraft at a specific drop location according to the flight mission requirements, and then the child aircraft cluster will reach the target area, which can significantly improve the utilization rate of the mother aircraft's payload and the effect of reaching the target. Related research began around the 1950s and has shown a rapid growth trend in the 21st century.

[0003] Existing research on the mother-and-child aircraft dispersion technology focuses on the dispersion separation method, the aerodynamic interference characteristics during the dispersion process, and the relative dispersion position of the child aircraft at the moment of dispersion. There is less research on the dispersion timing. Usually, the dispersion timing is set according to the expected dispersion position program before launch, which does not have the flexibility to face typical targets. In fact, the mother aircraft is affected by the guidance accuracy, control accuracy, external interference, etc. during the flight, and the actual trajectory will deviate from the expected trajectory, causing the actual dispersion position of the child aircraft cluster to deviate from the expected dispersion position, and the dispersion position of the child aircraft cluster seriously affects the effect of the child aircraft reaching the target. In addition, in order to complete the expected flight mission in various flight environments, the dispersion timing needs to be quickly solved online.

[0004] Therefore, an online solution method for the mother-and-child aircraft scattering timing based on reinforcement learning is proposed, which is used to realize the online rapid solution of the mother-and-child aircraft cluster scattering timing towards a typical arrival target. Summary of the invention

[0005] The technical problem to be solved by the present invention is: to overcome the deficiencies of the prior art, to solve the problem that the actual trajectory of the mother aircraft deviates from the expected trajectory due to the influence of guidance accuracy, control accuracy, external interference, etc. during flight, resulting in the actual scattering position of the sub-aircraft cluster deviating from the expected scattering position, thereby seriously affecting the effect of the sub-aircraft reaching the target; and to solve the problem that the scattering timing needs to be quickly solved online in order to complete the expected flight mission in various flight environments.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] An online calculation method for the timing of mother-and-child aircraft release based on reinforcement learning, comprising:

[0008] Predict the trajectory of the mother aircraft based on analytical / quasi-analytical solutions;

[0009] Evaluate the flight capability of the sub-aircraft under different initial states based on the optimization method;

[0010] Establish a flight capability data set of the sub-aircraft under different initial states;

[0011] Establish a Markov decision process model for the mother-and-child aircraft;

[0012] The timing of scattering is quickly calculated online based on a reinforcement learning algorithm that integrates neural network optimization.

[0013] In one embodiment of the present invention, the trajectory of the mother aircraft itself is predicted based on the analytical solution / quasi-analytical solution as follows:

[0014] Some necessary and reasonable assumptions are made based on the type of mother aircraft and the typical flight environment, such as assuming that the earth is a sphere, assuming that aerodynamic parameters are constant within a certain range and do not change with speed or angle of attack, and assuming that the mathematical relationship between atmospheric density and altitude is an exponential model.

[0015] On this basis, the analytical solution / quasi-analytical solution for the mother aircraft to predict its own trajectory is derived, so as to realize the online rapid prediction of the mother aircraft's own trajectory.

[0016] In one embodiment of the present invention, the flight capability of the sub-aircraft under different initial states is evaluated based on the optimization method as follows:

[0017] The flight capability of the sub-vehicle under different initial states can be evaluated by solving the reachable domain of the sub-vehicle under constraints such as thermal flow constraints, dynamic pressure constraints, and overload constraints.

[0018] Based on the optimization method, the reachable domain is solved by generating specific trajectories by changing the objective function multiple times. Take any initial state of the sub-aircraft as the initial value of the optimized state, set the objective function to the maximum range and the minimum range, solve the corresponding trajectory and record the maximum and minimum range values ​​L max With L min After that, set the objective function to L max With L min The middle amount: L i =L min +w i (L max -L min ), for the factor w i A series of values ​​are taken between (0,1) to obtain a corresponding series of optimization trajectories.

[0019] In one embodiment of the present invention, the flight capability data set of the sub-aircraft under different initial states is established as follows:

[0020] The terminal state of a series of optimized trajectories corresponding to a certain initial state of the sub-aircraft evaluated based on the optimization method affects the effect of the sub-aircraft reaching the target. Therefore, the terminal state of a series of optimized trajectories corresponding to a certain initial state of the sub-aircraft is defined as the flight capability of the sub-aircraft in this initial state. The terminal state of a series of optimized trajectories corresponding to different initial states of the sub-aircraft is calculated offline to establish a flight capability data set of the aircraft in different initial states.

[0021] In one embodiment of the present invention, the Markov decision process model for establishing the mother-child aircraft is specifically as follows:

[0022] The initial state of the sub-aircraft cluster is determined by the state of the mother aircraft when it is released, that is, the position, attitude, speed and other states of the sub-aircraft cluster are determined by the position, attitude, speed and other states of the mother aircraft when it is released. The mother aircraft's own trajectory prediction and the flight capabilities of the sub-aircraft in different initial states determine the release timing of the sub-aircraft cluster, and the release timing of the sub-aircraft cluster determines the effect of the sub-aircraft cluster reaching the target. Among them, the effect of the sub-aircraft cluster reaching the target includes the area, speed, angle, etc. of the sub-aircraft cluster reaching the target.

[0023] The Markov decision process model is the main body of the reinforcement learning algorithm. The five elements of the Markov decision process model that define the timing of the mother-and-child aircraft release are:

[0024] ① State set. The trajectory parameters of the mother aircraft predicted by the analytical solution / quasi-analytical solution, the flight capability data set of the sub-aircraft cluster in different initial states, and the expected effect of the sub-aircraft cluster reaching the target are used as the state set in the Markov decision process model.

[0025] ②Action set: The timing of dispersing the sub-aircraft cluster is used as the component element of the action set in the Markov decision process model.

[0026] ③ State transition probability. The effect of the sub-aircraft cluster reaching the target is used as the basis for action selection in the Markov decision process model. Divide the various intervals of the trajectory parameters of the mother aircraft itself predicted by the analytical solution / quasi-analytical solution, and the various intervals of the flight capability data set of the sub-aircraft cluster in different initial states, and determine the probability of each interval. Determine the probability of the sub-aircraft cluster reaching the target based on each interval combination.

[0027] ④ Reward function: The function of the difference between the target effect at the next moment after the mother aircraft performs an action in the current state and the expected target effect, which is used as the reward function in the Markov decision process model.

[0028] ⑤ Action value function with discount factor. Consider the action learning within a certain period of time from the current moment, and accumulate the reward value at each moment in the period with a certain discount, that is, the discount factor is used to reflect that the farther the reward is from the current moment, the less feedback contribution it makes to the current moment. The accumulated sum is used as the action value function with discount factor in the Markov decision process model, which is the Bellman equation.

[0029] In one embodiment of the present invention, the online rapid calculation of the scattering timing based on the reinforcement learning algorithm of fusion neural network optimization is specifically as follows:

[0030] Based on the established Markov decision process model of the mother-and-child aircraft, starting from an arbitrary strategy, the action value function under the strategy is calculated, and then the strategy is adjusted and improved according to the action value function to make it better, and this process is continuously iterated until the strategy converges. In order to quickly solve the timing of the release online, the characteristics of the neural network, such as fast learning speed, good convergence characteristics, and strong generalization ability, are used to accelerate the generation of the optimal action strategy based on the neural network, thereby realizing the online rapid solution of the timing of the release of the mother-and-child aircraft.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) The existing method of setting the release timing before launch does not have the flexibility to face typical targets. The online solution method for the release timing of the mother-and-child aircraft based on reinforcement learning established in the present invention can quickly solve it online according to the flight environment, and has high flexibility.

[0033] (2) The existing method of setting the scattering timing before launch cannot solve the problem that the mother aircraft deviates from the expected trajectory due to the influence of guidance accuracy, control accuracy, external interference, etc. during flight, thereby causing the actual scattering position of the child aircraft cluster to deviate from the expected scattering position. The online calculation method for the scattering timing of the mother and child aircraft based on reinforcement learning established in the present invention can quickly update the calculation timing of the scattering online according to the actual position of the mother aircraft, and correct the adverse effects caused by the deviation of the flight trajectory of the mother aircraft.

[0034] (3) The key elements of solving the timing of dispersing a cluster of sub-aircraft are efficiency and speed. Conventional methods based on mathematical models usually need to be solved by numerical integration methods, which greatly wastes computing resources and increases computing time. They are not suitable for efficient and fast online solutions. The online solution method for the timing of dispersing mother-child aircraft based on reinforcement learning established in the present invention establishes a direct relationship between the effect of the sub-aircraft cluster arriving at the target and the mother aircraft's own trajectory prediction and the flight capability of the sub-aircraft cluster through long-term interaction between the intelligent agent and the environment, thereby avoiding the integral solution link of the mathematical model and having the advantage of efficient and fast online solution.

[0035] (4) The existing method of setting the scattering timing before launch is not capable of coping with various flight missions, and it is also difficult to adjust and update the scattering timing according to the flight environment. The online calculation method for the scattering timing of the mother-and-child aircraft based on reinforcement learning established by the present invention can quickly adjust and update the scattering timing online for various flight missions in various flight environments, and has strong applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0037] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0038] During the flight, the mother aircraft is affected by the guidance accuracy, control accuracy, external interference, etc., and the actual trajectory deviates from the expected trajectory, causing the actual scattering position of the sub-aircraft cluster to deviate from the expected scattering position, thereby seriously affecting the effect of the sub-aircraft reaching the target. In addition, in order to complete the expected flight mission under various flight environments, it is necessary to be able to quickly solve the scattering timing online. In response to the above problems, the present invention proposes an online solution method for the scattering timing of mother-and-child aircraft based on reinforcement learning. First, the mother aircraft's own trajectory is predicted based on analytical solutions / quasi-analytical solutions, and the flight capabilities of the sub-aircraft under different initial states are evaluated based on optimization methods. The flight capability data set of the sub-aircraft under different initial states is established, and then a Markov decision process model of the mother-and-child aircraft is established. Finally, a reinforcement learning algorithm based on fusion neural network optimization is used to achieve online rapid solution of the scattering timing.

[0039] The present invention proposes an online calculation method for the timing of the mother-and-child aircraft release based on reinforcement learning, such as Figure 1 As shown, the specific steps are as follows:

[0040] (1) Predict the trajectory of the mother aircraft based on analytical / quasi-analytical solutions

[0041] Some necessary and reasonable assumptions are made based on the type of mother aircraft and the typical flight environment, such as assuming that the earth is a sphere, assuming that aerodynamic parameters are constant within a certain range and do not change with speed or angle of attack, and assuming that the mathematical relationship between atmospheric density and altitude is an exponential model.

[0042] On this basis, the analytical solution / quasi-analytical solution for the mother aircraft to predict its own trajectory is derived, so as to realize the online rapid prediction of the mother aircraft's own trajectory.

[0043] (2) Evaluate the flight capability of the sub-aircraft under different initial conditions based on the optimization method

[0044] The flight capability of the sub-aircraft under different initial states can be evaluated by solving the reachable domain of the sub-aircraft under constraints such as thermal flow constraints, dynamic pressure constraints, and overload constraints. The reachable domain is solved by generating specific trajectories by changing the objective function multiple times based on the optimization method.

[0045] Take any initial state of the sub-aircraft as the initial value of the optimized state, set the objective function to the maximum range and the minimum range, solve the corresponding trajectory and record the maximum and minimum range values ​​L max With L min After that, set the objective function to L max With L min The middle amount: L i =L min +w i (L max -L min ), for the factor w i A series of values ​​are taken between (0,1) to obtain a corresponding series of optimization trajectories.

[0046] (3) Establishing a flight capability dataset of the sub-aircraft under different initial conditions

[0047] The terminal state of a series of optimized trajectories corresponding to a certain initial state of the sub-aircraft evaluated based on the optimization method affects the effect of the sub-aircraft reaching the target. Therefore, the terminal state of a series of optimized trajectories corresponding to a certain initial state of the sub-aircraft is defined as the flight capability of the sub-aircraft in this initial state. The terminal state of a series of optimized trajectories corresponding to different initial states of the sub-aircraft is calculated offline to establish a flight capability data set of the aircraft in different initial states.

[0048] (4) Establishing a Markov decision process model for the mother-child aircraft

[0049] The initial state of the sub-aircraft cluster is determined by the state of the mother aircraft when it is released, that is, the position, attitude, speed and other states of the sub-aircraft cluster are determined by the position, attitude, speed and other states of the mother aircraft when it is released. The mother aircraft's own trajectory prediction and the flight capabilities of the sub-aircraft in different initial states determine the release timing of the sub-aircraft cluster, and the release timing of the sub-aircraft cluster determines the effect of the sub-aircraft cluster reaching the target. Among them, the effect of the sub-aircraft cluster reaching the target includes the area, speed, angle, etc. of the sub-aircraft cluster reaching the target.

[0050] The Markov decision process model is the main body of the reinforcement learning algorithm. The five elements of the Markov decision process model that define the timing of the mother-and-child aircraft release are:

[0051] ① State set. The trajectory parameters of the mother aircraft predicted by the analytical solution / quasi-analytical solution, the flight capability data set of the sub-aircraft cluster in different initial states, and the expected effect of the sub-aircraft cluster reaching the target are used as the state set in the Markov decision process model.

[0052] ②Action set: The timing of dispersing the sub-aircraft cluster is used as the component element of the action set in the Markov decision process model.

[0053] ③ State transition probability. The effect of the sub-aircraft cluster reaching the target is used as the basis for action selection in the Markov decision process model. Divide the various intervals of the trajectory parameters of the mother aircraft itself predicted by the analytical solution / quasi-analytical solution, and the various intervals of the flight capability data set of the sub-aircraft cluster in different initial states, and determine the probability of each interval. Determine the probability of the sub-aircraft cluster reaching the target based on each interval combination.

[0054] ④ Reward function: The function of the difference between the target effect at the next moment after the mother aircraft performs an action in the current state and the expected target effect, which is used as the reward function in the Markov decision process model.

[0055] ⑤ Action value function with discount factor. Consider the action learning within a certain period of time from the current moment, and accumulate the reward value at each moment in the period with a certain discount, that is, the discount factor is used to reflect that the farther the reward is from the current moment, the less feedback contribution it makes to the current moment. The accumulated sum is used as the action value function with discount factor in the Markov decision process model, which is the Bellman equation.

[0056] (5) Online rapid calculation of the timing of scattering based on the reinforcement learning algorithm that integrates neural network optimization

[0057] Based on the established Markov decision process model of the mother-and-child aircraft, starting from an arbitrary strategy, the action value function under the strategy is calculated, and then the strategy is adjusted and improved according to the action value function to make it better, and this process is continuously iterated until the strategy converges. In order to quickly solve the timing of the release online, the characteristics of the neural network, such as fast learning speed, good convergence characteristics, and strong generalization ability, are used to accelerate the generation of the optimal action strategy based on the neural network, thereby realizing the online rapid solution of the timing of the release of the mother-and-child aircraft.

[0058] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

[0059] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for calculating the timing of mother-and-child aircraft release based on reinforcement learning, characterized in that: include: Predict the trajectory of the mother aircraft based on analytical / quasi-analytical solutions; Evaluate the flight capability of the sub-aircraft under different initial states based on the optimization method; Establish a flight capability data set of the sub-aircraft under different initial states; Establish a Markov decision process model for the mother-and-child aircraft; The timing of scattering is quickly calculated online based on a reinforcement learning algorithm that integrates neural network optimization.

2. The method for calculating the timing of the mother-and-child aircraft release according to claim 1, characterized in that: Predicting the trajectory of the mother aircraft based on analytical / quasi-analytical solutions includes: Make necessary assumptions based on the type of mother aircraft and typical flight environment, and then derive the analytical solution / quasi-analytical solution for the mother aircraft to predict its own trajectory, thereby achieving online rapid prediction of the mother aircraft's own trajectory.

3. The method for calculating the timing of the mother-and-child aircraft scattering according to claim 2, characterized in that: Necessary assumptions include assuming that the earth is a sphere, assuming that aerodynamic parameters are constant within a preset range and do not change with speed or angle of attack, and assuming that the mathematical relationship between atmospheric density and altitude is an exponential model.

4. The method for online calculation of the timing of dispersing the mother-and-child aircraft according to claim 1, characterized in that: The flight capabilities of the sub-aircraft under different initial states are evaluated based on the optimization method, including: The flight capability of the sub-aircraft under different initial states is evaluated by solving the reachable domain of the sub-aircraft under thermal flow constraints, dynamic pressure constraints, and overload constraints; based on the optimization method, the reachable domain is solved by generating a specific trajectory by changing the objective function multiple times. Any initial state of the sub-aircraft is used as the initial value of the optimized state, and the objective function is set to the maximum range and the minimum range. The corresponding trajectory is solved and the maximum and minimum range values ​​L are recorded. max With L min After that, set the objective function to L max With L min The middle amount: L i =L min +w i (L max -L min ), for the factor w i A series of values ​​are taken between (0,1) to obtain a corresponding series of optimization trajectories.

5. The method for calculating the timing of the mother-and-child aircraft release on-line according to claim 4, characterized in that: The flight capability data set of the sub-aircraft under different initial states includes: The terminal state of a series of optimized trajectories corresponding to a certain initial state of the sub-aircraft is defined as the flight capability of the sub-aircraft in this initial state. The flight capability dataset of the aircraft in different initial states is established by offline calculating the terminal state of a series of optimized trajectories corresponding to different initial states of the sub-aircraft.

6. The method for online calculation of the timing of dispersing the mother-and-child aircraft according to claim 1, characterized in that: The Markov decision process model for the mother-and-child aircraft includes: The initial state of the sub-aircraft cluster is determined by the state of the mother aircraft when it is released, that is, the position, attitude and speed of the sub-aircraft cluster are determined by the position, attitude and speed of the mother aircraft when it is released; the mother aircraft's own trajectory prediction and the flight capabilities of the sub-aircraft in different initial states determine the release timing of the sub-aircraft cluster, and the release timing of the sub-aircraft cluster determines the effect of the sub-aircraft cluster reaching the target; among which, the effect of the sub-aircraft cluster reaching the target includes the area, speed and angle of the sub-aircraft cluster reaching the target; The Markov decision process model is the main body of the reinforcement learning algorithm. The five elements of the Markov decision process model for solving the timing of the mother-and-child aircraft release are defined as the state set, the action set, the state transition probability, the reward function, and the action value function with a discount factor.

7. The method for online calculation of the timing of dispersing the mother-and-child aircraft according to claim 6, characterized in that: The trajectory parameters of the mother aircraft predicted by the analytical solution / quasi-analytical solution, the flight capability data set of the sub-aircraft cluster in different initial states, and the expected arrival effect of the sub-aircraft cluster at the target are used as the state set in the Markov decision process model; The timing of dispersing the sub-aircraft cluster is used as the component element of the action set in the Markov decision process model; The state transition probability refers to taking the effect of the sub-aircraft cluster reaching the target as the basis for action selection in the Markov decision process model; dividing the various intervals of the trajectory parameters of the mother aircraft itself predicted by the analytical solution / quasi-analytical solution, and the various intervals of the flight capability data set of the sub-aircraft cluster in different initial states, and determining the probability of each interval; determining the probability of the sub-aircraft cluster reaching the target according to each interval combination; The function of the difference between the target effect and the expected target effect when the mother aircraft reaches the next state after executing the action in the current state is used as the reward function in the Markov decision process model; Considering the action learning within a certain period of time from the current moment, the reward value returned at each moment in the period is accumulated with a certain discount, that is, the discount factor is used to reflect that the farther the reward is from the current moment, the less feedback contribution it makes to the current moment. The accumulated sum is used as the action value function with a discount factor in the Markov decision process model.

8. The method for online calculation of the timing of dispersing the mother-and-child aircraft according to claim 1, characterized in that: The reinforcement learning algorithm based on fusion neural network optimization is used to quickly calculate the timing of scattering online, including: Based on the established Markov decision process model of the mother-and-child aircraft, starting from an arbitrary strategy, the action-value function under the strategy is calculated, and then the strategy is adjusted and improved according to the action-value function to make it better, and this process is continuously iterated until the strategy converges.

9. The method for online calculation of the timing of dispersing the mother-and-child aircraft according to claim 8, characterized in that: The generation of the optimal action strategy is accelerated based on the neural network, so as to realize the online rapid solution of the timing of the mother-and-child aircraft to be released.

10. A computer program product stored on a non-transitory computer readable medium, the computer program product comprising a program code for carrying out the method according to any one of claims 1 to 9.

Images