Formation flight control method and system based on cost sensitive element reinforcement learning

Abstract

The invention relates to the technical field of formation flight control, and provides a formation flight control method and system based on cost sensitive element reinforcement learning. The method mainly comprises the following steps: establishing a formation flight three-dimensional kinematics model; constructing a set target of the formation flight control method; constructing a formation flight control model according to the formation flight three-dimensional kinematics model and a set target of the formation flight control method; and training the formation flight control model according to cost sensitive element reinforcement learning to obtain final element strategy parameters. According to the method, a hierarchical optimization strategy is adopted in element reinforcement learning, so that rapid adaptation and security guarantee in a complex formation control dynamic environment are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of formation flight control, and provides a formation flight control method and system based on cost-sensitive meta-reinforcement learning. Background Art

[0002] In today's aerospace field, multi-aircraft formation flight has become a key technology that has attracted much attention and is widely used in many scenarios such as military reconnaissance, disaster relief, and geographic mapping. However, traditional formation flight control methods are increasingly unable to meet the complex and changing needs of reality. On the one hand, the flight environment is becoming more and more complex and harsh, and uncertain factors such as airflow turbulence, electromagnetic interference, and sudden obstacles frequently appear; on the other hand, mission requirements are becoming increasingly diverse and sophisticated, requiring the formation to be able to change formations quickly, flexibly, and accurately. Traditional control methods are stretched to the limit, and traditional methods based on fixed models are difficult to cope with such complex situations.

[0003] With the development of artificial intelligence technology, meta-reinforcement learning can be used to accurately solve control solutions that meet multiple constraints such as aircraft performance, obstacle avoidance, and communication to ensure the stability of the formation. In the face of the need to change formations, reinforcement learning can quickly iterate strategies, quickly adjust and coordinate responses, and reduce dependence on communication links. It has excellent scalability and provides an innovative solution to the problem of multi-aircraft formation flight. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention provides a formation flight control method and system based on cost-sensitive meta-reinforcement learning, which optimizes the energy consumption of control surface operation while ensuring the safety of formation flight, and realizes the high efficiency and economy of formation control.

[0005] The present invention provides a formation flight control method based on cost-sensitive meta-reinforcement learning, comprising: S1: Establish a three-dimensional kinematic model of formation flying; S2: Setting goals for constructing formation flight control methods; S3: constructing a formation flight control model according to the formation flight three-dimensional kinematic model and the set target of the formation flight control method; S4: Train the formation flight control model based on cost-sensitive meta-reinforcement learning to obtain the final meta-strategy parameters; S5: Controlling formation flight according to the final meta-strategy parameters.

[0006] According to a formation flight control method based on cost-sensitive meta-reinforcement learning provided by the present invention, step S1 is specifically as follows: S11: Definition In the aircraft formation system, there is a leader aircraft and There are a number of follower aircraft, and the state vectors of each follower aircraft in formation flying are designed as follows: in, For the The state vector of the following vehicle, To follow the aircraft sequence, , is the total number of aircraft, For the The horizontal coordinate of the following aircraft in the inertial coordinate system, For the The ordinate of the following aircraft in the inertial coordinate system, No. The vertical coordinate of the following aircraft in the inertial coordinate system, Indicates The speed of the following aircraft, Indicates The flight inclination angle of the following aircraft, Indicates The flight deflection angle of the following aircraft; S12: Design the control vectors of each follower aircraft in formation flying: in, For the A control vector for the following vehicle, For the Follow the aircraft in Overload in the axial direction, For the Follow the aircraft in Overload in the axial direction, For the Follow the aircraft in Overload in the axial direction; Transpose the matrix; S13: Constructing the formation flight three-dimensional kinematic model according to the state vector and the control vector : in, is the acceleration due to gravity, is the linearization matrix for the flight formation control, A basic model for aircraft formation control.

[0007] According to a formation flight control method based on cost-sensitive meta-reinforcement learning provided by the present invention, step S2 is specifically as follows: For any formation, the initial conditions are: in, is the position vector of the leader vehicle, For the The position vector of the following vehicle, It is the leading aircraft and the The desired distance between the following aircraft, is the velocity vector of the leading vehicle, For the A velocity vector of the following vehicle, is the transformation matrix from the ballistic coordinate system of the leading aircraft to the inertial reference coordinate system, is the position vector allowable error, is the allowable error of the velocity vector.

[0008] According to a formation flight control method based on cost-sensitive meta-reinforcement learning provided by the present invention, step S3 is specifically as follows: S31: Define the adjacency matrix of the leader aircraft: in, For the The communication status between the follower aircraft and the leader aircraft, if it is in the communication state If the communication status is not available , is a diagonal matrix; S32: Design Follow the control commands of the aircraft : in, To define a symbolic function, is the first argument of the symbolic function, To define the second argument of the symbolic function, is the symbolic function, To take the absolute value, is the ordinal number of the adjacent following aircraft of the following aircraft, For the The set of adjacent following aircraft of a following aircraft, It is The expected position of the follower vehicle in the leader's trajectory coordinate system, Indicates The position vector of the following vehicle, is the first feedback gain of the formation flight controller, is the second feedback gain of the formation flight controller, is the first-order gain of the formation flight controller, is the second-order gain of the formation flight controller, is the formation flight controller damping gain; It is a modulo operation; for The first derivative of for The first derivative of for The first derivative of for The first derivative of for The second derivative of S33: Design The control vector of the following vehicle : in, is the first feedback matrix of the formation flight controller state, is the second feedback matrix of the formation flight controller state; According to the control vector calculate and .

[0009] According to a formation flight control method based on cost-sensitive meta-reinforcement learning provided by the present invention, step S4 includes: S41: Randomly generate meta-strategy parameters ; S42: Setting local policy parameters Maximum number of iterations for local update The local strategy parameters are updated by the local update. ; S43: Update the meta-strategy parameters to obtain updated meta-strategy parameters ; S44: Determine whether the update times of the meta-strategy parameter reaches the iteration times threshold If not, return to step S42; if reached, execute step S45; S45: Output the final meta-strategy parameters .

[0010] According to a formation flight control method based on cost-sensitive meta-reinforcement learning provided by the present invention, the local update in step S42 includes the following steps: S421: For each time step , select and record the action space , state space , the state space of the next time step , the reward function and constraint cost : in, is the position tracking error of the ith following vehicle, is the reward function error threshold, is the reward threshold of the reward function, is the penalty threshold of the reward function, is the cost function threshold, is the horizontal coordinate of the leading aircraft in the inertial coordinate system, is the ordinate of the leading aircraft in the inertial coordinate system, is the vertical coordinate of the leader vehicle in the inertial coordinate system; S422: Based on the action space , state space , the state space of the next time step , the reward function , constraint cost Building a training dataset : S423: Calculate the gradient of the reward advantage function and the cost advantage function gradient ; in, For the The sub-local policy function, From the state space Status starts following policy The expected cumulative rewards obtained; From the state space , action space Status starts following policy The expected cumulative reward obtained is To represent in state space Take action space The associated cost function, is the state space The cost value function associated with the security constraint, for The gradient and The gradient parameter for The partial derivative when ; for The gradient and The gradient parameter for The partial derivative when ; S424: Update The second local strategy function is The local policy function of the iteration in, As constraints, Indicates taking the maximum value, is the Wasserstein distance between the new and old strategies, is the residual, for The decision cost, is the safety threshold.

[0011] According to a formation flight control method based on cost-sensitive meta-reinforcement learning provided by the present invention, step S43 includes: S431: Calculate the objective function gradient of the meta-strategy parameters and the constraint function gradient : in, is the total number of tasks, For the The expected value of the reward function after local update of the task, It is The expected value of the cost function of each task after local update; S432: Update the meta-policy parameters to obtain updated meta-policy parameters : in, is the residual of the meta-update cost constraint, is the Hessian matrix, is the size of the allowed meta-update trust domain, To obtain the Hessian matrix. (Here argmax means to maximize this function) The present invention also provides a formation flight control system based on cost-sensitive meta-reinforcement learning, comprising: a preprocessing module: establishing a three-dimensional kinematic model of formation flight, constructing a set target of a formation flight control method, and constructing a formation flight control model according to the three-dimensional kinematic model of formation flight and the set target of the formation flight control method; Meta-strategy parameter training module: Train the formation flight control model based on cost-sensitive meta-reinforcement learning to obtain the final meta-strategy parameters.

[0012] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: The present invention provides a formation flight control method and system based on cost-sensitive meta-reinforcement learning. By combining cost-sensitive meta-reinforcement learning with multi-aircraft formation control, the aircraft can perceive in real time and autonomously learn to cope with complex environmental interference and accumulate flight experience; through meta-reinforcement learning iterative update and combined with Wasserstein distance constraints, the constraint control of the safety distance can be met while accurately solving the reasonable formation control scheme of the aircraft; in addition, by adding BLO (Bi-Level Optimization) to the meta-reinforcement learning algorithm for end-to-end gradient calculation, the meta-strategy parameters can be optimized, which can further improve the generalization ability of the present invention across tasks. It provides an innovative solution to the problem of multi-aircraft formation flight.

[0013] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 It is a flow chart of a formation flight control method based on cost-sensitive meta-reinforcement learning provided by the present invention; Figure 2 is a schematic diagram of position tracking error according to an embodiment of the present invention; Figure 3 is a schematic diagram of velocity error of an aircraft according to an embodiment of the present invention; Figure 4 is a schematic diagram of the trajectory inclination error of an aircraft according to an embodiment of the present invention; Figure 5 is a schematic diagram of the trajectory deviation error of an aircraft according to an embodiment of the present invention; Figure 6 is a schematic diagram of the flight trajectory of an aircraft according to an embodiment of the present invention; Figure 7 It is a structural block diagram of a formation flight control system based on cost-sensitive meta-reinforcement learning provided by the present invention.

[0016] Reference numerals: 101. Preprocessing module; 102. Meta-strategy parameter training module; 103. Formation flying module. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme in the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0018] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0019] Combine the following Figures 1 to 7 The present invention is described.

[0020] Example like Figure 1 As shown, an embodiment of the present invention provides a formation flight control method based on cost-sensitive meta-reinforcement learning, comprising the following steps: S1: Establish a three-dimensional kinematic model of formation flying; S2: Setting goals for constructing formation flight control methods; S3: constructing a formation flight control model according to the formation flight three-dimensional kinematic model and the set target of the formation flight control method; S4: Train the formation flight control model based on cost-sensitive meta-reinforcement learning to obtain the final meta-strategy parameters; S5: Controlling formation flight according to the final meta-strategy parameters.

[0021] Specifically, step S1 includes: S11: Design the state vector of each aircraft in formation flying: in, For the The state vector of the aircraft, is the aircraft ordinal number, , is the total number of aircraft, For the The horizontal coordinate of the aircraft in the inertial coordinate system, For the The vertical coordinate of the aircraft in the inertial coordinate system, No. The vertical coordinate of the aircraft in the inertial coordinate system, Indicates The speed of the aircraft, Indicates The flight inclination angle of the aircraft, Indicates The flight angle of an aircraft.

[0022] S12: Design the control vectors of each follower aircraft in formation flying: in, For the A control vector for the following vehicle, For the Follow the aircraft in Overload in the axial direction, For the Follow the aircraft in Overload in the axial direction, For the Follow the aircraft in Overload in the axial direction; Transpose the matrix; S13: Constructing the formation flight three-dimensional kinematic model according to the state vector and the control vector : in, is the acceleration due to gravity, is the linearization matrix for the flight formation control, A basic model for aircraft formation control.

[0023] Specifically, step S2 includes: Considering a formation system containing N aircraft, we can define a leader aircraft and N−1 follower aircraft. The leader aircraft usually provides a reference state, while the follower aircraft adjusts according to the state of the leader aircraft to maintain a reasonable formation structure.

[0024] For any formation, the initial conditions are: in, is the position vector of the leader vehicle, For the The position vector of the following vehicle, It is the leading aircraft and the The desired distance between the following aircraft, is the velocity vector of the leading vehicle, For the A velocity vector of the following vehicle, is the transformation matrix from the ballistic coordinate system of the leading aircraft to the inertial reference coordinate system, is the position vector allowable error, is the allowable error of the velocity vector.

[0025] Specifically, step S3 includes: It is known that the formation control system contains a leading aircraft and N-1 following aircraft. In the inertial coordinate system, since each channel is independent of each other, the control laws in the x, y, and z directions need to be designed separately. The control instructions are designed for the i-th following aircraft based on the position and velocity information. S31: Define the adjacency matrix of the leader aircraft: in, For the The communication status between the follower aircraft and the leader aircraft, if it is in the communication state If the communication status is not available , is a diagonal matrix; S32: Design Follow the control commands of the aircraft : in, To define a symbolic function, is the first argument of the symbolic function, To define the second argument of the symbolic function, is the symbolic function, To take the absolute value, is the ordinal number of the adjacent following aircraft of the following aircraft, For the The set of adjacent following aircraft of a following aircraft, It is The expected position of the follower vehicle in the leader's trajectory coordinate system, Indicates The position vector of the following vehicle, is the first feedback gain of the formation flight controller, is the second feedback gain of the formation flight controller, is the first-order gain of the formation flight controller, is the second-order gain of the formation flight controller, is the formation flight controller damping gain; the damping gain can help absorb oscillations in the system and improve the stability of the system. In aircraft control, this can reduce unstable behaviors caused by rapid maneuvers or environmental changes; It is a modulo operation; for The first derivative of for The first derivative of for The first derivative of for The first derivative of for The second-order derivative of ; all function derivatives are derivatives with respect to time; S33: Design The control vector of the following vehicle : in, is the first feedback matrix of the formation flight controller state, is the second feedback matrix of the formation flight controller state, According to the control vector calculate and .

[0026] In reinforcement learning, the PPO (Proximal Policy Optimization) algorithm, represented by the online policy algorithm, requires a large number of samples for learning, and has the problem of low sample efficiency. The DDPG algorithm (deep deterministic policy gradient) is an offline policy algorithm. Although it improves the utilization rate of samples, DDPG is a deterministic strategy, that is, it only considers the best action in each state. It has the problems of insufficient exploration ability, unstable training, poor convergence, sensitivity to hyperparameters, and difficulty in adapting to different complex environments. Compared with ordinary reinforcement learning, the CSMRL (Cost-sensitive Meta-reinforcement learning) algorithm of the embodiment of the present invention shows significant advantages in many aspects: it realizes the ability to quickly adapt to new tasks through the meta-learning framework, ensuring efficient learning in a dynamically changing environment; at the same time, the constraint policy optimization mechanism integrated in the algorithm naturally considers the security constraints in the policy optimization process, ensuring that the learning strategy will not violate the preset security restrictions while pursuing performance improvement, and overcomes the end-to-end differentiability problem in traditional cost-sensitive reinforcement learning, making the entire learning process smoother and more efficient.

[0027] Specifically, step S4 includes the following steps: S41: Randomly generate meta-strategy parameters ; S42: Setting local policy parameters Maximum number of iterations for local update The local strategy parameters are updated by the local update. ; S43: Update the meta-strategy parameters to obtain updated meta-strategy parameters ; S44: Determine whether the update times of the meta-strategy parameter reaches the iteration times threshold If not, return to step S42; if reached, execute step S45; S45: Output the final meta-strategy parameters .

[0028] The local update in step S42 includes the following steps: S421: For each time step , select and record the action space , state space , the state space of the next time step , the reward function and constraint cost : in, is the position tracking error of the ith following vehicle, is the reward function error threshold, is the reward threshold of the reward function, is the penalty threshold of the reward function, is the cost function threshold, is the horizontal coordinate of the leading aircraft in the inertial coordinate system, is the ordinate of the leading aircraft in the inertial coordinate system, is the vertical coordinate of the leader vehicle in the inertial coordinate system; S422: Based on the action space , state space , the state space of the next time step , the reward function , constraint cost Building a training dataset : S423: Calculate the gradient of the reward advantage function and the cost advantage function gradient ; in, For the The sub-local policy function, From the state space Status starts following policy The expected cumulative rewards obtained; From the state space , action space Status starts following policy The expected cumulative reward obtained is To represent in state space Take action space The associated cost function, is the state space The cost value function associated with the security constraint, for The gradient and The gradient parameter for The partial derivative when ; for The gradient and The gradient parameter for The partial derivative when ; S424: Update The second local strategy function is The local policy function of the iteration , in, As constraints, is the matrix transpose, Indicates taking the maximum value, The Wasserstein distance between the new and old strategies is used to measure the step size of the strategy update to ensure that the update does not deviate too far from the current strategy. is the maximum allowed Wasserstein distance that defines the size of the trust region. is the residual, for The decision cost (in Decision strategy , the expected value of the cost function of the task), is the safety threshold.

[0029] The local update mainly applies the BLO (Bi-Level Optimization) method. BLO is used to calculate gradients, which allows the algorithm to optimize the meta-strategy parameters through end-to-end gradient calculations and local update results of all tasks, thereby achieving two-level optimization. Wasserstein distance is used as a metric in local updates to avoid instability that may be caused by overly aggressive updates. In local updates, cost-sensitive reinforcement learning methods optimize the objective function to maximize rewards while ensuring the safety and feasibility of the strategy through constraints.

[0030] Step S43 includes the following steps: S431: Calculate the objective function gradient of the meta-strategy parameters and the constraint function gradient : in, is the total number of tasks, For the The expected value of the reward function after local update of the task, It is The expected value of the cost function of each task after local update; S432: Update the meta-policy parameters to obtain updated meta-policy parameters : in, is the residual of the meta-update cost constraint, is the Hessian matrix, which is or about The second-order partial derivative matrix of is the size of the allowed meta-update trust domain, To obtain the Hessian matrix.

[0031] Within the meta-update, the BLO method is mainly applied. BLO is used to calculate the gradient of the meta-policy parameters, which allows the algorithm to optimize the meta-policy parameters through end-to-end gradient calculation and local update results of all tasks, thereby achieving two-level optimization. In the meta-update, adjustments are made based on the local update results of all tasks to improve the generalization ability across tasks. When evaluating the performance and cost on different tasks, the meta-update provides gradient information to BLO to help optimize the meta-policy.

[0032] S5: Based on the final meta-strategy parameters Controlling formation flying. Final meta-strategy parameters Including formation flight tasks, including speed, flight inclination and flight deviation control values. Specifically, the final meta-strategy parameters trained by the embodiment of the present invention are The effects on the flight formation are as follows: First, in the simulation environment, the local strategy parameters for each test task are initialized to the final meta-strategy parameters that have been obtained. , and then fine-tune the strategy to suit specific formation flying requirements. Perform formation flying missions in a simulated environment, control their speed, flight inclination and flight angle, while monitoring the coordination and formation accuracy of the aircraft to ensure that all aircraft comply with safety constraints. Evaluate the strategy performance based on the test results and further adjust the strategy if necessary.

[0033] In the embodiment of the present invention, as shown in Table 1 and Table 2, Table 1 is the Monte Carlo simulation environment parameters, and Table 2 is the setting of network parameters in the intelligent network CSMRL algorithm: Table 1: Monte Carlo simulation environment parameters in, To lead the aircraft speed, It is the first parameter that leads the aircraft speed. The second parameter of the leading aircraft speed, The third parameter of the leading aircraft speed, The first parameter of the leading aircraft angle, The second parameter of the leading aircraft angle, The third parameter of the leading aircraft angle, The fourth parameter of the leading aircraft angle, is the desired distance between the leader and the first follower, is the expected distance ordinate between the leader and the first follower, is the vertical coordinate of the expected distance between the leader and the first follower; is the desired distance between the leader and the second follower, is the desired distance ordinate between the leader and the second follower, is the vertical coordinate of the desired distance between the leader vehicle and the second follower vehicle.

[0034] Table 2 CSMRL network parameter settings

[0035] like Figure 2 As shown, Figure 2 is a schematic diagram of position tracking error according to an embodiment of the present invention. Figure 2 It can be seen that after training, the tracking error is less than 1.5 meters, and the error decreases very quickly, taking about 1500 seconds.

[0036] Figure 3 is a schematic diagram of the speed error of the aircraft according to an embodiment of the present invention. Figure 3 It can be seen that after about 1500 seconds, the speed change pattern and period of the two follower aircraft are similar to those of the leading aircraft.

[0037] Figure 4 is a schematic diagram of the trajectory inclination error of an aircraft according to an embodiment of the present invention. Figure 4 It can be seen that after about 1500 seconds, the inclination variation patterns and periods of the two follower aircraft are similar to those of the leading aircraft.

[0038] Figure 5 is a schematic diagram of the trajectory deviation error of an aircraft according to an embodiment of the present invention. Figure 5It can be seen that after about 1500 seconds, the variation pattern and period of the elevation angles of the two follower aircraft are similar to those of the leading aircraft.

[0039] Figure 6 is a schematic diagram of the flight trajectory of an aircraft according to an embodiment of the present invention; Figure 6 It can be seen that the flight trajectories of the two follower aircraft quickly become consistent with the leading aircraft after flying for a period of time.

[0040] like Figure 7 As shown, Figure 7 The structure diagram of a formation flight control system based on cost-sensitive meta-reinforcement learning provided by the present invention is used to execute a formation flight control method based on cost-sensitive meta-reinforcement learning, including: Preprocessing module 101: establishing a three-dimensional kinematic model of formation flight, constructing a set target of a formation flight control method, and constructing a formation flight control model according to the three-dimensional kinematic model of formation flight and the set target of the formation flight control method; Meta-strategy parameter training module 102: training the formation flight control model according to cost-sensitive meta-reinforcement learning to obtain final meta-strategy parameters; Formation flying module 103: controls formation flying according to the final meta-strategy parameters.

[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0042] It should be noted that the embodiments of the present disclosure may be implemented by hardware, software, or a combination of software and hardware. The hardware portion may be implemented using dedicated logic: the software portion may be stored in a memory and executed by an appropriate instruction execution system such as a microprocessor or dedicated design hardware. Those skilled in the art will appreciate that the above-described apparatus and methods may be implemented using computer executable instructions and / or contained in a processor control code, such as a programmable memory or a data carrier such as an optical or electronic signal carrier providing such code.

[0043] In addition, although the operation of the method of the present disclosure is described in a particular order in the accompanying drawings, this does not require or imply that these operations must be performed in this particular order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flow chart can change the order of execution. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution. It should also be noted that the features and functions of two or more devices according to the present disclosure can be embodied in one device. Conversely, the features and functions of a device described above can be further divided into being embodied by multiple devices.

Claims

1. A formation flight control method based on cost-sensitive meta-reinforcement learning, characterized in that: include: S1: Establish a three-dimensional kinematic model of formation flying; S2: Setting goals for constructing formation flight control methods; S3: constructing a formation flight control model according to the formation flight three-dimensional kinematic model and the set target of the formation flight control method; S4: Train the formation flight control model based on cost-sensitive meta-reinforcement learning to obtain the final meta-strategy parameters; S5: Controlling formation flight according to the final meta-strategy parameters.

2. A formation flight control method based on cost-sensitive meta-reinforcement learning according to claim 1, characterized in that: Step S1 is specifically as follows: S11: Definition In a formation system of aircraft, there is a leading aircraft and There are a number of follower aircraft, and the state vectors of each follower aircraft in formation flying are designed as follows: in, For the The state vector of the following vehicle, To follow the aircraft sequence, , is the total number of aircraft, For the The horizontal coordinate of the following aircraft in the inertial coordinate system, For the The ordinate of the following aircraft in the inertial coordinate system, No. The vertical coordinate of the following aircraft in the inertial coordinate system, Indicates The speed of the following aircraft, Indicates The flight inclination angle of the following aircraft, Indicates The flight deflection angle of the following aircraft; S12: Design the control vectors of each follower aircraft in formation flying: in, For the A control vector for the following vehicle, For the Follow the aircraft in Overload in the axial direction, For the Follow the aircraft in Overload in the axial direction, For the Follow the aircraft in Overload in the axial direction; Transpose the matrix; S13: Constructing the formation flight three-dimensional kinematic model according to the state vector and the control vector : in, is the acceleration due to gravity, is the linearization matrix for the flight formation control, A basic model for aircraft formation control.

3. A formation flight control method based on cost-sensitive meta-reinforcement learning according to claim 2, characterized in that: Step S2 is specifically as follows: For any formation, the initial conditions are: in, is the position vector of the leader vehicle, For the The position vector of the following vehicle, It is the leading aircraft and the The desired distance between the following aircraft, is the velocity vector of the leading vehicle, For the A velocity vector of the following vehicle, is the transformation matrix from the ballistic coordinate system of the leading aircraft to the inertial reference coordinate system, is the position vector allowable error, is the allowable error of the velocity vector.

4. A formation flight control method based on cost-sensitive meta-reinforcement learning according to claim 3, characterized in that: Step S3 is specifically as follows: S31: Define the adjacency matrix of the leader aircraft: in, For the The communication status between the follower aircraft and the leader aircraft, if it is in a communication state, If it is in a non-communication state, , is a diagonal matrix; S32: Design Follow the control commands of the aircraft : in, To define a symbolic function, is the first argument of the symbolic function, To define the second argument of the symbolic function, is the symbolic function, To take the absolute value, is the ordinal number of the adjacent following aircraft of the following aircraft, For the The set of adjacent following aircraft of a following aircraft, It is The expected position of the follower vehicle in the leader's trajectory coordinate system, Indicates The position vector of the following vehicle, is the first feedback gain of the formation flight controller, is the second feedback gain of the formation flight controller, is the first-order gain of the formation flight controller, is the second-order gain of the formation flight controller, is the formation flight controller damping gain; It is a modulo operation; for The first derivative of for The first derivative of for The first derivative of for The first derivative of for The second derivative of S33: Design The control vector of the following vehicle : in, is the first feedback matrix of the formation flight controller state, is the second feedback matrix of the formation flight controller state; According to the control vector calculate and .

5. A formation flight control method based on cost-sensitive meta-reinforcement learning according to claim 4, characterized in that: Step S4 includes: S41: Randomly generate meta-strategy parameters ; S42: Setting local policy parameters Maximum number of iterations for local update The local strategy parameters are updated by the local update. ; S43: Update the meta-strategy parameters to obtain updated meta-strategy parameters ; S44: Determine whether the update times of the meta-strategy parameter reaches the iteration times threshold If not, return to step S42; if reached, execute step S45; S45: Output the final meta-strategy parameters .

6. A formation flight control method based on cost-sensitive meta-reinforcement learning according to claim 5, characterized in that: The local update in step S42 includes the following steps: S421: For each time step , select and record the action space , state space , the state space of the next time step , the reward function and constraint cost : in, is the position tracking error of the ith following vehicle, is the reward function error threshold, is the reward threshold of the reward function, is the penalty threshold of the reward function, is the cost function threshold, is the horizontal coordinate of the leading aircraft in the inertial coordinate system, is the ordinate of the leading aircraft in the inertial coordinate system, is the vertical coordinate of the leader vehicle in the inertial coordinate system; S422: Based on the action space , state space , the state space of the next time step , the reward function , constraint cost Building a training dataset : S423: Calculate the gradient of the reward advantage function and the cost advantage function gradient ; in, For the The sub-local policy function, From the state space Status starts following policy The expected cumulative rewards obtained; From the state space , Action Space Status starts following policy The expected cumulative reward obtained is To represent in state space Take action space The associated cost function, is the state space The cost value function associated with the security constraint, for The gradient and The gradient parameter for The partial derivative when ; for The gradient and The gradient parameter for The partial derivative when ; S424: Update The local strategy function is The local policy function of the iteration , in, As constraints, Indicates taking the maximum value, is the Wasserstein distance between the new and old strategies, is the maximum allowed Wasserstein distance, is the residual, for The decision cost, is the safety threshold.

7. The formation flight control method based on cost-sensitive meta-reinforcement learning according to claim 5, characterized in that: Step S43 includes: S431: Calculate the objective function gradient of the meta-strategy parameters and the constraint function gradient : in, is the total number of tasks, For the The expected value of the reward function after local update of the task, It is The expected value of the cost function of each task after local update; S432: Update the meta-policy parameters to obtain updated meta-policy parameters : in, is the residual of the meta-update cost constraint, is the Hessian matrix, is the size of the allowed meta-update trust domain, To obtain the Hessian matrix.

8. A formation flight control system based on cost-sensitive meta-reinforcement learning, used to execute a formation flight control method based on cost-sensitive meta-reinforcement learning as claimed in any one of claims 1 to 7, characterized in that: include: Preprocessing module: establishing a three-dimensional kinematic model of formation flight, constructing a set target of a formation flight control method, and constructing a formation flight control model according to the three-dimensional kinematic model of formation flight and the set target of the formation flight control method; Meta-strategy parameter training module: trains the formation flight control model based on cost-sensitive meta-reinforcement learning to obtain the final meta-strategy parameters; Formation flying module: controls the formation flying according to the final meta-strategy parameters.

Images