Fixed-wing unmanned aerial vehicle formation control method and device based on deep reinforcement learning

Abstract

The invention discloses a fixed-wing unmanned aerial vehicle formation control method and device based on deep reinforcement learning, and aims to solve the problem of fixed-wing unmanned aerial vehicle cluster formation configuration under complex flight conditions. The method comprises the following steps: establishing a kinematic model of a fixed-wing unmanned aerial vehicle, and constructing an unmanned aerial vehicle training environment; designing an Actor network and a Critic network of a reinforcement learning algorithm; constructing a joint state space and an action space of the unmanned aerial vehicle according to the characteristics of the unmanned aerial vehicle; designing a reward function of the unmanned aerial vehicle formation according to the flight mission target; designing a training process of an algorithm according to functions of the Actor network and the Critic network; the invention provides a novel unmanned aerial vehicle formation control strategy, and formation control of an unmanned aerial vehicle cluster under complex and changeable flight conditions can be realized.

Description

Technical Field

[0001] The present invention relates to the field of fixed-wing UAV control, and in particular to a fixed-wing UAV formation control method and device based on deep reinforcement learning. Background Art

[0002] Fixed-wing UAVs have attracted great attention due to their high flight speed and long flight distance. In the face of increasingly complex environments, a single UAV can no longer meet mission requirements, so multiple UAVs need to cooperate. Therefore, the present invention focuses on the coordinated formation control of fixed-wing UAVs.

[0003] Traditional formation control methods, such as PID control and LQR control, are difficult to cope with the dynamic changes of fixed-wing UAVs in complex environments. Summary of the invention

[0004] The purpose of the present invention is to provide a fixed-wing UAV formation control method based on deep reinforcement learning, so as to solve the problem of difficult formation control of fixed-wing UAVs under complex conditions in continuous space in the prior art and improve the stability and efficiency of the formation.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A fixed-wing UAV formation control method based on deep reinforcement learning comprises the following steps:

[0007] S1: Establish a kinematic model of a fixed-wing UAV and a simulation training environment for the UAV. For the control instructions input to the model, the closed-loop control is completed through the underlying PID controller.

[0008] S2: Construct Actor network and Critic network through deep neural network;

[0009] S3: The following UAV and the pilot UAV collect their own state information, including position, speed, track angle, heading angle and remaining fuel ratio information, to form a joint state space; the following UAV outputs the roll angle according to its control command to construct the action space;

[0010] S4: Design a reinforcement learning reward function based on the characteristics of the drone and the formation task to guide the drone to obtain the maximum reward;

[0011] S5: During the interaction between the drone and the environment, the parameters of the Actor network and the Critic network are continuously updated using historical data, thereby training and forming the final Actor network model; the historical data includes the current state, the action performed, the reward generated by the action, and the next state reached after the action is performed;

[0012] S6: When performing a flight mission, the follower UAV collects the current status information of itself and the pilot UAV in real time, and combines the information into a joint state, and inputs the joint state into the final Actor network model to generate a roll angle control instruction for the follower UAV.

[0013] According to the characteristics of fixed-wing UAVs and formation tasks, the present invention designs a set of reinforcement learning reward functions that meet the fixed-wing UAV formation. The function includes five parts: velocity vector alignment reward function: driving the velocity vector of the formation members to quickly align with the desired geometric configuration point; distance reward function: driving the formation members to quickly approach the desired position. When the formation members are closer to the desired spatial position, a positive reward is applied to the UAV, and vice versa, a negative reward is applied; heading angle reward function: in order to further reduce the formation formation time, the direction angle of the velocity vector of the following UAV is constrained so that the velocity vector of the following UAV is consistent with the velocity vector of the leading UAV, thereby forming a stable formation; process reward function: used to constrain the process state variables of the UAV; energy consumption reward function: the energy consumption reward function is used to optimize the energy consumption in the formation process.

[0014] The present invention takes into account that in actual situations, the fuel carried by fixed-wing UAVs when performing tasks is limited. Therefore, the present invention uses the remaining fuel ratio as part of the state input when constructing the UAV reinforcement learning algorithm. At the same time, a reward and punishment mechanism is also designed for the state of the remaining fuel in the reward function. By training the algorithm in this way, the UAV can achieve the effect of saving fuel and outputting the best control instructions during the formation process.

[0015] The present invention also provides a fixed-wing UAV formation control device based on deep reinforcement learning, comprising:

[0016] Learning and training module: Establish the kinematic model of the fixed-wing UAV, construct the Actor network and the Critic network through the deep neural network, and use the historical data to continuously update the parameters of the Actor network and the Critic network during the interaction between the follower UAV and the environment, so as to train and form the final Actor network model; these historical data include the current state, the actions performed, the rewards generated by the actions, and the next state reached after the actions are performed; during the interaction process, the follower UAV collects the current state information of itself and the pilot UAV to form a joint state; the Actor network outputs the control instructions of the follower UAV according to the joint state, and calculates the state of each UAV at the next moment through the kinematic model of the fixed-wing UAV;

[0017] Mission execution module: When performing a flight mission, the follower UAV collects the current status information of itself and the pilot UAV in real time, combines it into a joint state, and inputs it into the trained Actor network model. The Actor network model generates control instructions for each follower UAV based on the joint state obtained in real time to adjust its roll angle.

[0018] In the fixed-wing UAV formation task, the present invention enables each follower UAV to know the status information of itself and the leading UAV, and the follower UAV then forms and maintains the formation according to the control instructions output by its own Actor network. This method solves the fixed-wing UAV formation problem under the Leader-Follower topology in the continuous state and action space, and improves the intelligence and flexibility of UAV task execution.

[0019] The present invention constructs an efficient, safe and flexible fixed-wing UAV formation control method based on deep reinforcement learning by integrating the perception and control of multi-machine formation collaborative tasks, the combination of UAV models and environmental disturbances, the construction of Actor and Critic networks, and the design of a perfect reward function. Its characteristics and advantages are that it can achieve efficient formation control without relying on accurate models, can adapt to formation tasks of different scales, and significantly improves the portability of control strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the implementation flow of the fixed-wing UAV formation coordination control method based on deep reinforcement learning in this embodiment.

[0021] Figure 2 It is a schematic diagram of the communication topology of the UAV in a specific application embodiment of the present invention.

[0022] Figure 3 It is a structural diagram of the Actor network in a specific application embodiment of the present invention.

[0023] Figure 4 It is a schematic diagram of the structure of the Critic network in a specific application embodiment of the present invention.

[0024] Figure 5 It is the pseudo code of the algorithm used in the present invention. DETAILED DESCRIPTION

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Figure 1 Flowchart of a fixed-wing UAV control method based on deep reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0029] S1: Establish a fixed-wing UAV kinematic model and a UAV simulation training environment. For the control instructions input to the model, complete closed-loop control through the underlying PID controller.

[0030] S2: Construct Actor network and Critic network through deep neural network.

[0031] S3: The follower UAV and the pilot UAV collect their respective state information, including position, speed, track angle, heading angle and remaining fuel ratio information, to form a joint state space; the follower UAV outputs the roll angle according to its control command to construct the action space.

[0032] S4: Design a reinforcement learning reward function based on the characteristics of the drone and the formation task to guide the drone to obtain the maximum reward.

[0033] S5: During the interaction between the drone and the environment, the parameters of the Actor network and the Critic network are continuously updated using historical data to train and form the final Actor network model; the historical data includes the current state, the action performed, the reward generated by the action, and the next state reached after the action is performed.

[0034] S6: When performing a flight mission, the follower drone collects the current status information of itself and the pilot drone in real time, and combines them into a joint state, inputs the joint state into the final Actor network model, and generates control instructions for the follower drone, which are used to adjust the roll angle.

[0035] In this embodiment, a preset formation configuration is given, and each follower drone knows the status information of itself and the leader drone. The follower drone then forms and maintains the formation according to the control instructions output by its own Actor network. This method solves the fixed-wing drone formation problem under the Leader-Follower topology in a continuous state and action space, greatly improving the intelligence and flexibility of drones in executing tasks.

[0036] The communication network topology of the drone group in this embodiment is fixed and connected. It is set manually. The purpose of the connection is to ensure that all drones can exchange information. The communication topology diagram is as follows: Figure 2 ; The target location is the location where the drone group will eventually arrive together, which is set manually.

[0037] This embodiment provides a fixed-wing UAV kinematic model as follows:

[0038]

[0039] Among them, r is the distance between the drone and the center of the earth, is the derivative of r; λ and φ are the longitude and latitude of the location of the UAV, respectively. and are the time derivatives of λ and φ respectively; V is the flight speed, is the time derivative of V; θ is the track angle, is the time derivative of θ; ψ is the heading angle, is the time derivative of ψ; D and L are drag acceleration and lift acceleration respectively, g is the local gravity acceleration, γ v is the roll angle of the drone, and the height of the drone from the ground h = rR 0 , R 0 is the radius of the earth; the expressions of drag acceleration and lift acceleration are:

[0040]

[0041] In the formula, C D and C L are the drag coefficient and lift coefficient, respectively, which are related to the angle of attack α and Mach number Ma of the UAV, ρ is the atmospheric density, S is the reference area of ​​the UAV, and m is the mass of the UAV.

[0042] In this embodiment, the roll angle command control channel of the UAV is a PID closed-loop control. The PID controller consists of three parts: proportional, integral and differential. Its control output c(t) represents the true value of the UAV. The calculation formula is as follows:

[0043]

[0044] Where e(t) is the difference between the target state and the current state at the current time t, K p is the proportional gain, K i is the integral gain, K d is the differential gain.

[0045] In this embodiment, the Actor network output is 1-dimensional, representing the roll angle control command, and the input dimension is a 13-dimensional joint state. The Critic network input is the sum of the state space dimension and the action space dimension, a total of 14 dimensions, and the output dimension is 1-dimensional. The specific structures of the Actor network and the Critic network are as follows: Figure 3 and Figure 4 .

[0046] The Actor network in this embodiment is specifically as follows: the Actor network is a feedforward neural network, comprising an input layer, two hidden layers and an output layer. The input layer receives the joint state vector s, the first hidden layer is a fully connected layer, the input dimension is 13 dimensions, the output dimension is 256, and the ReLU activation function is used. The second hidden layer is also a fully connected layer, the input dimension is 256, the output dimension is 256, and the ReLU activation function is used. The output layer is a fully connected layer, the input dimension is 256, the output dimension is the dimension 1 of the action space, and the Tanh activation function is used to generate the control input of the drone. The network structure extracts the features of the input state vector through two fully connected hidden layers, and increases the nonlinear expression capability through the ReLU activation function, and finally outputs the action vector through the Tanh activation function, limiting the value to the range of [-1,1], and then undergoes a linear transformation to adapt to the actual control requirements of the drone.

[0047] The Critic network in this embodiment is specifically as follows: The Critic network is a feedforward neural network, comprising an input layer, two hidden layers and an output layer. The input layer receives the state vector s and the action vector a, the input dimension is the sum of the dimensions of the state space and the action space, the first hidden layer is a fully connected layer, the input dimension is 14, the output dimension is 256, and the ReLU activation function is used. The second hidden layer is also a fully connected layer, the input dimension is 256, the output dimension is 256, and the ReLU activation function is used. The output layer is a fully connected layer, the input dimension is 256, the output dimension is 1, and it is used to generate a Q value. The network structure extracts the features of the input state and action vector through two fully connected hidden layers, and increases the nonlinear expression capability through the ReLU activation function, and finally generates a Q value through the output layer, which is used to evaluate the quality of a given state and action pair.

[0048] The construction process of the Actor network and the Critic network of the present invention can be described as follows:

[0049] S201: Given the current UAV joint state s, the Actor network generates an action a = μ(s|θ μ ), the Actor network is updated by maximizing the Q value output by the Critic network, and the formula is:

[0050]

[0051] in, is the Actor network policy gradient, μ is the mapping function represented by the Actor network, θ μ are the parameters of the Actor network, Represents the gradient of the Q value of the Critic network to the action, represents the experience replay pool, θ Q are the parameters of the Critic network, represents the expected reward when in state s in the experience pool;

[0052] S202: The Critic network evaluates the value of the action generated by the Actor network, i.e., the Q value. The input is the state s and the action a, and the output is the corresponding Q value Q(s,a|θ Q ), updated by minimizing the TD error, the formula is:

[0053]

[0054] Where L(θ Q ) represents the loss function of the Critic network, represents the expected value, which is the average of the samples in the experience replay pool, Q(s,a|θ Q ) represents the current Q value output by the Critic network, and the parameter is θQ , y represents the target Q value, y = r + γQ′(s′,μ′(s′|θ μ′ )|θ Q′ ), where r represents the current reward, γ represents the discount factor, which is used to weigh the impact of current rewards and future rewards, Q′(s′,μ′(s′|θ μ′ ) is the Q value output by the target Actor network, and the parameter is θ Q′ ; μ′(s′|θ μ′ ) The action output by the target Actor network, with parameter θ μ′ .

[0055] The Actor network relies on the Critic network to evaluate the quality of the actions it generates, and the Critic network relies on the Actor network to generate actions and calculate the Q-value function.

[0056] The joint state between the current leading UAV and the following UAV constructed in this embodiment is expressed as: s = (L, F), where F = (r f ,x f ,φ f ,V f ,θ f ,ψ f ,η f ), is the state information of the following drone; L=(r l ,x l ,φ l ,V l ,θ l ,ψ l ), which is the state information of the pilot drone itself. f ,x f ,φ f ,V f ,θ f ,ψ f and l ,x l ,φ l ,V l ,θ l ,ψ l Respectively represent the altitude, longitude, latitude, speed, track angle and heading angle of the follower UAV and the pilot UAV from the center of the earth, η f Indicates the remaining fuel ratio of the following drone. The drone control output action command is the roll angle γ command, which is in the range of [γ min ,γ max ]between.

[0057] In this embodiment, the reward function reward obtained by executing the action is calculated specifically according to the following formula:

[0058]

[0059] Among them, reward 1 (t) is the velocity vector alignment reward function, which is used to drive the velocity vectors of formation members to quickly align with the desired geometric configuration points. and are the velocity vector, position vector and expected position vector of the formation members, ω 1 >0 is the weight coefficient.

[0060] reward 2 (t) is the distance reward function. When the formation members are closer to the expected spatial position at adjacent moments, a positive reward is given to the drone, otherwise, a negative reward is given. 2 >0 is the weight coefficient, R(t-1) and R(t) are the distances of the drone relative to the desired position at time t-1 and t, divided by the velocity vector of the formation member It plays a normalizing role.

[0061] reward 3 (t) is the heading angle reward function. In order to further reduce the formation time, in addition to reducing the distance of the following drone relative to the expected position, the direction angle of the following drone’s velocity vector should also be constrained so that the velocity vector of the following drone is consistent with the velocity vector of the leading drone, thereby forming a stable formation. 3 >0 is the weight coefficient, θ(t), ψ(t) and θ 0 (t), ψ 0 (t) are the track angle and heading angle of the following UAV at time t and the track angle and heading angle of the leading UAV at time t respectively.

[0062] reward 4 (t) is the process reward function, which is used to reward the drone process state variables, R max The maximum distance that a formation member can deviate from the lead drone.

[0063] reward 5 (t) is the energy consumption reward function, which is used to optimize the energy consumption in the formation process; ω 5 >0 is the weight coefficient, η 0 is a positive constant, and η(t) is the remaining fuel ratio of the UAV.

[0064] In this embodiment, the process of training the Actor network and the Critic network is to first determine the training hyperparameters, and then the drone continuously updates the Actor network and the Critic network from historical data. The details are as follows:

[0065] S501: Determine training hyperparameters, as follows: Determine a discount factor for discounting future rewards; a soft update factor for soft updating of the target network. Determine the size of the experience replay pool to store enough experience samples so that samples can be drawn from the experience replay pool for training. In order to maintain exploration during training, exploration noise is added to the action selection process, usually Gaussian noise.

[0066] S502: Continuously collect data during the interaction between the drone and the environment and store it in an experience data pool. The historical data includes a four-tuple (s t a t r t s t+1 ), where s t is the current state, a t Output action for the controller, r t To obtain the return, s t+1 is the state reached after executing the action; here, the controller output action plus the disturbance, the disturbance obeys the zero-mean normal distribution, which is σ γ is the standard deviation.

[0067] S503: Random batch sampling is performed from the experience data pool, and the sampled data is used to continuously update the network parameters of the Actor network and the Critic network.

[0068] S504: When the training reaches the preset condition, the training process ends, and the Actor network model is finally formed and saved.

[0069] In this embodiment, in order to achieve a more realistic flight environment, certain interference is added to the output command of the drone controller to simulate the uncertainty interference caused by the complex environment encountered during the flight of the drone. At the same time, considering that the drone actuator executes the control command with damping and boundary properties, a closed-loop PID control is used at the bottom of the control mechanism to simulate the deviation caused by the actuator.

[0070] Figure 5 This is the pseudo code of the reinforcement learning algorithm used in this embodiment.

[0071] In this embodiment, the specific implementation steps of the drone when executing the task are as follows:

[0072] S601: Follow the drone to obtain its own status information F = (r f ,x f ,φ f ,V f ,θ f ,ψ f ,η f ) and the pilot drone status information L = (rl ,x l ,φ l ,V l ,θ l ,ψ l ), construct the joint state s = (L, F) between the current follower drone and the pilot drone;

[0073] S602: Load the joint state s into the trained Actor network model. The Actor outputs the optimal roll angle control command of the following UAV in the current state according to the current system joint state. The Actor network model runs on the onboard processor of the following UAV, and the output roll angle command is updated every specified time.

[0074] S603: The pilot UAV and the follower UAV complete the underlying closed-loop control by the PID controller according to their respective roll angle control instructions.

[0075] S604: Repeat the above process until the flight formation mission is completed and the UAV target mission is executed.

[0076] The present invention provides a multi-fixed-wing UAV formation control method based on reinforcement learning, which can enable a multi-UAV cluster to complete a flight mission of a predetermined formation configuration when an accurate environmental model cannot be established, and the method has certain robustness.

[0077] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A fixed-wing UAV formation control method based on deep reinforcement learning, It is characterized in that The following steps are involved: S1: Establish a kinematic model of a fixed-wing UAV and a simulation training environment for the UAV. For the control instructions input to the model, the closed-loop control is completed through the underlying PID controller. S2: Construct Actor network and Critic network through deep neural network; S3: The following UAV and the pilot UAV collect their own state information, including position, speed, track angle, heading angle and remaining fuel ratio information, to form a joint state space; the following UAV outputs the roll angle according to its control command to construct the action space; S4: Design a reinforcement learning reward function based on the characteristics of the drone and the formation task to guide the drone to obtain the maximum reward; S5: During the interaction between the drone and the environment, the parameters of the Actor network and the Critic network are continuously updated using historical data, thereby training and forming the final Actor network model; the historical data includes the current state, the action performed, the reward generated by the action, and the next state reached after the action is performed; S6: When performing a flight mission, the follower UAV collects the current status information of itself and the pilot UAV in real time, and combines the information into a joint state, and inputs the joint state into the final Actor network model to generate a roll angle control instruction for the follower UAV.

2. According to claim 1, the fixed-wing UAV formation control method based on deep reinforcement learning, It is characterized in that The kinematic model of the fixed-wing UAV described in S1 is as follows: Among them, r is the distance between the drone and the center of the earth, is the derivative of r; λ and φ are the longitude and latitude of the location of the UAV, respectively. and are the time derivatives of λ and φ respectively; V is the flight speed, is the time derivative of V; θ is the track angle, is the time derivative of θ; ψ is the heading angle, is the time derivative of ψ; D and L are drag acceleration and lift acceleration respectively, g is the local gravity acceleration, γ v is the roll angle of the drone, and the height of the drone from the ground h = rR 0 , R 0 is the radius of the earth; the expressions of drag acceleration and lift acceleration are: Among them C D and C L are the drag coefficient and lift coefficient, respectively, which are related to the angle of attack α and Mach number Ma of the UAV, ρ is the atmospheric density, S is the reference area of ​​the UAV, and m is the mass of the UAV.

3. The fixed-wing UAV formation control method based on deep reinforcement learning according to claim 1, It is characterized in that The PID controller consists of three parts: proportional, integral and differential. Its control output c(t) represents the true value of the drone. The calculation formula is as follows: Where e(t) is the difference between the target state and the current state at the current time t, K p is the proportional gain, K i is the integral gain, K d is the differential gain.

4. The fixed-wing UAV formation control method based on deep reinforcement learning according to claim 1, It is characterized in that The construction process of the Actor network and Critic network described in S2 is as follows: S201: Given the current UAV joint state s, the Actor network generates an action a = μ(s|θ μ ), the Actor network is updated by maximizing the Q value output by the Critic network, and the formula is: in, is the Actor network policy gradient, μ is the mapping function represented by the Actor network, θ μ are the parameters of the Actor network, Represents the gradient of the Q value of the Critic network to the action, represents the experience replay pool, θ Q are the parameters of the Critic network, represents the expected reward when in state s in the experience pool; S202: The Critic network evaluates the value of the action generated by the Actor network, i.e., the Q value. The input is the state s and the action a, and the output is the corresponding Q value Q(s,a|θ Q ), updated by minimizing the TD error, the formula is: Where L(θ Q ) represents the loss function of the Critic network, represents the expected value, which is the average of the samples in the experience replay pool, Q(s,a|θ Q ) represents the current Q value output by the Critic network, and the parameter is θ Q , y represents the target Q value, y = r + γQ′(s′,μ′(s′|θ μ′ )|θ Q′ ), where r represents the current reward, γ represents the discount factor, which is used to weigh the impact of current rewards and future rewards, Q′(s′,μ′(s′|θ μ′ ) is the Q value output by the target Actor network, and the parameter is θ Q′ ; μ′(s′|θ μ′ ) The action output by the target Actor network, with parameter θ μ′ .

5. The fixed-wing UAV formation control method based on deep reinforcement learning according to claim 1, It is characterized in that The specific method for the actuator in S3 to construct the joint state space and action space is: S301: Follow the drone's own status information F = (r f ,x f ,φ f ,V f ,θ f ,ψ f ,η f ), the pilot drone's own state information L = (r l ,x l ,φ l ,V l ,θ l ,ψ l ), where r f ,x f ,φ f ,V f ,θ f ,ψ f and l ,x l ,φ l ,V l ,θ l ,ψ l Respectively represent the altitude, longitude, latitude, speed, track angle and heading angle of the follower UAV and the pilot UAV from the center of the earth, η f represents the remaining fuel ratio of the following UAV; thereby constructing the joint state s=(L, F) between the current leading UAV and the following UAV; S302: The UAV control output action command is a roll angle γ command, which is within the range of [γ min ,γ max ]between.

6. The fixed-wing UAV formation control method based on deep reinforcement learning according to claim 1, It is characterized in that The form of the reward function described in S4 is as follows: Among them, reward 1 (t) is the velocity vector alignment reward function, which is used to drive the velocity vectors of formation members to quickly align with the desired geometric configuration point; and are the velocity vector, position vector and expected position vector of the formation members, ω 1 >0 is the weight coefficient; reward 2 (t) is the distance reward function. When the formation members are closer to the expected spatial position at adjacent moments, a positive reward is given to the drone. Otherwise, a negative reward is given. 2 >0 is the weight coefficient, R(t-1) and R(t) are the distances of the drone relative to the desired position at time t-1 and t, divided by the velocity vector of the formation member Play a normalizing role; reward 3 (t) is the heading angle reward function, which is used to constrain the direction angle of the velocity vector of the following drone, so that the velocity vector of the following drone is consistent with the velocity vector of the leading drone, thereby forming a stable formation; ω 3 >0 is the weight coefficient, θ(t), ψ(t) and θ 0 (t), ψ 0 (t) are the track angle and heading angle of the following UAV at time t and the track angle and heading angle of the leading UAV at time t respectively; reward 4 (t) is the process reward function, which is used to reward the drone process state variables, R max The maximum distance that a formation member can deviate from the lead drone. reward 5 (t) is the energy consumption reward function, which is used to optimize the energy consumption in the formation process; ω 5 >0 is the weight coefficient, η 0 is a positive constant, and η(t) is the remaining fuel ratio of the UAV.

7. The fixed-wing UAV formation control method based on deep reinforcement learning according to claim 1, It is characterized in that The method described in S5 for the drone to continuously update the Actor network and Critic network from historical data is: S501: Continuously collect data during the interaction between the drone and the environment and store it in an experience data pool. The historical data includes a four-tuple (s t a t r t s t+1 ), where s t is the current state, a t Output action for the controller, r t To obtain the return, s t+1 is the state achieved after performing the action; S502: Random batch sampling is performed from the experience data pool, and the sampled data is used to continuously update the network parameters of the Actor network and the Critic network; S503: When the training reaches the preset conditions, the training process ends, and the Actor network model is finally formed and saved.

8. The fixed-wing UAV formation control method based on deep reinforcement learning according to claim 7, It is characterized in that Add disturbance to the controller output action, and the disturbance obeys zero-mean normal distribution, which is σ γ is the standard deviation.

9. The fixed-wing UAV formation control method based on deep reinforcement learning according to claim 1, It is characterized in that The specific method of the drone in S6 when performing the task is: S601: Collect the status information of the following drone F = (r f ,x f ,φ f ,V f ,θ f ,ψ f ,η f ), the pilot drone's own status information L = (r l ,x l ,φ l ,V l ,θ l ,ψ l ), thereby obtaining the joint state s = (L, F) between the current leading UAV and the following UAV; S602: Load the joint state s into the trained Actor network, and output the optimal roll angle control command under the current state of the UAV; The execution of the UAV target mission can be completed by continuously looping the above two steps.

10. A fixed-wing UAV formation control device based on deep reinforcement learning. It is characterized in that include: Learning and training module: Establish a kinematic model of a fixed-wing UAV, construct an Actor network and a Critic network through a deep neural network, and use historical data to continuously update the parameters of the Actor network and the Critic network in the process of following the interaction between the UAV and the environment, so as to train and form the final Actor network model; these historical data include the current state, the action performed, the reward generated by the action, and the next state reached after the action is performed; During the interaction, the follower drone collects the current status information of itself and the pilot drone to form a joint state. The Actor network outputs the control instructions of the follower drone based on the joint state, and calculates the state of each drone at the next moment through the kinematic model of the fixed-wing drone. Mission execution module: When performing a flight mission, the follower UAV collects the current status information of itself and the pilot UAV in real time, combines it into a joint state, and inputs it into the trained Actor network model. The Actor network model generates control instructions for each follower UAV based on the joint state obtained in real time to adjust its roll angle.

Images