An unmanned aerial vehicle formation control method and system based on cluster center state prediction and a storage medium

Abstract

The application discloses a kind of unmanned aerial vehicle formation control method and system based on cluster center state prediction and storage medium, it is related to unmanned aerial vehicle cluster control field, to solve the problem that existing method is difficult to realize the control consistency and stable maintenance of unmanned aerial vehicle cluster formation in unstable communication environment unmanned aerial vehicle cluster formation control.For solving, including: step S1, based on the observation state of unmanned aerial vehicle cluster control strategy and the control instruction of unmanned aerial vehicle cluster control strategy, construct unmanned aerial vehicle cluster control strategy network;Step S2, generate the training data set of unmanned aerial vehicle cluster center state end-to-end prediction network using the trained unmanned aerial vehicle cluster control strategy network;Step S3, construct unmanned aerial vehicle cluster center state end-to-end prediction network and train;Step S4, deploy the unmanned aerial vehicle cluster control strategy network and unmanned aerial vehicle cluster center state end-to-end prediction network trained on unmanned aerial vehicle, control unmanned aerial vehicle cluster to realize formation stable maintenance in unstable communication environment.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) swarm control technology, and more specifically, to a UAV formation control method, system, and storage medium based on swarm center state prediction. Background Technology

[0002] With the rapid advancement of drone technology, drone swarm control has become a hot research area. Drone swarm control technology aims to enable coordinated flight among multiple drones to complete specific tasks or form predetermined flight patterns. This technology has broad application prospects in military, agriculture, logistics, and many other fields.

[0003] In recent years, numerous researchers have dedicated themselves to exploring different UAV formation control algorithms. Some studies focus on the formation control problem of fixed-wing UAVs, considering how to maintain these UAVs at a fixed distance and angle during flight. Other studies focus on optimizing UAV formation flight in specific environments, such as confined spaces. To enhance the stability and safety of formations, research has also involved obstacle avoidance, adaptive leader-follower models, and distributed coordinated control based on different roles. To better cope with unknown disturbances and motion constraints, some advanced methods, such as the variable-step-size MPC-APCMPIO algorithm, have been proposed. These methods not only improve the control accuracy of UAV formations but also enhance the robustness of the system. Meanwhile, some studies have also verified the effectiveness of the proposed algorithms through actual flight experiments.

[0004] Furthermore, advanced technologies such as reinforcement learning, graph theory methods, and bio-inspired algorithms are widely used in UAV swarm control. These methods enable UAVs to perform formation control in GPS-free environments and maintain system stability under conditions of network latency and communication instability. In addition, researchers have proposed various task-specific optimization strategies to address different application requirements, such as energy conservation, obstacle avoidance, and dynamic task allocation.

[0005] While there is already a wealth of research on the control of drone swarms, when communication distances are limited and the communication environment is unstable, drones can only acquire intermittent, localized swarm status. Furthermore, coordinated and consistent control of drone swarms requires the support of global swarm information; without such information, stable maintenance of drone swarm formations is impossible. Summary of the Invention

[0006] The technical problem to be solved by this invention is:

[0007] Existing methods lack global information for UAV swarm formation control in environments with limited communication distance and unstable communication, making it difficult to achieve control consistency and stability of UAV swarm formation.

[0008] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0009] This invention provides a method for UAV formation control based on cluster center state prediction, comprising the following steps:

[0010] Step S1: Based on the observation status and control commands of the UAV swarm control strategy, construct the UAV swarm control strategy network and train the UAV swarm control strategy network using deep reinforcement learning.

[0011] Step S2: Use the trained UAV swarm control strategy network to generate a training dataset for the end-to-end prediction network of the UAV swarm center state in the simulation environment.

[0012] Step S3: Construct an end-to-end prediction network for the central state of the UAV cluster, and train the end-to-end prediction network for the central state of the UAV cluster based on the training dataset.

[0013] Step S4: Deploy the trained UAV swarm control strategy network and the UAV swarm center state end-to-end prediction network on the UAV to control the UAV swarm to maintain stable formation in an unstable communication environment.

[0014] This invention first employs a centralized training and distributed execution learning architecture in deep reinforcement learning to train a UAV swarm control strategy network. Then, based on the trained UAV swarm control strategy network, training data for a UAV swarm center state prediction network is generated in a simulation environment. Next, an end-to-end prediction network for the UAV swarm center state is established, and its parameters are trained using supervised learning. Finally, the trained UAV swarm control strategy network and the UAV swarm center state prediction network are deployed on UAVs to control the UAV swarm to fly in a preset formation. This invention establishes a UAV swarm behavior prediction network and a distributed control strategy for the UAV swarm. When swarm communication is unstable, it predicts the state of the UAV swarm center by processing local temporal information observed by the UAVs, thus solving the problem of maintaining UAV swarm formation in unstable communication environments.

[0015] Furthermore, the observation state of the UAV swarm control strategy described in step S1 is as follows: at time t, the observation state of any UAV in the UAV swarm is... in, This indicates the position information of the target point of the drone swarm relative to the current drone; This indicates the status of the ad hoc drones that can communicate with the current drone, including relative position and relative speed information; This indicates obstacle information within the current drone's perception range, including the relative position information between the drone and the obstacles. This indicates the desired position of the current drone relative to the center of the drone swarm within a pre-defined drone swarm formation. This indicates the current state of the drone relative to the center of the drone swarm, including relative position and relative speed information;

[0016] The control command of the UAV swarm control strategy is the deviation a = (Δv) between the UAV control command and the current actual state of the UAV. H , Δχ, Δh), where Δv H Δχ represents the deviation between the speed control command and the actual speed of the drone, Δh represents the deviation between the heading angle control command and the actual heading angle of the drone, and Δh represents the deviation between the altitude control command and the actual altitude of the drone.

[0017] Furthermore, the UAV swarm control strategy network described in step S1 includes three layers of processing units. The first and second layers each have 128 nonlinear processing units with ReLU activation function. The third layer has two outputs. The first output has three nonlinear processing units with Tanh activation function, and the output value is the average value of the UAV control commands. The second output has three linear processing units with a linear activation function, and the output value is the logarithmic standard deviation σ of the UAV control commands. log .

[0018] Furthermore, step S1 also includes constructing a reward function for drone swarm formation control:

[0019] r t =r t g +r t f +r t cmd +r t col

[0020] In the formula, r t r represents the reward value received by the drone at time t. t g r represents the reward value associated with the target point. t f r represents the reward value associated with maintaining formation. t col r represents the reward value associated with safe collision avoidance by drones. tcmd This represents the reward value associated with the smoothing of drone commands;

[0021] Where, r t g Calculated by the following formula:

[0022]

[0023] In the formula, This represents the relative distance between the UAV and the target point at time t. This represents the relative distance between the UAV and the target point at time t-1. Δh represents the relative azimuth angle between the UAV and the target point at time t. t g R represents the relative altitude difference between the UAV and the target point at time t. g Indicates the radius of the target region;

[0024] r t f Calculated by the following formula:

[0025]

[0026] In the formula, This represents the heading angle deviation of the current drone relative to the center of the drone swarm at time t. This represents the speed deviation of the current drone relative to the center of the drone swarm at time t. This represents the deviation of the current position of the UAV relative to the expected position of the preset formation at time t;

[0027] r t col Calculated by the following formula:

[0028]

[0029] In the formula, This represents the minimum distance between the current drone and other drones in the cluster. This indicates the minimum distance between the drone and the obstacle.

[0030] r t cmd Calculated by the following formula:

[0031]

[0032] In the formula, This represents the control command for the UAV at time t. χ c (t-1) and h c (t-1) represents the control command for the UAV at time t-1.

[0033] Furthermore, the training of the UAV swarm control strategy network based on the deep reinforcement learning method in step S1 specifically involves: assuming that each UAV can obtain the true value of the UAV swarm center state, and letting the UAV swarm control strategy network be π. φ φ represents the network parameters, and the action-value network for the UAV swarm control strategy is Q. θ (s t a t ), where θ is the network parameter. According to the reinforcement learning algorithm, the network parameters of the UAV swarm control strategy, the action-value network parameters of the UAV swarm control strategy, and the temperature regulation coefficient α are updated by minimizing the following objective function:

[0034]

[0035] In the formula, This represents a data experience pool that stores the interaction experience of drone swarms. This represents the target action-value network, where γ represents the discount factor. This represents the target entropy.

[0036] Further, step S2 includes the following steps:

[0037] Step S2-1: Randomly initialize the simulation environment and load the trained UAV swarm control strategy network parameters φ;

[0038] Step S2-2: The state observed by drone i in the drone swarm The states of the most recent multiple drones communicating with the current drone for k consecutive frames from time t-k+1 to time t. And the deviation between the global cluster center state and the current local cluster center state.

[0039] Step S2-3: Each drone in the cluster outputs according to the drone cluster control strategy. When performing maneuvers, the simulation state is recursively calculated according to the set step size;

[0040] Step S2-4: Pair the data Store the dataset Θ;

[0041] Step S2-5: Repeat steps S2-1 to S2-4 until the required number of end-to-end prediction network training data for the UAV cluster center state are collected.

[0042] Further, step S3 includes the following steps:

[0043] Step S3-1: Construct an end-to-end prediction network for the central state of the UAV swarm, letting the input state of the end-to-end prediction network for the central state of the UAV swarm be... Output status is Let the end-to-end prediction network for the central state of the drone swarm be For network parameters;

[0044] Step S3-2: Randomly sample a dataset containing K from the dataset Θ. sl 100 empirical data points Batch data;

[0045] Step S3-3: Update the end-to-end prediction network parameters of the UAV cluster center state using the following loss function.

[0046]

[0047] Step S3-4: Repeat steps S3-2 to S3-3 until the end-to-end prediction network training in the center state of the UAV cluster converges.

[0048] Furthermore, the end-to-end prediction network for the central state of the UAV cluster described in step S3-1 is based on a long short-term memory network and includes a 5-layer network. The first layer is an LSTM unit used to process time-series observation information, the second to fourth layers are fully connected layers with ReLU activation function, and the last layer is a fully connected layer with tanh activation function.

[0049] This invention provides a UAV formation control system based on cluster center state prediction. The system has a program module corresponding to the steps of any of the above-described technical solutions, and executes the steps in the above-described UAV formation control method based on cluster center state prediction when running.

[0050] The present invention also provides a computer-readable storage medium storing a computer program configured to, when invoked by a processor, implement the steps of the UAV formation control method based on cluster center state prediction as described in any of the above technical solutions.

[0051] Compared with the prior art, the beneficial effects of the present invention are:

[0052] The present invention constructs an end-to-end prediction network for the center state of a UAV swarm in unstable communication environments. By processing local temporal information observed by UAVs, it infers the state of the UAV swarm center, ensuring stable information interaction between UAVs in unstable communication environments and solving the problem of lack of global information about the UAV swarm in formation control. 2) Through a centralized training and distributed execution architecture in deep reinforcement learning, the trained UAV swarm formation control strategy network and the end-to-end prediction network for the center state of the UAV swarm can control the UAV swarm to maintain stable formation in unstable communication environments, improving the formation maintenance effect of the UAV swarm under adverse communication conditions, and has significant advantages in terms of swarm control consistency and formation maintenance capabilities.

[0053] Through simulation experiments, the method of this invention can endow UAVs with the ability to acquire the state of the UAV swarm center during the training process, enabling the training of the UAV swarm control strategy network to be fast, stable and converge. The UAV swarm center state end-to-end prediction network can accurately predict the state of the UAV swarm center in unstable environments. The UAV swarm control strategy network combined with the UAV swarm center state end-to-end prediction network has obvious advantages in terms of swarm control consistency and formation maintenance capabilities. Attached Figure Description

[0054] Figure 1 This is a flowchart of the UAV formation control method based on cluster center state prediction in an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the network structure of the distributed control strategy network and Q-value network for the UAV swarm in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the network structure of the UAV cluster center state prediction network in an embodiment of the present invention;

[0057] Figure 4 This is a graph showing the change in reward value during the training process of the UAV swarm formation control strategy in this embodiment of the invention.

[0058] Figure 5 This is a graph showing the change in loss value during the training process of the UAV cluster center state prediction network in this embodiment of the invention.

[0059] Figure 6 This is a 3D trajectory diagram of the drone swarm formation in the test case of this invention embodiment;

[0060] Figure 7 This is a graph showing the communication distance variation curve of UAV No. 1 in this embodiment of the invention and the prediction curve of the cluster center status.

[0061] Figure 8This is a comparison of the average formation maintenance deviation of the UAV swarm under different communication distance distributions in this embodiment of the invention. Detailed Implementation

[0062] To enable those skilled in the art to better understand the present invention, exemplary embodiments or examples of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments or examples are merely some, not all, of the embodiments or examples of the present invention. All other embodiments or examples obtained by those skilled in the art based on the embodiments or examples of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0063] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0064] In a typical implementation, combined with Figure 1 As shown, this invention provides a UAV formation control method based on cluster center state prediction, comprising the following steps:

[0065] Step S1: Based on the observation status and control commands of the UAV swarm control strategy, construct the UAV swarm control strategy network and train the UAV swarm control strategy network using deep reinforcement learning.

[0066] Step S2: Use the trained UAV swarm control strategy network to generate a training dataset for the end-to-end prediction network of the UAV swarm center state in the simulation environment.

[0067] Step S3: Construct an end-to-end prediction network for the center state of the UAV cluster. Based on the training dataset, train the end-to-end prediction network for the center state of the UAV cluster using supervised learning.

[0068] Step S4: Deploy the trained UAV swarm control strategy network and the UAV swarm center state end-to-end prediction network on the UAV to control the UAV swarm to maintain stable formation in an unstable communication environment.

[0069] In a typical implementation, preferably, step S1 specifically includes the following steps:

[0070] Step S1-1: Determine the observation state of the UAV swarm control strategy. At time t, determine the observation state of any UAV in the swarm. It contains 5 parts, among which, This indicates the position information of the target point of the drone swarm relative to the current drone; This indicates the status of the ad hoc drones that can communicate with the current drone, including relative position and relative speed information; This indicates obstacle information within the current drone's perception range, including the relative position information between the drone and the obstacles. This indicates the desired position of the current drone relative to the center of the drone swarm within a pre-defined drone swarm formation. This indicates the current state of the drone relative to the center of the drone swarm, including relative position and relative speed information.

[0071] Step S1*2: Determine the drone control commands to maintain drone swarm formation. Based on the drone's motion characteristics, the drone control commands are the drone speed control commands. Heading angle control command χ c and altitude control command h c The output value of the UAV swarm control strategy network is the deviation (Δv) between the UAV control command and the current actual state of the UAV. H ,Δχ,Δh), that is χ c =χ+Δχ,h c =h+Δh,(v H , χ, h) represent the current actual state of the UAV. Therefore, the control command of the UAV swarm control strategy is expressed as a = (Δv, χ, h). H ,Δχ,Δh).

[0072] Steps S1-3: Based on the observation status and control commands of the UAV swarm control strategy, construct the network structure for the UAV swarm control strategy. The UAV swarm control strategy network is as follows: Figure 2 As shown in (a), the network consists of three layers. The first and second layers each have 128 nonlinear processing units with ReLU activation function. The third layer has two outputs. The first output has three nonlinear processing units with Tanh activation function, and its output value is the average of the UAV control commands. The second output has three linear processing units with a linear activation function, and the output value is the logarithmic standard deviation σ of the UAV control commands. log The control commands for the drone are in and σ log Constructed normal distribution Sampling is performed on top, that is

[0073] Steps S1-4: Construct the reward function for UAV swarm formation control. The goal of the UAV swarm control strategy is to maintain a preset formation and fly to the target point while ensuring safety. Therefore, the reward function is constructed in the following form:

[0074] r t =r t g +r t f +r t cmd +r t col

[0075] In the formula: r t r represents the reward value received by the drone at time t. t g r represents the reward value associated with the target point. t f r represents the reward value associated with maintaining formation. t col r represents the reward value associated with safe collision avoidance by drones. t cmd This represents the reward value associated with smooth drone command delivery.

[0076] Where, r t g Calculated by the following formula:

[0077]

[0078] In the formula: This represents the relative distance between the UAV and the target point at time t. This represents the relative distance between the UAV and the target point at time t-1. This represents the relative azimuth angle between the UAV and the target point at time t. R represents the relative altitude difference between the UAV and the target point at time t. g Indicates the radius of the target region.

[0079] r t f Calculated by the following formula:

[0080]

[0081] In the formula, This represents the heading angle deviation of the current drone relative to the center of the drone swarm at time t. This represents the speed deviation of the current drone relative to the center of the drone swarm at time t. This represents the deviation of the current position of the UAV relative to the expected position of the preset formation at time t.

[0082] r t col Calculated by the following formula:

[0083]

[0084] In the formula, This represents the minimum distance between the current drone and other drones in the cluster. This indicates the minimum distance between the drone and the obstacle.

[0085] r t cmd Calculated by the following formula:

[0086]

[0087] In the formula, This represents the control command for the UAV at time t. χ c (t-1) and h c (t-1) represents the control command for the UAV at time t-1.

[0088] Steps S1-5: Based on reinforcement learning algorithms, train the UAV swarm control strategy in a simulation environment. This utilizes the advantage of obtaining global information during centralized training, assuming that each UAV can obtain the true value of the UAV swarm's central state during training. Let the UAV swarm control strategy network be π. φ φ represents the network parameters; let the action-value network of the UAV swarm control strategy be Q. θ (s t a t ), where θ is the network parameter, and the network structure is as follows: Figure 2 As shown in (b), according to the SAC algorithm in reinforcement learning, the policy network parameters, action-value function network parameters, and temperature regulation coefficient α can be updated by minimizing the objective function shown in the following equation:

[0089]

[0090] In the formula, This represents a data experience pool that stores the interaction experience of drone swarms. This represents the target action-value network, where γ (0 < γ < 1) represents the discount factor. This represents the target entropy.

[0091] In a typical implementation, preferably, step S2 specifically includes the following steps:

[0092] Step S2-1: Randomly initialize the simulation environment and load the trained UAV swarm control strategy network parameters φ.

[0093] Step S2-2: The state observed by drone i in the drone swarm The states of the most recent multiple drones communicating with the current drone for k consecutive frames from time t-k+1 to time t. And the deviation between the global cluster center state and the current local cluster center state. For any drone i in the drone swarm, the observed state is... This refers to the observation state of the UAV swarm control strategy in step S1-1. It is assumed that the UAVs can obtain the global UAV swarm center state. Let (t-k+1) represent the states of the four most recent UAVs communicating with the current UAV for k consecutive frames from time t to t. This represents the positional deviation between drone j and the local cluster center position observed by the current drone. This represents the speed deviation between UAV j and the speed of the local cluster center observed by the current UAV. This represents the heading deviation of UAV j from the velocity of the local swarm center observed by the current UAV; state. in, This indicates the current state of the local drone swarm center as observed by the drone. Indicates the global drone swarm center status. and These represent the position and velocity normalization parameters, respectively.

[0094] Step S2-3: Each drone in the cluster outputs according to the drone cluster control strategy. When performing maneuvers, the simulation state is recursively calculated according to the set step size;

[0095] Step S2-4: Pair the data Store the dataset Θ;

[0096] Step S2-5 repeats steps S2-1 to S2-4 until the required number of end-to-end prediction network training data for the drone cluster center state are collected.

[0097] In a typical implementation, preferably, step S3 specifically includes the following steps:

[0098] Step S3-1: Input state of the end-to-end prediction network based on the central state of the UAV swarm and output status End-to-end prediction network for the central state of the drone swarm Network structure such as Figure 3 As shown, For network parameters. In the designed network structure, in order to effectively process the temporal information observed by the UAV, an end-to-end prediction network for the central state of the UAV swarm is constructed based on a Long Short-Term Memory (LSTM) network. The network consists of 5 layers. The first layer is composed of LSTM units to process the temporal observation information. The next 3 layers are fully connected layers with ReLU activation function. The last layer is a fully connected layer with tanh activation function, the purpose of which is to limit the output value to [-1, 1].

[0099] Step S3-2: Randomly sample a dataset containing K from the dataset Θ. sl 100 empirical data points Batch data;

[0100] Step S3-3: Update the end-to-end prediction network parameters of the UAV swarm center state using the following loss function.

[0101]

[0102] Step S3-4: Repeat steps S3-2 to S3-3 until the end-to-end prediction network training of the UAV cluster center state converges.

[0103] The UAV formation control method (algorithm) based on cluster center state prediction proposed in this invention is the underlying technical core of this invention, and various products can be derived based on the algorithm.

[0104] Based on the method proposed in this invention, a UAV formation control system based on cluster center state prediction is developed using a programming language. This system has program modules corresponding to the steps of the above technical solution, and executes the steps in the above-mentioned UAV formation control method based on cluster center state prediction when running.

[0105] The developed system (software) computer program is stored on a computer-readable storage medium, and the computer program is configured to implement the steps of the above-described UAV formation control method based on cluster center state prediction when called by a processor. In other words, the invention is materialized on a carrier, becoming a computer program product.

[0106] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] The computational programs (also referred to as programs, software, software applications, or code) of this invention include machine instructions of a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device PLD) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0108] The beneficial effects of the present invention will be described below with reference to specific embodiments.

[0109] Example 1

[0110] (1) Experimental Environment

[0111] This embodiment uses digital simulation to verify the correctness and rationality of the invention. First, a drone swarm formation flight control simulation environment is constructed in a Python environment. The drone swarm consists of 5 isomorphic drones, and the environment contains 8 randomly distributed cylindrical obstacles. The radius of the drone swarm's target area is set to R. g =50m. Assume the communication distance of each drone follows a normal distribution. It means that, where d com This represents the average communication distance of the UAV, set to 50m in the simulation experiment, σ com The standard deviation of the communication distance of the UAV is represented by a value set to [0, 50] m in the simulation experiment. com σ represents the instability of the communication environment. comThe larger the value, the more unstable the communication environment. The maximum and minimum horizontal speeds of the drones were set to 20 m / s and 12 m / s, respectively; the climb rate and descent rate were set to 5 m / s and -5 m / s, respectively; and the obstacle detection range was set to 300 m. The neural network was built using PyTorch. At the start of each simulation interaction, the target point of the drones was randomly set within 1000 m of the initial point of the drone swarm.

[0112] The simulation test environment in this embodiment is Windows 10 + Python 3.7, and the hardware environment is AMD Ryzen 77840HSw / Radeon 780M Graphics 3.80GHz.

[0113] (2) Experimental Results and Analysis

[0114] This embodiment first verifies whether the training process of the UAV swarm control strategy network and the UAV swarm central state end-to-end prediction network can converge. Figure 4 The reward curves of two UAV swarm control strategies during the training process are shown: one lacking swarm center state prediction capability and the other possessing swarm center state prediction capability. The standard deviation σ of the UAV communication distance is set as the threshold. com =30m. As shown in the figure, in the established unstable UAV communication environment, the UAV swarm control strategy with swarm center state prediction capability can obtain a larger reward value and basically converges after 5000 training cycles. In contrast, when lacking swarm center state prediction capability, the corresponding UAV swarm control strategy obtains a much smaller reward value than the one with swarm center state prediction capability, and the reward curve fluctuates continuously during training, failing to converge. These results demonstrate that the UAV formation control method combined with swarm center state prediction proposed in this invention can improve the stability of the UAV swarm control strategy during training, making the training process easier to converge and obtain a higher reward value.

[0115] Figure 5 The figure shows the loss value variation curve during the training process of the UAV swarm center state prediction network. As shown in the figure, during the training of the UAV swarm center state prediction network using the supervised learning method, the network's loss value decreases rapidly and stabilizes, indicating that the training process converges quickly.

[0116] Figure 6 This demonstrates the effect of a test experiment (setting the standard deviation σ of the drone communication distance). com =30m) The three-dimensional trajectory of the drone swarm formation. As can be seen from the figure, the trained drone swarm control strategy can control the drone swarm to fly safely to the target area (the green sphere is the target area of ​​the drone swarm). Figure 7 As shown Figure 6The communication distance variation curve of UAV No. 1 in the test case and the prediction curve of the cluster center state. (From...) Figure 7 As shown in the first subgraph, the communication distance of UAV No. 1 is dynamically changing, indicating that its communication environment is unstable. Figure 7 The second sub-graph shows the predicted location of the drone swarm center and the actual location of the drone swarm center, both for drone #1. The dashed line represents the predicted value, and the solid line represents the actual value. As can be seen from the graph, drone #1 can predict the location of the global drone swarm center relatively accurately through local interaction information. Figure 7 The third sub-graph shows the predicted velocity of the drone swarm center and the actual velocity of the drone swarm center, both from drone number 1. The dashed line represents the predicted value, and the solid line represents the actual value. As can be seen from the graph, drone number 1 can predict the velocity of the global drone swarm center relatively accurately through local interaction information. Figure 7 The fourth sub-graph shows the predicted heading angle of the drone swarm center from UAV 1 and the actual heading angle of the drone swarm center. The dashed line represents the predicted value, and the solid line represents the actual value. As can be seen from the graph, UAV 1 can predict the heading angle of the global drone swarm center relatively accurately through local interactive information.

[0117] Figure 8 The figure shows a comparison of the average formation maintenance deviation of UAV swarms under different communication distance distributions. Each set of results is a statistical result obtained from 100 random simulation experiments. As can be seen from the figure, compared to UAV swarm control strategies lacking swarm center state prediction capability, UAV swarm control strategies with swarm center state prediction capability can better maintain formation based on the predicted UAV swarm center state, controlling the average formation maintenance deviation of the UAV swarm within 10m and achieving a better formation maintenance effect.

[0118] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A method for UAV formation control based on cluster center state prediction, characterized in that: Includes the following steps: Step S1: Based on the observation status and control commands of the UAV swarm control strategy, construct the UAV swarm control strategy network and train the UAV swarm control strategy network using deep reinforcement learning. Step S2: Use the trained UAV swarm control strategy network to generate a training dataset for the end-to-end prediction network of the UAV swarm center state in the simulation environment. Step S3: Construct an end-to-end prediction network for the central state of the UAV cluster, and train the end-to-end prediction network for the central state of the UAV cluster based on the training dataset. Step S4: Deploy the trained UAV swarm control strategy network and the UAV swarm center state end-to-end prediction network on the UAV to control the UAV swarm to maintain stable formation in an unstable communication environment.

2. The UAV formation control method based on cluster center state prediction according to claim 1, characterized in that: The observation state of the UAV swarm control strategy described in step S1 is as follows: at time t, the observation state of any UAV in the UAV swarm is... in, This indicates the position information of the target point of the drone swarm relative to the current drone; This indicates the status of the ad hoc drones that can communicate with the current drone, including relative position and relative speed information; This indicates obstacle information within the current drone's perception range, including the relative position information between the drone and the obstacles. This indicates the desired position of the current drone relative to the center of the drone swarm within a pre-defined drone swarm formation. This indicates the current state of the drone relative to the center of the drone swarm, including relative position and relative speed information; The control command of the UAV swarm control strategy is the deviation a = (Δv) between the UAV control command and the current actual state of the UAV. H , Δχ, Δh), where Δv H Δχ represents the deviation between the speed control command and the actual speed of the drone, Δh represents the deviation between the heading angle control command and the actual heading angle of the drone, and Δh represents the deviation between the altitude control command and the actual altitude of the drone.

3. The UAV formation control method based on cluster center state prediction according to claim 2, characterized in that: The UAV swarm control strategy network described in step S1 includes three layers of processing units. The first and second layers each have 128 nonlinear processing units with ReLU activation function. The third layer has two outputs. The first output has three nonlinear processing units with Tanh activation function, and the output value is the average of the UAV control commands. The second output has three linear processing units with a linear activation function, and the output value is the logarithmic standard deviation σ of the UAV control commands. log .

4. The UAV formation control method based on cluster center state prediction according to claim 3, characterized in that: Step S1 also includes constructing a reward function for drone swarm formation control: r t ＝r t g +r t + +r t cmd +r t col In the formula, r t r represents the reward value received by the drone at time t. t g r represents the reward value associated with the target point. t f r represents the reward value associated with maintaining formation. t col r represents the reward value associated with safe collision avoidance by drones. t cmd This represents the reward value associated with the smoothing of drone commands; Where, r t g Calculated by the following formula: In the formula, This represents the relative distance between the UAV and the target point at time t. This represents the relative distance between the UAV and the target point at time t-1. This represents the relative azimuth angle between the UAV and the target point at time t. R represents the relative altitude difference between the UAV and the target point at time t. g Indicates the radius of the target region; r t f Calculated by the following formula: In the formula, This represents the heading angle deviation of the current drone relative to the center of the drone swarm at time t. This represents the speed deviation of the current drone relative to the center of the drone swarm at time t. This represents the deviation of the current position of the UAV relative to the expected position of the preset formation at time t; r t col Calculated by the following formula: In the formula, This represents the minimum distance between the current drone and other drones in the cluster. This indicates the minimum distance between the drone and the obstacle. r t cmd Calculated by the following formula: In the formula, This represents the control command for the UAV at time t. χ c (t-1) and h c (t-1) represents the control command for the UAV at time t-1.

5. The UAV formation control method based on cluster center state prediction according to claim 4, characterized in that: The step S1, which trains the UAV swarm control policy network using deep reinforcement learning, specifically involves: assuming each UAV can obtain the true value of the UAV swarm center state, let the UAV swarm control policy network be π. φ φ represents the network parameters, and the action-value network for the UAV swarm control strategy is Q. θ (s t a t ), where θ is the network parameter. According to the reinforcement learning algorithm, the network parameters of the UAV swarm control strategy, the action-value network parameters of the UAV swarm control strategy, and the temperature regulation coefficient α are updated by minimizing the following objective function: In the formula, This represents a data experience pool that stores the interaction experience of drone swarms. This represents the target action-value network, where γ represents the discount factor. This represents the target entropy.

6. The UAV formation control method based on cluster center state prediction according to claim 5, characterized in that: Step S2 includes the following steps: Step S2-1: Randomly initialize the simulation environment and load the trained UAV swarm control strategy network parameters φ; Step S2-2: The state observed by drone i in the drone swarm The states of the most recent multiple drones communicating with the current drone for k consecutive frames from time t-k+1 to time t. And the deviation between the global cluster center state and the current local cluster center state. Step S2-3: Each drone in the cluster outputs according to the drone cluster control strategy. When performing maneuvers, the simulation state is recursively calculated according to the set step size; Step S2-4: Pair the data Store the dataset Θ; Step S2-5: Repeat steps S2-1 to S2-4 until the required number of end-to-end prediction network training data for the UAV cluster center state are collected.

7. The UAV formation control method based on cluster center state prediction according to claim 6, characterized in that: Step S3 includes the following steps: Step S3-1: Construct an end-to-end prediction network for the central state of the UAV swarm, letting the input state of the end-to-end prediction network for the central state of the UAV swarm be... Output status is Let the end-to-end prediction network for the central state of the drone swarm be For network parameters; Step S3-2: Randomly sample a dataset containing K from the dataset Θ. sl 100 empirical data points Batch data; Step S3-3: Update the end-to-end prediction network parameters of the UAV cluster center state using the following loss function. Step S3-4: Repeat steps S3-2 to S3-3 until the end-to-end prediction network training in the center state of the UAV cluster converges.

8. The UAV formation control method based on cluster center state prediction according to claim 7, characterized in that: The end-to-end prediction network for the central state of the UAV cluster described in step S3-1 is based on a long short-term memory network and includes a 5-layer network. The first layer is an LSTM unit used to process time-series observation information. The second to fourth layers are fully connected layers with ReLU activation function, and the last layer is a fully connected layer with tanh activation function.

9. A UAV formation control system based on cluster center state prediction, characterized in that: The system has a program module corresponding to the steps of the method described in any one of claims 1 to 8, and executes the steps in the above-described UAV formation control method based on cluster center state prediction when it is run.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to, when invoked by a processor, implement the steps of the UAV formation control method based on cluster center state prediction as described in any one of claims 1 to 8.

Images