Unmanned aerial vehicle cluster intelligent formation method, device and equipment based on distributed reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle cluster intelligent formation method, device and equipment based on distributed reinforcement learning, and the method comprises the steps: constructing a reinforcement learning environment of an unmanned aerial vehicle cluster, which comprises a state space, an action space and a reward function, according to a flight task scene of an unmanned aerial vehicle, the flight mission scene comprises execution of air refueling and long-distance formation transportation; building a training framework of a distributed reinforcement learning algorithm based on the reinforcement learning environment, and performing network construction based on the training framework to obtain a neural network including an action decision network and a formation target evaluation network; performing distributed training on the neural network by adjusting hyper-parameters of the neural network and utilizing large-scale distributed data to obtain a formation model; and loading the formation model to an actual unmanned aerial vehicle cluster system, and adjusting a formation strategy according to a real-time flight environment and task requirements to carry out a formation flight task. The cooperative flight capability and the self-adaptability of the unmanned aerial vehicle cluster can be improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle technology, and in particular to a method, device and equipment for intelligent formation of unmanned aerial vehicle clusters based on distributed reinforcement learning. Background Art

[0002] The traditional centralized formation control method relies on a central control node, which is responsible for collecting the status information of all drones, making global plans, and issuing control instructions. Although this control mode ensures the stability and consistency of formation flight to a certain extent, it has the following main problems: Large amount of calculation: The central control node needs to process the real-time status data from each drone and perform complex calculations to generate the global optimal flight path and control instructions. As the number of drones increases, the amount of calculation increases sharply, placing extremely high demands on the processing power and stability of the control node.

[0003] High communication pressure: In the centralized control mode, each drone needs to maintain real-time communication with the central control node to transmit status information and receive control instructions. This leads to heavy load on the communication link, which is prone to communication delays and packet loss in complex environments, affecting the stability and real-time performance of formation flight.

[0004] Poor flexibility: The centralized control method lacks flexibility. Once the central control node fails or is disturbed, the entire formation will lose control, resulting in mission failure. In addition, in a dynamically changing flight environment, the centralized control method is difficult to quickly respond to environmental changes and adjust the formation strategy. Summary of the invention

[0005] In view of this, the purpose of the present invention is to propose a method, device and equipment for intelligent formation of drone swarms based on distributed reinforcement learning, aiming to solve the problems of large computational complexity, high communication pressure and poor flexibility in traditional centralized formation control methods.

[0006] To achieve the above object, the present invention provides a method for intelligent formation of drone swarm based on distributed reinforcement learning, the method comprising: Construct a reinforcement learning environment for drone swarms that includes state space, action space, and reward function based on the flight mission scenarios of drones. The flight mission scenarios include aerial refueling and long-distance formation transportation. Building a training framework for a distributed reinforcement learning algorithm based on the reinforcement learning environment, and constructing a network based on the training framework to obtain a neural network including an action decision network and a formation target evaluation network; A formation model is obtained by adjusting hyperparameters of the neural network and performing distributed training on the neural network using large-scale distributed data; The formation model is loaded into an actual UAV cluster system, and the formation strategy is adjusted according to the real-time flight environment and mission requirements to perform formation flight missions.

[0007] Preferably, the reinforcement learning environment of the drone cluster including the state space, action space and reward function is constructed according to the specific application scenario of the drone, including: Defining the state space, including the position information, speed information, formation information, environmental perception information and energy state information of the UAV; The action space is defined, including a transverse rod, a longitudinal rod, a pedal and a throttle rod, and the flight attitude and position of the UAV are adjusted by the transverse rod, the longitudinal rod and the pedal, and the flight speed of the UAV is adjusted by the throttle rod; The reward function is defined, including a formation stability reward, a task completion reward, an obstacle avoidance capability reward, and an energy efficiency reward.

[0008] Preferably, the training framework includes an Actor sampling module, a Learner training module, and a scheduling management and storage module; wherein the Actor sampling module is responsible for interacting with the reinforcement learning environment to collect samples; the Learner training module is responsible for updating the parameters of the neural network; the scheduling management and storage module is responsible for scheduling and managing communications between different nodes, and the scheduling management and storage module includes local scheduling, object storage, and global control storage.

[0009] Preferably, the network construction based on the training framework to obtain a neural network including an action decision network and a formation target evaluation network includes: The neural network is constructed by adopting the value function method; wherein the action decision network is implemented based on the DRQN ​​network, the input of the action decision network includes the local observation information of each UAV, and the output includes the value function estimation value of each action obtained by the UAV under the current observation; the formation target evaluation network is implemented based on the hybrid network, the input of the formation target evaluation network includes the local action value function estimation value of each UAV and the current global state information, and the output includes the value function estimation value of the formation target state.

[0010] Preferably, the formation model is obtained by adjusting the hyperparameters of the neural network and performing distributed training on the neural network using large-scale distributed data, including: Initializing the action decision network, the formation target assessment network, and the experience pool; The action decision network of each sampling node outputs a formation strategy according to the current state and environmental information of the drone cluster, and sends control instructions to the drone to adjust the flight state and position of the drone. Through the interaction between the Actor sampling module of the drone and the reinforcement learning environment, large-scale distributed data including samples is generated, and the samples are sent to the global controller in an asynchronous manner and stored in the experience pool, wherein the samples include reward values, state values, and observation values; Updating the parameters of the neural network includes: in each training round, obtaining batch samples from the experience pool through the training sampling node to calculate and update the parameters, and sending the updated parameters to the sampling node through the global controller to update the parameters of the action decision network; and, Using the TD-Error loss function to update the parameters of the action decision network, and updating the parameters of the formation target evaluation network by minimizing the error between the reward value of the formation target state and the true reward; When the formation target state of the drone cluster, including flight stability and obstacle avoidance capability, does not reach the preset target state, the neural network continues to be trained and the formation strategy is adjusted.

[0011] Preferably, the step of loading the formation model into an actual drone cluster system includes: The formation model is loaded into an actual drone cluster system for deployment in an executable format or source code form, wherein the executable format includes exporting the computational graph of the formation model into an executable format of a TensorFlow Lite model or a TensorFlow.js model for execution on the client; the source code form includes exporting the computational graph of the formation model into source code in other languages ​​for compilation and execution on the client.

[0012] To achieve the above object, the present invention also provides a UAV swarm intelligent formation device based on distributed reinforcement learning, the device comprising: An environment construction unit, used to construct a reinforcement learning environment for a drone cluster including a state space, an action space, and a reward function according to a flight mission scenario of the drone, wherein the flight mission scenario includes performing aerial refueling and long-distance formation transportation; A network construction unit, used to build a training framework of a distributed reinforcement learning algorithm based on the reinforcement learning environment, and to construct a network based on the training framework to obtain a neural network including an action decision network and a formation target evaluation network; A network training unit, configured to obtain a formation model by adjusting hyperparameters of the neural network and performing distributed training on the neural network using large-scale distributed data; The formation unit is used to load the formation model into the actual UAV cluster system and adjust the formation strategy to perform formation flight missions according to the real-time flight environment and mission requirements.

[0013] In order to achieve the above-mentioned objectives, the present invention also proposes a drone swarm intelligent formation device based on distributed reinforcement learning, comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a drone swarm intelligent formation method based on distributed reinforcement learning as described in the above-mentioned embodiment.

[0014] In order to achieve the above-mentioned objectives, the present invention also proposes a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of a drone swarm intelligent formation method based on distributed reinforcement learning as described in the above-mentioned embodiment.

[0015] In order to achieve the above objectives, the present invention also proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a drone swarm intelligent formation method based on distributed reinforcement learning as described in the above embodiment.

[0016] Beneficial effects: The above scheme, by constructing a UAV cluster formation method based on distributed reinforcement learning, can flexibly respond to complex and changeable flight environments and mission requirements, and improve the autonomous decision-making ability of UAV clusters; adopts a distributed reinforcement learning algorithm and trains through large-scale distributed data, which significantly reduces the computational burden of a single computing node and improves the overall computing efficiency; adjusts the formation strategy according to the real-time flight environment and mission requirements, so that the UAV cluster can respond quickly to environmental changes and complete formation flight missions, thereby improving the collaborative flight capability and adaptability of the UAV cluster in complex environments.

[0017] The above scheme, by defining the state space, action space and reward function in the reinforcement learning environment, introduces time series information or historical state information in the state space, so that the drone can understand the current environmental state more comprehensively and improve the accuracy and stability of decision-making. Through the detailed definition of the action space (including the lateral rod, longitudinal rod, pedal and throttle rod), the precise control of the flight attitude and position of the drone is achieved. Combining specific application scenarios and dynamic environments to build reward functions can better adapt to environmental changes or external factors.

[0018] The above solution achieves efficient distributed training and data management by constructing the Actor sampling module, Learner training module, and scheduling management and storage module in the training framework; through modular design, the system can be easily expanded to adapt to larger-scale drone clusters and more complex mission requirements.

[0019] The above scheme adopts the value function method to construct a neural network. The action decision network and the formation target evaluation network are responsible for accurate action decision-making and formation target state evaluation, respectively, which improves the performance of the overall formation flight. The action decision network makes decisions based on local observation information, while the formation target evaluation network combines the local action value function estimate and the global state information for evaluation, realizing the effective fusion of local and global information.

[0020] The above scheme improves the efficiency of sample collection and utilization by sending samples to the global controller in an asynchronous manner and storing them in the experience pool. It uses the TD-Error loss function to update the parameters of the action decision network and the formation target evaluation network to ensure the continuous optimization of the model during the training process. When the formation target state does not reach the preset target, the neural network continues to be trained and the formation strategy is adjusted, so that the UAV cluster can continuously adapt to environmental changes and improve the stability and safety of formation flight.

[0021] The above solutions provide multiple deployment methods (executable format or source code form), so that the formation model can be easily applied to different platforms and devices; by exporting the calculation graph of the formation model into source code or executable format in different languages, cross-platform compatibility and a wide range of applications are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic flow chart of a method for intelligent formation of drone swarms based on distributed reinforcement learning provided in accordance with an embodiment of the present invention.

[0024] Figure 2 A schematic diagram of the framework of a distributed reinforcement learning algorithm provided by an embodiment of the present invention.

[0025] Figure 3 A schematic diagram of a neural network training process provided by an embodiment of the present invention.

[0026] Figure 4A schematic diagram of the structure of a drone swarm intelligent formation device based on distributed reinforcement learning provided by one embodiment of the present invention.

[0027] The realization of the purpose, functional features and advantages of the invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0029] The present invention is described in detail below with reference to the embodiments.

[0030] Reference Figure 1 The figure is a flow chart of a method for intelligent formation of drone swarms based on distributed reinforcement learning provided by one embodiment of the present invention.

[0031] In this embodiment, the method includes: S11, constructing a reinforcement learning environment for a drone cluster including a state space, an action space, and a reward function according to the flight mission scenarios of the drones, wherein the flight mission scenarios include performing aerial refueling and long-distance formation transportation.

[0032] In step S11, the reinforcement learning environment of the drone cluster including the state space, action space and reward function is constructed according to the specific application scenario of the drone, including: Defining the state space, including the position information, speed information, formation information, environmental perception information and energy state information of the UAV; The action space is defined, including a transverse rod, a longitudinal rod, a pedal and a throttle rod, and the flight attitude and position of the UAV are adjusted by the transverse rod, the longitudinal rod and the pedal, and the flight speed of the UAV is adjusted by the throttle rod; The reward function is defined, including a formation stability reward, a task completion reward, an obstacle avoidance capability reward, and an energy efficiency reward.

[0033] In this embodiment, a reinforcement learning algorithm environment for drone clusters is constructed according to specific application scenarios, including state space, action space, and reward function. The state space can introduce time series information (including the position of the first time step, the position of the second time step, etc.) or historical state information. The processing method of the state space includes processing the original situation through feature selection, feature combination, one-hot encoding (one-hot), feature normalization, feature dimension reduction, etc. to form a vector; the action space includes a transverse rod, a longitudinal rod, a pedal and a throttle lever. By controlling the drone, the flight attitude and position of the drone are adjusted. Among them, the transverse rod, the longitudinal rod, and the pedal cooperate with each other to realize complex actions such as forward, backward, ascending, descending, turning left, and turning right, thereby adjusting the flight attitude and position of the drone; the throttle lever can change the flight speed of the drone; the construction of the reward function is combined with the specific application scenario, and should be When considering dynamic environments (i.e. complex environments, including natural environments, flight mission scenarios, etc., where natural environments: such as severe weather conditions (such as strong winds, heavy rains, lightning, etc.), complex terrain (such as mountains, waters, urban buildings, etc.), and possible natural disasters (such as earthquakes, volcanic eruptions, etc.), these natural environmental factors will directly affect the flight performance and formation stability of drones; flight mission scenarios: when performing tasks such as aerial refueling and long-distance formation transportation, it is necessary to consider the formation stability of drone formations under communication constraints, the problem of internal collision avoidance in the formation, and the timing and deadlock problems of task execution, etc.), design reward functions that can adapt to environmental changes or external factors to ensure that the agent can continue to obtain effective learning signals. Among them, the state space includes: (1) UAV location information: including the coordinates of the UAV in three-dimensional space (x, y, z); (2) Velocity information: velocity vector of the UAV (linear velocity and angular velocity); (3) Formation information: the difference between the current relative position of the UAVs and the preset formation shape; (4) Environmental perception information: obstacle location, location of other drones or objects, wind direction and speed; (5) Energy status information: the remaining fuel amount of the UAV.

[0034] The reward function includes: (1) Formation stability reward: encourages drones to maintain a stable formation shape and gives negative rewards based on the position error value; (2) Mission Completion Rewards: Based on the objectives of the formation mission (reaching a designated location, performing a specific task), corresponding positive rewards are given; (3) Obstacle avoidance ability reward: A drone that successfully avoids obstacles will be given a positive reward, and a drone that collides with an obstacle will be given a negative reward; (4) Energy efficiency reward: negative rewards are given for fuel consumption.

[0035] S12, building a training framework for a distributed reinforcement learning algorithm based on the reinforcement learning environment, and constructing a network based on the training framework to obtain a neural network including an action decision network and a formation target evaluation network.

[0036] Furthermore, the training framework includes an Actor sampling module, a Learner training module, and a scheduling management and storage module; wherein the Actor sampling module is responsible for interacting with the reinforcement learning environment to collect samples; the Learner training module is responsible for updating the parameters of the neural network; the scheduling management and storage module is responsible for scheduling and managing communications between different nodes, and the scheduling management and storage module includes local scheduling, object storage, and global control storage.

[0037] In step S12, the network is constructed based on the training framework to obtain a neural network including an action decision network and a formation target evaluation network, including: The neural network is constructed by adopting the value function method; wherein the action decision network is implemented based on the DRQN ​​network, the input of the action decision network includes the local observation information of each UAV, and the output includes the value function estimation value of each action obtained by the UAV under the current observation; the formation target evaluation network is implemented based on the hybrid network, the input of the formation target evaluation network includes the local action value function estimation value of each UAV and the current global state information, and the output includes the value function estimation value of the formation target state.

[0038] Reference Figure 2As shown. In this embodiment, the training framework of the distributed reinforcement learning algorithm is designed in a modular way, and its training framework includes: (1) Actor sampling module: responsible for interacting with the reinforcement learning environment to collect samples; (2) Learner training module: responsible for updating the network; (3) Scheduling management and storage module: responsible for scheduling and managing the communication between different nodes (such as sampling nodes and training nodes), which is divided into local scheduling, object storage and global control storage. Each node is equipped with a local scheduler. When the user submits a task, the local scheduler will evaluate the task requirements and decide whether to submit the task to the global controller for distribution to other working nodes for execution; object storage is a memory object storage system that allows efficient communication and data exchange between different nodes; global control storage is responsible for tracking and managing the status of all nodes, the status of tasks, the status of objects, etc. The sampling node simulates or actually controls the drone, performs actions and observes the results (such as flight attitude, position change, reward value, etc.) under specific flight environments and mission requirements. That is, the sampling node is responsible for collecting the interaction data of the drone in the reinforcement learning environment. These observation results and action information are recorded as sample data for subsequent neural network training; the sampling node sends the collected sample data asynchronously to the global controller and stores it in the experience pool. The training node obtains these sample data from the experience pool to update the parameters of the neural network. The architecture of the training framework supports multiple GPU communication backends, including MPI, NCCL and GLOO. By building a distributed reinforcement learning training framework, intelligent formation control of drone clusters is achieved, which improves the stability of the formation and the autonomy of formation transformation.

[0039] Since complex environments may cause communication interference or interruption, this embodiment can distribute the training tasks of the drone cluster to multiple nodes through distributed computing and scheduling management functions, reducing the communication pressure of a single node. In addition, the neural network parameter sharing and gradient synchronization of the distributed training framework can reduce communication redundancy, transmit information in the form of parameters, and achieve smooth real-time communication between drone clusters.

[0040] S13, obtaining a formation model by adjusting the hyperparameters of the neural network and performing distributed training on the neural network using large-scale distributed data.

[0041] In step S13, the formation model is obtained by adjusting the hyperparameters of the neural network and performing distributed training on the neural network using large-scale distributed data, including: S13-1, initializing the action decision network, the formation target evaluation network and the experience pool; S13-2, outputting a formation strategy according to the current state and environmental information of the drone cluster through the action decision network of each sampling node, and sending control instructions to the drone to adjust the flight state and position of the drone, generating large-scale distributed data including samples through the interaction between the Actor sampling module of the drone and the reinforcement learning environment, sending the samples to the global controller in an asynchronous manner and storing them in the experience pool, wherein the samples include reward values, state values ​​and observation values; S13-3, updating the parameters of the neural network, including: in each training round, obtaining batch samples from the experience pool through the training sampling node to calculate and update the parameters, and sending the updated parameters to the sampling node through the global controller to update the parameters of the action decision network; and S13-4, using the TD-Error loss function to update the parameters of the action decision network, and updating the parameters of the formation target evaluation network by minimizing the error between the reward value of the formation target state and the true reward; S13-5, when the formation target state of the drone cluster including flight stability and obstacle avoidance capability is evaluated and does not reach the preset target state, the neural network continues to be trained and the formation strategy is adjusted.

[0042] In this embodiment, the neural network is constructed using the value function method, including an action decision network and a formation target evaluation network. The action decision network is implemented by the DRQN ​​(Deep Recurrent Q-Network) network, which replaces the fully connected layer in the traditional DQN network with a recurrent neural network (such as GRU or LSTM). The recurrent neural network can memorize historical observation information (including local observation information and global state information. Local observation information and global state information are both information in the state space. Local observation information refers to the information set obtained by a single drone based on its own radar perception range, that is, the state details that the drone can directly observe within its local observation perspective (or radar horizon). For example, when the drone can detect an enemy target, its local observation information contains the relative position information of the enemy target. Conversely, if it cannot be directly observed, the corresponding information is missing; and global state information refers to data obtained from a global perspective (or panoramic perspective) that describes the overall state of the system in a global and complete manner. This type of information is not limited by the observation capability of a single drone and can fully reflect the status of all relevant elements in the system, including but not limited to the drone itself, enemy targets, and other factors that may affect task execution.) to process partially observable Markov decision processes (POMDPs). The input of the action decision network is the local observation information of each drone, and the output is the estimated value function of each action obtained by the drone under the current observation (referring to the local observation perspective of the drone itself, not the global perspective). The formation target evaluation network is implemented through a hybrid network (the Mixing network of the QMIX algorithm. The hybrid network is a network structure used in the QMIX algorithm to integrate the local Q value functions of each intelligent agent to obtain the global joint Q value function. It adopts a special design, that is, the weights and biases in the network are dynamically generated by the super network according to the current global state information. This design ensures the monotonic relationship between the joint Q value function and each local Q value function.) The input is the estimated value of the local action value function of each drone and the current global state information (referring to the global state information under the global perspective at the current moment), and the output is the estimated value function of the formation target state.

[0043] Distributed training mainly adjusts the hyperparameters of the neural network and uses large-scale distributed data (referring to the data in the experience pool, including state values, observation values, reward values, etc., that is, all data obtained by interacting with the reinforcement learning environment) for training to achieve intelligent formation control of drone clusters. Figure 3 The specific training process includes: First, initialize the action decision network, formation target evaluation network and experience pool.

[0044] Secondly, collect experience: the action decision network of each sampling node outputs the optimal formation strategy (the higher the estimated score, the better the strategy) according to the current state and environmental information of the drone cluster, and sends control instructions to the drone to adjust the flight attitude and position of the drone. In the process of continuous interaction between the sampling module and the simulation environment (reinforcement learning environment), a large number of samples (including reward values, state values, observation values, etc.) are generated, and the samples are sent to the global controller asynchronously and saved in the experience pool.

[0045] Then update the parameters of the neural network: in each training round, the training sampling node obtains batch samples from the experience pool for calculation and parameter update, and then sends the updated parameters to the sampling node through the global controller to update the parameters of the action decision network. TD-Error (Temporal Difference Error) is used as the loss function to update the parameters of the action decision network, and the parameters of the hybrid network (formation target evaluation network) are updated by minimizing the error between the reward value of the formation target state and the actual reward. Among them, ; Where Q(s,a) is the predicted Q value of taking action a in the current state s, i.e., the estimated value of the value function mentioned above; r is the immediate reward obtained from the environment after executing action a; γ is the discount factor, which is used to balance the importance of immediate rewards and future rewards, and its value range is usually between 0.9 and 0.99; max Q(s t+1 ,a t+1 ) represents the maximum Q value of all possible actions calculated by the network in the next state, that is, the estimated value of the action value function at the next moment.

[0046] Finally, the formation effect (formation target status) of the drone cluster is regularly evaluated, including flight stability and obstacle avoidance capability. If the target status is not reached, the neural network will continue to be trained and the formation strategy will be adjusted.

[0047] In the overall training process, the sampling process is run on the CPU, each sampling node has an independent sampling module and simulation environment, and the training process is run on the GPU, using the Horovod method and the Ring-Allreduce algorithm to achieve efficient computing and improve the efficiency of distributed devices. The distributed learning framework supports hybrid resource simultaneous training, which can maximize the use of CPU and GPU resources and improve the gradient update speed. Through distributed reinforcement learning training, drone clusters can learn the optimal formation strategy, improve autonomous decision-making capabilities and adaptability, and the use of a distributed architecture can reduce communication volume and computational complexity; by continuously iterating and optimizing the formation strategy, drone clusters can maintain a stable formation structure and good flight performance in a complex environment. The training framework of the present invention has good scalability and upgradeability, and can adapt to the formation mission requirements of drone clusters of different sizes and types.

[0048] How to avoid collisions between drones during dense formation flight is an important issue, especially in complex environments, where drones need to control flight trajectories and speeds more accurately to ensure the safety and stability of the formation. However, this embodiment can significantly increase the number of samples by multi-node parallel sampling processing in distributed training, effectively utilize more data to train the model, and thus train a stable model that can effectively avoid collisions, so as to effectively solve the problem of collision avoidance. In addition, complex environments may cause the drone formation to be subject to various external interferences, thereby affecting the stability and consistency of the formation, and there is a problem of maintaining the formation. However, this embodiment uses distributed reinforcement learning calculations, and each drone can perceive the position and state of the surrounding drones in real time based on the information in the state space, and adjust its own flight attitude and speed based on this information to maintain the stability and consistency of the formation.

[0049] S14, loading the formation model into an actual UAV cluster system, and adjusting the formation strategy to perform a formation flight mission according to the real-time flight environment and mission requirements.

[0050] In step S14, the formation model is loaded into the actual drone cluster system, including: The formation model is loaded into an actual drone cluster system for deployment in an executable format or source code form, wherein the executable format includes exporting the computational graph of the formation model into an executable format of a TensorFlow Lite model or a TensorFlow.js model for execution on the client; the source code form includes exporting the computational graph of the formation model into source code in other languages ​​for compilation and execution on the client.

[0051] In this embodiment, the trained formation model is loaded into the actual drone cluster system for deployment and application, and the formation strategy is adjusted autonomously according to environmental changes and mission requirements to perform formation flight missions. The deployment method includes executable format and source code form, where the executable format refers to exporting the calculation graph of the formation model into a certain executable format, such as TensorFlow Lite model or TensorFlow.js model, and then executing it on the client device; the source code form refers to exporting the calculation graph of the formation model into source code in other languages, such as C++ code, and then compiling and executing it on the client.

[0052] Reference Figure 4 Shown is a structural schematic diagram of a drone swarm intelligent formation device based on distributed reinforcement learning provided by an embodiment of the present invention.

[0053] In this embodiment, the device 20 includes: An environment construction unit 21 is used to construct a reinforcement learning environment of a drone cluster including a state space, an action space and a reward function according to a flight mission scenario of the drone, wherein the flight mission scenario includes performing aerial refueling and long-distance formation transportation; A network construction unit 22 is used to build a training framework of a distributed reinforcement learning algorithm based on the reinforcement learning environment, and to construct a network based on the training framework to obtain a neural network including an action decision network and a formation target evaluation network; A network training unit 23, configured to obtain a formation model by adjusting the hyperparameters of the neural network and performing distributed training on the neural network using large-scale distributed data; The formation unit 24 is used to load the formation model into the actual UAV cluster system and adjust the formation strategy according to the real-time flight environment and mission requirements to perform formation flight missions.

[0054] Each unit module of the device 20 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the above corresponding steps for details.

[0055] The embodiment of the present invention also provides a UAV swarm intelligent formation device based on distributed reinforcement learning, the device includes the UAV swarm intelligent formation device based on distributed reinforcement learning as described above, wherein the UAV swarm intelligent formation device based on distributed reinforcement learning can adopt Figure 4 The structure of the embodiment can be executed accordingly. Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0056] The device includes: a mobile phone, a digital camera, a tablet computer, or other devices with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0057] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0058] The input unit can be used to receive input digital or character or image information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment includes not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0059] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel, and optionally, the display panel may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface may cover the display panel, and when the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event, and then the processor provides a corresponding visual output on the display panel according to the type of touch event.

[0060] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not installed in a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1The UAV swarm intelligent formation method based on distributed reinforcement learning is shown. The computer readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0061] The embodiment of the present invention also provides a computer program product, including a computer program / instruction, which is loaded and executed by a processor to implement Figure 1 A distributed reinforcement learning-based intelligent formation method for drone swarms is shown.

[0062] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, equipment embodiment and storage medium embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0063] Furthermore, in this document, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0064] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A method for intelligent formation of drone swarms based on distributed reinforcement learning, characterized in that: The method comprises: Construct a reinforcement learning environment for drone swarms that includes state space, action space, and reward function based on the drone’s flight mission scenarios. The flight mission scenarios include aerial refueling and long-distance formation transportation. Building a training framework for a distributed reinforcement learning algorithm based on the reinforcement learning environment, and constructing a network based on the training framework to obtain a neural network including an action decision network and a formation target evaluation network; The formation model is obtained by adjusting the hyperparameters of the neural network and performing distributed training on the neural network using large-scale distributed data; further comprising: Initializing the action decision network, the formation target assessment network, and the experience pool; The action decision network of each sampling node outputs a formation strategy according to the current state and environmental information of the drone cluster, and sends control instructions to the drone to adjust the flight state and position of the drone. Through the interaction between the Actor sampling module of the drone and the reinforcement learning environment, large-scale distributed data including samples is generated, and the samples are sent to the global controller in an asynchronous manner and stored in the experience pool, wherein the samples include reward values, state values, and observation values; Updating the parameters of the neural network includes: in each training round, obtaining batch samples from the experience pool through the training sampling node to calculate and update the parameters, and sending the updated parameters to the sampling node through the global controller to update the parameters of the action decision network; and, Using the TD-Error loss function to update the parameters of the action decision network, and updating the parameters of the formation target evaluation network by minimizing the error between the reward value of the formation target state and the true reward; The formation model is loaded into an actual UAV cluster system, and the formation strategy is adjusted according to the real-time flight environment and mission requirements to perform formation flight missions.

2. The method for intelligent formation of drone swarm based on distributed reinforcement learning according to claim 1, characterized in that: The reinforcement learning environment of the drone cluster including the state space, action space and reward function is constructed according to the specific application scenario of the drone, including: Defining the state space, including the position information, speed information, formation information, environmental perception information and energy state information of the UAV; The action space is defined, including a transverse rod, a longitudinal rod, a pedal and a throttle rod, and the flight attitude and position of the UAV are adjusted by the transverse rod, the longitudinal rod and the pedal, and the flight speed of the UAV is adjusted by the throttle rod; The reward function is defined, including a formation stability reward, a task completion reward, an obstacle avoidance capability reward, and an energy efficiency reward.

3. The method for intelligent formation of drone swarm based on distributed reinforcement learning according to claim 1, characterized in that: The training framework includes an Actor sampling module, a Learner training module, and a scheduling management and storage module; wherein the Actor sampling module is responsible for interacting with the reinforcement learning environment to collect samples; the Learner training module is responsible for updating the parameters of the neural network; the scheduling management and storage module is responsible for scheduling and managing the communication between different nodes, and the scheduling management and storage module includes local scheduling, object storage, and global control storage.

4. The method for intelligent formation of drone swarm based on distributed reinforcement learning according to claim 1, characterized in that: The network is constructed based on the training framework to obtain a neural network including an action decision network and a formation target evaluation network, including: The neural network is constructed by adopting the value function method; wherein the action decision network is implemented based on the DRQN ​​network, the input of the action decision network includes the local observation information of each UAV, and the output includes the value function estimation value of each action obtained by the UAV under the current observation; the formation target evaluation network is implemented based on the hybrid network, the input of the formation target evaluation network includes the local action value function estimation value of each UAV and the current global state information, and the output includes the value function estimation value of the formation target state.

5. The method for intelligent formation of drone swarm based on distributed reinforcement learning according to claim 1, characterized in that: The formation model is obtained by adjusting the hyperparameters of the neural network and performing distributed training on the neural network using large-scale distributed data, including: When the formation target state of the drone cluster, including flight stability and obstacle avoidance capability, does not reach the preset target state, the neural network continues to be trained and the formation strategy is adjusted.

6. The method for intelligent formation of drone swarm based on distributed reinforcement learning according to claim 1, characterized in that: The step of loading the formation model into an actual drone cluster system includes: The formation model is loaded into an actual drone cluster system for deployment in an executable format or source code form, wherein the executable format includes exporting the computational graph of the formation model into an executable format of a TensorFlow Lite model or a TensorFlow.js model for execution on the client; the source code form includes exporting the computational graph of the formation model into source code in other languages ​​for compilation and execution on the client.

7. A UAV swarm intelligent formation device based on distributed reinforcement learning, characterized in that: The device comprises: An environment construction unit, used to construct a reinforcement learning environment for a drone cluster including a state space, an action space, and a reward function according to a flight mission scenario of the drone, wherein the flight mission scenario includes performing aerial refueling and long-distance formation transportation; A network construction unit, used to build a training framework of a distributed reinforcement learning algorithm based on the reinforcement learning environment, and to construct a network based on the training framework to obtain a neural network including an action decision network and a formation target evaluation network; A network training unit, used to obtain a formation model by adjusting the hyperparameters of the neural network and performing distributed training on the neural network using large-scale distributed data; further comprising: Initializing the action decision network, the formation target assessment network, and the experience pool; The action decision network of each sampling node outputs a formation strategy according to the current state and environmental information of the drone cluster, and sends control instructions to the drone to adjust the flight state and position of the drone. Through the interaction between the Actor sampling module of the drone and the reinforcement learning environment, large-scale distributed data including samples is generated, and the samples are sent to the global controller in an asynchronous manner and stored in the experience pool, wherein the samples include reward values, state values, and observation values; Updating the parameters of the neural network includes: in each training round, obtaining batch samples from the experience pool through the training sampling node to calculate and update the parameters, and sending the updated parameters to the sampling node through the global controller to update the parameters of the action decision network; and, Using the TD-Error loss function to update the parameters of the action decision network, and updating the parameters of the formation target evaluation network by minimizing the error between the reward value of the formation target state and the true reward; The formation unit is used to load the formation model into the actual UAV cluster system and adjust the formation strategy to perform formation flight missions according to the real-time flight environment and mission requirements.

8. A drone swarm intelligent formation device based on distributed reinforcement learning, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a drone swarm intelligent formation method based on distributed reinforcement learning as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program / instructions, wherein the computer program / instructions are executed by a processor to implement the steps of a drone swarm intelligent formation method based on distributed reinforcement learning as described in any one of claims 1 to 6.

Images