A method and device for cooperative control of multiple unmanned aerial vehicles

Abstract

The present invention relates to the field of drone control technology, and specifically to a multi-drone collaborative control method and device, the method comprising: defining the observation space and action space of each drone; constructing a multi-drone collaborative control network framework based on the observation space and action space; constructing a control strategy for each drone in the multi-drone collaborative control network framework, including an executor network and an evaluator network; defining an objective function based on the executor network and the evaluator network; constructing a strategy network model, and training the strategy network model; outputting the flight actions of each drone through the trained strategy network model to control each drone. The above scheme enables each drone to operate in a dynamic environment, can effectively identify obstacles or drones in the same array, enhances the autonomy and flexibility of drone operations, and thus maximizes the collaborative operation capability of the drone formation.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle control, and in particular to a method and device for cooperatively controlling multiple unmanned aerial vehicles. Background Art

[0002] In complex air or ground environments, multi-UAV systems need to work together efficiently and accurately to complete tasks such as target tracking, environmental monitoring, and material delivery.

[0003] In the field of UAV cooperative control and path planning, the design of the algorithm directly affects the efficiency, safety and coordination of the UAV formation when performing tasks. The multi-UAV path planning algorithm needs to be able to flexibly and effectively plan paths in dynamic and complex environments while avoiding collisions with other UAVs or obstacles. At present, in the existing multi-UAV path planning algorithms, UAVs usually need to search for navigation paths in a wider search area, or can only perform tasks under a pre-designed perception network.

[0004] The above solutions are usually difficult to adapt to dynamic and complex environmental changes, and cannot effectively identify obstacles or drones in the same array, thus causing mutual interference and failing to maximize the collaborative operation capabilities of the drone formation. Summary of the invention

[0005] In view of this, the present invention provides a multi-UAV collaborative control method and device to solve the problem that existing solutions are difficult to adapt to dynamic and complex environmental changes, and cannot effectively identify obstacles or UAVs in the same array, thereby causing mutual interference and failing to maximize the collaborative operation capabilities of the UAV formation.

[0006] In a first aspect, the present invention provides a multi-UAV cooperative control method, the method comprising:

[0007] Define the observation space and action space of each drone;

[0008] Based on the observation space and the action space, a multi-UAV cooperative control network framework is constructed; the control strategy of each UAV in the multi-UAV cooperative control network framework includes an executor network and an evaluator network;

[0009] Based on the executor network and the evaluator network, an objective function is defined;

[0010] Constructing a policy network model, and training the policy network model based on the objective function;

[0011] The flight actions of the various UAVs are outputted through the trained strategy network model to coordinate and control the various UAVs.

[0012] The above scheme can effectively realize information sharing and decision-making coordination among multiple drones by defining the observation space and action space of each drone and constructing a multi-drone collaborative control network framework based on the executor network and the evaluator network, greatly improving the collaborative efficiency and operational capability of the drone swarm when performing tasks under dynamic and complex environmental changes, so that the drone swarm can complete the established goals more quickly and accurately; the executor network is responsible for generating control instructions based on the currently observed environmental information, while the evaluator network evaluates the pros and cons of these control instructions and feeds them back to the executor network for optimization. Through the reinforcement learning mechanism based on deep learning, the drone can learn and optimize its control strategy through continuous trial and error, and achieve real results. The adaptive adjustment to complex environments improves the robustness and intelligence of the system; by constructing a clear objective function and training the policy network model based on the objective function, it can ensure that the drone swarm can follow the optimal decision-making path when working collaboratively, reduce redundant operations and conflicts, and thus significantly improve the quality and efficiency of task completion; through collaborative control and intelligent decision-making, the drone swarm can better avoid potential risks and conflicts and ensure safe flight in complex environments. At the same time, the mutual backup and collaborative support between multiple drones also improve the reliability and stability of the entire system, reduce the risk of mission failure due to the failure of a single drone, and maximize the collaborative operation capabilities of the drone formation.

[0013] In an optional implementation, the observation space is a set of finite state information obtained by the drone through sensors, and the action space is a set of all flight actions taken by the drone.

[0014] In an optional implementation, the step of defining the observation space and action space of each drone includes:

[0015] Acquiring environmental information of the corresponding drone through sensors of each drone;

[0016] Separating the environmental information into different channels to form a multi-channel matrix;

[0017] Based on the multi-channel matrix, obtaining local observations of each UAV to generate an observation space;

[0018] Discretize the action space of each drone and obtain a discretization result;

[0019] The discretization processing result is converted into discrete maneuver control commands to generate an action space.

[0020] The above solution integrates accurate environmental perception and discrete action space, allowing multiple drones to work together better. Each drone has a clear understanding of its own local environment and can jointly achieve complex tasks through a collaborative control network framework.

[0021] In an optional implementation, the multi-UAV cooperative control network framework is constructed based on the observation space and the action space, including:

[0022] Based on the observation space and action space, each UAV is equipped with an executor network and an evaluator network to construct a multi-UAV collaborative control network framework; the executor network is used to receive the local observation as input and output discretized actions; the evaluator network is used to receive the flight actions of the executor network and the states of neighboring UAVs, and output a value function to adjust the executor network.

[0023] The above scheme is a multi-UAV collaborative control network framework constructed based on observation space and action space. Through the synergy of executor network and evaluator network, it effectively improves the collaborative decision-making ability, adaptability and efficiency of the multi-UAV system, and is suitable for application scenarios in various complex tasks and environments.

[0024] In an optional implementation, the objective function is defined based on the executor network and the evaluator network, including:

[0025] Learning an approximate value of the value function through the evaluator network, and minimizing the temporal difference error as the learning goal of the evaluator network;

[0026] Obtaining rewards and penalties for each drone from the environmental information, and taking the cumulative maximization of the rewards and penalties as the learning goal of the executor network;

[0027] The objective function is constructed by combining the learning objectives of the evaluator network and the learning objectives of the executor network.

[0028] By defining and optimizing a comprehensive objective function, the above scheme can realize efficient and intelligent task execution in various complex scenarios and achieve better overall performance through the multi-UAV collaborative control system based on the executor network and the evaluator network.

[0029] In an optional implementation, the constructing of the policy network model includes:

[0030] The multi-channel matrix of each drone and the state of the neighboring drone are used as the input of the policy network model, and the action probability distribution of each drone is used as the output of the policy network model to construct a policy network model; the structure of the policy network model includes a convolutional layer, a fully connected layer and an xLSTM unit.

[0031] The policy network model in the above scheme can effectively integrate the perception information of each drone and the status of neighboring drones, thereby improving the entire system's ability to understand and respond to the environment. This information sharing helps each drone make more intelligent and coordinated decisions; through the output of action probability distribution, the policy network enables drones to adjust their actions in real time and perform collaborative operations based on the dynamic status of neighboring drones, which can effectively reduce collision risks and improve task execution efficiency.

[0032] In an optional implementation, the training of the policy network model based on the objective function includes:

[0033] Initializing model parameters of the strategy network model; the model parameters include evaluator network parameters and executor network parameters;

[0034] Generate flight actions of each drone through an executor network of each drone according to the current state of each drone and environmental noise;

[0035] Control each drone to actively interact with the environment according to the flight actions generated by the strategy network, and obtain rewards, penalties and the next state;

[0036] The current state, the flight action, the reward and punishment, and the next state are stored as experience in the experience pool of the multi-UAV cooperative control network framework;

[0037] Randomly sampling samples from the experience pool, and training the policy network model based on the sampled samples;

[0038] Obtaining the temporal differential error and policy gradient during the training process, and updating the evaluator network parameters according to the temporal differential error, and updating the executor network parameters according to the policy gradient;

[0039] After updating the network parameters, continue to train the policy network model to optimize the objective function, and obtain the trained policy network model.

[0040] Through experience replay and parameter update, the above scheme enables the policy network model to learn the optimal strategy in various situations, thereby improving the robustness and adaptability of the system in complex environments; the policy network model can guide the UAV to adjust its flight actions according to real-time needs and environmental conditions, thereby optimizing resource utilization and task execution efficiency. Moreover, during the training process of the policy network model, the UAVs share experience and influence each other through the experience pool, which can enhance their collaborative work ability, reduce conflicts and improve task coordination.

[0041] In a second aspect, the present invention provides a multi-UAV cooperative control device, the device comprising:

[0042] The space definition module is used to define the observation space and action space of each drone;

[0043] A network framework construction module is used to construct a multi-UAV cooperative control network framework based on the observation space and the action space; the control strategy of each UAV in the multi-UAV cooperative control network framework includes an executor network and an evaluator network;

[0044] An objective function definition module, used to define an objective function based on the executor network and the evaluator network;

[0045] A model training module, used to construct a policy network model and train the policy network model based on the objective function;

[0046] The coordination control module is used to output the flight actions of each UAV through the trained strategy network model to coordinate and control each UAV.

[0047] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute a multi-UAV collaborative control method according to the first aspect or any corresponding embodiment thereof.

[0048] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute a multi-UAV collaborative control method according to the first aspect or any corresponding embodiment thereof.

[0049] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute a multi-UAV collaborative control method according to the first aspect or any corresponding embodiment thereof.

[0050] The technical solution provided by the present invention may include the following beneficial effects:

[0051] The present invention defines the observation space and action space of each drone and constructs a multi-drone collaborative control network framework based on the executor network and the evaluator network, which can effectively realize information sharing and decision-making coordination among multiple drones, greatly improving the collaborative efficiency and operational capability of the drone group when performing tasks under dynamic and complex environmental changes, so that the drone group can complete the established goals more quickly and accurately; the executor network generates control instructions based on the currently observed environmental information, while the evaluator network evaluates the quality of the control instructions and feeds them back to the executor network for optimization. Through the reinforcement learning mechanism based on deep learning, the drone can learn and optimize its control strategy in continuous trial and error, and achieve complex Adaptive adjustment of the environment improves the robustness and intelligence of the system; the present invention also constructs a clear objective function and trains the policy network model based on the objective function, which can ensure that the drone swarm can follow the optimal decision-making path when working collaboratively, reduce redundant operations and conflicts, and thus significantly improve the quality and efficiency of task completion; through collaborative control and intelligent decision-making, the drone swarm can better avoid potential risks and conflicts and ensure safe flight in complex environments. At the same time, the mutual backup and collaborative support between multiple drones also improve the reliability and stability of the entire system, reduce the risk of mission failure due to the failure of a single drone, and maximize the collaborative operation capability of the drone formation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 It is a schematic diagram of reinforcement learning of related technologies;

[0054] Figure 2 is a flow chart of a multi-UAV cooperative control method according to an embodiment of the present invention;

[0055] Figure 3 is a flow chart of another multi-UAV cooperative control method according to an embodiment of the present invention;

[0056] Figure 4 2 is a schematic diagram of a multi-UAV cooperative control network framework according to an embodiment of the present invention;

[0057] Figure 5 is a schematic diagram of the structure of a policy network model according to an embodiment of the present invention;

[0058] Figure 6 is a structural block diagram of a multi-UAV cooperative control device according to an embodiment of the present invention;

[0059] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0061] It should be noted that the present invention uses the deep deterministic policy gradient algorithm (DDPG) to construct a collaborative control algorithm, combines it with a distributed execution framework for solving, and realizes algorithm training and evaluation of multiple UAVs. It mainly expands the use of a multi-agent actor-evaluator (AC) algorithm, combining value-based and policy-based algorithms, where the actor represents the policy function for generating actions, and the evaluator represents the value approximator for evaluating action rewards. In terms of network design, a deep reinforcement learning network is used to optimize the path planning of multiple UAVs, mainly to improve the navigation efficiency of UAVs in complex environments and enhance the collaborative operation capabilities of multiple UAVs.

[0062] Among them, reinforcement learning is a method of training machine learning models to make a series of decisions through trial and error in a dynamic environment. Figure 1 A schematic diagram of reinforcement learning of related technologies is shown, such as Figure 1 As shown in Figure 1, the agent needs to learn to achieve goals by programming them with rewards or penalties in an uncertain and potentially complex environment. The robot selects actions as outputs, recognizes the environment, and receives the state of the environment. The state of the behavior is changed, and the state transition is transmitted to the agent as a reinforcement signal.

[0063] According to an embodiment of the present invention, an embodiment of a multi-UAV cooperative control method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0064] In this embodiment, a multi-UAV cooperative control method is provided, which can be used for a multi-UAV cooperative control system. Figure 2is a flow chart of a multi-UAV cooperative control method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0065] Step S201, defining the observation space and action space of each drone.

[0066] Furthermore, each drone has its own observation space to obtain environmental status information, such as position, speed, and the position of surrounding drones; at the same time, each drone also has its own action space, including flight speed, steering angle, and other possible actions. In order for each drone to determine the environmental state it can perceive (observation space) and all possible actions it can perform (action space), this embodiment first defines the observation space and action space of each drone. The observation space is a collection of environmental information that the drone can perceive, including the drone's own state (such as position, speed, attitude, etc.) and information about the surrounding environment (such as obstacle position, state of other drones, etc.). The action space is a collection of all possible actions that a drone can take, such as changing speed, adjusting direction, ascending or descending, etc.

[0067] Step S202: construct a multi-UAV cooperative control network framework based on the observation space and the action space; the control strategy of each UAV in the multi-UAV cooperative control network framework includes an executor network and an evaluator network.

[0068] Furthermore, this embodiment constructs a network framework suitable for multi-UAV cooperative control based on the observation space and action space defined in the previous step. The multi-UAV cooperative control network framework may include multiple sub-networks, each sub-network corresponds to a control strategy of a UAV, and each sub-network is interconnected to achieve cooperative control. The multi-UAV cooperative control network framework is an overall control framework for coordinating and managing the behavior of multiple UAVs, which can be distributed, allowing each UAV to make decisions based on its own observations.

[0069] In this embodiment, each drone is equipped with two key neural networks to achieve intelligent decision-making. The actor network is used to generate action decisions, that is, it is responsible for outputting the drone's actions based on the currently observed environmental information; the critic network is used to evaluate the potential value of the action selected by the actor network, that is, to predict the cumulative reward that the action can obtain in the future, so as to evaluate the quality of the action selected by the actor network, thereby helping the actor network improve its decision-making.

[0070] Step S203: defining an objective function based on the executor network and the evaluator network.

[0071] Furthermore, the objective function in this embodiment defines the goal of UAV collaborative control, and the objective function is used to guide the learning process of the executor network and the evaluator network, ensuring the correct optimization of the executor network and the evaluator network to guide and evaluate the behavior strategy of the UAV.

[0072] Step S204: construct a policy network model, and train the policy network model based on the objective function.

[0073] Furthermore, this embodiment uses the executor network and the evaluator network to construct an overall policy network model, and then trains the model by optimizing the objective function so that it can adapt to different environments and task requirements.

[0074] Step S205: outputting the flight actions of each UAV through the trained strategy network model to coordinate and control each UAV.

[0075] Furthermore, the trained policy network model will output the flight actions of each drone, which are the results of optimization based on the current environment and objective function. In this way, each drone can perform tasks under collaborative control to achieve overall coordination and efficiency of the system. In other words, after training, the policy network model can output the flight actions of the drone based on the current observation information. These actions are executed by the drone's control system, thereby achieving coordinated control of the drone. Since the control strategies of multiple drones are interrelated, each drone can work together to complete the task together.

[0076] In summary, this embodiment defines the observation space and action space of each drone and constructs a multi-drone collaborative control network framework based on the executor network and the evaluator network, which can effectively realize information sharing and decision-making coordination among multiple drones, greatly improve the collaborative efficiency and operational capability of the drone swarm when performing tasks under dynamic and complex environmental changes, and enable the drone swarm to complete the established goals more quickly and accurately; the executor network generates control instructions based on the currently observed environmental information, while the evaluator network evaluates the pros and cons of the control instructions and feeds them back to the executor network for optimization. Through the reinforcement learning mechanism based on deep learning, the drone can learn and optimize its control strategy through continuous trial and error, and achieve Adaptive adjustment in complex environments improves the robustness and intelligence of the system; this embodiment also constructs a clear objective function and trains the policy network model based on the objective function, which can ensure that the drone swarm can follow the optimal decision-making path when working collaboratively, reduce redundant operations and conflicts, and thus significantly improve the quality and efficiency of task completion; through collaborative control and intelligent decision-making, the drone swarm can better avoid potential risks and conflicts and ensure safe flight in complex environments. At the same time, the mutual backup and collaborative support between multiple drones also improve the reliability and stability of the entire system, reduce the risk of mission failure due to the failure of a single drone, and maximize the collaborative operation capabilities of the drone formation.

[0077] In this embodiment, another multi-UAV cooperative control method is provided, which can be used in a multi-UAV cooperative control system. Figure 3 is a flow chart of another multi-UAV cooperative control method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0078] Step S301, defining the observation space and action space of each drone. The observation space is a set of finite state information obtained by the drone through sensors, and the action space is a set of all flight actions taken by the drone.

[0079] In an optional implementation, step S301 includes:

[0080] Acquire environmental information of the corresponding drone through the sensors of the respective drones;

[0081] Separate the environmental information into different channels to form a multi-channel matrix;

[0082] Based on the multi-channel matrix, local observations of each UAV are obtained to generate an observation space;

[0083] Discretize the action space of each UAV and obtain the discretization result;

[0084] The discretization results are converted into discrete maneuver control commands to generate an action space.

[0085] Furthermore, before defining the observation space and action space of each drone, this embodiment first defines the following symbolic representation:

[0086] : The i-th drone, 0≦i≦ N ;

[0087] State Space( S ): the set of all possible drone states, S’ Indicates updated S .

[0088] Environmental transfer matrix ( T ): The transition probability that describes how the drone state changes over time.

[0089] Action Space( A ): The set of actions that each drone can take.

[0090] Observation space ( O ): A collection of finite state information obtained by the drone through sensors.

[0091] Strategy( ): The action selection strategy adopted by each UAV is a deterministic strategy based on local observations.

[0092] Reward function ( R ): The reward each drone gets for each state and action combination gather.

[0093] Initial state distribution ( ): describes the probability distribution of the initial state of the drone.

[0094] The expected revenue per drone is:

[0095] ;

[0096] in, Indicates i The expected revenue of a drone is represents the discount factor, express t The discount factor at the time, t Indicates the time, Indicates i A drone in t The rewards of the moment, M Represents the total time series.

[0097] Value Function Q : A function used by reinforcement learning to measure the value of an action in the decision-making process, expressing the expected value of the final reward of the action under a specific environmental state. Q’ is the updated value function Q .

[0098] TD error: Temporal Difference Error, used to measure the difference between the actual reward received and the expected reward received.

[0099] Furthermore, after defining the symbolic representation, this embodiment also needs to define the observation space and action space of each drone. First, the drone obtains environmental information through sensors such as cameras. In the limited field of view of the drone, the available information is separated into different channels to simplify the learning task of the drone. The set of all possible actions that the drone can take is the action space. For a continuous action space such as a drone, the action space is too large and the high-dimensional continuous optimization requires an exponential growth in the memory space required to store the action values. Therefore, the continuous action is discretized and converted into discrete maneuver control. This implementation uses the drone's motion direction and flight attitude as the origin of the relative coordinate system to always keep the drone flying at a safe speed.

[0100] When defining the observation space, each drone is equipped with corresponding sensors to capture relevant information about its surrounding environment, such as position, speed, position of surrounding objects, etc., and the environmental information obtained by the sensor is separated into different channels to form a multi-channel matrix (such as a four-channel matrix). Each channel may correspond to a specific type of environmental information, such as position coordinates, speed, angle, information about surrounding objects, etc. Then, based on the multi-channel matrix, each drone can generate its local observation, which contains the environmental information of the drone at the current time step and reflects the environmental conditions around it.

[0101] When defining the action space, this embodiment discretizes the action space of each drone, thereby dividing the continuous action space into a finite number of discrete actions for the convenience of reinforcement learning algorithm processing and optimization. The result of the discretization process is a finite set, which includes discrete actions that the drone can choose. For example, including turning left, turning right, accelerating, decelerating, etc., and then mapping the results of the discretization process to specific control commands, namely discrete maneuver control commands. These control commands are specific action instructions executed by the drone, which can directly affect its movement and behavior in the environment.

[0102] Step S302: Based on the observation space and the action space, each UAV is equipped with an executor network and an evaluator network to construct a multi-UAV collaborative control network framework.

[0103] In an optional embodiment, the control strategy of each UAV in the multi-UAV collaborative control network framework includes an executor network and an evaluator network, wherein the executor network is used to receive the local observation as input and output a discretized action; the evaluator network is used to receive the flight action of the executor network and the status of neighboring UAVs, and output a value function to adjust the executor network.

[0104] Furthermore, in the entire multi-UAV system, each UAV independently runs an actor network and an evaluator network. These networks together form a multi-UAV collaborative control framework, which enables each UAV to make appropriate action decisions based on local observations and obtain feedback information about its action selection through the evaluator network. The output of the evaluator network is not only used to evaluate the effect of the current action, but also can be used as a feedback signal to improve the action selection strategy by adjusting the actor network. This feedback mechanism can help UAVs learn and optimize their behavior in continuous interaction to achieve better overall collaborative control effects.

[0105] For further information, see Figure 4 The schematic diagram of the multi-UAV cooperative control network framework is shown in the figure. The deep reinforcement learning strategy of this embodiment enables UAVs to operate in a dynamic environment, enhancing the autonomy and flexibility of UAV operations. This is mainly reflected in the structure of the multi-UAV cooperative control network framework. The control of each UAV is composed of an executor network and an evaluator network. The relevant parameters of the network can be dynamically adjusted and updated by environmental parameters and experience pools, so that the UAV has better environmental adaptability. Among them, the executor network and the evaluator network together constitute the control network of the UAV, which is equivalent to Figure 1 The agent in the network interacts with the environment to learn "experience". The executor network is used to accept updates from the evaluator network and output actions to the environment, such as flying at a certain speed and pitch angle. The experience pool is a reinforcement learning playback buffer. When the policy network is trained, a batch of experiences will be randomly drawn from this buffer to update the network. The parameters of the target executor and evaluator networks will slowly track the parameters of the corresponding networks, which are used to calculate the stable target of the expected return, further stabilizing the training process of the policy network.

[0106] Figure 4 middle, Indicates flying action. Indicates the updated flight action, that is, the updated , j for i arrive N i Integers between express and A flying action between Indicates flight status. Indicates updated , that is, the updated flight status, ( S,A,R,S' ) represents the Markov decision process in reinforcement learning, which is a four-tuple. Figure 4 As shown, each drone has an equal status and senses the surrounding environment and the status of other drones through sensors. Contains i drones Parameters , the actor network is based on the first i drones Status As input, i drones Deterministic discretization of actions is the output. The evaluator network is used to calculate the value function ,in, Indicates i drones The state set matrix of all drones in the neighborhood of Indicates that the parameter is To the input state The set of actions obtained by the target actor network, Indicates i drones The strategies of all neighboring drones, represents a set of action strategies, represents the local strategy, i.e. i drones The strategy for taking a certain action, for The evaluator network takes the flight actions of the performer network and the states of all neighboring drones obtained from the environment as input, and outputs the value function value to the drone to adjust the performer network.

[0107] Step S303: learning the approximate value of the value function through the evaluator network, and minimizing the time difference error is used as the learning goal of the evaluator network.

[0108] Furthermore, the evaluator network is a neural network used to approximately estimate the future cumulative rewards and penalties (also called value function) under a given state or a given state-action pair. The TD error is used to measure the difference between the predicted value of the evaluator network and the actual rewards and penalties. By minimizing this difference (i.e., the temporal difference error), the evaluator network can more accurately predict future rewards and penalties, thereby optimizing the approximation of its value function. The learning goal of the evaluator network is to adjust its network parameters to minimize the temporal difference error.

[0109] Step S304, obtaining rewards and penalties for each drone from the environmental information, and taking the cumulative maximization of the rewards and penalties as the learning goal of the executor network.

[0110] Furthermore, environmental information is information about the surrounding environment and its own state obtained by the drone through its sensors, which is used to evaluate the effect of the drone's behavior and give rewards and punishments accordingly. Rewards and punishments are an immediate feedback from the environment on the drone's behavior, which are used to guide the drone's learning process, including positive rewards and punishments (indicating that the behavior is encouraged) or negative rewards and punishments (indicating that the behavior is punished). The actor network is a neural network that is used to generate the drone's actions based on the current observation information (i.e., the result of the drone's processing of environmental information). The learning goal of the actor network is to adjust its network parameters to maximize the cumulative value of rewards and punishments obtained from the environment, so that the drone can obtain as many positive rewards and punishments as possible and as few negative rewards and punishments as possible in the long run.

[0111] Step S305: Combine the learning objectives of the evaluator network and the executor network to construct an objective function, which includes the prediction error of the value function and the reward and punishment of the flight strategy selection.

[0112] Furthermore, the evaluator network learns how to more accurately predict future rewards and punishments by minimizing the temporal difference error, while the executor network uses the value estimate (or gradient information) provided by the evaluator network to adjust its policy parameters to maximize the cumulative value of rewards and punishments. The two processes are parallel and interdependent. The final objective function is a function that comprehensively considers the learning objectives of the evaluator network and the executor network. The evaluator network provides a long-term effect evaluation of the action, and the executor network selects the action based on the current environmental information and reward and punishment signals. In this way, the system can adjust the behavior of the drone in real time in a dynamic environment to optimize the performance and efficiency of the overall system.

[0113] Furthermore, the critic network is used to learn the value function The learning goal is to minimize the time difference error, and the time difference error TD is defined as:

[0114] ;

[0115] ;

[0116] in, represents the approximate action-value function, represents the TD error, Represents the updated value function.

[0117] No. i drones After performing an action, you get rewards and punishments from the environment , define rewards and penalties for:

[0118] ;

[0119] in, Indicates the target position of the drone, It means t The first i drones status.

[0120] Updated value function Predicted by the target evaluator network, For the updated i drones The state set matrix of all drones in the neighborhood of The updated parameters are The updated input state is The target actor network obtains a set of actions, parameters is the discount factor.

[0121] The loss of the critic network can be described as , by minimizing the loss of the critic network, the parameters are updated as The reviewer network:

[0122] ;

[0123] in, Indicates that the parameter is The network of evaluators, Represents the expectation of the value of the action in that state, that is, the state-value function.

[0124] Executor Network It is a continuous deterministic strategy that aims to maximize the cumulative rewards and penalties. Its expression is as follows:

[0125] ;

[0126] in, Indicates cumulative rewards and penalties,G represents a communication link diagram, G i surface Shidi i drones Communication link diagram.

[0127] The expression of policy gradient is as follows:

[0128] ;

[0129] in, represents the global policy gradient, Expressed as S i The action strategy in the state, represents the action function gradient, Represents policy gradient and action value function The outer product of the gradient.

[0130] Step S306, taking the multi-channel matrix of each UAV and the state of the neighboring UAV as the input of the policy network model, taking the action probability distribution of each UAV as the output of the policy network model, and constructing a policy network model.

[0131] In an optional implementation, the structure of the policy network model includes a convolutional layer, a fully connected layer, and an xLSTM unit.

[0132] For further information, see Figure 5 The structural diagram of the strategy network model is shown. In this embodiment, a deep neural network is used to output the motion control strategy of the UAV. The strategy neural network has two inputs: a behavior matrix and a neighboring UAV state matrix, which are preprocessed before being connected to the neural network (the behavior matrix and the neighboring UAV state matrix represent local observations, and the preprocessed behavior matrix is ​​a four-channel 10×10×4 tensor), as shown in FIG. Figure 5 As shown in the figure, the four-channel matrix (10×10×4 tensor) representing the local observation is passed through three convolutions and two stages of maximum pooling before being fed into the last convolution layer. At the same time, the vector and value of the target position are passed through a fully connected layer. Then, the concatenation of these two preprocessed inputs passes through two fully connected layers and is finally fed into the xLSTM unit. In ReLU (rectified linear unit), the residual structure connects the output of the connection layer to the input layer of the xLSTM. The output layer consists of a policy neuron with a softmax transfer function (softmax) activation and a feature layer with value output. xLSTM is an extension of the LSTM algorithm, introducing exponential gating and parallelizable matrix memory structures, integrating sLSTM and mLSTM into the residual block, and building its architecture through stacking to support large-scale parallel processing.

[0133] That is to say, in this embodiment, for each drone, a multi-channel matrix is ​​used to represent its local observation. The multi-channel matrix is ​​a tensor of size 10×10×4, which represents the environmental state around the drone. The state of neighboring drones contains information such as relative position and speed with other drones. In view of the design of the strategy network model structure, this embodiment accepts the multi-channel matrix of each drone and the state of its neighboring drones at the input layer, uses convolutional layers and pooling layers to extract features from the input, and sends the output of the convolutional layer to the fully connected layer for further processing and conversion of features. Each node in the output layer corresponds to a drone, and each node outputs the action probability distribution of the drone. The activation function used in this embodiment is softmax, which converts the output into an action probability distribution, which is suitable for multi-classification tasks.

[0134] Step S307: training the policy network model based on the objective function.

[0135] In an optional implementation, step S307 includes:

[0136] Initialize the model parameters of the strategy network model; the model parameters include the evaluator network parameters and the executor network parameters;

[0137] Generate flight actions of each drone through an executor network of each drone according to the current state of each drone and the environmental noise;

[0138] Control each drone to actively interact with the environment according to the flight actions generated by the strategy network, and obtain rewards, penalties and the next state;

[0139] The current state, the flight action, the reward and punishment, and the next state are stored as experience in the experience pool of the multi-UAV cooperative control network framework;

[0140] Randomly sampling samples from the experience pool, and training the policy network model based on the sampled samples;

[0141] Obtaining the time difference error and the policy gradient during the training process, and updating the evaluator network parameters according to the time difference error, and updating the executor network parameters according to the policy gradient;

[0142] After updating the network parameters, continue to train the policy network model to optimize the objective function and obtain the trained policy network model.

[0143] Furthermore, this embodiment first initializes the hyperparameters in the policy network model, puts the N drones in a set of random states, and enters the following simulation cycle:

[0144] The executor network generates drone actions for thei drones , selects the execution action from the current executor network by the following formula:

[0145] ;

[0146] in, represents artificial environmental noise, Indicates i drones exist t Execution action at the moment.

[0147] This embodiment uses Rayleigh noise, which obeys the following action probability distribution:

[0148] ;

[0149] in, , is a positive integer, , , represents the action probability distribution.

[0150] Next, the flight action interacts with the environment and executes a set of flight actions obtained in the previous step , all drones interact with the environment and obtain their respective rewards r i and the next state from the environment S’ , and store playback experience to the experience pool; update the drone status and train data sampling, each drone randomly samples experience samples from the experience pool.

[0151] This embodiment updates the parameters by minimizing the loss function defined by the formula , update the parameters by sampling policy gradients through the formula ;

[0152] Update the target executor network parameters and target evaluator network parameters through the following formula:

[0153] ;

[0154] ;

[0155] in, represents the updated target executor network parameters, represents the soft update coefficient, represents the target executor network parameters, represents the updated target evaluator network parameters, Represents the target executor network parameters.

[0156] The process repeats M rounds until the end, and a strategy network that can be used for the coordinated control of multiple UAVs is obtained.

[0157] That is to say, in the training process of this embodiment, the hyper parameters in the strategy network are first initialized so that the N drones are in random initial states. i drones , flight actions are selected from the current actor network, which are generated by adding artificial environmental noise that obeys Rayleigh noise. After that, this embodiment executes the selected action, each drone interacts with the environment, and obtains the corresponding reward and next state, and stores the experience of each drone (including current state, action, reward and next state) in the experience pool for subsequent training, and then updates the state of each drone to the new state to prepare for the next action selection.

[0158] This embodiment randomly samples experience samples from the experience pool to obtain training data for each drone, and uses the sampled experience samples to calculate the temporal difference error for evaluating the performance of the strategy. By minimizing the defined loss function, the parameters of the evaluator network are updated to optimize the estimation of each state value function or action value function, and the sampled policy gradient is used to update the parameters of the executor network to optimize the strategy for generating actions.

[0159] This embodiment updates the parameters of the target executor network and the target evaluator network, and repeats the above steps M times until the end condition of the training is reached. Through this process, the policy network model is continuously optimized, and finally an effective policy for multi-UAV collaborative control is generated, so that each UAV can coordinate movement and decision-making in a complex environment.

[0160] Step S308: output the flight actions of each UAV through the trained strategy network model to coordinate and control each UAV.

[0161] Furthermore, for each drone, this embodiment obtains its current state information, such as position, speed, direction, etc., uses the trained executor network to generate actions for each drone, and applies the generated actions to each drone to achieve specific flight missions or collaborative actions, and monitor the real-time status of each drone and the effect of the executed actions. As needed, it can be fed back to the policy network for dynamic adjustment and optimization to adapt to environmental changes or mission requirements. Throughout the process, ensure that the actions of each drone are coordinated with each other. This may include control strategies in multiple aspects such as obstacle avoidance, path planning, and fixed formations to achieve the expected collective behavior.

[0162] In summary, this embodiment defines the observation space and action space of each drone and constructs a multi-drone collaborative control network framework based on the executor network and the evaluator network, which can effectively realize information sharing and decision-making coordination among multiple drones, greatly improve the collaborative efficiency and operational capability of the drone swarm when performing tasks under dynamic and complex environmental changes, and enable the drone swarm to complete the established goals more quickly and accurately; the executor network generates control instructions based on the currently observed environmental information, while the evaluator network evaluates the pros and cons of the control instructions and feeds them back to the executor network for optimization. Through the reinforcement learning mechanism based on deep learning, the drone can learn and optimize its control strategy through continuous trial and error, and achieve Adaptive adjustment in complex environments improves the robustness and intelligence of the system; this embodiment also constructs a clear objective function and trains the policy network model based on the objective function, which can ensure that the drone swarm can follow the optimal decision-making path when working collaboratively, reduce redundant operations and conflicts, and thus significantly improve the quality and efficiency of task completion; through collaborative control and intelligent decision-making, the drone swarm can better avoid potential risks and conflicts and ensure safe flight in complex environments. At the same time, the mutual backup and collaborative support between multiple drones also improve the reliability and stability of the entire system, reduce the risk of mission failure due to the failure of a single drone, and maximize the collaborative operation capabilities of the drone formation.

[0163] In this embodiment, a multi-UAV cooperative control device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0164] This embodiment provides a multi-UAV cooperative control device, such as Figure 6 As shown, including:

[0165] The space definition module 601 is used to define the observation space and action space of each drone;

[0166] A network framework construction module 602 is used to construct a multi-UAV cooperative control network framework based on the observation space and the action space; the control strategy of each UAV in the multi-UAV cooperative control network framework includes an executor network and an evaluator network;

[0167] An objective function definition module 603 is used to define an objective function based on the executor network and the evaluator network;

[0168] A model training module 604 is used to construct a policy network model and train the policy network model based on the objective function;

[0169] The coordination control module 605 is used to output the flight actions of each UAV through the trained strategy network model to coordinate and control each UAV.

[0170] In some optional implementations, the observation space is a set of finite state information obtained by the drone through sensors, and the action space is a set of all flight actions taken by the drone.

[0171] In some optional implementations, the space definition module 601 is further used to:

[0172] Acquire environmental information of the corresponding drone through the sensors of the respective drones;

[0173] Separate the environmental information into different channels to form a multi-channel matrix;

[0174] Based on the multi-channel matrix, local observations of each UAV are obtained to generate an observation space;

[0175] Discretize the action space of each UAV and obtain the discretization result;

[0176] The discretization results are converted into discrete maneuver control commands to generate an action space.

[0177] In some optional implementations, the network framework construction module 602 is further used to:

[0178] Based on the observation space and action space, each UAV is equipped with an executor network and an evaluator network to construct a multi-UAV collaborative control network framework; the executor network is used to receive the local observation as input and output discretized actions; the evaluator network is used to receive the flight actions of the executor network and the states of neighboring UAVs, and output a value function to adjust the executor network.

[0179] In some optional implementations, the objective function definition module 603 is further used to:

[0180] An approximate value of the value function is learned through the evaluator network, and minimizing the time difference error is used as the learning goal of the evaluator network;

[0181] Obtain rewards and penalties for each drone from the environmental information, and take the cumulative maximization of the rewards and penalties as the learning goal of the executor network;

[0182] The objective function is constructed by combining the learning objectives of the evaluator network and the executor network.

[0183] In some optional implementations, the model training module 604 is further used to:

[0184] The multi-channel matrix of each UAV and the state of the neighboring UAV are used as the input of the policy network model, and the action probability distribution of each UAV is used as the output of the policy network model to construct a policy network model; the structure of the policy network model includes a convolutional layer, a fully connected layer and an xLSTM unit.

[0185] In some optional implementations, the model training module 604 is further used to:

[0186] Initialize the model parameters of the strategy network model; the model parameters include the evaluator network parameters and the executor network parameters;

[0187] Generate flight actions of each drone through an executor network of each drone according to the current state of each drone and the environmental noise;

[0188] Control each drone to actively interact with the environment according to the flight actions generated by the strategy network, and obtain rewards, penalties and the next state;

[0189] The current state, the flight action, the reward and punishment, and the next state are stored as experience in the experience pool of the multi-UAV cooperative control network framework;

[0190] Randomly sampling samples from the experience pool, and training the policy network model based on the sampled samples;

[0191] Obtaining the time difference error and the policy gradient during the training process, and updating the evaluator network parameters according to the time difference error, and updating the executor network parameters according to the policy gradient;

[0192] After updating the network parameters, continue to train the policy network model to optimize the objective function and obtain the trained policy network model.

[0193] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0194] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0195] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0196] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0197] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0198] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0199] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0200] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0201] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0202] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the defined scope.

Claims

1. A multi-UAV cooperative control method, characterized in that: The method comprises: Define the observation space and action space of each drone; Based on the observation space and the action space, a multi-UAV cooperative control network framework is constructed; the control strategy of each UAV in the multi-UAV cooperative control network framework includes an executor network and an evaluator network; Based on the executor network and the evaluator network, defining an objective function; Constructing a policy network model, and training the policy network model based on the objective function; Outputting the flight actions of each UAV through the trained strategy network model to coordinate and control each UAV; The definition of the observation space and action space of each drone includes: Acquiring environmental information of the corresponding drone through sensors of each drone; Separating the environmental information into different channels to form a multi-channel matrix; Based on the multi-channel matrix, obtaining local observations of each UAV to generate an observation space; Discretize the action space of each drone and obtain a discretization result; Converting the discretization processing result into discrete maneuver control commands to generate an action space; The multi-UAV cooperative control network framework is constructed based on the observation space and the action space, including: Based on the observation space and action space, each UAV is equipped with an executor network and an evaluator network to construct a multi-UAV collaborative control network framework; the executor network is used to receive the local observation as input and output a discretized action; the evaluator network is used to receive the flight action of the executor network and the state of the neighboring UAVs, and output a value function to adjust the executor network; The objective function is defined based on the executor network and the evaluator network, including: Learning an approximate value of the value function through the evaluator network, and minimizing the temporal difference error as the learning goal of the evaluator network; Obtaining rewards and penalties for each drone from the environmental information, and taking the cumulative maximization of the rewards and penalties as the learning goal of the executor network; Combining the learning objectives of the evaluator network and the executor network, an objective function is constructed; the objective function is used to guide the learning process of the executor network and the evaluator network to guide and evaluate the behavior strategy of the UAV; The constructing of the strategy network model includes: The multi-channel matrix of each drone and the state of the neighboring drones are used as inputs of a policy network model, and the action probability distribution of each drone is used as output of the policy network model to construct a policy network model.

2. The method according to claim 1, characterized in that The observation space is a set of finite state information obtained by the drone through sensors, and the action space is a set of all flight actions taken by the drone.

3. The method according to claim 1, characterized in that The structure of the strategy network model includes a convolutional layer, a fully connected layer and an xLSTM unit.

4. The method according to claim 3, characterized in that The training of the policy network model based on the objective function includes: Initializing model parameters of the strategy network model; the model parameters include evaluator network parameters and executor network parameters; Generate flight actions of each drone through an executor network of each drone according to the current state of each drone and environmental noise; Control each drone to actively interact with the environment according to the flight actions generated by the strategy network, and obtain rewards, penalties and the next state; The current state, the flight action, the reward and punishment, and the next state are stored as experience in the experience pool of the multi-UAV cooperative control network framework; Randomly sampling samples from the experience pool, and training the policy network model based on the sampled samples; Obtaining the temporal differential error and policy gradient during the training process, and updating the evaluator network parameters according to the temporal differential error, and updating the executor network parameters according to the policy gradient; After updating the network parameters, continue to train the policy network model to optimize the objective function, and obtain the trained policy network model.

5. A multi-UAV cooperative control device, characterized in that: The device comprises: The space definition module is used to define the observation space and action space of each drone; A network framework construction module is used to construct a multi-UAV cooperative control network framework based on the observation space and the action space; the control strategy of each UAV in the multi-UAV cooperative control network framework includes an executor network and an evaluator network; An objective function definition module, used to define an objective function based on the executor network and the evaluator network; A model training module, used to construct a policy network model and train the policy network model based on the objective function; A coordination control module, used to output the flight actions of each UAV through the trained strategy network model, so as to coordinate and control each UAV; The space definition module is also used for: Acquiring environmental information of the corresponding drone through sensors of each drone; Separating the environmental information into different channels to form a multi-channel matrix; Based on the multi-channel matrix, obtaining local observations of each UAV to generate an observation space; Discretize the action space of each drone and obtain a discretization result; Converting the discretization processing result into discrete maneuver control commands to generate an action space; The network framework building module is also used to: Based on the observation space and action space, each UAV is equipped with an executor network and an evaluator network to construct a multi-UAV collaborative control network framework; the executor network is used to receive the local observation as input and output a discretized action; the evaluator network is used to receive the flight action of the executor network and the state of the neighboring UAVs, and output a value function to adjust the executor network; The objective function definition module is also used for: Learning an approximate value of the value function through the evaluator network, and minimizing the temporal difference error as the learning goal of the evaluator network; Obtaining rewards and penalties for each drone from the environmental information, and taking the cumulative maximization of the rewards and penalties as the learning goal of the executor network; Combining the learning objectives of the evaluator network and the executor network, an objective function is constructed; the objective function is used to guide the learning process of the executor network and the evaluator network to guide and evaluate the behavior strategy of the UAV; The model training module is also used for: The multi-channel matrix of each drone and the state of the neighboring drones are used as inputs of a policy network model, and the action probability distribution of each drone is used as output of the policy network model to construct a policy network model.

6. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multi-UAV cooperative control method according to any one of claims 1 to 4 by executing the computer instructions.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute a multi-UAV cooperative control method according to any one of claims 1 to 4.

Images