Robust safety control method, system and medium for multi-UAV collaboration

Abstract

A method, system, and medium for robust and safe control of multi-UAV coordination. The method includes training a UAV cluster based on a constructed multi-UAV coordinated flight simulation model under the interference of environmental uncertainties to update the flight strategy of the multiple UAVs; simultaneously establishing an experience pool and updating the states of the multiple UAVs; extracting samples from the experience pool, calculating the expected benefits of the robustness of the multi-UAV coordinated control, and updating the critic network based on the expected benefits of the multi-UAV coordinated control robustness; constructing a deep reinforcement learning algorithm model based on robustness of the motion, performing safety gradient descent to update the action network; updating the target policy network; adding different perturbations to the updated state of the target policy network, observing the reward values ​​under different motion perturbations, and saving the current policy if the reward value meets convergence; otherwise, returning to the experience pool and re-extracting samples. The present invention improves the safety of multi-UAV coordinated control.

Description

Technical Field

[0001] The present invention relates to the field of multi-UAV collaborative control, and in particular to a method, system and medium for robust and safe motion control of multi-UAV collaboration. Background Art

[0002] In recent years, with the increasing application of multi-UAV technology in wireless communications, UAVs have garnered widespread attention in areas such as disaster relief and data collection and transmission. UAVs are characterized by high flexibility, strong maneuverability, and the ability to carry a variety of devices. Through collaborative control strategies, multiple UAVs can be scheduled based on the real-time data transformation and spatial distribution of related devices. This ensures data timeliness, broad and equitable communication coverage, and efficient data acquisition by UAVs, while also achieving energy savings.

[0003] Existing technologies for multi-UAV coordinated control based on deep reinforcement learning require consistent data between the training and test domains. However, in complex real-world environments, uncertain variables can arise in the multi-UAV movement domain, causing UAVs to misjudge actions that differ from their ideal behavior. This can lead to incorrect actions and potentially lead to safety issues such as collisions between the UAVs. Research has shown that by analyzing the uncertainty in UAV transition probabilities and solving a maximization-minimization problem, the probability of the worst-case outcome can be reduced, thereby improving the safety of multi-UAV coordinated control. In this context, a method for perturbing the movements of multiple UAVs during training is needed. By observing the UAVs' movements, the probabilities of each action can be determined when faced with environmental uncertainty, thereby reducing the probability of executing the most dangerous actions.

[0004] Existing deep reinforcement learning-based approaches, when training multi-UAV collaborative control, fail to consider the potential for uncertainty in complex real-world environments to interfere with the correct action judgments of multiple UAVs. This can lead to incorrect UAV actions and even safety issues. For example, the method described in the article "AoI-minimal UAV Crowdsensing by Model-based Graph Convolutional Reinforcement Learning" ignores the potential for uncertainty in the collaborative control process, which can induce UAVs to misjudge their actions. For example, patent application CN118591030A, "Unmanned Swarm Network Topology Intelligent Decision-Making Method, Apparatus, Device, and Medium," proposes an intelligent decision-making method for unmanned swarm network topology. This method enables UAVs to autonomously learn and optimize network topology, allowing them to adjust themselves in complex and dynamic environments based on real-time feedback. It can autonomously move nodes within the cluster according to networking requirements, adjust the network topology, and enhance network adaptability. Patent application publication number CN118502466A, "A Control Method, System, Electronic Device, and Storage Medium for a Swarm of Unmanned Aerial Vehicles," proposes a control method for a swarm of unmanned aerial vehicles (UAVs) aimed at rationally planning flight paths and improving mission safety. However, during the training phase of multi-UAV coordinated control, these methods fail to consider how the uncertainty variables inherent in complex real-world environments can interfere with the decision-making of the swarm during mission execution, potentially leading to erroneous decisions and potential safety hazards during mission execution. Patent application publication number CN116931543A, "A Safe Flight Control Method, System, Device, and Medium for Multi-UAV Data Collection," proposes a safe flight control method for multi-UAV data collection that addresses the technical issue of inconsistent physical perception during multi-UAV data collection and improves the security and robustness of multi-UAV flight data collection. However, this solution addresses the issue of UAV state uncertainty and does not address the disturbances in the movements of multiple UAVs caused by uncertain variables in complex environments, thus failing to guarantee safety and robustness. Summary of the Invention

[0005] The purpose of the present invention is to address the problems in the above-mentioned prior art and provide a method, system and medium for robust safety control of multi-UAV collaboration, so as to reduce the safety problems arising during the collaborative control of multiple UAVs.

[0006] In order to achieve the above object, the present invention has the following technical solutions:

[0007] In a first aspect, a method for robust safety control of multi-UAV coordinated motion is provided, comprising:

[0008] Obtain the objective function of multiple UAVs and construct a multi-UAV collaborative flight simulation model;

[0009] Initialize the states of multiple drones, set up a random action exploration process, select the initial actions of multiple drones, and initialize the policy network and critic network;

[0010] Under the interference of environmental uncertainties, based on the constructed multi-UAV cooperative flight simulation model, the UAV cluster is trained to update the flight strategy of multiple UAVs; at the same time, an experience pool is established to update the status of multiple UAVs;

[0011] Draw samples from the experience pool, calculate the expected benefits of multi-UAV collaborative control robustness, and update the critic network based on the expected benefits of multi-UAV collaborative control robustness;

[0012] Build an action-robust deep reinforcement learning algorithm model and perform secure gradient descent to update the action network;

[0013] Update target policy network;

[0014] Add different disturbances to the updated state of the target policy network and observe the reward values ​​under different action disturbances. If the reward value meets the convergence requirement, save the current policy; otherwise, return to re-sample the sample in the experience pool.

[0015] As a preferred solution, the objective function expression of the multi-UAV is as follows:

[0016]

[0017] Where, Indicates the The distance between the drone and the target point, Indicates the maximum value of the total distance between all drones and the target point, N is the total number of drones in the drone cluster.

[0018] As a preferred solution, the step of constructing a multi-UAV collaborative flight simulation model includes constructing a multi-UAV flight model, a multi-UAV energy consumption model, UAV swarm multi-agent deep deterministic policy gradient algorithm MADDPG learning parameters, and a threat model;

[0019] Methods for constructing a multi-UAV flight model include:

[0020] Building a drone group and ground user groups , assuming that the UAV is flying at a constant altitude, the UAV The position coordinates are ; Coordinates and The coordinates are updated in the following ways:

[0021]

[0022]

[0023] Where, Indicates time slot, Indicates the next time slot, represents the flight speed of the UAV u, Indicates the direction angle of the UAV u during flight;

[0024] The methods for constructing a multi-UAV energy consumption model include:

[0025] The communication power from the UAV to the ground is ,in, is the rate of achievement Energy required;

[0026] The power and induced power from the drone to the ground user are:

[0027]

[0028]

[0029] Where, is the weight of the drone, is the rotor radius, is the angular velocity, ρ is the air density, represents the fracturing coefficient, represents the rotor disk area, is the blade chord, is the number of leaves, is a fixed constant;

[0030] Ground users The expected unloading delay is ,in is the task size;

[0031] The learning parameters of the MADDPG algorithm for constructing a multi-agent deep deterministic policy gradient algorithm for drone swarms include:

[0032] Set the state space of multiple drones to S, the action space to A, and the policy parameters of each drone to:

[0033] ,remember is the strategy set of all agents;

[0034] Methods for constructing threat models include:

[0035] The attacker interferes with the communication between the drone and the ground user group, modifies the communication data and instructions, and causes the drone to adopt different strategies; assuming that the attack strength of the drone is , the greater the attack strength, The bigger;

[0036] The strategies used by the attacker to interfere with the drone are:

[0037]

[0038] Where, is the mixed strategy after the attacker interferes, For drone strategies, is the attacker's strategy, and P is the transition probability.

[0039] As a preferred solution, the steps of initializing the states of multiple drones, setting a random action exploration process, selecting the initial actions of multiple drones, and initializing the policy network and the critic network include: according to the expression Select the drone action, where For drone action, It is the deterministic policy in the multi-agent deep deterministic policy gradient algorithm MADDPG, is the observation status of the UAV, The random action exploration process of the UAV during the time slot.

[0040] As a preferred solution, under the interference of environmental uncertainty factors, based on the constructed multi-UAV cooperative flight simulation model, the UAV cluster is trained to update the flight strategy of multiple UAVs; at the same time, an experience pool is established and the steps of updating the status of multiple UAVs include:

[0041] The UAV swarm encounters interference from the model generation of environmental uncertainties during flight;

[0042] pass Calculate the reward of the system, where is the task size, is the angular velocity, is the expected offloading delay rate of ground user g, is the reward value of the drone;

[0043] Will Write into the experience pool F, is the state of the drone in the time slot, is the action performed by the drone in the time slot, is the reward for the drone in the time slot, is the state of the drone in the next time slot;

[0044] Experience pool F contains All drone experiences are recorded, Indicates the status of multiple drone groups in the time slot, Indicates the state of the multi-UAV group in the next time slot, represents the actions of the drone group in the time slot, represents the reward of the drone group in the time slot;

[0045] Use the new state for the next time slot Replace the old state .

[0046] As a preferred solution, the steps of extracting samples from the experience pool, calculating the expected benefits of the multi-UAV collaborative control robustness, and updating the critic network according to the expected benefits of the multi-UAV collaborative control robustness include:

[0047] The expected return gradient of multiple drones is calculated as follows:

[0048]

[0049] Where, o is the observation status of the drone, x Represents the observation results of all drones, is the strategy set of multiple drones, is the strategy of drone u under the deterministic strategy, is the centralized action value function under the deterministic strategy, which contains all the drone actions and state information to estimate the expected return, and F is the experience pool;

[0050] Update the critic network based on the expected return to minimize the error between the drone flight monitoring value and the target value:

[0051]

[0052]

[0053] Where, is the loss function of UAV u, n is the size of the flight monitoring value dataset, y is the true value of flight monitoring, To predict the target value, represents the discount factor, , For the reward of drone u, is the predicted target value for the next time slot.

[0054] As a preferred solution, the steps of constructing an action-robust deep reinforcement learning algorithm model and performing secure gradient descent to update the action network include:

[0055] Construct an action-robust deep reinforcement learning algorithm model as follows:

[0056]

[0057] The initial state of the drone is , select an action in this state :

[0058]

[0059] Where, is the current status of a single drone, are the UAV’s strategy parameters, For this moment, the opponent's strategy parameters are: is the probability of taking adversarial actions, ;

[0060] Update the policy parameters of drone u under interference according to the policy gradient network:

[0061]

[0062] is the strategy parameter of UAV u, are the opponent’s strategy parameters, is the attacker strength, It is the drone cluster state. is the centralized action function of the UAV, is the observation status result of multiple UAVs, is the deterministic strategy of UAV u, Hybrid strategies for UAVs under jamming;

[0063] Update the opponent's policy parameters based on the gradient information to minimize the opponent's long-term return:

[0064]

[0065] In the formula, the updated opponent's strategy parameters are , the opponent parameters before update are , express The dot product with its corresponding gradient, which is used to guide the direction and magnitude of the update, The current opponent's strategy;

[0066] For the critic network, the updated critic network in the action-robust deep reinforcement learning PR-MDP algorithm is expressed as:

[0067]

[0068] Where, For drone target network, is the adversary’s target network, is the reward value of the drone, represents the discount factor, , is the probability of taking adversarial actions, Indicates that the drone is in status The expected return under Indicates that the opponent is Expected reward under the state;

[0069] The action network of the drone and the opponent is updated by performing a safety gradient descent through the action-robust deep reinforcement learning PR-MDP algorithm to calculate the safety gradient descent minimization. The calculation expression is as follows:

[0070]

[0071]

[0072] .

[0073] As a preferred solution, in the step of updating the target policy network, according to the expression To update the target policy network; where, Indicates the current strategy; Indicates new strategies; is the learning rate, which represents the weight of the new parameter to the current parameter;

[0074] When adding different disturbances to the updated state of the target policy network, set the disturbance values ​​to 0, 0.1, 0.2, and 0.3 respectively; observe the fluctuations in the reward value under different action disturbances. If the reward value fluctuation is less than 0.1, convergence is satisfied.

[0075] Secondly, a robust safety control system for multi-UAV collaboration is provided, including:

[0076] The simulation model construction module is used to obtain the objective functions of multiple UAVs and construct a multi-UAV collaborative flight simulation model;

[0077] The initialization module is used to initialize the states of multiple drones, set a random action exploration process, select the initial actions of multiple drones, and initialize the policy network and critic network;

[0078] The flight strategy update module is used to train the drone cluster based on the constructed multi-drone collaborative flight simulation model to update the flight strategy of multiple drones under the interference of environmental uncertainties; at the same time, it establishes an experience pool and updates the status of multiple drones;

[0079] The expected return calculation module is used to extract samples from the experience pool, calculate the expected return of the multi-UAV collaborative control robustness, and update the critic network based on the expected return of the multi-UAV collaborative control robustness;

[0080] The action network update module is used to build an action-robust deep reinforcement learning algorithm model and perform safe gradient descent to update the action network;

[0081] A target policy network update module, used to update the target policy network;

[0082] The strategy screening module is used to add different disturbances to the updated state of the target strategy network and observe the reward values ​​under different action disturbances. If the reward value meets the convergence requirement, the current strategy is saved; otherwise, the strategy is returned and samples are re-extracted from the experience pool.

[0083] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and wherein when the computer program is executed by a processor, the method for robust safety control of actions for multi-UAV collaboration is implemented.

[0084] Compared with the prior art, the present invention has at least the following beneficial effects:

[0085] The present invention addresses the inconsistency between the training domain of multi-UAV collaborative control and the actual complex environment. It simulates a realistic complex environment during multi-UAV collaborative control training and, by providing attackers for the multiple UAVs during the collaborative control process, obtains the probability of each UAV action occurring when faced with environmental uncertainty. Under the interference of environmental uncertainty factors, the present invention trains a UAV cluster based on a constructed multi-UAV collaborative flight simulation model to update the flight strategy of the multiple UAVs. Furthermore, it constructs a deep reinforcement learning algorithm model based on robust motion, performs safety gradient descent to update the motion network, and then updates the strategy to reduce the probability of executing the most dangerous actions, thereby improving the robustness of the multi-UAV collaborative control system and reducing the occurrence of safety accidents. The present invention effectively addresses the problem of inconsistent test data and training data due to uncertain variables in the real complex environment during the multi-UAV collaborative control process, resulting in misjudgment of UAV actions, thereby improving the safety of multi-UAV collaborative control.

[0086] Furthermore, the multi-UAV collaborative flight simulation model constructed by the present invention includes a multi-UAV flight model, a multi-UAV energy consumption model, learning parameters for a drone swarm multi-agent deep deterministic policy gradient algorithm (MADDPG), and a threat model. The threat model is used to provide the multi-UAV swarm with attackers to simulate uncertainty in the simulated environment, and the MADDPG learning parameters are used to train the multi-UAV swarm to form a deterministic strategy to minimize the error between the drone flight evaluation value and the target value when facing the attacker. Simultaneously, a robust deep reinforcement learning (PR-MDP) algorithm is used to perform safety gradient descent to calculate the safety gradient descent minimization to update the action networks of the drones and the adversary. This allows the multi-UAV swarm to continuously update its strategy based on the actual situation, based on the deterministic strategy formed by the action decisions based on the MADDPG algorithm, under the perturbations of multiple environmental uncertainties, to improve the safety and robustness of multi-UAV collaborative control. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0088] Figure 1 Flowchart of a method for robust safety control of multi-UAV collaboration in an embodiment of the present invention. DETAILED DESCRIPTION

[0089] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, ordinary technicians in this field can also derive other embodiments without making any creative work.

[0090] See also Figure 1 The embodiment of the present invention proposes a robust and safe control method for multi-UAV collaborative actions, which aims to effectively solve the problem that in the multi-UAV collaborative control process, uncertain variables in the real complex environment lead to inconsistencies between test data and training data, causing UAVs to make misjudgments of actions, thereby improving the safety of multi-UAV collaborative control.

[0091] The method for robust safety control of multi-UAV coordination in an embodiment of the present invention mainly includes the following steps:

[0092] Step 1): Obtain the objective function of multiple UAVs and construct a multi-UAV collaborative flight simulation model;

[0093] Step 2): Initialize the state of multiple drones, set up a random action exploration process, select the initial action a of multiple drones, and initialize the policy network and critic network;

[0094] Step 3): Based on the UAV collaborative flight simulation model constructed in Step 1), the drone cluster is trained using the MADDPG learning method to update the flight strategy table for multiple drones under the influence of environmental uncertainties. Rewards are calculated, and an experience pool table containing the old state, action, reward, and new state is established to update the states of multiple drones.

[0095] Step 4): Extract a small number of samples from the experience pool, calculate the expected benefits of multi-UAV collaborative control robustness, and update the critic network based on the expected benefits;

[0096] Step 5): Build a deep reinforcement learning algorithm based on action robustness, perform the improved gradient descent step through the robust policy iteration algorithm, and update the action network;

[0097] Step 6): Update the target policy network;

[0098] Step 7): Add different disturbances to this state and observe the reward values ​​under different action disturbances. If the reward value meets the convergence requirement, execute step 8). Otherwise, go back to step 3).

[0099] Step 8): Save the updated policy in step 6).

[0100] In a possible implementation, in step 1), the objective function expression of the multi-UAV is as follows:

[0101]

[0102] Where, Indicates the The distance between the drone and the target point, Indicates the maximum value of the total distance between all drones and the target point, N is the total number of drones in the drone cluster.

[0103] In a possible implementation, the multi-UAV cooperative flight simulation model of an embodiment of the present invention includes a UAV group flight model, an energy consumption model, and a simulation model of environmental uncertainty factors.

[0104] Step 1.1) The method of constructing a multi-UAV flight model includes:

[0105] Building a drone group and ground user groups , assuming that the UAV is flying at a constant altitude, the UAV The position coordinates are ; Coordinates and The coordinates are updated in the following ways:

[0106]

[0107]

[0108] Where, Indicates time slot, Indicates the next time slot, Indicates the flight speed of the drone. Indicates the direction angle of the drone during flight;

[0109] Step 1.2) The method for constructing a multi-UAV energy consumption model includes:

[0110] The communication power from the UAV to the ground is ,in, is the rate of achievement Energy required;

[0111] The power and induced power from the drone to the ground user are:

[0112]

[0113]

[0114] Where, is the weight of the drone, is the rotor radius, is the angular velocity, ρ is the air density, represents the fracturing coefficient, represents the rotor disk area, is the blade chord, is the number of leaves, is a fixed constant;

[0115] Ground users The expected unloading delay is ,in is the task size;

[0116] Step 1.3) Build the drone cluster MADDPG learning parameters include:

[0117] Set the state space of multiple drones to S, the action space to A, and the policy parameters of each drone to:

[0118] ,remember is the strategy set of all agents;

[0119] Step 1.4) Methods for constructing a threat model include:

[0120] The attacker interfered with the communication between the drone and the ground user group, modified the communication data and instructions, and caused the drone to adopt different strategies.

[0121] Assume that the attack strength of the drone is , the greater the attack strength, The bigger;

[0122] The strategies used by the attacker to interfere with the drone are:

[0123]

[0124] Where, is the mixed strategy after the attacker interferes, For drone strategies, is the attacker's strategy, and P is the transition probability.

[0125] In one possible implementation, the embodiment of the present invention sets the initial state of the drone to s, the random action exploration process to x, initializes the state of the drone, sets a random action exploration process, selects the initial action of the drone, and initializes the critic network and the policy network. Specifically, the embodiment of the present invention uses the expression Select the drone action, where For drone action, It is the deterministic policy in the multi-agent deep deterministic policy gradient algorithm MADDPG, is the observation status of the UAV, The random action exploration process of the UAV during the time slot.

[0126] In a possible implementation manner, step 3) of the embodiment of the present invention specifically includes:

[0127] 3.1) The UAV swarm encounters interference from the model generation of environmental uncertainties during flight;

[0128] 3.2) Pass Calculate the rewards of the system, is the task size, is the angular velocity, is the expected offloading delay rate of ground user g, is the reward value of the drone;

[0129] 3.3) Write into the experience pool F, is the state of the drone in the time slot, is the action performed by the drone in the time slot, is the reward for the drone in the time slot, is the state of the drone in the next time slot;

[0130] 3.4) Experience pool F contains All drone experiences are recorded, Indicates the status of multiple drone groups in the time slot, Indicates the state of the multi-UAV group in the next time slot, represents the actions of the drone group in the time slot, represents the reward of the drone group in the time slot;

[0131] 3.5) Use the new state of the next time slot Replace the old state .

[0132] In a possible implementation manner, step 4) of the embodiment of the present invention specifically includes:

[0133] Step 4.1) Calculate the expected benefit gradient of multi-UAV data collection as follows:

[0134]

[0135] Where, o is the observation status of the drone, x Represents the observation results of all drones, is the strategy set of multiple drones, is the strategy of drone u under the deterministic strategy, is the centralized action value function under the deterministic strategy, which contains all the drone actions and state information to estimate the expected return, and F is the experience pool;

[0136] Step 4.2) Update the critic network based on the expected return to minimize the error between the drone flight evaluation value and the target value:

[0137]

[0138]

[0139] Where, is the loss function of UAV u, n is the size of the flight monitoring value dataset, y is the true value of flight monitoring, To predict the target value, represents the discount factor, , For the reward of drone u, is the predicted target value for the next time slot.

[0140] In a possible implementation manner, step 5) of the embodiment of the present invention specifically includes:

[0141] Step 5.1) Build an action-robust deep reinforcement learning algorithm model as follows:

[0142]

[0143] The initial state of the drone is , select an action in this state :

[0144]

[0145] Where, is the current status of a single drone, are the UAV’s strategy parameters, For this moment, the opponent's strategy parameters are: is the probability of taking adversarial actions, ;

[0146] Update the policy parameters of drone u under interference according to the policy gradient network:

[0147]

[0148] is the strategy parameter of UAV u, are the opponent’s strategy parameters, is the attacker strength, It is the drone cluster state. is the centralized action function of the UAV, is the observation status result of multiple UAVs, is the deterministic strategy of UAV u, Hybrid strategies for UAVs under jamming;

[0149] Update the opponent's policy parameters based on the gradient information to minimize the opponent's long-term return:

[0150]

[0151] In the formula, the updated opponent's strategy parameters are , the opponent parameters before update are , express The dot product with its corresponding gradient, which is used to guide the direction and magnitude of the update, The current opponent's strategy.

[0152] For the critic network, in the action-robust deep reinforcement learning PR-MDP (Probabilistic Action Robust Markov Decision Process) algorithm, the updated critic network can be expressed as

[0153]

[0154] Where, For drone target network, is the adversary’s target network, is the reward value of the drone, represents the discount factor, , is the probability of taking adversarial actions, Indicates that the drone is in status The expected return under Indicates that the opponent is Expected reward under the state;

[0155] Step 5.2) Perform safety gradient descent through the action-robust deep reinforcement learning PR-MDP algorithm to calculate the safety gradient descent minimization to update the action network of the drone and the opponent. The calculation expression is as follows:

[0156]

[0157]

[0158] .

[0159] In one possible implementation, step 6 of the embodiment of the present invention) is based on the expression To update the target policy network; where, Indicates the current strategy; Indicates new strategies; is the learning rate, which represents the weight of the new parameter on the current parameter.

[0160] In one possible implementation, in step 7) of the embodiment of the present invention, the disturbance values ​​are set to 0, 0.1, 0.2, and 0.3, respectively; the fluctuation of the reward value under different disturbances is observed; if the fluctuation is less than 0.1, convergence is satisfied, and step 8) is executed. If not, the process goes to step 4) to re-sample from the experience pool and calculate the expected return.

[0161] The robust safety control method for multi-UAV collaborative actions in this embodiment of the present invention addresses the inconsistency between the training domain of multi-UAV collaborative control and the actual complex environment. It applies an environmental simulation threat model to simulate the complex real-world environment during multi-UAV collaborative control training. By providing attackers for multiple UAVs during the collaborative control process, the probability of each UAV action occurring when faced with environmental uncertainty is determined. Based on an action decision system based on the MADDPG algorithm, a modified safety gradient descent is performed using the PR-MDP algorithm to minimize the calculated gradient descent and update the UAV and adversary action networks. This strategy is then updated to reduce the probability of executing the most dangerous actions, thereby improving the robustness of the multi-UAV collaborative control system and reducing the occurrence of safety incidents.

[0162] The embodiment of the present invention further proposes a multi-UAV coordinated motion robust safety control system, comprising:

[0163] The simulation model construction module is used to obtain the objective functions of multiple UAVs and construct a multi-UAV collaborative flight simulation model;

[0164] The initialization module is used to initialize the states of multiple drones, set a random action exploration process, select the initial actions of multiple drones, and initialize the policy network and critic network;

[0165] The flight strategy update module is used to train the drone cluster based on the constructed multi-drone collaborative flight simulation model to update the flight strategy of multiple drones under the interference of environmental uncertainties; at the same time, it establishes an experience pool and updates the status of multiple drones;

[0166] The expected return calculation module is used to extract samples from the experience pool, calculate the expected return of the multi-UAV collaborative control robustness, and update the critic network based on the expected return of the multi-UAV collaborative control robustness;

[0167] The action network update module is used to build an action-robust deep reinforcement learning algorithm model and perform safe gradient descent to update the action network;

[0168] A target policy network update module, used to update the target policy network;

[0169] The strategy screening module is used to add different disturbances to the updated state of the target strategy network and observe the reward values ​​under different action disturbances. If the reward value meets the convergence requirement, the current strategy is saved; otherwise, the strategy is returned and samples are re-extracted from the experience pool.

[0170] Another embodiment of the present invention further provides an electronic device, comprising:

[0171] a memory storing at least one instruction; and

[0172] A processor executes instructions stored in the memory to implement the robust safety control method for multi-UAV collaboration.

[0173] Another embodiment of the present invention further proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method for robust safety control of actions for multi-UAV collaboration.

[0174] Exemplarily, the instructions stored in the memory can be divided into one or more modules / units, which are stored in a computer-readable storage medium and executed by the processor to implement the robust and safe motion control method for multi-UAV collaboration described in the present invention. The one or more modules / units can be a series of computer-readable instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program in the server.

[0175] The electronic device may be a computing device such as a smartphone, laptop, PDA, or cloud server. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the electronic device may include more or fewer components, or a combination of certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, and the like.

[0176] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0177] The memory may be an internal storage unit of the server, such as a hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the server. Furthermore, the memory may include both an internal storage unit of the server and an external storage device. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store data that has been output or is about to be output.

[0178] It should be noted that the information interaction, execution process, etc. between the above-mentioned module units are based on the same concept as the method embodiment. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0179] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0180] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0181] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0182] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A robust safety control method for multi-UAV collaboration, characterized by: include: Obtain the objective function of multiple UAVs and construct a multi-UAV collaborative flight simulation model; Initialize the states of multiple drones, set up a random action exploration process, select the initial actions of multiple drones, and initialize the policy network and critic network; Under the interference of environmental uncertainties, based on the constructed multi-UAV cooperative flight simulation model, the UAV cluster is trained to update the flight strategy of multiple UAVs; at the same time, an experience pool is established to update the status of multiple UAVs; Draw samples from the experience pool, calculate the expected benefits of multi-UAV collaborative control robustness, and update the critic network based on the expected benefits of multi-UAV collaborative control robustness; Build an action-robust deep reinforcement learning algorithm model and perform secure gradient descent to update the action network; Update target policy network; Add different disturbances to the updated state of the target policy network and observe the reward values ​​under different action disturbances. If the reward value meets the convergence requirement, save the current policy; otherwise, return to re-sample from the experience pool. The steps of initializing the states of multiple drones, setting a random action exploration process, selecting the initial actions of multiple drones, and initializing the policy network and the critic network include: according to the expression Select the drone action, where a u For drone action, is the deterministic policy in the multi-agent deep deterministic policy gradient algorithm MADDPG, O u is the observation status of the UAV, The random action exploration process of the UAV during the time slot; The steps of constructing an action-robust deep reinforcement learning algorithm model and performing secure gradient descent to update the action network include: Construct an action-robust deep reinforcement learning algorithm model as follows: The initial state of the drone is s, and action a is selected in this state. t , then the UAV executes the ideal action f(s;θ) without interference with a probability of 1-α π ), the UAV’s policy parameter θ π Control; the drone performs actions under adversary attack with probability α The strategy parameters of the adversary horizontal line control; α is the probability of taking an adversarial action, α∈(0,1]; Update the policy parameters of drone u under interference according to the policy gradient network: θ u is the strategy parameter of UAV u, is the opponent's strategy parameter, α is the probability of taking adversarial actions, S is the state of the drone cluster, is the centralized action function of the UAV, x={o1,...,o u } is the observation status result of multiple UAVs, μ u (a u |o u ) is the deterministic strategy of UAV u, Hybrid strategies for UAVs under jamming; Update the opponent's policy parameters based on the gradient information to minimize the opponent's long-term return: In the formula, the updated opponent's strategy parameters are The opponent parameters before the update are express The dot product with its corresponding gradient, which is used to guide the direction and magnitude of the update, The current opponent's strategy; φ is the critic network, and updating the critic network in the action-robust deep reinforcement learning PR-MDP algorithm is expressed as: Where θ is the UAV target network, is the opponent's target network, r is the reward value of the drone, γ is the discount factor, 0<γ<1, α∈(0,1) is the probability of taking an adversarial action, Q(s';f(s';θ)) is the expected return of the drone in state S', represents the expected reward of the opponent in state S'; The action network of the drone and the opponent is updated by performing a safety gradient descent through the action-robust deep reinforcement learning PR-MDP algorithm to calculate the safety gradient descent minimization. The calculation expression is as follows:

2. The method for robust safety control of multi-UAV collaboration according to claim 1 is characterized in that: The objective function expression of the multi-UAV is as follows: Where Δd i represents the distance between the i-th UAV and the target point, MAX represents the maximum value of the sum of the distances between all UAVs and the target point, and N is the total number of UAVs in the UAV cluster.

3. The method for robust safety control of multi-UAV collaboration according to claim 1 is characterized in that: The step of constructing a multi-UAV collaborative flight simulation model includes constructing a multi-UAV flight model, a multi-UAV energy consumption model, UAV swarm multi-agent deep deterministic policy gradient algorithm MADDPG learning parameters and a threat model; Methods for constructing a multi-UAV flight model include: Construct the UAV group U = {u = 1, 2, ..., U} and the ground user group G = {g | g = 1, 2, ..., G}. Assuming that the UAV flies at a constant altitude, the position coordinates of the UAV u are (x u ,y u ); the x-coordinate and y-coordinate are updated as follows: Where k represents the time slot, k+1 represents the next time slot, and υ u represents the flight speed of the UAV u, Indicates the direction angle of the UAV u during flight; The methods for constructing a multi-UAV energy consumption model include: The communication power from the UAV to the ground is P c =υ u [k]×p u,g , where p u,g is the energy required to achieve rate v; The power and induced power from the drone to the ground user are: P l =ωR0W×λ Where W is the weight of the drone, R0 is the rotor radius, ω is the angular velocity, ρ is the air density, represents the fracturing coefficient, Indicates the rotor disc area, C b is the blade chord, N b is the number of blades, and λ is a fixed constant; The expected offloading delay for ground user g is Where T g is the task size; The learning parameters of the MADDPG algorithm for constructing a multi-agent deep deterministic policy gradient algorithm for drone swarms include: Set the state space of multiple drones to S, the action space to A, and the policy parameters of each drone to: θ={θ1,...,θ N }, let π={π1,...,π N } is the strategy set of all agents; Methods for constructing threat models include: The attacker interferes with the communication between the drone and the ground user group, modifies the communication data and instructions, and causes the drone to adopt different strategies. Assume that the probability of the drone taking adversarial actions is α∈(0,1). The greater the attack intensity, the greater α. The strategies used by the attacker to interfere with the drone are: Where, is the mixed strategy after the attacker’s interference, π is the UAV’s strategy, is the attacker's strategy, and P is the transition probability.

4. The method for robust safety control of multi-UAV coordination according to claim 3 is characterized in that: The steps of training a drone cluster to update the flight strategy of multiple drones based on the constructed multi-drone collaborative flight simulation model under the interference of environmental uncertainties and simultaneously establishing an experience pool and updating the status of multiple drones include: The UAV swarm encounters interference from the model generation of environmental uncertainties during flight; pass Calculate the reward of the system, where T g is the task size, ω u [k] is the angular velocity, m u,g [k] is the expected unloading delay rate of ground user g, j u [k] is the reward value of the drone; (s k ,a k ,r k ,s k+1 ) is written into the experience pool F, s k is the state of the drone in the time slot, a k is the action performed by the UAV in the time slot, r k is the reward of the drone in the time slot, s k+1 is the state of the drone in the next time slot; The experience pool F contains (S, S', a1..., a u ,r1,...,r u ) records the experience of all drones, S={s1,...s u } represents the state of the multi-UAV group in the time slot, S'={s1',...,s u '} represents the state of the multi-UAV group in the next time slot, a1,...a u represents the actions of the drone group in the time slot, r1,...,r u represents the reward of the drone group in the time slot; The old state S is replaced by the new state S' for the next time slot.

5. The method for robust safety control of multi-UAV coordination according to claim 4 is characterized in that: The steps of extracting samples from the experience pool, calculating the expected benefits of the multi-UAV collaborative control robustness, and updating the critic network according to the expected benefits of the multi-UAV collaborative control robustness include: The expected return gradient of multiple drones is calculated as follows: Where o is the observation state of the UAV, x represents the observation results of all UAVs, {θ1,...,θ u } is the strategy set of multiple drones, μ u is the strategy of drone u under the deterministic strategy, Q μ (x,a1,...,a u ) is the centralized action value function under the deterministic strategy, which contains all drone actions and state information to estimate the expected return, and F is the experience pool; Update the critic network based on the expected return to minimize the error between the drone flight monitoring value and the target value: Where, l(θ u ) is the loss function of UAV u, n is the size of the flight monitoring value dataset, y is the true value of flight monitoring, To predict the target value, γ represents the discount factor, 0<γ<1, r u For the reward of drone u, is the predicted target value for the next time slot.

6. The method for robust safety control of multi-UAV coordination according to claim 5 is characterized in that: In the step of updating the target policy network, according to the expression θ′ u ←τθ u +(1-τ)θ′ u To update the target policy network; where θ u represents the current strategy; θ' u represents the new strategy; τ is the learning rate, which represents the weight of the new parameter to the current parameter; When adding different perturbations to the updated state of the target policy network, the perturbation values ​​are set to 0, 0.1, 0.2, and 0.3 respectively; Observe the fluctuation of reward value under different action disturbances. If the fluctuation of reward value is less than 0.1, convergence is satisfied.

7. A robust safety control system for multi-UAV collaboration, characterized by: include: The simulation model construction module is used to obtain the objective functions of multiple UAVs and construct a multi-UAV collaborative flight simulation model; The initialization module is used to initialize the states of multiple drones, set a random action exploration process, select the initial actions of multiple drones, and initialize the policy network and critic network; The flight strategy update module is used to train the drone cluster based on the constructed multi-drone collaborative flight simulation model to update the flight strategy of multiple drones under the interference of environmental uncertainties; at the same time, it establishes an experience pool and updates the status of multiple drones; The expected return calculation module is used to extract samples from the experience pool, calculate the expected return of the multi-UAV collaborative control robustness, and update the critic network based on the expected return of the multi-UAV collaborative control robustness; The action network update module is used to build an action-robust deep reinforcement learning algorithm model and perform secure gradient descent to update the action network; A target policy network update module, used to update the target policy network; The strategy screening module is used to add different disturbances to the updated state of the target strategy network and observe the reward values ​​under different action disturbances. If the reward value meets the convergence requirement, the current strategy is saved; otherwise, the strategy is returned to resample from the experience pool. The steps of initializing the states of multiple drones, setting a random action exploration process, selecting the initial actions of multiple drones, and initializing the policy network and the critic network include: according to the expression Select the drone action, where a u For drone action, is the deterministic policy in the multi-agent deep deterministic policy gradient algorithm MADDPG, O u is the observation status of the UAV, The random action exploration process of the UAV during the time slot; The steps of constructing an action-robust deep reinforcement learning algorithm model and performing secure gradient descent to update the action network include: Construct an action-robust deep reinforcement learning algorithm model as follows: The initial state of the drone is s, and action a is selected in this state. t , then the UAV executes the ideal action f(s;θ) without interference with a probability of 1-α π ), the UAV’s policy parameter θ π Control; the drone performs actions under adversary attack with probability α The strategy parameters of the adversary horizontal line control; α is the probability of taking an adversarial action, α∈[0,1]; Update the policy parameters of drone u under interference according to the policy gradient network: θ u is the strategy parameter of UAV u, is the opponent's strategy parameter, α is the probability of taking adversarial actions, S is the state of the drone cluster, is the centralized action function of the UAV, x={o1,...,o u } is the observation status result of multiple UAVs, μ u (a u |o u ) is the deterministic strategy of UAV u, Hybrid strategies for UAVs under jamming; Update the opponent's policy parameters based on the gradient information to minimize the opponent's long-term return: In the formula, the updated opponent's strategy parameters are The opponent parameters before the update are express The dot product with its corresponding gradient, which is used to guide the direction and magnitude of the update, The current opponent's strategy; φ is the critic network, and updating the critic network in the action-robust deep reinforcement learning PR-MDP algorithm is expressed as: Where θ is the UAV target network, is the opponent's target network, r is the reward value of the drone, γ is the discount factor, 0<γ<1, α∈(0,1) is the probability of taking an adversarial action, Q(s';f(s';θ)) is the expected return of the drone in state S', represents the expected reward of the opponent in state S'; The action network of the drone and the opponent is updated by performing a safety gradient descent through the action-robust deep reinforcement learning PR-MDP algorithm to calculate the safety gradient descent minimization. The calculation expression is as follows:

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for robust safety control of multi-UAV collaboration is implemented as described in any one of claims 1 to 6.

Images