Action robust safety control method and system for cooperation of multiple unmanned aerial vehicles, and medium

Abstract

The invention discloses an action robust safety control method and system for multi-unmanned aerial vehicle cooperation and a medium, and the method comprises the steps: training an unmanned aerial vehicle cluster to update a flight strategy of multiple unmanned aerial vehicles based on a constructed multi-unmanned aerial vehicle cooperation flight simulation model under the interference of environment uncertainty factors; meanwhile, establishing an experience pool, and updating the states of the multiple unmanned aerial vehicles; samples in the experience pool are extracted, the expected profit of the multi-unmanned-aerial-vehicle cooperative control robustness is calculated, and the reviewer network is updated according to the expected profit of the multi-unmanned-aerial-vehicle cooperative control robustness; constructing a deep reinforcement learning algorithm model based on action robustness, and executing security gradient descent to update the action network; updating the target policy network; and adding different disturbances in a state after the target strategy network is updated, observing reward values under different action disturbances, if the reward values meet convergence, storing the current strategy, otherwise, returning to re-extract the samples in the experience pool. According to the invention, the safety of cooperative control of multiple unmanned aerial vehicles is improved.

Description

Technical Field

[0001] The present invention relates to the field of multi-UAV cooperative control, and in particular to a method, system and medium for robust safety control of multi-UAV cooperative actions. Background Art

[0002] In recent years, with the increasing application of multi-UAV technology in the field of wireless communications, the application of UAVs in disaster relief, data collection and transmission has attracted widespread attention. UAVs have the characteristics of high flexibility, strong maneuverability and the ability to carry a variety of equipment. Through collaborative control strategies, multiple UAVs can be scheduled based on the real-time data transformation and spatial distribution of related equipment to ensure the timeliness of data, the breadth and fairness of communication coverage, and the efficiency of UAV data acquisition. At the same time, multiple UAVs can also achieve energy saving.

[0003] In the existing technology, the collaborative control of multiple UAVs based on deep reinforcement learning requires the consistency of training and test domain data. However, in the actual complex environment, uncertain variables will appear in the action field of multiple UAVs, causing the UAVs to misjudge actions that are different from the ideal actions, resulting in wrong results of UAV actions, and burying hidden dangers for safety issues such as collisions between multiple UAVs. Studies have shown that a maximization and minimization problem can be solved by analyzing the uncertainty in the UAV transfer probability to reduce the possibility of the worst result, thereby improving the safety of multi-UAV collaborative control. In this case, a method for perturbing actions during multi-UAV training is needed. By observing the actions of the UAVs, the probability of each action of the UAV in the face of environmental uncertainty can be obtained to reduce the probability of executing the most dangerous actions.

[0004] When training the coordinated control of multiple UAVs, the existing deep reinforcement learning-based solutions do not take into account the situation where the uncertainty variables in the real complex environment interfere with the correct action judgment of multiple UAVs, which will cause UAVs to act incorrectly and even cause safety problems. For example, the method mentioned in the article "AoI-minimal UAV Crowdsensing by Model-based Graph Convolutional Reinforcement Learning" ignores the situation that the uncertain variables of multiple UAVs may induce UAVs to make misjudgments of actions during the coordinated control process. For example, the patent application "Unmanned Cluster Network Topology Structure Intelligent Decision Method, Device, Equipment and Medium" with publication number CN118591030A proposes an unmanned cluster network topology structure intelligent decision method, which enables UAVs to autonomously learn and optimize the network topology structure, allowing UAVs to self-adjust according to real-time feedback in complex and dynamic environments, autonomously move the node positions in the cluster according to networking requirements, adjust the network topology structure, and enhance the adaptability of the network topology. The patent application with publication number CN118502466A, "A control method, system, electronic device and storage medium for a drone cluster", proposes a control method for a drone cluster, aiming to rationally plan the flight path of the drone cluster and improve the safety of the mission. However, in the training stage of multi-drone collaborative control, the above methods do not consider the interference of uncertain variables in the real complex environment on the decision-making of the drone cluster in performing tasks, induce drones to make wrong decisions, and cause safety hazards when the drone cluster performs tasks. The patent application with publication number CN116931543A, "A safe flight control method, system, equipment and medium for multi-drone data collection", proposes a safe flight control method for multi-drone data collection, which solves the technical problem of inconsistent physical perception faced in the process of multi-drone data collection, and has the characteristics of improving the safety and robustness of data collection during multi-drone flight. However, this solution is aimed at the problem of drone state uncertainty, and cannot solve the problem of disturbance of multi-drone actions caused by uncertain variables in complex environments, and safety and robustness cannot be guaranteed. Summary of the invention

[0005] The purpose of the present invention is to address the problems in the above-mentioned prior art and provide a method, system and medium for robust safety control of multi-UAV collaborative actions, so as to reduce the safety problems arising in the process of multi-UAV collaborative control.

[0006] In order to achieve the above object, the present invention has the following technical solutions: In a first aspect, a method for robust safety control of multi-UAV coordinated actions is provided, comprising: Obtain the objective function of multiple UAVs and construct a multi-UAV collaborative flight simulation model; Initialize the states of multiple drones, set up a random action exploration process, select the initial actions of multiple drones, and initialize the policy network and critic network; Under the interference of environmental uncertainties, based on the constructed multi-UAV cooperative flight simulation model, the UAV cluster is trained to update the flight strategy of multiple UAVs; at the same time, an experience pool is established to update the status of multiple UAVs; Extract samples from the experience pool, calculate the expected benefits of multi-UAV collaborative control robustness, and update the critic network based on the expected benefits of multi-UAV collaborative control robustness; Build an action-robust deep reinforcement learning algorithm model and perform secure gradient descent to update the action network; Update target policy network; Add different disturbances to the updated state of the target policy network and observe the reward values ​​under different action disturbances. If the reward value meets the convergence requirement, save the current policy; otherwise, return to re-sample the experience pool.

[0007] As a preferred solution, the objective function expression of the multi-UAV is as follows:

[0008] In the formula, Indicates The distance between the drone and the target point, Indicates the maximum value of the sum of distances between all drones and the target point. N is the total number of drones in the drone cluster.

[0009] As a preferred solution, the step of constructing a multi-UAV collaborative flight simulation model includes constructing a multi-UAV flight model, a multi-UAV energy consumption model, a UAV cluster multi-agent deep deterministic policy gradient algorithm MADDPG learning parameters and a threat model; Methods for constructing a multi-UAV flight model include: Building a drone group and ground user groups , assuming that the drone flies at a constant altitude, the drone The position coordinates are ; Coordinates and The coordinates are updated in the following ways:

[0010]

[0011] In the formula, Indicates time slot, Indicates the next time slot, represents the flight speed of UAV u, Indicates the direction angle of the UAV u when flying; The methods for constructing a multi-UAV energy consumption model include: The communication power from the UAV to the ground is ,in, is the rate of achievement Energy required; The power and induction power from the drone to the ground user are:

[0012]

[0013] In the formula, is the weight of the drone, is the rotor radius, is the angular velocity, ρ is the air density, represents the fracturing coefficient, represents the rotor disk area, is the blade chord, is the number of leaves, is a fixed constant; Ground users The expected unloading delay is ,in is the task size; The learning parameters of the MADDPG algorithm for constructing a drone swarm multi-agent deep deterministic policy gradient algorithm include: Set the state space of multiple drones to S, the action space to A, and the policy parameters of each drone to: ,remember is the strategy set of all agents; Methods for constructing threat models include: The attacker interferes with the communication between the drone and the ground user group, modifies the communication data and instructions, and causes the drone to adopt different strategies; assuming that the attack strength of the drone is , the greater the attack strength, The bigger; The strategies by which drones are interfered with by attackers are:

[0014] In the formula, is the mixed strategy after the attacker interferes, Strategies for drones, is the attacker’s strategy, and P is the transition probability.

[0015] As a preferred solution, the steps of initializing the states of multiple drones, setting a random action exploration process, selecting the initial actions of multiple drones, and initializing the policy network and the critic network include: according to the expression Select the drone action, where For drone action, It is the deterministic policy in the multi-agent deep deterministic policy gradient algorithm MADDPG. is the observation status of the drone, The random action exploration process of the drone during the time slot.

[0016] As a preferred solution, under the interference of environmental uncertainty factors, based on the constructed multi-UAV cooperative flight simulation model, the UAV cluster is trained to update the flight strategy of multiple UAVs; at the same time, an experience pool is established, and the steps of updating the status of multiple UAVs include: The UAV swarm encounters interference from the model generation of environmental uncertainties during flight; pass Calculate the system's reward, where is the task size, is the angular velocity, is the expected unloading delay rate of ground user g, is the reward value of the drone; Will Write into the experience pool F, is the state of the drone in the time slot, is the action performed by the drone in the time slot, The reward for the drone in the time slot, is the state of the drone in the next time slot; Experience pool F contains All drone experiences are recorded, Indicates the status of multiple drone groups in time slots, Indicates the state of the multi-UAV group in the next time slot, represents the actions of the drone group in the time slot, represents the reward of the drone group in the time slot; Use the new state for the next time slot Replace the old state .

[0017] As a preferred solution, the steps of extracting samples from the experience pool, calculating the expected benefits of multi-UAV collaborative control robustness, and updating the critic network according to the expected benefits of multi-UAV collaborative control robustness include: The expected return gradient of multiple drones is calculated as follows:

[0018] In the formula, o is the observation status of the drone, x represents the observation results of all drones, is the strategy set of multiple drones, is the strategy of drone u under the deterministic strategy, is the centralized action value function under the deterministic strategy, which contains all the drone actions and state information to estimate the expected return, and F is the experience pool; Update the critic network based on the expected return to minimize the error between the drone flight monitoring value and the target value:

[0019]

[0020] In the formula, is the loss function of drone u, n is the size of the flight monitoring value dataset, y is the true value of flight monitoring, To predict the target value, represents the discount factor, , For the reward of drone u, is the predicted target value for the next time slot.

[0021] As a preferred solution, the steps of constructing an action-robust deep reinforcement learning algorithm model and performing safety gradient descent to update the action network include: Construct the action-robust deep reinforcement learning algorithm model as follows:

[0022] The initial state of the drone is , select an action in this state :

[0023] In the formula, is the current state of a single drone, is the UAV’s strategy parameter, For the opponent's strategy parameters at this moment, is the probability of taking adversarial actions, ; Update the policy parameters of drone u under interference according to the policy gradient network:

[0024] is the strategy parameter of UAV u, are the opponent’s strategy parameters, is the attacker strength, The drone cluster state. is the centralized action function of the drone, is the observation status result of multiple UAVs, is the deterministic strategy of UAV u, A hybrid strategy for UAVs under jamming; Update the opponent's strategy parameters based on the gradient information to minimize the opponent's long-term return:

[0025] In the formula, the updated opponent's strategy parameters are , the opponent parameters before update are , express The dot product with its corresponding gradient, which is used to guide the direction and magnitude of the update, The strategy of the current opponent; For the critic network, updating the critic network in the action-robust deep reinforcement learning PR-MDP algorithm is expressed as:

[0026] In the formula, For drone target network, is the target network of the adversary, is the reward value of the drone, represents the discount factor, , is the probability of taking adversarial actions, Indicates that the drone is in status The expected return under Indicates that the opponent is Expected return under the state; The safety gradient descent is performed through the action-robust deep reinforcement learning PR-MDP algorithm to calculate the safety gradient descent minimization to update the action network of the drone and the opponent. The calculation expression is as follows:

[0027]

[0028] .

[0029] As a preferred solution, in the step of updating the target policy network, according to the expression To update the target policy network; where, Indicates the current strategy; Indicates new strategies; is the learning rate, which represents the weight of the new parameter on the current parameter; When adding different disturbances to the updated state of the target policy network, the disturbance values ​​are set to 0, 0.1, 0.2, and 0.3 respectively; the reward value fluctuations under different action disturbances are observed. If the reward value fluctuation is less than 0.1, convergence is satisfied.

[0030] In the second aspect, a robust safety control system for multi-UAV coordination is provided, including: The simulation model building module is used to obtain the objective functions of multiple UAVs and construct a multi-UAV collaborative flight simulation model; The initialization module is used to initialize the states of multiple drones, set a random action exploration process, select the initial actions of multiple drones, and initialize the policy network and the critic network; The flight strategy update module is used to train the drone cluster to update the flight strategy of multiple drones based on the constructed multi-drone collaborative flight simulation model under the interference of environmental uncertainties; at the same time, an experience pool is established to update the status of multiple drones; The expected benefit calculation module is used to extract samples from the experience pool, calculate the expected benefits of multi-UAV collaborative control robustness, and update the critic network according to the expected benefits of multi-UAV collaborative control robustness; The action network update module is used to build an action-robust deep reinforcement learning algorithm model and perform safety gradient descent to update the action network; A target policy network update module, used for updating the target policy network; The strategy screening module is used to add different disturbances to the updated state of the target strategy network and observe the reward values ​​under different action disturbances. If the reward value meets the convergence requirement, the current strategy is saved; otherwise, the sample in the experience pool is re-extracted.

[0031] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and wherein the computer program, when executed by a processor, implements the method for robust safety control of actions for multi-UAV collaboration.

[0032] Compared with the prior art, the present invention has at least the following beneficial effects: The robust safety control method for multi-UAV collaboration of the present invention takes into account the problem that the training domain of multi-UAV collaborative control is inconsistent with the actual complex environment, simulates the real complex environment in the training of multi-UAV collaborative control, and obtains the probability of each action of the UAV when facing the uncertainty of the environment by providing attackers for the multi-UAV in the collaborative control process. Under the interference of environmental uncertainty factors, the present invention trains the UAV cluster based on the constructed multi-UAV collaborative flight simulation model to update the flight strategy of the multi-UAV, and constructs a deep reinforcement learning algorithm model based on action robustness, performs safety gradient descent to update the action network, and then updates the strategy to reduce the execution probability of the most dangerous action, thereby improving the robustness of the multi-UAV collaborative control system and reducing the occurrence of safety accidents. The present invention effectively solves the problem that the uncertainty variables in the real complex environment in the multi-UAV collaborative control process cause the test data and training data to be inconsistent, resulting in the misjudgment of the UAV action, and improves the safety of multi-UAV collaborative control.

[0033] Furthermore, the multi-UAV cooperative flight simulation model constructed by the present invention includes a multi-UAV flight model, a multi-UAV energy consumption model, a UAV cluster multi-agent deep deterministic policy gradient algorithm MADDPG learning parameters and a threat model. Among them, the threat model is used to provide attackers to the multi-UAV cluster to simulate the uncertainty in the simulation environment, and the UAV cluster multi-agent deep deterministic policy gradient algorithm MADDPG learning parameters are used to train the multi-UAV cluster to form a deterministic strategy in the face of attackers to minimize the error between the UAV flight evaluation value and the target value. At the same time, the action robust deep reinforcement learning PR-MDP algorithm is used to perform security gradient descent to calculate the security gradient descent minimization to update the action network of the UAV and the opponent, so as to realize the multi-UAV cluster under the disturbance of multiple environmental uncertainties. On the basis of the deterministic strategy formed by the action decision based on the multi-agent deep deterministic policy gradient algorithm MADDPG, the strategy is continuously updated according to the actual situation to improve the safety and robustness of multi-UAV collaborative control. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 Flowchart of a method for robust safety control of multi-UAV coordinated actions according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, ordinary technicians in this field can also obtain other embodiments without making creative work.

[0037] See also Figure 1 The embodiment of the present invention proposes a method for robust safety control of multi-UAV collaborative actions, which aims to effectively solve the problem that in the process of multi-UAV collaborative control, the uncertainty variables in the real complex environment lead to inconsistency between test data and training data, resulting in misjudgment of UAV actions, thereby improving the safety of multi-UAV collaborative control.

[0038] The method for robust safety control of multi-UAV coordination in an embodiment of the present invention mainly includes the following steps: Step 1): Obtain the objective function of multiple UAVs and construct a multi-UAV collaborative flight simulation model; Step 2): Initialize the state of multiple drones, set a random action exploration process, select the initial action a of multiple drones, and initialize the policy network and critic network; Step 3): Under the interference of environmental uncertainties, based on the UAV collaborative flight simulation model built in step 1), the MADDPG learning method is used to train the UAV cluster to update the flight strategy table of multiple UAVs. Calculate rewards, establish an experience pool table containing old states, actions, rewards, and new states, and update the states of multiple UAVs; Step 4): Extract a small number of samples from the experience pool, calculate the expected benefits of multi-UAV collaborative control robustness, and update the critic network based on the expected benefits; Step 5): Build a deep reinforcement learning algorithm based on action robustness, perform the improved gradient descent step through the robust policy iteration algorithm, and update the action network; Step 6): Update the target policy network; Step 7): Add different disturbances to this state and observe the reward values ​​under different action disturbances. If the reward value meets the convergence requirement, execute step 8), otherwise go back to step 3); Step 8): Save the updated policy in step 6).

[0039] In a possible implementation, in step 1), the objective function expression of the multi-UAV is as follows:

[0040] In the formula, Indicates The distance between the drone and the target point, Indicates the maximum value of the sum of distances between all drones and the target point. N is the total number of drones in the drone cluster.

[0041] In a possible implementation, the multi-UAV cooperative flight simulation model of the embodiment of the present invention includes a UAV group flight model, an energy consumption model, and a simulation model of environmental uncertainty factors.

[0042] Step 1.1) The method of constructing a multi-UAV flight model includes: Building a drone group and ground user groups , assuming that the drone flies at a constant altitude, the drone The position coordinates are ; Coordinates and The coordinates are updated in the following ways:

[0043]

[0044] In the formula, Indicates time slot, Indicates the next time slot, Indicates the flight speed of the drone. Indicates the direction angle of the drone during flight; Step 1.2) The method of constructing a multi-UAV energy consumption model includes: The communication power from the UAV to the ground is ,in, is the rate of achievement Energy required; The power and induction power from the drone to the ground user are:

[0045]

[0046] In the formula, is the weight of the drone, is the rotor radius, is the angular velocity, ρ is the air density, represents the fracturing coefficient, represents the rotor disk area, is the blade chord, is the number of leaves, is a fixed constant; Ground users The expected unloading delay is ,in is the task size; Step 1.3) Build the drone cluster MADDPG learning parameters include: Set the state space of multiple drones to S, the action space to A, and the policy parameters of each drone to: ,remember is the strategy set of all agents; Step 1.4) Methods for constructing a threat model include: The attacker interfered with the communication between the drone and the ground user group, modified the communication data and instructions, and caused the drone to adopt different strategies.

[0047] Assume that the attack strength of the drone is , the greater the attack strength, The bigger; The strategies by which drones are interfered with by attackers are:

[0048] In the formula, is the mixed strategy after the attacker interferes, Strategies for drones, is the attacker’s strategy, and P is the transition probability.

[0049] In a possible implementation, the embodiment of the present invention sets the initial state of the drone to s, the random action exploration process to x, initializes the state of the drone, sets a random action exploration process, selects the initial action of the drone, and initializes the critic network and the policy network. Specifically, the embodiment of the present invention uses the expression Select the drone action, where For drone action, It is the deterministic policy in the multi-agent deep deterministic policy gradient algorithm MADDPG. is the observation status of the drone, The random action exploration process of the drone during the time slot.

[0050] In a possible implementation manner, step 3) of the embodiment of the present invention specifically includes: 3.1) The UAV swarm encounters interference from the model generation of environmental uncertainties during flight; 3.2) By Calculate the rewards of the system, is the task size, is the angular velocity, is the expected unloading delay rate of ground user g, is the reward value of the drone; 3.3) Write into the experience pool F, is the state of the drone in the time slot, is the action performed by the drone in the time slot, The reward for the drone in the time slot, is the state of the drone in the next time slot; 3.4) Experience pool F contains All drone experiences are recorded, Indicates the status of multiple drone groups in time slots, Indicates the state of the multi-UAV group in the next time slot, represents the actions of the drone group in the time slot, represents the reward of the drone group in the time slot; 3.5) Use the new state of the next time slot Replace the old state .

[0051] In a possible implementation manner, step 4) of the embodiment of the present invention specifically includes: Step 4.1) Calculate the expected benefit gradient of multi-UAV data collection as follows:

[0052] In the formula, o is the observation status of the drone, x represents the observation results of all drones, is the strategy set of multiple drones, is the strategy of drone u under the deterministic strategy, is the centralized action value function under the deterministic strategy, which contains all the drone actions and state information to estimate the expected return, and F is the experience pool; Step 4.2) Update the critic network based on the expected return to minimize the error between the drone flight evaluation value and the target value:

[0053]

[0054] In the formula, is the loss function of drone u, n is the size of the flight monitoring value dataset, y is the true value of flight monitoring, To predict the target value, represents the discount factor, , For the reward of drone u, is the predicted target value for the next time slot.

[0055] In a possible implementation manner, step 5) of the embodiment of the present invention specifically includes: Step 5.1) Construct an action-robust deep reinforcement learning algorithm model as follows:

[0056] The initial state of the drone is , select an action in this state :

[0057] In the formula, is the current state of a single drone, is the UAV’s strategy parameter, For this moment, the opponent's strategy parameters are: is the probability of taking adversarial actions, ; Update the policy parameters of drone u under interference according to the policy gradient network:

[0058] is the strategy parameter of UAV u, are the opponent’s strategy parameters, is the attacker strength, It is the drone cluster state. is the centralized action function of the drone, is the observation status result of multiple UAVs, is the deterministic strategy of UAV u, A hybrid strategy for UAVs under jamming; Update the opponent's strategy parameters based on the gradient information to minimize the opponent's long-term return:

[0059] In the formula, the updated opponent's strategy parameters are , the opponent parameters before update are , express The dot product with its corresponding gradient, which is used to guide the direction and magnitude of the update, The current opponent's strategy.

[0060] is the critic network. In the action-robust deep reinforcement learning PR-MDP (Probabilistic Action Robust Markov Decision Process) algorithm, the updated critic network can be expressed as

[0061] In the formula, For drone target network, is the target network of the adversary, is the reward value of the drone, represents the discount factor, , is the probability of taking adversarial actions, Indicates that the drone is in status The expected return under Indicates that the opponent is Expected return under the state; Step 5.2) Perform safety gradient descent through the action-robust deep reinforcement learning PR-MDP algorithm to calculate the safety gradient descent minimization to update the action network of the drone and the opponent. The calculation expression is as follows:

[0062]

[0063] .

[0064] In a possible implementation manner, step 6 of the present embodiment) is based on the expression To update the target policy network; where, Indicates the current strategy; Indicates new strategies; is the learning rate, which represents the weight of the new parameter on the current parameter.

[0065] In a possible implementation, in step 7) of the embodiment of the present invention, the disturbance values ​​are set to 0, 0.1, 0.2 and 0.3 respectively; the fluctuation of the reward value under different disturbances is observed, and if the fluctuation is less than 0.1, the convergence is satisfied, and step 8) is executed at this time; if it is not satisfied, go to step 4) to re-draw samples from the experience pool to calculate the expected return.

[0066] The method for robust safety control of multi-UAV collaborative actions in the embodiment of the present invention takes into account the inconsistency between the training domain of multi-UAV collaborative control and the actual complex environment, applies an environmental simulation threat model, simulates the actual complex environment in the training of multi-UAV collaborative control, and obtains the probability of each action of the UAV when facing environmental uncertainty by providing attackers for the multi-UAVs in the collaborative control process. On the basis of the action decision system based on the MADDPG algorithm, the PR-MDP algorithm is used to perform an improved safety gradient descent to calculate its minimization to update the action network of the UAV and the opponent, and then the strategy is updated to reduce the execution probability of the most dangerous action, thereby improving the robustness of the multi-UAV collaborative control system and reducing the occurrence of safety accidents.

[0067] The embodiment of the present invention further proposes a multi-UAV coordinated motion robust safety control system, comprising: The simulation model building module is used to obtain the objective functions of multiple UAVs and construct a multi-UAV collaborative flight simulation model; The initialization module is used to initialize the states of multiple drones, set a random action exploration process, select the initial actions of multiple drones, and initialize the policy network and the critic network; The flight strategy update module is used to train the drone cluster to update the flight strategy of multiple drones based on the constructed multi-drone collaborative flight simulation model under the interference of environmental uncertainties; at the same time, an experience pool is established to update the status of multiple drones; The expected benefit calculation module is used to extract samples from the experience pool, calculate the expected benefits of multi-UAV collaborative control robustness, and update the critic network according to the expected benefits of multi-UAV collaborative control robustness; The action network update module is used to build an action-robust deep reinforcement learning algorithm model and perform safety gradient descent to update the action network; A target policy network update module, used for updating the target policy network; The strategy screening module is used to add different disturbances to the updated state of the target strategy network and observe the reward values ​​under different action disturbances. If the reward value meets the convergence requirement, the current strategy is saved; otherwise, the sample in the experience pool is re-extracted.

[0068] Another embodiment of the present invention further provides an electronic device, comprising: a memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the robust safety control method for multi-UAV collaboration.

[0069] Another embodiment of the present invention further proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for robust safety control of actions for multi-UAV collaboration is implemented.

[0070] Exemplarily, the instructions stored in the memory may be divided into one or more modules / units, which are stored in a computer-readable storage medium and executed by the processor to complete the robust safety control method for multi-UAV collaboration of the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the server.

[0071] The electronic device may be a computing device such as a smart phone, a notebook, a PDA, and a cloud server. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the electronic device may also include more or fewer components, or a combination of certain components, or different components, for example, the electronic device may also include an input / output device, a network access device, a bus, etc.

[0072] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0073] The memory may be an internal storage unit of the server, such as a hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server. Furthermore, the memory may include both an internal storage unit of the server and an external storage device. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store data that has been output or is to be output.

[0074] It should be noted that the information interaction, execution process and other contents between the above-mentioned module units are based on the same concept as the method embodiment. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0075] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0076] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0077] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0078] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for robust safety control of multi-UAV coordination, characterized in that: include: Obtain the objective function of multiple UAVs and construct a multi-UAV collaborative flight simulation model; Initialize the states of multiple drones, set up a random action exploration process, select the initial actions of multiple drones, and initialize the policy network and critic network; Under the interference of environmental uncertainties, based on the constructed multi-UAV cooperative flight simulation model, the UAV cluster is trained to update the flight strategy of multiple UAVs; at the same time, an experience pool is established to update the status of multiple UAVs; Extract samples from the experience pool, calculate the expected benefits of multi-UAV collaborative control robustness, and update the critic network based on the expected benefits of multi-UAV collaborative control robustness; Build an action-robust deep reinforcement learning algorithm model and perform secure gradient descent to update the action network; Update target policy network; Add different disturbances to the updated state of the target policy network and observe the reward values ​​under different action disturbances. If the reward value meets the convergence requirement, save the current policy; otherwise, return to re-sample the experience pool.

2. The method for robust safety control of multi-UAV coordination according to claim 1 is characterized in that: The objective function expression of the multi-UAV is as follows: In the formula, Indicates The distance between the drone and the target point, Indicates the maximum value of the sum of distances between all drones and the target point. N is the total number of drones in the drone cluster.

3. The method for robust safety control of multi-UAV coordination according to claim 1 is characterized in that: The step of constructing a multi-UAV collaborative flight simulation model includes constructing a multi-UAV flight model, a multi-UAV energy consumption model, a UAV cluster multi-agent deep deterministic policy gradient algorithm MADDPG learning parameters and a threat model; Methods for constructing a multi-UAV flight model include: Building a drone group and ground user groups , assuming that the drone flies at a constant altitude, the drone The position coordinates are ; Coordinates and The coordinates are updated in the following ways: In the formula, Indicates time slot, Indicates the next time slot, represents the flight speed of UAV u, Indicates the direction angle of the UAV u when flying; The methods for constructing a multi-UAV energy consumption model include: The communication power from the UAV to the ground is ,in, is the rate of achievement Energy required; The power and induction power from the drone to the ground user are: In the formula, is the weight of the drone, is the rotor radius, is the angular velocity, ρ is the air density, represents the fracturing coefficient, represents the rotor disk area, is the blade chord, is the number of leaves, is a fixed constant; Ground users The expected unloading delay is ,in is the task size; The learning parameters of the MADDPG algorithm for constructing a drone swarm multi-agent deep deterministic policy gradient algorithm include: Set the state space of multiple drones to S, the action space to A, and the policy parameters of each drone to: ,remember is the strategy set of all agents; Methods for constructing threat models include: The attacker interferes with the communication between the drone and the ground user group, modifies the communication data and instructions, and causes the drone to adopt different strategies; assuming that the attack strength of the drone is , the greater the attack strength, The bigger; The strategies by which drones are interfered with by attackers are: In the formula, is the mixed strategy after the attacker interferes, Strategies for drones, is the attacker’s strategy, and P is the transition probability.

4. The method for robust safety control of multi-UAV coordination according to claim 1 is characterized in that: The steps of initializing the states of the multiple drones, setting a random action exploration process, selecting the initial actions of the multiple drones, and initializing the policy network and the critic network include: according to the expression Select the drone action, where For drone action, It is the deterministic policy in the multi-agent deep deterministic policy gradient algorithm MADDPG. is the observation status of the drone, The random action exploration process of the drone during the time slot.

5. The method for robust safety control of multi-UAV coordination according to claim 1 is characterized in that: The steps of training the drone cluster to update the flight strategy of multiple drones based on the constructed multi-drone collaborative flight simulation model under the interference of environmental uncertainty factors and establishing an experience pool to update the status of multiple drones include: The UAV swarm encounters interference from the model generation of environmental uncertainties during flight; pass Calculate the system's reward, where is the task size, is the angular velocity, is the expected unloading delay rate of ground user g, is the reward value of the drone; Will Write into the experience pool F, is the state of the drone in the time slot, is the action performed by the drone in the time slot, The reward for the drone in the time slot, is the state of the drone in the next time slot; Experience pool F contains All drone experiences are recorded, Indicates the status of multiple drone groups in time slots, Indicates the state of the multi-UAV group in the next time slot, represents the actions of the drone group in the time slot, represents the reward of the drone group in the time slot; Use the new state for the next time slot Replace the old state .

6. The method for robust safety control of multi-UAV coordination according to claim 1 is characterized in that: The steps of extracting samples from the experience pool, calculating the expected benefits of the multi-UAV collaborative control robustness, and updating the critic network according to the expected benefits of the multi-UAV collaborative control robustness include: The expected return gradient of multiple drones is calculated as follows: In the formula, o is the observation status of the drone, x represents the observation results of all drones, is the strategy set of multiple drones, is the strategy of drone u under the deterministic strategy, is the centralized action value function under the deterministic strategy, which contains all the drone actions and state information to estimate the expected return, and F is the experience pool; Update the critic network based on the expected return to minimize the error between the drone flight monitoring value and the target value: In the formula, is the loss function of drone u, n is the size of the flight monitoring value dataset, y is the true value of flight monitoring, To predict the target value, represents the discount factor, , For the reward of drone u, is the predicted target value for the next time slot.

7. The method for robust safety control of multi-UAV coordination according to claim 1 is characterized in that: The steps of constructing an action-robust deep reinforcement learning algorithm model and performing a safety gradient descent to update the action network include: Construct the action-robust deep reinforcement learning algorithm model as follows: The initial state of the drone is , select an action in this state : In the formula, is the current state of a single drone, are the UAV’s strategy parameters, For the opponent's strategy parameters at this moment, is the probability of taking adversarial actions, ; Update the policy parameters of drone u under interference according to the policy gradient network: is the strategy parameter of UAV u, are the opponent’s strategy parameters, is the attacker strength, It is the drone cluster state. is the centralized action function of the drone, is the observation status result of multiple UAVs, is the deterministic strategy of UAV u, A hybrid strategy for UAVs under jamming; Update the opponent's strategy parameters based on the gradient information to minimize the opponent's long-term return: In the formula, the updated opponent's strategy parameters are , the opponent parameters before update are , express The dot product with its corresponding gradient, which is used to guide the direction and magnitude of the update, The strategy of the current opponent; For the critic network, updating the critic network in the action-robust deep reinforcement learning PR-MDP algorithm is expressed as: In the formula, For drone target network, is the target network of the adversary, is the reward value of the drone, represents the discount factor, , is the probability of taking adversarial actions, Indicates that the drone is in status The expected return under Indicates that the opponent is Expected return under the state; The safety gradient descent is performed through the action-robust deep reinforcement learning PR-MDP algorithm to calculate the safety gradient descent minimization to update the action network of the drone and the opponent. The calculation expression is as follows: 。 8. The method for robust safety control of multi-UAV coordination according to claim 1 is characterized in that: In the step of updating the target policy network, according to the expression To update the target policy network; where, Indicates the current strategy; Indicates new strategies; is the learning rate, which represents the weight of the new parameter on the current parameter; When adding different disturbances to the updated state of the target policy network, the disturbance values ​​are set to 0, 0.1, 0.2, and 0.3 respectively; Observe the fluctuation of reward value under different action disturbances. If the fluctuation of reward value is less than 0.1, convergence is satisfied.

9. A robust safety control system for multi-UAV coordination, characterized in that: include: The simulation model building module is used to obtain the objective functions of multiple UAVs and construct a multi-UAV collaborative flight simulation model; The initialization module is used to initialize the states of multiple drones, set a random action exploration process, select the initial actions of multiple drones, and initialize the policy network and the critic network; The flight strategy update module is used to train the drone cluster to update the flight strategy of multiple drones based on the constructed multi-drone collaborative flight simulation model under the interference of environmental uncertainties; at the same time, an experience pool is established to update the status of multiple drones; The expected benefit calculation module is used to extract samples from the experience pool, calculate the expected benefits of multi-UAV collaborative control robustness, and update the critic network according to the expected benefits of multi-UAV collaborative control robustness; The action network update module is used to build an action-robust deep reinforcement learning algorithm model and perform safety gradient descent to update the action network; A target policy network update module, used for updating the target policy network; The strategy screening module is used to add different disturbances to the updated state of the target strategy network and observe the reward values ​​under different action disturbances. If the reward value meets the convergence requirement, the current strategy is saved; otherwise, the sample in the experience pool is re-extracted.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for robust safety control of actions for multi-UAV collaboration as described in any one of claims 1 to 8 is implemented.

Images