Multi-unmanned aerial vehicle planning method and device based on toughness improvement

Abstract

The invention relates to a multi-unmanned aerial vehicle planning method and device based on toughness improvement, and the method comprises the following steps: S1, obtaining a state set, an observation set and an action set of unmanned aerial vehicles, the state set including the state space of each unmanned aerial vehicle, the observation set including the observation of each unmanned aerial vehicle, and the action set including the action of each unmanned aerial vehicle; and S2, according to the state space and observation of each unmanned aerial vehicle, selecting an action as an optimal action decision through an MADDPG algorithm, calculating a state-value function value of the optimal action decision of each unmanned aerial vehicle, and updating parameters of the MADDPG algorithm according to the state-value function value. By designing a toughness index capable of reflecting the anti-interference capability of a multi-unmanned aerial vehicle network face when the network face is disturbed, the multi-unmanned aerial vehicle is planned through an MADDPG algorithm with the goal of improving toughness. The purpose of improving the toughness of the multi-unmanned aerial vehicle network when the multiple unmanned aerial vehicles in the multi-unmanned aerial vehicle network are planned is achieved.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a multi-UAV planning method and apparatus based on resilience enhancement. Background Technology

[0002] With the development of drone technology, drones, as aerial access devices with advantages of high flexibility and low-cost rapid deployment, are expected to become the core infrastructure of aerial networks in the future, and can be widely used in scenarios such as e-commerce delivery, medical supply transportation, and emergency disaster relief. Drones can quickly adapt to different mission requirements and environmental changes, efficiently completing tasks. Their relatively low deployment cost allows them to be used without large-scale construction and application. During the autonomous flight missions of drones, path planning technology has become a core technology, aiming to generate safe and efficient flight paths for drones in complex environments, which directly affects the network's operational efficiency and cost.

[0003] Resilience is widely used in the transportation and urban development sectors. It describes a system's ability to maintain its basic functions and recover to its original state or reach a new stable state after being subjected to adverse conditions such as disturbances, shocks, or stress. In the transportation sector, resilience is used to assess the ability of transportation systems to recover from damage. By improving the resilience of transportation systems, their stability, predictability, and controllability are enhanced, enabling them to cope more effectively with disturbances such as congestion and accidents, ensuring smooth traffic flow. In the urban development sector, resilience can also be used to assess urban disaster resilience, psychological resilience, and economic resilience. This allows for the construction of actionable implementation systems and the development of resilience enhancement strategies, enabling cities to effectively respond to natural disasters, socio-economic changes, and other situations.

[0004] Existing UAV path planning methods include graph search-based algorithms like Dijkstra's and A*, biomimetic algorithms such as Particle Swarm Optimization (PSO) and Ant Colony Optimization, and intelligent algorithms that have emerged in recent years, such as Deep Reinforcement Learning (DRL) and neural networks. These algorithms can solve path optimization problems in static environments. However, most of these path planning methods focus on the path planning of a single UAV, neglecting path collaboration in multi-UAV cooperative scenarios and the resilience of the entire UAV network. Consequently, when multiple UAVs and multiple tasks exist in the network, each UAV only considers its own work efficiency and cannot cooperate with other UAVs to reduce the network burden, leading to a decrease in network resilience. Summary of the Invention

[0005] Therefore, the purpose of this invention is to overcome the defects or deficiencies of existing technologies and solve the problem of decreased network efficiency caused by the fact that existing UAV path planning methods do not consider task allocation and path coordination in multi-UAV collaborative scenarios. The cause of decreased network efficiency is usually due to unpredictable disturbances within the system, such as the emergence of new tasks, traffic congestion, or partial UAV failures. This invention first designs a network resilience index to evaluate the network's ability to recover from disturbances.

[0006] Figure 1 This is a schematic diagram illustrating how network performance changes over time when a disturbance occurs. Represents network performance. t The time frame represents the entire process from the occurrence of a disturbance to the network's recovery to normal levels, including three stages and five states: the disturbance defense stage, the disturbance persistence stage, the disturbance recovery stage, and the initial state, degradation state, steady state, recovery state, and termination state. to This is the disturbance defense phase, at which time... The network remained stable during this period. When the disturbance occurred, to This is the stage where the disturbance continues to have an impact; to of Started to fall and At this point, network performance begins to decline; to This belongs to the steady-state stage after degradation, at which point... Maintain stability; to This is the recovery phase after degradation. Q(t) Gradually recovering to the state before the disturbance occurred The level. The next phase is the disturbance recovery phase.

[0007] In this invention, network resilience is used G The following expression is used to evaluate the network's resilience to disturbances:

[0008] In the formula, G Representing network resilience, Represents network performance. This represents the starting point of the phase where the disturbance continues to have an impact. This represents the end point of the phase in which the disturbance continues to have an impact.

[0009] Specifically, the performance of the network The following formula is used to calculate:

[0010] in, For time base parameters, This indicates the data transmission time between the drone and the terminal. Indicates terminal i The amount of data, Indicates drone n To the terminal i The data transmission rate.

[0011] Furthermore, drones n To the terminal i Data transmission rate Calculated using Shannon's formula: ; in, B For bandwidth, For the first n The drone and the first i The signal-to-noise ratio between individual terminals is calculated using the following formula:

[0012] in For drones n To the terminal i The transmission power, The power of Gaussian white noise. For drones n To the terminal i The path loss is calculated using the following expression:

[0013] in, For drones n To the terminal i The line-of-sight path loss is calculated using the following expression:

[0014] For drones n To the terminal i The non-line-of-sight path loss is calculated as follows:

[0015] In the formula, For drones n With terminal i The distance between them For communication frequency, At the speed of light, and These represent signal attenuation due to shadow fading, refraction, and scattering in non-line-of-sight and line-of-sight transmissions, respectively. and These are the probabilities of sight distance and non-sight distance, respectively, and their relationship is expressed as follows:

[0016] Meanwhile, to ensure that the UAV is both efficient and safe when performing tasks, this invention defines the following constraints to limit the UAV's planning and constructs the following conditional optimization model to maximize the resilience of the UAV during planning. The expression of the conditional optimization model is as follows:

[0017] in, C1 Define a set to restrict the join variables. For drones n With terminal i The connection variable between them takes a value of 0 or 1, representing time. t Inside, drones n Connect to terminal? i . C2 and C3 The connection restrictions between the drone and the terminal indicate the limitations for each drone. n and time t At least one terminal i Connect to the drone; simultaneously, for each terminal i and time t Only one drone can be connected to the terminal at a time to ensure a valid connection between the drone and the terminal. C4 restricts the drone's flight range. For drones n In time t The location coordinates must be located within the preset flight area. S Within the safe zone, ensure that the equipment operates. C5 Due to the data transmission rate requirements from the terminal to the over-the-air access device, For drones n With terminal i In time t The data transmission rate must meet or exceed the required minimum rate. . C6 Indicates the altitude limit for drones. The drone's flight altitude needs to be controlled within its maximum flight altitude. Inside.

[0018] This invention provides a multi-UAV planning method based on resilience enhancement, comprising the following steps: S1: Obtain the state set, observation set, and action set of the drone. The state set contains the state space of each drone, the observation set contains the observations of each drone, and the action set contains the actions of each drone. S2: Based on the state space and observations of each UAV, select an action as the optimal action decision using a MADDPG algorithm, calculate the state-value function value of the optimal action decision for each UAV, and update the parameters of the MADDPG algorithm based on the state-value function value. The expression for the state-value function is:

[0019] in, Indicates discount factor, Indicates the power parameter of the discount factor. Indicates drone n reward function value, Indicates drone n Input status, Indicates drone n The action performed.

[0020] Furthermore, the calculation expression for the reward function is as follows:

[0021] in, G For resilience, Indicates the penalty factor. p1, p2, p3, p4 All of these are penalty terms, and the calculation expressions for each penalty term are as follows:

[0022] in, For drones n With terminal i The connection variable between them takes a value of 0 or 1; For drones n In time t Position coordinates; For drones n With terminal i In time t Data transmission rate; This refers to the flight altitude of the drone; The discriminant function is expressed as follows: .

[0023] Furthermore, drones n To the terminal iThe data transmission rate is calculated using Shannon's formula:

[0024] in, B For bandwidth, For the first n The drone and the first i The signal-to-noise ratio between individual terminals is calculated using the following formula:

[0025] in, For drones n To the terminal i The transmission power, The power of Gaussian white noise. For drones n To the terminal i The path loss is calculated using the following expression:

[0026] in, For drones n To the terminal i The line-of-sight path loss is calculated using the following expression:

[0027] For drones n To the terminal i The non-line-of-sight path loss is calculated as follows:

[0028] In the formula, For drones n With terminal i The distance between them For communication frequency, At the speed of light, and These represent signal attenuation due to shadow fading, refraction, and scattering in non-line-of-sight and line-of-sight transmissions, respectively. and Let be the probabilities of sight distance and non-sight distance, respectively, and their relationship is expressed as follows: .

[0029] Further, in step S1: Transform the drone planning problem into a problem with N The distributed part of the Markov game for each drone is observable, and the set of states is defined. S include N The state space of a drone The observation set B include N Observation of a drone The set of actions A include N The operational space of a drone ； The state space ,in and For the drone n exist t The horizontal and vertical coordinates of time; For the drone n At time t and the terminal i The connection variable takes a value of 1 or 0; For drones n exist t Flight altitude at any given moment; The action space ,in For drones n exist t The distance of flight at any given moment For drones n exist t Flight direction at any moment For drones n exist t Time and Terminal i The join variable, For drones n exist t The flight altitude at any given moment.

[0030] Furthermore, step S2 includes the following steps; S21: Based on the state space and observations of each UAV, select an action as the optimal action decision using a MADDPG algorithm; S22: Calculate the state-value function value for the optimal action decision of each UAV, and update the parameters of the MADDPG algorithm based on the state-value function value; The parameters in step S22 include the parameters of the Actor evaluation network. Parameters of the Critic evaluation network Parameters of the Actor target network Parameters of the Critic target network ; The evaluation network parameters of the Critic network Updates are performed using a loss function; the expression for the loss function is:

[0031] in, For state-value functions; For drones n The l Next transition state; For drones n The l Next state transition action; The evaluation network parameters of the Actor network By maximizing the update of the policy objective function, the expression of the policy objective function is:

[0032] in, For evaluating the Actor network; For the first n The actions of the drone; For the first n The first drone l The second observation; The target network parameters of the Critic network are updated through a first function, the expression of which is:

[0033] The target network parameters of the Actor network are updated through a second function, the expression of which is:

[0034] In the above formula, It is a positive constant close to 0.

[0035] Furthermore, in step S22, an experience pool is provided to store the state, actions, and state-value functions of each drone, and experience samples are formed in the experience pool to update the parameters of the Actor evaluation network. Parameters of the Critic evaluation network Parameters of the Actor target network Parameters of the Critic target network .

[0036] A multi-UAV planning device based on resilience enhancement includes: The UAV state acquisition module acquires the UAV's state set, observation set, and action set. The state set contains the state space of each UAV, the observation set contains the observations of each UAV, and the action set contains the actions of each UAV. The UAV planning module is used to select an action as the optimal action decision based on the state space and observation of each UAV, calculate the state-value function value of the optimal action decision for each UAV, and update the parameters of the UAV planning module based on the state-value function value. The expression for the state-value function is:

[0037] in, Indicates discount factor, Indicates the power parameter of the discount factor. Indicates drone n reward function value, Indicates drone n Input status, Indicates drone n The action performed.

[0038] Furthermore, the calculation expression for the reward function is as follows:

[0039] in, G For resilience, Indicates the penalty factor. p1, p2, p3, p4 All of these are penalty terms, and the calculation expressions for each penalty term are as follows:

[0040] In the above formula, where, For drones n With terminal i The connection variable between them takes a value of 0 or 1; For drones n In time t Position coordinates; For drones n With terminal i In time t Data transmission rate; This refers to the flight altitude of the drone; The discriminant function is expressed as follows: .

[0041] Furthermore, drones n To the terminal i The data transmission rate is calculated using Shannon's formula: ; in, B For bandwidth, For the firstn The drone and the first i The signal-to-noise ratio between individual terminals is calculated using the following formula:

[0042] in For drones n To the terminal i The transmission power, The power of Gaussian white noise. For drones n To the terminal i The path loss is calculated using the following expression:

[0043] in, For drones n To the terminal i The line-of-sight path loss is calculated using the following expression:

[0044] For drones n To the terminal i The non-line-of-sight path loss is calculated as follows:

[0045] In the formula, For drones n With terminal i The distance between them For communication frequency, At the speed of light, and These represent signal attenuation due to shadow fading, refraction, and scattering in non-line-of-sight and line-of-sight transmissions, respectively. and Let be the probabilities of sight distance and non-sight distance, respectively, and their relationship is expressed as follows: .

[0046] Furthermore, the UAV status acquisition module transforms the UAV planning problem into one with... N The distributed part of the Markov game for each drone is observable, and the set of states is defined. S include N The state space of a drone The observation set B include N Observation of a drone The set of actions A include N The operational space of a drone The state space ,in and For the drone n exist t The horizontal and vertical coordinates of time; For the drone n At time t and the terminal i The connection variable takes a value of 1 or 0; For drones n exist t Flight altitude at any given moment; the aforementioned action space ,in For drones n exist t The distance of flight at any given moment For drones n exist t Flight direction at any moment For drones n exist t Time and Terminal i The join variable, For drones n exist t Flight altitude at any given moment; The drone planning module includes: The action decision unit is used to select an action as the optimal action decision based on the state space and observations of each UAV. The network update unit is used to calculate the state-value function value of the optimal action decision for each UAV, and update the parameters of the action decision unit based on the state-value function value; The parameters of the action decision unit include the parameters of the Actor evaluation network. Critic evaluates the parameters of the network. Parameters of the Actor target network Parameters of the Critic target network ; The evaluation network parameters of the Critic network Updates are performed using a loss function; the expression for the loss function is:

[0047] in, For state-value functions; For drones n The l Next transition state; For drones n The l Next state transition action; The evaluation network parameters of the Actor network By maximizing the update of the policy objective function, the expression of the policy objective function is:

[0048] in, For evaluating the Actor network; For the first n The actions of the drone; For the first n The first drone l The second observation; The target network parameters of the Critic network are updated through a first function, the expression of which is:

[0049] The target network parameters of the Actor network are updated through a second function, the expression of which is:

[0050] In the above formula, It is a positive constant that is close to 0.

[0051] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described resilient-enhanced multi-UAV planning method.

[0052] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0053] Figure 1 This diagram illustrates the connection relationships between drones, satellites, and terminals in a network. Figure 2 This is a schematic diagram illustrating the changes in the toughness of a drone. Figure 3 This is a structural diagram of the multi-UAV planning device based on resilience enhancement according to the present invention; Figure 4 This is a flowchart of the multi-UAV planning method based on resilience enhancement of the present invention. Detailed Implementation

[0054] Please see Figure 1This diagram illustrates the connection relationships between drones, satellites, and terminals in a network. Each drone is equipped with two communication interfaces: one for communication with the satellite and the other for communication with the terminal. Existing drone path planning methods fail to consider task allocation and path coordination in multi-drone collaborative scenarios, leading to decreased network efficiency. This efficiency degradation is typically caused by unpredictable disturbances within the system, such as new tasks, traffic congestion, or partial drone malfunctions. This invention first designs a network resilience index to evaluate the network's ability to recover from disturbances.

[0055] Specifically, please refer to Figure 2 This is a schematic diagram illustrating how network performance changes over time when a disturbance occurs. Represents network performance. t The time frame represents the entire process from the occurrence of a disturbance to the network's recovery to normal levels, including three stages and five states: the disturbance defense stage, the disturbance persistence stage, the disturbance recovery stage, and the initial state, degradation state, steady state, recovery state, and termination state. to This is the disturbance defense phase, at which time... The network remained stable during this period. When the disturbance occurred, to This is the stage where the disturbance continues to have an impact; to of Started to fall and At this point, network performance begins to decline; to This belongs to the steady-state stage after degradation, at which point... Maintain stability; to This is the recovery phase after degradation. Q(t) Gradually recovering to the state before the disturbance occurred The level. The next phase is the disturbance recovery phase.

[0056] In this invention, network resilience is used G The following expression is used to evaluate the network's resilience to disturbances:

[0057] In the formula, G Representing network resilience, Represents network performance. This represents the starting point of the phase where the disturbance continues to have an impact. This represents the end point of the phase in which the disturbance continues to have an impact.

[0058] Specifically, the performance of the network The following formula is used to calculate:

[0059] in, For time base parameters, This indicates the data transmission time between the drone and the terminal. Indicates terminal i The amount of data, Indicates drone n To the terminal i The data transmission rate.

[0060] Furthermore, drones n To the terminal i Data transmission rate Calculated using Shannon's formula: ; in, B For bandwidth, For the first n The drone and the first i The signal-to-noise ratio between individual terminals is calculated using the following formula:

[0061] in For drones n To the terminal i The transmission power, The power of Gaussian white noise. For drones n To the terminal i The path loss is calculated using the following expression:

[0062] in, For drones n To the terminal i The line-of-sight path loss is calculated using the following expression:

[0063] For drones n To the terminal i The non-line-of-sight path loss is calculated as follows:

[0064] In the formula, For drones n With terminal i The distance between them For communication frequency, At the speed of light, and These represent signal attenuation due to shadow fading, refraction, and scattering in non-line-of-sight and line-of-sight transmissions, respectively. and These are the probabilities of sight distance and non-sight distance, respectively, and their relationship is expressed as follows:

[0065] Meanwhile, to ensure that the UAV is both efficient and safe when performing tasks, this invention defines the following constraints to limit the UAV's planning and constructs the following conditional optimization model to maximize the resilience of the UAV during planning. The expression of the conditional optimization model is as follows:

[0066] in, C1 Define a set to restrict the join variables. For drones n With terminal i The connection variable between them takes a value of 0 or 1, representing time. t Inside, drones n Connect to terminal? i . C2 and C3 The connection restrictions between the drone and the terminal indicate the limitations for each drone. n and time t At least one terminal i Connect to the drone; simultaneously, for each terminal i and time t Only one drone can be connected to the terminal at a time to ensure a valid connection between the drone and the terminal. C4 restricts the drone's flight range. For drones n In time t The location coordinates must be located within the preset flight area. S Within the safe zone, ensure that the equipment operates. C5 Due to the data transmission rate requirements from the terminal to the over-the-air access device, For drones n With terminal i In time t The data transmission rate must meet or exceed the required minimum rate. . C6 Indicates the altitude limit for drones. The drone's flight altitude needs to be controlled within its maximum flight altitude. Inside.

[0067] The goal of this invention is to improve network resilience through a multi-UAV planning method and apparatus based on resilience enhancement. It utilizes the MADDPG algorithm for multi-UAV planning. (See also...) Figure 3 and Figure 4 ,in Figure 3 This is a structural diagram of the multi-UAV planning device based on resilience enhancement according to the present invention. Figure 4 The flowchart of the multi-UAV planning method based on resilience enhancement of the present invention is shown below; the multi-UAV planning device based on resilience enhancement includes a UAV status acquisition module 10 and a UAV planning module 20.

[0068] The UAV state acquisition module 10 is used to perform step S1: acquire the UAV state set, observation set, and action set. The state set contains the state space of each UAV, the observation set contains the observations of each UAV, and the action set contains the actions of each UAV.

[0069] Specifically, in this embodiment, the UAV planning problem is first transformed into a distributed partially observable Markov game with N UAVs, and the state set is defined. S include N The state space of a drone The observation set B include N Observation of a drone The set of actions A include N The operational space of a drone .

[0070] The state space ,in and For the drone n exist t The horizontal and vertical coordinates of time; For the drone n At time t and the terminal i The connection variable takes a value of 1 or 0; For drones n exist t The flight altitude at any given moment.

[0071] The action space ,in For drones n exist t The distance of flight at any given moment For drones n exist t Flight direction at any moment For drones n exist tTime and Terminal i The join variable, For drones n exist t The flight altitude at any given moment.

[0072] The UAV planning module 20 is used to execute step S2: based on the state space and observations of each UAV, select an action as the optimal action decision using a MADDPG algorithm, calculate the state-value function value of the optimal action decision for each UAV, and update the parameters of the MADDPG algorithm in the UAV planning module 20 based on the state-value function value.

[0073] The drone planning module 20 includes an action decision unit 21. In this embodiment, the action decision unit 21 is an Actor-Critic network.

[0074] The action decision unit 21 is used to execute step S21: based on the state space and observation of each UAV, select an action as the optimal action decision through a MADDPG algorithm.

[0075] Specifically, in step S21, in the first t Within each time slot, the set of states comprised of all drones is as follows:

[0076] For each time slot t Each drone, based on its corresponding observation... and the current state Strategies are formulated through Actor networks. Select the time slot from its action space. t China makes N A drone towards I The strategy that maximizes the total rate at which information is collected by all ground IoT devices is selected as the optimal strategy, and the corresponding action within the action space is executed.

[0077] Then the first N The actions of a drone can be represented as:

[0078] Preferably, the UAV planning module 20 further includes a network update unit 22.

[0079] The network update unit 22 is used to perform step S22: calculate the state-value function value of the optimal action decision for each UAV, and update the parameters of the MADDPG algorithm of the action decision unit 21 in step S21 according to the state-value function value.

[0080] Specifically, in step S22, the Actor-Critic network includes an Actor target network, an Actor evaluation network, a Critic target network, and a Critic evaluation network.

[0081] The actions in the policy selected by the Actor network are evaluated using the state-value function of the Critic network.

[0082] The expression for the state-value function is:

[0083] in Indicates discount factor, Indicates the power parameter of the discount factor. Indicates drone n reward function value, Indicates drone n Input status, Indicates drone n The action performed.

[0084] The reward function is used to measure the effectiveness of a drone taking an action in a given state. The calculation expression of the reward function is as follows:

[0085] in, G For resilience, Indicates the penalty factor. p1, p2, p3, p4 All of these are penalty terms, and the calculation expressions for each penalty term are as follows:

[0086] in, For drones n With terminal i The connection variable between them takes a value of 0 or 1; For drones n In time t Position coordinates; For drones n With terminal i In time t Data transmission rate; This refers to the flight altitude of the drone; The discriminant function is expressed as follows:

[0087] In the formula, K When =1, it represents a penalty term. p1The corresponding reward function value; K When =2, it represents a penalty term. p2 The corresponding reward function value; K When =3, it represents a penalty term. p3 The corresponding reward function value; K When =4, it represents a penalty term. p4 The corresponding reward function value. The penalty term. p1 The optimization conditions corresponding to the aforementioned conditional optimization model C1-C3 The penalty item p2 The optimization conditions corresponding to the aforementioned conditional optimization model C4 The penalty item p3 The optimization conditions corresponding to the aforementioned conditional optimization model C5 The penalty item p4 The optimization conditions corresponding to the aforementioned conditional optimization model C6 .

[0088] In step S22, the parameters that need to be updated include the parameters of the Actor evaluation network. Parameters of the Critic evaluation network Parameters of the Actor target network Parameters of the Critic target network .

[0089] The evaluation network parameters of the Critic network Updates are performed using a loss function; the expression for the loss function is:

[0090] in, For state-value functions; For drones n The l Next transition state; For drones n The l Next state transition action.

[0091] The evaluation network parameters of the Actor network By maximizing the update of the policy objective function, the expression of the policy objective function is:

[0092] in, For evaluating the Actor network; For the first n The actions of the drone; For the first n The first dronel This is the second observation.

[0093] The target network parameters of the Critic network are updated through a first function, the expression of which is:

[0094] The target network parameters of the Actor network are updated through a second function, the expression of which is:

[0095] In the above formula, It is a positive constant close to 0.

[0096] Preferably, since the state, action data, and state-value function generated by the UAV are sequentially continuous, directly using this continuous data to update the network parameters would cause the samples to not satisfy the assumption of independent and identically distributed distribution. Furthermore, to improve data utilization, in step S22, an experience pool is provided to store the state, action, and state-value function of each UAV. Experience samples are formed in the experience pool to update the parameters of the Actor evaluation network. Parameters of the Critic evaluation network Parameters of the Actor target network Parameters of the Critic target network .

[0097] Specifically, the size of the experience pool is D First, a small batch of experience samples is randomly drawn from the experience pool and input into the drone one by one. The Actor and Critic networks update their parameters based on each input mini-batch of samples. In the l-th transition, the Critic network adjusts its evaluation network parameters using a loss function, expressed as:

[0098] in, For state-value functions; For drones n The l Next transition state; For drones n The l Next state transition action. When When the loss function is continuous and differentiable, the Critic network adjusts itself based on the gradient of the loss function.

[0099] Since the Actor Network requires an action decision for each observation, each drone aims to maximize its cumulative reward. Therefore, the network parameters of the Actor Network are updated by maximizing the policy objective function. The policy objective function is expressed as: .

[0100] in, Let Actor be the evaluation network function of the Actor network, which represents the deterministic policy. ; For the first n The actions of the drone; For the first n The first drone l This is the second observation. Since the action space is continuous, D Continuously differentiable, meaning it can be updated Adjust its descent direction.

[0101] The target network parameters of the Critic network are updated through a first function, the expression of which is:

[0102] The target network parameters of the Actor network are updated through a second function, the expression of which is:

[0103] In the above formula, It is a positive constant close to 0.

[0104] Compared to existing multi-UAV planning methods, the multi-UAV planning method and apparatus based on resilience enhancement of this invention designs a resilience index that reflects the anti-interference ability of a multi-UAV network when facing disturbances, and plans multiple UAVs using the MADDPG algorithm with the goal of improving resilience. This achieves the goal of improving the resilience of a multi-UAV network when planning multiple UAVs within the network.

[0105] Based on the same inventive concept, this application also provides an electronic device, which can be a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). The device includes one or more processors and a memory, wherein the processor is used to execute a program to implement the resilient-enhanced multi-UAV planning method of the embodiments of the present invention; the memory is used to store computer programs executable by the processor.

[0106] Based on the same inventive concept, this application also provides a computer-readable storage medium corresponding to the aforementioned embodiment of a multi-UAV planning method based on resilience enhancement. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the multi-UAV planning method based on resilience enhancement described in any of the above embodiments.

[0107] This application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0108] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and the present invention also intends to include these modifications and variations.

Claims

1. A multi-UAV planning method based on resilience enhancement, characterized in that: Includes the following steps: S1: Obtain the state set, observation set, and action set of the drone. The state set contains the state space of each drone, the observation set contains the observations of each drone, and the action set contains the actions of each drone. S2: Based on the state space and observations of each UAV, select an action as the optimal action decision using a MADDPG algorithm, calculate the state-value function value of the optimal action decision for each UAV, and update the parameters of the MADDPG algorithm based on the state-value function value. The expression for the state-value function is: in, Indicates discount factor, Indicates the power parameter of the discount factor. Indicates drone n reward function value, Indicates drone n Input status, Indicates drone n The action performed.

2. The multi-UAV planning method based on resilience enhancement according to claim 1, characterized in that: The calculation expression for the reward function is as follows: in, G For resilience, Indicates the penalty factor. p1, p2, p3, p4 All of these are penalty terms, and the calculation expressions for each penalty term are as follows: in, For drones n With terminal i The connection variable between them takes a value of 0 or 1; For drones n In time t Position coordinates; For drones n With terminal i In time t The data transmission rate; This refers to the flight altitude of the drone; The discriminant function is expressed as follows: 。 3. The multi-UAV planning method based on resilience enhancement according to claim 2, characterized in that: drones n To the terminal i The data transmission rate is calculated using Shannon's formula: in, B For bandwidth, For the first n The drone and the first i The signal-to-noise ratio between individual terminals is calculated using the following formula: in, For drones n To the terminal i The transmission power, The power of Gaussian white noise. For drones n To the terminal i The path loss is calculated using the following expression: in, For drones n To the terminal i The line-of-sight path loss is calculated using the following expression: For drones n To the terminal i The non-line-of-sight path loss is calculated as follows: In the formula, For drones n With terminal i The distance between them For communication frequency, At the speed of light, and These represent signal attenuation due to shadow fading, refraction, and scattering in non-line-of-sight and line-of-sight transmissions, respectively. and Let be the probabilities of sight distance and non-sight distance, respectively, and their relationship is expressed as follows: .

4. The multi-UAV planning method based on resilience enhancement according to any one of claims 1-3, characterized in that: In step S1: Transform the drone planning problem into a problem with N The distributed part of the unmanned aerial vehicle (UAV) is an observable Markov game, and the set of states is defined. S include N The state space of a drone The observation set B include N Observation of a drone The set of actions A include N The operational space of a drone ； The state space ,in and For the drone n exist t The horizontal and vertical coordinates of time; For the drone n At time t and the terminal i The connection variable takes a value of 1 or 0; For drones n exist t Flight altitude at any given moment; The action space ,in For drones n exist t The distance of flight at any given moment For drones n exist t Flight direction at any moment For drones n exist t Time and Terminal i The join variable, For drones n exist t The flight altitude at any given moment.

5. The multi-UAV planning method based on resilience enhancement according to claim 4, characterized in that: Step S2 includes the following steps; S21: Based on the state space and observations of each UAV, select an action as the optimal action decision using a MADDPG algorithm; S22: Calculate the state-value function value for the optimal action decision of each UAV, and update the parameters of the MADDPG algorithm based on the state-value function value; The parameters in step S22 include the parameters of the Actor evaluation network. Parameters of the Critic evaluation network Parameters of the Actor target network Parameters of the Critic target network ; The evaluation network parameters of the Critic network Updates are performed using a loss function; the expression for the loss function is: in, For state-value functions; For drones n The l Next transition state; For drones n The l Next state transition action; The evaluation network parameters of the Actor network By maximizing the update of the policy objective function, the expression of the policy objective function is: in, For evaluating the Actor network; For the first n The actions of the drone; For the first n The first drone l The second observation; The target network parameters of the Critic network are updated through a first function, the expression of which is: The target network parameters of the Actor network are updated through a second function, the expression of which is: In the above formula, It is a positive constant close to 0.

6. The multi-UAV planning method based on resilience enhancement according to claim 5, characterized in that: In step S22, an experience pool is also provided to store the state, actions, and state-value function of each drone. Experience samples are formed in the experience pool to update the parameters of the Actor evaluation network. Parameters of the Critic evaluation network Parameters of the Actor target network Parameters of the Critic target network .

7. A multi-UAV planning device based on resilience enhancement, characterized in that: include: The UAV state acquisition module acquires the UAV's state set, observation set, and action set. The state set contains the state space of each UAV, the observation set contains the observations of each UAV, and the action set contains the actions of each UAV. The UAV planning module is used to select an action as the optimal action decision based on the state space and observation of each UAV, calculate the state-value function value of the optimal action decision for each UAV, and update the parameters of the UAV planning module based on the state-value function value. The expression for the state-value function is: in, Indicates discount factor, Indicates the power parameter of the discount factor. Indicates drone n reward function value, Indicates drone n Input status, Indicates drone n The action performed.

8. The multi-UAV planning device based on resilience enhancement according to claim 7, characterized in that: The calculation expression for the reward function is as follows: in, G For resilience, Indicates the penalty factor. p1, p2, p3, p4 All of these are penalty terms, and the calculation expressions for each penalty term are as follows: In the above formula, where, For drones n With terminal i The connection variable between them takes a value of 0 or 1; For drones n In time t Position coordinates; For drones n With terminal i In time t The data transmission rate; This refers to the flight altitude of the drone; The discriminant function is expressed as follows: 。 9. The multi-UAV planning device based on resilience enhancement according to claim 8, characterized in that: drones n To the terminal i The data transmission rate is calculated using Shannon's formula: in, B For bandwidth, For the first n The drone and the first i The signal-to-noise ratio between individual terminals is calculated using the following formula: in For drones n To the terminal i The transmission power, The power of Gaussian white noise. For drones n To the terminal i The path loss is calculated using the following expression: in, For drones n To the terminal i The line-of-sight path loss is calculated using the following expression: For drones n To the terminal i The non-line-of-sight path loss is calculated as follows: In the formula, For drones n With terminal i The distance between them For communication frequency, At the speed of light, and These represent signal attenuation due to shadow fading, refraction, and scattering in non-line-of-sight and line-of-sight transmissions, respectively. and Let be the probabilities of sight distance and non-sight distance, respectively, and their relationship is expressed as follows: .

10. The multi-UAV planning device based on resilience enhancement according to any one of claims 7-9, characterized in that: The UAV status acquisition module transforms the UAV planning problem into a function with... N The distributed part of the unmanned aerial vehicle (UAV) is an observable Markov game, and the set of states is defined. S include N The state space of a drone The observation set B include N Observation of a drone The set of actions A include N The operational space of a drone The state space ,in and For the drone n exist t The horizontal and vertical coordinates of time; For the drone n At time t and the terminal i The connection variable takes a value of 1 or 0; For drones n exist t Flight altitude at any given moment; the aforementioned action space ,in For drones n exist t The distance of flight at any given moment For drones n exist t Flight direction at any moment For drones n exist t Time and Terminal i The join variable, For drones n exist t Flight altitude at any given moment; The drone planning module includes: The action decision unit is used to select an action as the optimal action decision based on the state space and observations of each UAV. The network update unit is used to calculate the state-value function value of the optimal action decision for each UAV, and update the parameters of the action decision unit based on the state-value function value; The parameters of the action decision unit include the parameters of the Actor evaluation network. Critic evaluates the parameters of the network. Parameters of the Actor target network Parameters of the Critic target network ; The evaluation network parameters of the Critic network Updates are performed using a loss function; the expression for the loss function is: in, For state-value functions; For drones n The l Next transition state; For drones n The l Next state transition action; The evaluation network parameters of the Actor network By maximizing the update of the policy objective function, the expression of the policy objective function is: in, For evaluating the Actor network; For the first n The actions of the drone; For the first n The first drone l The second observation; The target network parameters of the Critic network are updated through a first function, the expression of which is: The target network parameters of the Actor network are updated through a second function, the expression of which is: In the above formula, It is a positive constant that is close to 0.

Images