Unmanned aerial vehicle cluster topology reconstruction method under emergency rescue task

Abstract

The invention relates to an unmanned aerial vehicle cluster topology reconstruction method under an emergency rescue task, belongs to the technical field of unmanned aerial vehicle communication, and solves the problem of low communication efficiency after existing topology reconstruction. Comprising the steps that a topology reconstruction network is constructed and trained, and the topology reconstruction network comprises a global hybrid network and local hybrid networks and proxy networks corresponding to all unmanned aerial vehicles; the proxy network receives local state information of the corresponding unmanned aerial vehicle at each moment and outputs an action function value of an action at each moment, the local hybrid network receives the action function value of the corresponding unmanned aerial vehicle and outputs a local joint action value, and the global hybrid network receives the local joint action values of all the unmanned aerial vehicles and outputs a cluster joint action value; when the unmanned aerial vehicle breaks down, local state information of the undamaged unmanned aerial vehicle and actions of the undamaged unmanned aerial vehicle at the previous moment are obtained and transmitted into the corresponding trained proxy network, and the corresponding actions are selected according to an epsilon-greedy strategy for topology reconstruction. And topology reconstruction with high communication efficiency is realized.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle communication technology, and in particular to a method for reconstructing the topology of an unmanned aerial vehicle cluster under an emergency rescue mission. Background Art

[0002] With the rapid development of drone technology, drones have been widely used in agricultural plant protection, express delivery, and disaster relief due to their strong adaptability, low cost, and flexible configuration. When disasters such as collapse, fire, and earthquake occur, the primary task is to obtain disaster information and use drones to establish a temporary communication network to provide information support for subsequent rescue. However, due to the complex and harsh post-disaster environment, drone nodes often fail. Not only will it affect the communication quality in the emergency rescue area, it will also divide the network into multiple unconnected partitions, affect data exchange between nodes in the network, and even cause network paralysis, seriously affecting the coordination ability of the entire drone cluster.

[0003] When the drone relay network is paralyzed due to node failure, topology reconstruction should be performed. By adjusting the position of the drone, the network topology from the drone to the ground station is reasonably optimized, and an information transmission path from each drone to the ground station is planned, the network has higher performance while restoring connectivity, reducing the impact of drone node failure on drone collaboration. Considering the large scale of the solution space caused by the dynamic topology of drones, the complex and numerous solution situations, and other problems, traditional heuristic algorithms are difficult to solve, so finding a suitable solution deserves close attention.

[0004] The existing topology reconstruction problem is mainly concentrated in the field of wireless sensors. Due to energy exhaustion and the influence of harsh environment, sensor nodes are prone to failure, resulting in network disconnection. According to the scale of network damage, they are divided into large-scale fault repair algorithms and small-scale fault repair algorithms. In small-scale fault repair algorithms, they can be divided into two categories: algorithms that distinguish node importance and algorithms that do not distinguish node importance. Both of them repair the network by changing the position of existing drones. The former focuses on restoring network coverage as soon as possible but increases information transmission overhead, while the latter focuses on repairing network connectivity but reduces efficiency. In large-scale fault repair algorithms, network repair is mainly completed by adding relay nodes. A lot of research focuses on how to reduce the number of relay nodes and how to improve the robustness of the network after reconnection.

[0005] At present, there are few studies specifically targeting UAV topology reconstruction technology. There are differences in application requirements and scenarios between UAV cluster communication networks and sensor networks. The dynamic topology and inter-machine collaboration of UAV networks make it difficult to directly apply wireless sensor network topology repair methods to UAV cluster communication networks. Therefore, it is particularly important to design a topology reconstruction method with high communication efficiency based on the characteristics of UAV clusters. Summary of the invention

[0006] In view of the above analysis, an embodiment of the present invention aims to provide a method for reconstructing the topology of a drone cluster under emergency rescue tasks, so as to solve the problem that the existing method does not consider the cooperative communication of drones, resulting in low communication efficiency after topology reconstruction.

[0007] The embodiment of the present invention provides a method for reconfiguring the topology of a drone cluster in an emergency rescue mission, comprising the following steps:

[0008] Based on the multi-agent reinforcement learning algorithm, a topology reconstruction network is constructed and trained. The topology reconstruction network includes: a global hybrid network and a local hybrid network and an agent network corresponding to each drone; the agent network is used to receive the local state information of the corresponding drone at each moment and the corresponding action at the previous moment and output the action function value of the action at each moment, the local hybrid network is used to receive the action function value of the corresponding drone and output the local joint action value, and the global hybrid network is used to receive the local joint action value of all drones and output the cluster joint action value;

[0009] When a drone fails, the local state information and the actions of the undamaged drones at the previous moment are obtained and passed into the corresponding trained proxy network. The action function value of each action of the drone at the current moment is output, and the corresponding action is selected according to the ε-greedy strategy for topology reconstruction.

[0010] Based on the further improvement of the above method, the proxy network is divided into a mobile sub-proxy network and a transmission sub-proxy network according to the actions of the UAV, which are used to select the moving direction of the UAV and the next-hop transmission node of the data packet respectively; the moving directions include: east, south, west, north and hovering, and the next-hop transmission node of the data packet is the neighboring node within the communication range of the UAV.

[0011] Based on the further improvement of the above method, the mobile subagent network and the transmission subagent network output two action function values, which are simultaneously transmitted into the local hybrid network of the corresponding UAV to output the local joint action value of the UAV; the network parameters of the local hybrid network are generated by a supernetwork that takes the local state information of the UAV as input.

[0012] Based on the further improvement of the above method, the network parameters of the global hybrid network are generated by a super network that takes the global state information of the drone cluster as input.

[0013] Based on the further improvement of the above method, the mobile subagent network and transmission subagent network of each drone have the same reward function, which is used to calculate the reward value obtained by each drone after performing an action; the reward value includes: data packet transmission reward value, constraint condition reward value, reconstruction cost reward value and transmission end reward value.

[0014] Based on the further improvement of the above method, the data packet transmission reward value is the inverse of the data packet transmission time of each hop, the constraint condition reward value is the preset negative reward value corresponding to the violated constraint condition, the reconstruction cost reward value is the preset negative reward value corresponding to the selected moving direction, and the transmission end reward value is the preset positive reward value for the data packet transmission to the ground base station.

[0015] Based on the further improvement of the above method, the packet transmission time is optimized with minimizing the packet transmission time, which is expressed as follows:

[0016]

[0017] in, Indicates that the data packet is from node k m To node k m+1 The transmission time of this hop, M represents the total number of hops of the data packet transmission path ξ, and W represents the set of flight positions of each UAV.

[0018] Based on the further improvement of the above method, data packet transmission also meets the following constraints: the position spacing of the UAV between adjacent moments is the product of the flight speed and the time interval; the next-hop transmission node of the data packet is within the communication range of the UAV; the remaining queue capacity of the next-hop transmission node of the data packet is greater than or equal to the size of the data packet; the sum of the energy consumed by the UAV in performing actions at each moment within the time period is less than or equal to the total energy of the UAV; the position of the UAV at each moment is within the preset flight area.

[0019] Based on the further improvement of the above method, the transmission time of each hop data packet is calculated by the following steps:

[0020] Calculate the path loss of the direct transmission link based on the next-hop transmission node type and transmission distance of the data packet;

[0021] The signal-to-noise ratio of the direct link is calculated based on the path loss value, the UAV transmission power and the noise power;

[0022] The transmission rate of the direct link is calculated based on the signal-to-noise ratio and the bandwidth of the drone;

[0023] The packet transmission time is calculated based on the size and transmission rate of the packet at each hop.

[0024] Based on the further improvement of the above method, the path loss of the direct transmission link is calculated according to the next hop transmission node type and transmission distance of the data packet, including:

[0025] If the next-hop transmission node type of the data packet is a ground base station, the probability of line-of-sight link and non-line-of-sight link is calculated according to the environment, drone altitude and transmission distance; the path loss of line-of-sight link and non-line-of-sight link is calculated according to the transmission distance, carrier frequency, speed of light and additional loss of non-line-of-sight link; the path loss of the direct link is calculated according to the probability and path loss of line-of-sight link and non-line-of-sight link;

[0026] If the next-hop transmission node type of the data packet is a drone, the path loss of the line-of-sight link is calculated based on the transmission distance, carrier frequency, and speed of light as the path loss of the direct transmission link.

[0027] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: for the scenario of emergency rescue of drone cluster communication network, the drone movement characteristics and communication task requirements are comprehensively considered, and based on the multi-agent reinforcement learning algorithm, the drone's agent network only needs local information, reducing the problem search space; the divide-and-conquer idea is used to decompose the agent network of each drone into two sub-agents to reduce the action space, one is responsible for the drone movement decision, and the other is responsible for the data packet next-hop transmission selection decision, so as to better realize the collaboration between drones; with minimizing the data packet transmission time as the optimization goal, the drone's action reward is calculated from four aspects: data packet transmission time, constraints, reconstruction cost and transmission end, and the drone mission requirements are considered while repairing the topology, so as to further optimize the communication network performance, reduce the impact of topology damage on drone collaborative communication, and improve the communication efficiency of topology reconstruction.

[0028] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings are only for the purpose of illustrating specific embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0030] Figure 1 This is a flow chart of a method for reconfiguring the topology of a drone cluster under an emergency rescue mission in an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of the structure of a topology reconstruction network in an embodiment of the present invention;

[0032] Figure 3 Schematic diagram of a drone cluster network in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0034] A specific embodiment of the present invention discloses a method for reconfiguring the topology of a drone cluster under an emergency rescue mission based on an emergency rescue environment. Figure 1 As shown, the following steps are included:

[0035] S11. Based on the multi-agent reinforcement learning algorithm, a topology reconstruction network is constructed and trained. The topology reconstruction network includes: a global hybrid network and a local hybrid network and an agent network corresponding to each drone; the agent network is used to receive the local state information of the corresponding drone at each moment and the corresponding action at the previous moment and output the action function value of the action at each moment, the local hybrid network is used to receive the action function value of the corresponding drone and output the local joint action value, and the global hybrid network is used to receive the local joint action values ​​of all drones and output the cluster joint action value.

[0036] It should be noted that, considering the need for drone clusters to complete tasks through collaborative communication, it is necessary not only to complete the repair of the drone cluster communication network in a short time, but also to ensure the collaborative ability of the drone communication network in the event of node failure. That is, drones must make decisions in a short time to ensure topology repair and select appropriate next-hop nodes to improve communication efficiency, which leads to an increase in the action space. This embodiment uses the idea of ​​divide and conquer to decompose the actions of drones into movement and data packet transmission. Correspondingly, the proxy network of each drone is divided into a mobile sub-agent network and a transmission sub-agent network, which are used to select the movement direction of the drone and the next-hop transmission node of the data packet, respectively, solving the problem of large drone action space and inter-machine collaboration, so that the drone cluster network still has high communication efficiency after topology reconstruction.

[0037] Specifically, the movement directions of the drone include: east, south, west, north and hovering, and the next-hop transmission node of the data packet is the neighboring node within the communication range of the drone.

[0038] The schematic diagram of the topology reconstruction network is as follows: Figure 2As shown in the figure, the two subagent networks of each drone are independent, and the parameters of the same type of subagent networks between different drones can be shared. The subagent network structure includes three layers in sequence: the input layer is a multi-layer perceptron neural network MLP, the middle layer is a gated recurrent neural network GRU, and the output layer is a multi-layer perceptron neural network MLP. It selects actions under the ε-greedy strategy and has the same reward function, which is used to calculate the reward value obtained after each drone performs an action. The local hybrid network receives the two action function values ​​output by the mobile subagent network and the transmission subagent network corresponding to each drone, and outputs the local joint action value after monotonically mixing; the local joint action value of all drones is output as the cluster joint action value through the global hybrid network. The parameters of the local hybrid network and the global hybrid network are generated by the super network, but the input of the super network of the local hybrid network is the local state information of the corresponding drone, and the input of the super network of the global hybrid network is the global state information of the drone cluster. The hypernetwork for generating weights in turn consists of a single linear layer followed by an absolute value activation function, ensuring that the weights of the local and global hybrid networks are non-negative. The biases are generated in the same way, but the hypernetwork for generating biases does not have an absolute value activation function and is generated by a 2-layer hypernetwork with ReLU nonlinearity.

[0039] It should be noted that the local status information includes: the location of the current UAV and the location of its neighboring nodes within its communication range, the remaining queue capacity of the current UAV and the remaining queue capacity of its neighboring nodes within its communication range, the remaining energy of the current UAV and its neighboring nodes, and the size of the data packets to be transmitted by the current UAV and its neighboring nodes; the global status information includes: the location of all UAVs, the remaining queue capacity, the remaining energy and the size of the data packets to be transmitted.

[0040] After the topology reconstruction network is constructed, it is trained through the following steps to obtain the trained topology reconstruction network:

[0041] ① Initialize the drone cluster network and obtain training samples based on the proxy network corresponding to each drone.

[0042] It should be noted that if Figure 3 As shown, the drone cluster network includes K drones, denoted as The drone is on a side with length L s km square area, all drones fly at a height of H, a uniform flight speed of V, and the drones hover near the ground base station and establish communication with the ground base station, which is the transmission end point of the data packet. In the preset time period, the number of time slots is N, and the time interval is the time slot length δ t , then the flight trajectory of the UAV, that is, the set of flight positions of each UAV, is expressed as ω k[n] represents the position of drone k in time slot n.

[0043] The data packets to be transmitted are randomly set in the drone cluster network; the experience pool is initialized to be empty. It should be noted that the experience pool is a global cache area.

[0044] The local state information of each UAV at each moment is collected in the order of time intervals within the preset time period, and is input into the mobile subagent network and transmission subagent network corresponding to the UAV together with the moving direction and the next-hop transmission node of the data packet selected by the mobile subagent network and the transmission subagent network at the previous moment. The moving direction and the next-hop transmission node of the data packet at each moment are selected by the ε-greedy strategy. Each UAV performs actions according to the moving direction and the next-hop transmission node of the data packet at each moment, calculates the reward value according to the reward function, and collects the corresponding local state information at the next moment to obtain the action at the next moment.

[0045] The local state information, moving direction, next-hop transmission node of the data packet, reward value and the corresponding local state information at the next moment of each drone are taken as a record and stored in the experience pool; the records of all drones at the same time constitute a data set, and multiple data sets are randomly sampled to obtain training samples.

[0046] Specifically, in the UAV cluster network, the record of UAV k at time t includes: local state information Direction of movement The next hop node of the data packet Reward value r k and the local state information at the next moment Each data set includes K records. If the drone has no data packets to be transmitted, the next hop transmission node of the data packet is the node where the drone itself is located.

[0047] The reward value of each record includes the data packet transmission reward value, the constraint reward value, the reconstruction cost reward value and the transmission end reward value, among which the data packet transmission reward value is the opposite of the data packet transmission time of each hop. The shorter the data packet transmission time, the greater the reward; the constraint reward value is the preset negative reward value corresponding to the violated constraint condition, the reconstruction cost reward value is the preset negative reward value corresponding to the selected moving direction, and the transmission end reward value is the preset positive reward value for the data packet transmission to the ground base station.

[0048] Furthermore, the packet transmission time is optimized to minimize the packet transmission time, which is expressed as follows:

[0049]

[0050] in, Indicates that the data packet is from node km To node k m+1 The transmission time of this hop, M represents the total number of hops of the data packet transmission path ξ.

[0051] The established optimization objectives also satisfy the following constraints:

[0052] 1) The distance between the positions of the drones at adjacent moments is the product of the flight speed and the time interval:

[0053] ||ω k [n+1]-ω k [n]||=Vδ t Formula (2)

[0054] 2) The next hop transmission node of the data packet is within the communication range of the drone:

[0055]

[0056] Where d represents the preset UAV communication range, and Respectively represent at node k m and node k m+1 The position of the drone in time slot n;

[0057] 3) The remaining queue capacity of the next-hop transmission node of the data packet is greater than or equal to the size of the data packet to avoid network congestion:

[0058]

[0059] in, Indicates that at the next hop node k m+1 The remaining queue capacity of the drone at position n in time slot n, μ represents the packet size;

[0060] 4) The sum of the energy consumed by the drone to perform actions at each moment in the time period is less than or equal to the total energy of the drone:

[0061]

[0062] in, Indicates that at node k m The energy consumed by the drone at position n to perform actions in time slot n, E max represents the total energy of the drone;

[0063] 5) The position of the drone at each moment is within the preset flight area:

[0064] φ l ≤ω k [n]≤φ u Formula (6)

[0065] Among them, φ l and φ u They represent the upper and lower boundary coordinates of the preset flight area, that is, the side length is L s The boundary coordinates of the square area of ​​km in the actual scene.

[0066] Each of the above constraints corresponds to a preset negative reward value. If any constraint is violated, a penalty will be given according to the preset negative reward value. When the mobile subagent network chooses a non-hovering moving direction, a penalty will be given according to the preset negative reward value. When the data packet is successfully transmitted to the ground base station, a reward will be given according to the preset positive reward value.

[0067] For example, in one-hop transmission, the packet transmission time is 10, and the packet transmission reward value is -10. However, the movement of the UAV exceeds the flight area, violating the fifth constraint condition, the preset negative reward value of which is -5. The selected moving direction is south, and the preset negative reward value is -2. Because the data packet is not transmitted to the ground base station, no positive reward value is obtained. The reward value obtained by the UAV after performing the action is -17.

[0068] Furthermore, in formula (1), the data packet is sent from node k m To node k m+1 The transmission time of this hop Calculate by following the steps below:

[0069] Calculate the path loss of the direct transmission link based on the next-hop transmission node type and transmission distance of the data packet;

[0070] The signal-to-noise ratio of the direct link is calculated based on the path loss value, the UAV transmission power and the noise power;

[0071] The transmission rate of the direct link is calculated based on the signal-to-noise ratio and the bandwidth of the drone;

[0072] The packet transmission time is calculated based on the size and transmission rate of the packet at each hop.

[0073] Specifically, the path loss of the direct transmission link is calculated based on the next-hop transmission node type and transmission distance of the data packet, including:

[0074] The transmission distance of each hop data packet is calculated by formula (7); if the next hop transmission node type of the data packet is a ground base station, the line-of-sight link probability is calculated by formula (8) according to the environment, drone altitude and transmission distance. The probability of non-line-of-sight link is calculated by formula (9): Based on the transmission distance, carrier frequency, speed of light and the additional loss of non-line-of-sight links, the path loss of the line-of-sight link is calculated using formula (10): The path loss of the non-line-of-sight link is calculated by formula (11): According to the probability and path loss of line-of-sight link and non-line-of-sight link, the path loss of direct link is calculated by formula (12):

[0075]

[0076]

[0077]

[0078] Among them, H represents the flight altitude of the drone, Represents node k m UAV and node k m+1 The distance between the base station / drone at time slot n, i.e., the data packet from node k m To node k m+1 The transmission distance of this hop, η1 and η2 represent probability parameters, which depend on the flight environment. For example, in a suburban environment, η1 is set to 4.88 and η2 is set to 0.43;

[0079]

[0080]

[0081]

[0082] Among them, f c represents the carrier frequency of the UAV communication system, c represents the speed of light, ψ NLOS Indicates the additional loss of the non-line-of-sight link. Its value depends on the environment. Exemplarily, it is set to 20 dB.

[0083] If the next-hop transmission node type of the data packet is a drone, the path loss of the line-of-sight link is calculated by formula (10) based on the transmission distance, carrier frequency and speed of light, and is used as the path loss of the direct transmission link.

[0084] Furthermore, according to the path loss value, the UAV transmission power and the noise power, the signal-to-noise ratio of the direct transmission link is calculated by formula (13):

[0085]

[0086] in, Indicates that at node k m The transmission power of the UAV at the location, σ 2 Represents the noise power.

[0087] Furthermore, according to the signal-to-noise ratio and the bandwidth of the UAV, the transmission rate of the direct link is calculated by formula (14):

[0088]

[0089] in, Indicates that at node k m The bandwidth of the drone at the location.

[0090] Finally, according to the size and transmission rate of each hop data packet, the data packet transmission time is calculated by formula (15):

[0091]

[0092] Where μ represents the size of the data packet.

[0093] Specifically, in formula (5) It is calculated by the following formula:

[0094]

[0095] c1=2v0 2 Formula (17)

[0096]

[0097] Among them, P0 represents the blade profile power when the drone is in the hovering state, P1 represents the induced power when the drone is in the hovering state, and U tip represents the tip speed of the UAV rotor blade, v0 represents the average rotation induced speed of the UAV, d0 represents the fuselage drag ratio, ρ is the air density, s0 represents the rotation stability, and A represents the rotor rotation area.

[0098] Compared with the existing technology, this step takes the packet transmission time as the optimization target, and calculates the drone's action reward from four aspects: packet transmission time, constraints, reconstruction cost, and transmission completion. This achieves the consideration of the drone's mission requirements while repairing the topology, further optimizing the communication network performance.

[0099] ② Build a network with the same structure as the topology reconstruction network as the corresponding target network.

[0100] It should be noted that the target network with the same structure as the topology reconstruction network is constructed to keep the target value stable and prevent overfitting, thereby improving the stability and convergence speed of the training process. During the training process, the parameters of the topology reconstruction network are updated instantly through gradients, while the parameters of the target network are not updated through gradients. Instead, the parameters of the topology reconstruction network are copied to the target network at regular intervals.

[0101] ③ In the iterative training process, based on the topology reconstruction network, the cluster joint action value of each training sample is obtained, and based on the target network, the maximum joint action value of each training sample is obtained; according to the maximum joint action value of each training sample, the reward value and attenuation factor of the corresponding training sample, the target joint action value of each training sample is obtained; the sum of the squares of the difference between the cluster joint action value and the target joint action value of the training sample is used as the error function, and the parameters of the topology reconstruction network are updated based on the gradient descent method in back propagation, and the parameters of the topology reconstruction network are periodically copied to the target network; when the maximum number of iterations is reached or the preset accuracy is reached, the trained topology reconstruction network is obtained.

[0102] Specifically, according to the K records of each randomly selected training sample and the records in the experience pool, the local state information o corresponding to each record in the training sample at the current moment is obtained. t and the previous action (a t-1,h and a t-1,d ), the corresponding mobile subagent network and transmission subagent network in the input topology reconstruction network (o t and a t-1,h Incoming transport sub-agent network, o t and a t-1,d The two actions of each record are selected through the ε-greedy strategy to obtain the two action function values ​​of each record, and the two action function values ​​of each record are passed to the corresponding local hybrid network in the topology reconstruction network to obtain the local joint action value of each record, which is then passed to the global hybrid network in the topology reconstruction network to obtain the cluster joint action value Q of each training sample. tot (o,a,s;θ); where o represents the set of local state information of the drone in the current state, a represents the set of actions of the drone in the current state, s represents the global state information in the current state, and θ represents the parameters of the topology reconstruction network.

[0103] Get the local state information o of each record in the training sample at the next moment t+1 and the current action (a t,h and a t,d ), and the corresponding mobile sub-agent network and transmission sub-agent network in the target network (o t+1 and a t,h Incoming transport sub-agent network, o t+1 and a t,d The two maximum action function values ​​of each record are obtained, and the two maximum action function values ​​of each record are passed into the corresponding local hybrid network in the target network to obtain the maximum local joint action value of each record, and then passed into the global hybrid network in the target network to obtain the maximum joint action value of each training sample. Among them, o' represents the set of local state information of the drone in the next state, a' represents the set of drone actions in the next state, s' represents the global state information in the next state, θ - Represents the parameters of the target network.

[0104] According to the maximum joint action value of each training sample, the reward value of the corresponding training sample and the attenuation factor, the target joint action value of each training sample is obtained by the following formula:

[0105]

[0106] in, represents the target joint action value of the i-th training sample, r i represents the average value of the reward value of each record in the i-th training sample, γ represents the decay factor, Represents the maximum joint action value of the i-th training sample.

[0107] According to the cluster joint action value and target joint action value of the training sample, the error function is calculated by the following formula:

[0108]

[0109] Where b represents the total number of training samples. Represents the cluster joint action value of the i-th training sample.

[0110] After the loss is obtained, the parameters of the global hybrid network, the local hybrid networks of all drones, and the proxy network are uniformly updated to maximize the local action value of each drone. When the maximum number of iterations is reached or the preset accuracy is achieved, the trained topology reconstruction network is obtained.

[0111] S12. When a drone fails, the local state information and the actions of the undamaged drones at the previous moment are obtained and passed into the corresponding trained proxy network. The action function values ​​of the actions of the drones at the current moment are output, and the corresponding actions are selected according to the ε-greedy strategy for topology reconstruction.

[0112] It should be noted that in actual situations, once a damaged drone appears, the network is reconstructed using the topology trained in step S11, and the local state information and the action at the previous moment of the undamaged drone are transmitted, so that the undamaged drone learns the action sequence according to the current local state information, and according to the action function value of each action, the corresponding action is selected according to the ε-greedy strategy to arrive at the corresponding position and transmit the data packet to the next-hop transmission node, thereby realizing the reconstruction of the network topology and reducing the impact of node failure on the drone communication network.

[0113] Compared with the prior art, the present embodiment provides a method for reconstructing the topology of a drone cluster under an emergency rescue mission. Aiming at the scenario of emergency rescue of a drone cluster communication network, the method comprehensively considers the movement characteristics of the drone and the communication mission requirements, and based on the multi-agent reinforcement learning algorithm, the drone's agent network only requires local information, reducing the problem search space; the divide-and-conquer idea is used to decompose the agent network of each drone into two sub-agents to reduce the action space, one is responsible for the drone movement decision, and the other is responsible for the next-hop transmission selection decision of the data packet, so as to better realize the collaboration between drones; with minimizing the transmission time of the data packet as the optimization goal, the drone's action reward is calculated from four aspects: data packet transmission time, constraints, reconstruction cost and transmission end, and the drone mission requirements are considered while repairing the topology, so as to further optimize the communication network performance, reduce the impact of topology damage on drone collaborative communication, and improve the communication efficiency of topology reconstruction.

[0114] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0115] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for reconstructing the topology of a drone cluster under emergency rescue missions, characterized in that: The following steps are involved: Based on the multi-agent reinforcement learning algorithm, a topology reconstruction network is constructed and trained, wherein the topology reconstruction network includes: a global hybrid network and a local hybrid network and an agent network corresponding to each drone; The proxy network is used to receive the local state information of the corresponding drone at each moment and the corresponding action at the previous moment and output the action function value of the action at each moment; the local hybrid network is used to receive the action function value of the corresponding drone and output the local joint action value; the global hybrid network is used to receive the local joint action values ​​of all drones and output the cluster joint action value; When a drone fails, the local state information and the actions of the undamaged drones at the previous moment are obtained and passed into the corresponding trained proxy network. The action function value of each action of the drone at the current moment is output, and the corresponding action is selected according to the ε-greedy strategy for topology reconstruction.

2. The method for reconfiguring the topology of a drone cluster under an emergency rescue mission according to claim 1, characterized in that: The proxy network is divided into a mobile sub-proxy network and a transmission sub-proxy network according to the actions of the drone, which are used to select the drone's moving direction and the next-hop transmission node of the data packet respectively; the moving directions include: east, south, west, north and hovering, and the next-hop transmission node of the data packet is a neighboring node within the communication range of the drone.

3. The method for reconfiguring the topology of a drone cluster under an emergency rescue mission according to claim 2, characterized in that: The mobile subagent network and the transmission subagent network output two action function values, which are simultaneously transmitted into the local hybrid network of the corresponding UAV, and the local joint action value of the UAV is output; the network parameters of the local hybrid network are generated by a super network that takes the local state information of the UAV as input.

4. The method for reconfiguring the topology of a drone cluster under an emergency rescue mission according to claim 1, characterized in that: The network parameters of the global hybrid network are generated by a super network that takes the global state information of the drone cluster as input.

5. The method for reconfiguring the topology of a drone cluster under an emergency rescue mission according to claim 2, characterized in that: The mobile subagent network and transmission subagent network of each drone have the same reward function, which is used to calculate the reward value obtained by each drone after performing an action; the reward value includes: data packet transmission reward value, constraint condition reward value, reconstruction cost reward value and transmission end reward value.

6. The method for reconfiguring the topology of a drone cluster under an emergency rescue mission according to claim 5, characterized in that: The data packet transmission reward value is the inverse of the data packet transmission time of each hop, the constraint condition reward value is the preset negative reward value corresponding to the violated constraint condition, the reconstruction cost reward value is the preset negative reward value corresponding to the selected moving direction, and the transmission end reward value is the preset positive reward value for data packet transmission to the ground base station.

7. The method for reconfiguring the topology of a drone cluster under an emergency rescue mission according to claim 6, characterized in that: The packet transmission time is optimized to minimize the packet transmission time, which is expressed as follows: in, Indicates that the data packet is from node k m To node k m+1 The transmission time of this hop, M represents the total number of hops of the data packet transmission path ξ, and W represents the set of flight positions of each UAV.

8. The method for reconfiguring the topology of a drone cluster under an emergency rescue mission according to claim 7, characterized in that: The data packet transmission also satisfies the following constraints: the position spacing between the drones at adjacent times is the product of the flight speed and the time interval; the next-hop transmission node of the data packet is within the communication range of the drone; the remaining queue capacity of the next-hop transmission node of the data packet is greater than or equal to the size of the data packet; the sum of the energy consumed by the drone to perform actions at each time within the time period is less than or equal to the total energy of the drone; The position of the drone at each moment is within the preset flight area.

9. The method for reconfiguring the topology of a drone cluster under an emergency rescue mission according to claim 7, characterized in that: The transmission time of each hop data packet is calculated by the following steps: Calculate the path loss of the direct transmission link based on the next-hop transmission node type and transmission distance of the data packet; The signal-to-noise ratio of the direct link is calculated based on the path loss value, the UAV transmission power and the noise power; The transmission rate of the direct link is calculated based on the signal-to-noise ratio and the bandwidth of the drone; The packet transmission time is calculated based on the size and transmission rate of the packet at each hop.

10. The method for reconfiguring the topology of a drone cluster under an emergency rescue mission according to claim 9, characterized in that: The step of calculating the path loss of the direct transmission link according to the next hop transmission node type and the transmission distance of the data packet includes: If the next-hop transmission node type of the data packet is a ground base station, the probability of line-of-sight link and non-line-of-sight link is calculated according to the environment, drone altitude and transmission distance; the path loss of line-of-sight link and non-line-of-sight link is calculated according to the transmission distance, carrier frequency, speed of light and additional loss of non-line-of-sight link; the path loss of the direct link is calculated according to the probability and path loss of line-of-sight link and non-line-of-sight link; If the next-hop transmission node type of the data packet is a drone, the path loss of the line-of-sight link is calculated based on the transmission distance, carrier frequency, and speed of light as the path loss of the direct transmission link.

Images