Search and rescue unmanned aerial vehicle cluster exploration method based on neighborhood random network distillation

Abstract

The invention provides a search and rescue unmanned aerial vehicle cluster exploration method based on neighborhood random network distillation, and the method comprises the steps: initializing the state of an unmanned aerial vehicle, and network parameters and hyper-parameters in the training of the unmanned aerial vehicle; joint observation of the unmanned aerial vehicle is constructed into neighborhood observation data, and neighborhood observation is converted into neighborhood representation through a coding layer; the neighborhood representation obtains an attention weight through an attention module in the neighborhood random network; the neighborhood representation obtains a communication weight through a communication module in the neighborhood random network; the attention weight and the communication weight obtain exploration representation of the unmanned aerial vehicle cluster through an output layer, and internal rewards are calculated; performing weighted calculation on the internal rewards and the environment rewards to obtain comprehensive rewards; and updating network parameters of unmanned aerial vehicle training, and performing next iteration to obtain an unmanned aerial vehicle cluster control model. According to the method, the unmanned aerial vehicle cluster can make full use of the neighborhood relation between the unmanned aerial vehicles in the search and rescue environment with sparse rewards to achieve efficient exploration of the environment, and intelligent control of the unmanned aerial vehicle cluster is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent control of unmanned aerial vehicle clusters, and in particular relates to a search and rescue unmanned aerial vehicle cluster exploration method based on neighborhood random network distillation. Background Art

[0002] In disaster relief, search and rescue drone swarm systems are extremely important due to their ability to be quickly deployed and efficiently explored. In difficult-to-reach and dangerous areas, drones can effectively increase the search and rescue range and speed, reduce the risks to search and rescue personnel, and save as many lives as possible. However, despite the significant progress made in search and rescue drone technology in recent years, there are still some technical challenges that need to be solved.

[0003] First, in the search and rescue process, the uncertainty of the search and rescue target and the sparsity of training data make the search and rescue process of drones extremely difficult. In addition, in the search and rescue mission, the time nodes before and after successfully finding the missing person are relatively short, so the reward signal obtained by the drone is sparse, and it is difficult to provide effective feedback for real-time decision-making. Secondly, although the currently commonly used multi-agent exploration method has achieved certain results, it often does not take into account the spatial neighborhood relationship between drones, and targeted dynamic coordination is required between each drone. In addition, the environment in the search and rescue mission may contain complex terrain changes, sudden weather conditions and a variety of obstacles, all of which may affect the exploration strategy and path planning of the drone.

[0004] CN111062491B discloses a method for exploring unknown environments by an intelligent agent based on reinforcement learning, which extracts local information, calculates the difference of local information, and constructs an internal reward function to give the intelligent agent a human-like selection ability. However, the fixed extraction rules make it difficult to adapt to dynamic sparse environments, and it relies on local features, and the overall exploration efficiency is low. CN115617034B discloses a multi-agent environmental exploration method, device, electronic device and storage medium, which obtains image and posture information, constructs and merges topological maps, generates a relationship matrix in a hierarchical network, predicts action distribution, and plans exploration paths. However, the application scenarios of this technology are limited, and the coordination between each intelligent agent is difficult. CN117350361A provides a multi-agent reinforcement learning method for decoupling exploration and utilization strategies, which decouples the exploration strategy and the utilization strategy, and trains them separately to balance the contradiction between exploration and utilization. However, after decoupling, the information sharing between the intelligent agents is insufficient, which affects the efficiency of each intelligent agent in exploring sparse environments.

[0005] In order to adapt to the above environment, the drone cluster needs to have a method between the neighborhoods that can quickly and dynamically explore the sparsely distributed target points in the search and rescue environment. In view of this, the present invention is committed to solving the technical problem of insufficient collaborative exploration efficiency of drone clusters in sparse reward environments. Summary of the invention

[0006] In order to solve the above technical problems, the purpose of the present invention is to provide a UAV swarm intelligent decision-making method based on neighborhood random network distillation, which can enable the UAV swarm to efficiently explore the environment and achieve high-quality and rapid completion of search and rescue missions.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is: a drone cluster intelligent decision-making method based on neighborhood random network distillation, comprising the following steps:

[0008] Step S1: Initialize the drone state, as well as the network parameters and hyperparameters in drone training;

[0009] Step S2: construct the joint observation of UAVs into neighborhood observation data, and transform the neighborhood observation into neighborhood representation through the encoding layer;

[0010] Step S3: The neighborhood representation obtains the attention weight through the attention module in the neighborhood random network;

[0011] Step S4: The neighborhood representation obtains the communication weight through the communication module in the neighborhood random network;

[0012] Step S5: The attention weight and communication weight are used to obtain the exploration representation of the drone cluster through the output layer, and the intrinsic reward is calculated;

[0013] Step S6: Perform weighted calculation on the intrinsic reward and the environmental reward to obtain a comprehensive reward;

[0014] Step S7: Update the network parameters in the UAV training and perform the next round of iteration to obtain the UAV cluster control model.

[0015] Preferably, step S1 initializes the drone state, as well as the network parameters and hyperparameters in drone training, specifically:

[0016] Step S1.1: Initialize the individual Q function of the drone in the reinforcement learning network [Q i ], individual advantage function [A i ], individual positive importance weight [λ i ], joint Q function Q tot The network parameters θ and the target network parameters θ of the reinforcement learning network - ;

[0017] Step S1.2: Initialize the target network parameters in the exploration module Prediction network parameters φ, and decoder network parameters η;

[0018] Step S1.3: Initialize training hyperparameters, including learning rate and number of iterations.

[0019] Preferably, step S2 constructs the joint observation of the drone into neighborhood observation data, and transforms the neighborhood observation into neighborhood representation through the coding layer, specifically:

[0020] Step S2.1: Construct the adjacency matrix M through the neighborhood relationship of the drone, expressed as:

[0021]

[0022] Among them, m ij Indicates whether drone i and drone j are adjacent. The judgment method is as follows:

[0023]

[0024] Where m represents the neighborhood threshold, dist(·) is the Euclidean distance calculation function in three-dimensional space;

[0025] Step S2.2: The matrix dot product of the adjacency matrix M of the drone cluster and the joint observation O is used as the joint neighborhood observation C = [c1, c2, ..., c n ], the calculation process is as follows:

[0026] C=M·O T

[0027] Among them, joint observation O = [o1, o2, ..., o n Each element o in ] i is the individual observation of the UAV, and the joint neighborhood observation C = [c1, c2, ..., c n Each element c in ] i Provides individual drone neighborhood observations;

[0028] Step S2.3: Input the individual drone neighborhood observations into the encoding layer in the exploration module prediction network Get the exploration module prediction network neighborhood representation

[0029] Step S2.4: Input the individual drone neighborhood observations into the encoding layer of the exploration module target network Get the exploration module target network neighborhood representation

[0030] Preferably, in step S3, the neighborhood representation obtains the attention weight through the attention module in the neighborhood random network, specifically:

[0031] Step S3.1: Calculate the attention vector between drone i and other neighboring drones in the prediction network The calculation is as follows:

[0032]

[0033] Among them, the query matrix and key matrix Respectively and Converted into query vector and key vector, B i represents the set of all drones except drone i;

[0034] Step S3.2: Pass Calculate the attention weight of drone i in the prediction network The calculation is as follows:

[0035]

[0036] Step S3.3: According to the method of steps 3.1 and 3.2, the target network is queried by the matrix Q φ , key matrix K φ Sum value matrix V φ ,Will Converted to attention weight

[0037] Preferably, in step S4, the neighborhood characterization obtains the communication weight through the communication module in the neighborhood random network, specifically:

[0038] Step S4.1: Compute the update gate in the prediction network gated recurrent unit (GRU) The calculation is as follows:

[0039]

[0040] in, and represents the weight matrix of the update gate in the prediction network, represents the bias vector of the update gate in the prediction network, Represents the hidden state of the previous moment;

[0041] In the same way, update the weight matrix of the gate in the target network and Bias Vector Hidden state at the last moment as well as Get the target network update gate

[0042] Step S4.2: Calculate the reset gate in the prediction network GRU The calculation is as follows:

[0043]

[0044] in, and represents the weight matrix of the reset gate in the prediction network, Represents the bias vector of the reset gate in the prediction network;

[0045] In the same way, reset the weight matrix of the gate in the target network and Bias Vector Hidden state at the last moment as well as Get the target network reset gate

[0046] Step S4.3: Calculate candidate values ​​in the prediction network GRU The calculation is as follows:

[0047]

[0048] in, and represents the weight matrix of candidate values ​​in the prediction network, Represents the bias vector of the candidate value in the prediction network, and W represents the matrix dot product;

[0049] In the same way, the weight matrix of the candidate value in the target network is and Bias Vector Hidden state at the last moment as well as Get the target network candidate value

[0050] Step S4.4: Generate hidden state The calculation is as follows:

[0051]

[0052] In the same way, update the gate in the target network and candidate values Get the target network hidden state

[0053] Preferably, in step S5, the attention weight and the communication weight are used to obtain the exploration representation of the drone cluster through the output layer, and the intrinsic reward is calculated, specifically:

[0054] Step S5.1: and Input to the output layer of the prediction network The prediction network exploration representation is obtained from

[0055] Will and Input to the output layer of the target network The target network exploration representation is obtained from

[0056] Step S5.2: Calculate the intrinsic reward r of drone i i int , calculated as follows:

[0057]

[0058] Preferably, step S6 performs weighted calculation of the intrinsic reward and the environmental reward to obtain a comprehensive reward, specifically:

[0059] Step S6.1: Calculate the total reward r of drone i i , is the environment exploration reward r i ext With intrinsic rewards i int The sum of:

[0060] r i =βr i int +r i ext

[0061] Among them, β represents the scaling factor that adjusts the ratio between rewards.

[0062] Preferably, step S7 updates the network parameters in the drone training and performs the next round of iteration to obtain the drone cluster control model, specifically:

[0063] Step S7.1: Update the prediction network parameters by predicting the network error The update method is as follows:

[0064]

[0065] in, is the learning rate of the prediction network;

[0066] Step S7.2: According to the state value function V of drone i i (·), advantage function A i (·), global state O, calculate the state value function V of drone i in the global state i(τ), advantage function A i (τ,a i ), calculated as follows:

[0067] V i (τ) = w i (τ)V i (τ i )+b i (τ)

[0068] A i (τ,a i )=w i (τ)A i (τ i ,a i )+b i (τ)

[0069] Among them, τ is the joint action-observation trajectory, τ i is the action-observation trajectory of aircraft i, w(·) is a positive weight function;

[0070] Step S7.3: Calculate the joint state value function V tot (τ) and the joint advantage function A tot (τ,a), is calculated as follows:

[0071]

[0072] Among them, λ i (τ,a) is calculated as follows:

[0073]

[0074] Among them, K represents the number of heads, φ i,k represents the attention weight activated by sigmoid, v k represents the positive bond of each head;

[0075] Step S7.4: Pass the joint state value function V of drone i tot (τ), joint advantage function A tot (τ, a) and the global state s, calculate the joint action value function Q tot (τ,a), is calculated as follows:

[0076]

[0077] Step S7.5: Calculate the reinforcement learning time difference error function L RL , calculated as follows:

[0078]

[0079] in, The calculation method is as follows:

[0080]

[0081] Step S7.6: Update reinforcement learning network parameters θ and soft update target network parameters θ - , the update method is as follows:

[0082]

[0083]

[0084] in, represents the soft update ratio, α θ Represents the learning rate.

[0085] Compared with the prior art, the present invention has the following beneficial effects:

[0086] (1) The present invention enables the drone cluster to fully extract the neighborhood features of each drone using the attention mechanism during task execution, and focus on the collaborative relationship between drones and neighboring drones.

[0087] (2) The present invention can guide UAVs to explore efficiently, use contrastive learning to enhance the role of attention mechanism in UAV exploration, and improve the exploration ability of UAV clusters in sparse reward environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is a flow chart of the search and rescue drone cluster exploration method based on neighborhood random network distillation of the present invention;

[0089] Figure 2 Schematic diagram of the neighborhood random network distillation method of the present invention;

[0090] Figure 3 A schematic diagram of the change of average success rate during training in one embodiment of the present invention;

[0091] Figure 4 It is a schematic diagram of the change of average collision rate during training in one embodiment of the present invention. DETAILED DESCRIPTION

[0092] Unless otherwise specified, any feature disclosed throughout this specification may be replaced by an alternative feature that is equivalent or has a similar purpose. That is, each feature is only an example in a series of equivalent or similar features.

[0093] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be described in more detail below in conjunction with the drawings of the specification. It should be noted that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0094] Example:

[0095] Based on previous work, this paper innovatively proposes a search and rescue drone cluster exploration method based on neighborhood random network distillation, which can be applied to drone cluster collaboration.

[0096] Reference Figure 1 According to the present invention, a search and rescue drone cluster exploration method based on neighborhood random network distillation includes:

[0097] Step S1 initializes the drone state, as well as the network parameters and hyperparameters in drone training, specifically:

[0098] Step S1.1: Initialize the individual Q function of the drone in the reinforcement learning network [Q i ], individual advantage function [A i ], individual positive importance weight [λ i ], joint Q function Q tot The network parameters θ and the target network parameters θ of the reinforcement learning network - .

[0099] Step S1.2: Initialize the target network parameters in the exploration module Prediction network parameters φ, and decoder network parameters η.

[0100] Step S1.3: Initialize training hyperparameters, including learning rate and number of iterations.

[0101] Reference Figure 2 , Steps S2 to S6 describe the process of generating intrinsic rewards by the neighborhood random network distillation method.

[0102] Step S2 constructs the joint observation of the UAV into neighborhood observation data, and transforms the neighborhood observation into neighborhood representation through the encoding layer, specifically:

[0103] Step S2.1: Construct the adjacency matrix M through the neighborhood relationship of the drone, expressed as:

[0104]

[0105] Among them, m ij Indicates whether drone i and drone j are adjacent. The judgment method is as follows:

[0106]

[0107] Where m represents the neighborhood threshold, and dist(·) is the Euclidean distance calculation function in three-dimensional space.

[0108] Step S2.2: The matrix dot product of the adjacency matrix M of the drone cluster and the joint observation O is used as the joint neighborhood observation C = [c1, c2, ..., c n ], the calculation process is as follows:

[0109] C=M·O T

[0110] Among them, joint observation O = [o1, o2, ..., o n Each element o in ] i is the individual observation of the UAV, and the joint neighborhood observation C = [c1, c2, ..., c n Each element c in ] i Observe the neighborhood of an individual drone.

[0111] Step S2.3: Input the individual drone neighborhood observations into the encoding layer in the exploration module prediction network Get the exploration module prediction network neighborhood representation

[0112] Step S2.4: Input the individual drone neighborhood observations into the encoding layer of the exploration module target network Get the exploration module target network neighborhood representation

[0113] Step S3: The neighborhood representation obtains the attention weight through the attention module in the neighborhood random network, specifically:

[0114] Step S3.1: Calculate the attention vector between drone i and other neighboring drones in the prediction network The calculation is as follows:

[0115]

[0116] Among them, the query matrix and key matrix Respectively and Converted into query vector and key vector, B i Represents the set of all drones except drone i.

[0117] Step S3.2: Pass Calculate the attention weight of drone i in the prediction network The calculation is as follows:

[0118]

[0119] Step S3.3: According to the method of steps 3.1 and 3.2, the target network is queried by the matrix Q φ , key matrix K φ Sum value matrix V φ ,Will Converted to attention weight

[0120] Step S4: The neighborhood characterization obtains the communication weight through the communication module in the neighborhood random network, specifically:

[0121] Step S4.1: Calculate the update gate in the prediction network gated recurrent unit GRU The calculation is as follows:

[0122]

[0123] in, and represents the weight matrix of the update gate in the prediction network, represents the bias vector of the update gate in the prediction network, Represents the hidden state of the previous moment. In the same way, the weight matrix of the gate is updated in the target network. and Bias Vector Hidden state at the last moment as well as Get the target network update gate

[0124] Step S4.2: Calculate the reset gate in the prediction network GRU The calculation is as follows:

[0125]

[0126] in, and represents the weight matrix of the reset gate in the prediction network, Represents the bias vector of the reset gate in the prediction network. In the same way, the weight matrix of the reset gate in the target network is and Bias Vector Hidden state at the last moment as well as Get the target network reset gate

[0127] Step S4.3: Calculate candidate values ​​in the prediction network GRU The calculation is as follows:

[0128]

[0129] in, and represents the weight matrix of candidate values ​​in the prediction network, Represents the bias vector of the candidate value in the prediction network, and W represents the matrix dot product. In the same way, the weight matrix of the candidate value in the target network is and Bias Vector Hidden state at the last moment as well as Get the target network candidate value

[0130] Step S4.4: Generate hidden state The calculation is as follows:

[0131]

[0132] In the same way, update the gate in the target network and candidate values Get the target network hidden state

[0133] Step S5: The attention weight and communication weight are used to obtain the exploration representation of the drone cluster through the output layer, and the intrinsic reward is calculated, which is:

[0134] Step S5.1: and Input to the output layer of the prediction network The prediction network exploration representation is obtained from Will and Input to the output layer of the target network The target network exploration representation is obtained from

[0135] Step S5.2: Calculate the intrinsic reward r of drone i i int , calculated as follows:

[0136]

[0137] Step S6 performs weighted calculation of the intrinsic reward and the environmental reward to obtain a comprehensive reward, which is specifically:

[0138] Step S6.1: Calculate the total reward r of drone i i , is the environment exploration reward r i extWith intrinsic rewards i int The sum of:

[0139] r i =βr i int +r i ext

[0140] Among them, β represents the scaling factor that adjusts the ratio between rewards.

[0141] Step S7 updates the network parameters in the drone training and performs the next round of iteration to obtain the drone cluster control model, which is specifically:

[0142] Step S7.1: Update the prediction network parameters by predicting the network error The update method is as follows:

[0143]

[0144] in, is the learning rate of the prediction network.

[0145] Step S7.2: According to the state value function V of drone i i (·), advantage function A i (·), global state O, calculate the state value function V of drone i in the global state i (τ), advantage function A i (τ,a i ), calculated as follows:

[0146] V i (τ) = w i (τ)V i (τ i )+b i (τ)

[0147] A i (τ,a i )=w i (τ)A i (τ i ,a i )+b i (τ)

[0148] Among them, τ is the joint action-observation trajectory, τ i is the action-observation trajectory of vehicle i, and w(·) is a positive weight function.

[0149] Step S7.3: Calculate the joint state value function V tot (τ) and the joint advantage function A tot(τ,a), is calculated as follows:

[0150]

[0151] Among them, λ i (τ,a) is calculated as follows:

[0152]

[0153] Among them, K represents the number of heads, φ i,k represents the attention weight activated by sigmoid, v k Represents the positive bonds of each head.

[0154] Step S7.4: Pass the joint state value function V of drone i tot (τ), joint advantage function A tot (τ, a) and the global state s, calculate the joint action value function Q tot (τ,a), is calculated as follows:

[0155]

[0156] Step S7.5: Calculate the reinforcement learning time difference error function L RL , calculated as follows:

[0157]

[0158] in, The calculation method is as follows:

[0159]

[0160] Step S7.6: Update reinforcement learning network parameters θ and soft update target network parameters θ - , the update method is as follows:

[0161]

[0162]

[0163] in, represents the soft update ratio, α θ Represents the learning rate.

[0164] After executing steps S1 to S7, a PX4-based UAV swarm collaborative task simulation environment is designed and built for verification, simulating the scenario of UAV swarm collaborative reconnaissance and rescue targets. Five UAVs and five target points are created in the simulation platform Gazebo, and spheres, cubes, and any other irregular obstacles are added to the training scene. In addition, collision properties are added to obstacles, target locations, and UAVs.

[0165] The external environment reward setting includes three aspects. The first is the inspection and processing of collision events. If a positive object (mission target) is collided with, a reward of 10 is given. When all drones reach the corresponding target position, a reward of 100 is given and this round is successful, otherwise it fails until the specified time is reached. If a negative object is collided with: if it collides with an obstacle, a reward of -10 is given, and the damage degree of the drone equipment is increased by 10. If the damage degree reaches 30, it fails; if it collides with other drones, a reward of -20 is given, and the damage degree of the drone equipment is increased by 20. If the damage degree reaches 30, it fails. The second is to control cross-border flight. According to the high-precision GPS positioning information carried by the drone, it is monitored in real time whether the current drone crosses the boundary. If it crosses the boundary, a reward of -10 is given. The third is power monitoring. If the drone's power is less than 10%, a reward of -5 is given, and it fails.

[0166] The simulation training paused every 2000 time steps, conducted 32 tests, and performed a total of 400,000 iterations. The learning rate of reinforcement learning was set to 0.0001, the soft update ratio was set to 0.01, and the proportional factor for adjusting the ratio between rewards was set to 0.5. In order to eliminate contingency, all algorithms repeated the simulation training 5 times and took the median. Under the sparse reward setting, simulation training was performed on the method proposed by the present invention and the method not improved by the method proposed by the present invention, and the average success rates were 91.3% and 71.9%, respectively, and the average collision rates were 0% and 0.8%, respectively. The changes in the average success rate and average collision rate of the two methods during the training process are shown in the figure below. Figure 3 and Figure 4 shown.

[0167] The above embodiments are only preferred embodiments of the present invention, and do not limit the scope of the present invention. For those skilled in the art, without departing from the design spirit of the present invention, the present invention may be subjected to several deformations, improvements and equivalent substitutions, and these deformations, improvements and equivalent substitutions are deemed to fall within the protection scope of the claims of the present invention.

Claims

1. A search and rescue drone cluster exploration method based on neighborhood random network distillation, It is characterized in that include: Step S1: Initialize the drone state, as well as the network parameters and hyperparameters in drone training; Step S2: construct the joint observation of UAVs into neighborhood observation data, and transform the neighborhood observation into neighborhood representation through the encoding layer; Step S3: The neighborhood representation obtains the attention weight through the attention module in the neighborhood random network; Step S4: The neighborhood representation obtains the communication weight through the communication module in the neighborhood random network; Step S5: The attention weight and communication weight are used to obtain the exploration representation of the drone cluster through the output layer, and the intrinsic reward is calculated; Step S6: Perform weighted calculation on the intrinsic reward and the environmental reward to obtain a comprehensive reward; Step S7: Update the network parameters in the UAV training and perform the next round of iteration to obtain the UAV cluster control model.

2. The search and rescue drone cluster exploration method based on neighborhood random network distillation according to claim 1, It is characterized in that Step S1 includes: Step S1.1: Initialize the individual Q function of the drone in the reinforcement learning network [Q i ], individual advantage function [A i ], individual positive importance weight [λ i ], joint Q function Q tot The network parameters θ and the target network parameters θ of the reinforcement learning network - ; Step S1.2: Initialize the target network parameters in the exploration module Prediction network parameters φ, and decoder network parameters η; Step S1.3: Initialize training hyperparameters, including learning rate and number of iterations.

3. The search and rescue drone cluster exploration method based on neighborhood random network distillation according to claim 1, It is characterized in that Step S2 includes: Step S2.1: Construct the adjacency matrix M through the neighborhood relationship of the drone, expressed as: Among them, m ij Indicates whether drone i and drone j are adjacent. The judgment method is as follows: Where m represents the neighborhood threshold, dist(·) is the Euclidean distance calculation function in three-dimensional space; Step S2.2: The matrix dot product of the adjacency matrix M of the drone cluster and the joint observation O is used as the joint neighborhood observation C = [c 1 ,c 2 ,…,c n ], the calculation process is as follows: C=M·O T Among them, joint observation O = [o 1 ,o 2 ,…,o n Each element o in ] i is the individual observation of the UAV, and the joint neighborhood observation C = [c 1 ,c 2 ,…,c n Each element c in ] i Provides individual drone neighborhood observations; Step S2.3: Input the individual drone neighborhood observations into the encoding layer in the exploration module prediction network Get the exploration module prediction network neighborhood representation Step S2.4: Input the individual drone neighborhood observations into the encoding layer of the exploration module target network Get the exploration module target network neighborhood representation 4. The search and rescue drone cluster exploration method based on neighborhood random network distillation according to claim 1, It is characterized in that Step S3 includes: Step S3.1: Calculate the attention vector between drone i and other neighboring drones in the prediction network The calculation is as follows: Among them, the query matrix and key matrix Respectively and Converted into query vector and key vector, B i represents the set of all drones except drone i; Step S3.2: Pass Calculate the attention weight of drone i in the prediction network The calculation is as follows: Step S3.3: According to the method of steps 3.1 and 3.2, the target network is queried by the matrix Q φ , key matrix K φ Sum value matrix V φ ,Will Converted to attention weight 5. The search and rescue drone cluster exploration method based on neighborhood random network distillation according to claim 1, It is characterized in that Step S4 includes: Step S4.1: Compute the update gate in the prediction network gated recurrent unit The calculation is as follows: in, and represents the weight matrix of the update gate in the prediction network, represents the bias vector of the update gate in the prediction network, Represents the hidden state of the previous moment; In the same way, update the weight matrix of the gate in the target network and Bias Vector Hidden state at the last moment as well as Get the target network update gate Step S4.2: Compute the reset gate in the prediction network gated recurrent unit The calculation is as follows: in, and represents the weight matrix of the reset gate in the prediction network, Represents the bias vector of the reset gate in the prediction network; In the same way, reset the weight matrix of the gate in the target network and Bias Vector Hidden state at the last moment as well as Get the target network reset gate Step S4.3: Calculate candidate values ​​in the gated recurrent unit of the prediction network The calculation is as follows: in, and represents the weight matrix of candidate values ​​in the prediction network, Represents the bias vector of the candidate value in the prediction network, and W represents the matrix dot product; In the same way, the weight matrix of the candidate value in the target network is and Bias Vector Hidden state at the last moment as well as Get the target network candidate value Step S4.4: Generate hidden state The calculation is as follows: In the same way, update the gate in the target network and candidate values Get the target network hidden state 6. The search and rescue drone cluster exploration method based on neighborhood random network distillation according to claim 1, It is characterized in that Step S5 includes: Step S5.1: and Input to the output layer of the prediction network The prediction network exploration representation is obtained from Will and Input to the output layer of the target network The target network exploration representation is obtained from Step S5.2: Calculate the intrinsic reward of drone i The calculation is as follows:

7. The search and rescue drone cluster exploration method based on neighborhood random network distillation according to claim 1, It is characterized in that Step S6 includes: Step S6.1: Calculate the total reward r of drone i i , is the environment exploration reward r i ext With intrinsic rewards i int The sum of: r i ＝βr i int +r i ext Among them, β represents the scaling factor that adjusts the ratio between rewards.

8. The search and rescue drone cluster exploration method based on neighborhood random network distillation according to claim 1, It is characterized in that Step S7 includes: Step S7.1: Update the prediction network parameters by predicting the network error The update method is as follows: in, is the learning rate of the prediction network; Step S7.2: According to the state value function V of drone i i (·), advantage function A i (·), global state O, calculate the state value function V of drone i in the global state i (τ), advantage function A i (τ,a i ), calculated as follows: V i (τ)=w i (t)V i (t i )+b i (t) A i (t,a i )=w i (t)A i (t i ,a i )+b i (t) Among them, τ is the joint action-observation trajectory, τ i is the action-observation trajectory of aircraft i, w(·) is a positive weight function; Step S7.3: Calculate the joint state value function V tot (τ) and the joint advantage function A tot (τ,a), is calculated as follows: Among them, λ i (τ,a) is calculated as follows: Among them, K represents the number of heads, φ i,k represents the attention weight activated by sigmoid, v k represents the positive bond of each head; Step S7.4: Pass the joint state value function V of drone i tot (τ), joint advantage function A tot (τ, a) and the global state s, calculate the joint action value function Q tot (τ,a), is calculated as follows: Step S7.5: Calculate the reinforcement learning time difference error function L RL , calculated as follows: in, The calculation method is as follows: Step S7.6: Update reinforcement learning network parameters θ and soft update target network parameters θ - , the update method is as follows: in, represents the soft update ratio, α θ Represents the learning rate.

Images