NRO-QMIX-based urban environment multi-unmanned aerial vehicle cooperative target search method

Abstract

The invention discloses an NRO-QMIX-based urban environment multi-unmanned aerial vehicle cooperative target search method, which comprises the following steps: constructing a multi-unmanned aerial vehicle search environment architecture based on a to-be-searched unknown region, designing an unmanned aerial vehicle search target detection rule and a reward function, establishing a region information credible graph for mapping the unknown region, and searching the unknown region according to the region information credible graph. According to the method, the optimization problem of fusing target search and information collection and realizing maximized updating of a regional credible graph is proposed, an improved NRO-QMIX algorithm is designed to solve the optimization problem, the optimal solution of the execution action of each unmanned aerial vehicle is obtained, and cooperative target search and information collection of multiple unmanned aerial vehicles in the urban complex environment are completed. According to the method, the problems that combined optimization tasks of target search and information collection are difficult to perform at the same time in a complex urban environment and the intelligent agent and environment exploration efficiency is low due to the fact that an existing reinforcement learning method uses a greedy strategy to select actions are solved, and multi-unmanned aerial vehicle cooperative target search and information collection in the urban environment are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle target search, and in particular to a multi-unmanned aerial vehicle collaborative target search method in an urban environment based on NRO-QMIX. Background Art

[0002] Faced with the complexity and uncertainty of urban environments, multi-UAV collaborative search technology plays an important role in target search and regional information collection tasks. Due to the complexity and variability of the scene, the current multi-UAV collaborative search solution methods can be roughly divided into two categories: rule-based search methods and mathematical programming-based search methods.

[0003] Rule-based search methods are usually used for large-scale coverage search tasks. This method pays more attention to the problems of missed search areas and resource consumption in existing coverage algorithms, and is committed to optimizing the search formation. Although it can effectively solve the defects of traditional coverage algorithms and perform well in coverage search environments and fixed drone formations, when the environment is complex or the number of people changes, the drone's search path needs to be replanned.

[0004] The method based on mathematical programming is used in more complex environments. It combines optimization theory to model the Multi-UAV Cooperative Target Search (MCTS) problem as an optimization problem with minimum search time or maximum coverage as the objective function. Swarm intelligence algorithms are usually used for search and solution. However, the time and space complexity of this method increases rapidly with the increase in the number of drones, resulting in unsatisfactory results.

[0005] With the rise of reinforcement learning, solving MCTS problems based on reinforcement learning has become widely popular and has become a new paradigm for solving urban multi-target collaborative search and information collection problems. However, in urban environments, MCTS tasks face high uncertainty in unknown areas, and complex terrain and obstacles significantly increase the difficulty of the task. Existing methods often cannot effectively handle uncertainty, which leads to low search efficiency or incomplete information collection. In MCTS tasks, the complexity of urban environments requires drones to quickly adapt to new situations and find the optimal strategy, while multi-agent reinforcement learning algorithms are less efficient in exploring new paths or strategies, resulting in slow convergence of search tasks or poor results. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a multi-UAV collaborative target search method in an urban environment based on NRO-QMIX, so as to improve the efficiency of multi-UAV collaborative target search and information collection tasks in an unknown urban environment.

[0007] Technical solution: To achieve the above purpose, the present invention provides a method for multi-UAV collaborative target search in an urban environment based on NRO-QMIX, comprising the following steps:

[0008] Step 1: Based on the area to be searched Build a multi-UAV search environment architecture;

[0009] Step 2: Design the drone search target detection rules and reward function;

[0010] Step 3: Create a mapping area The regional information credible graph is constructed, and an optimization problem is proposed to integrate target search and information collection to achieve maximum update of the regional credible graph.

[0011] Step 4: Design an improved NRO-QMIX algorithm to solve the optimization problem, obtain the optimal solution for the execution action of each UAV, and complete the multi-UAV collaborative target search and information collection in the urban environment;

[0012] The improved NRO-QMIX algorithm includes processing each UAV observation information separately to generate a local The policy network and local Perform a hybrid to generate a global A hybrid network, by building a global With local to establish a global constraint relationship between Maximum and local The mapping relationship between the maximum values ​​ensures that the local optimal action of each drone is also the global optimal action; among them, the output layer of the policy network is the NROWAN noise network.

[0013] The multi-UAV search environment architecture described in step 1 includes an environment model, a target and threat area model, and a UAV model;

[0014] The method for constructing the environment model is as follows: After rasterization, it is divided into multiple squares. Represent the number of rows and columns of the grid respectively; define the grid as , and the grid center is represented as ,in, , ;

[0015] The target and threat zone model is constructed as follows: Assuming that each target occupies only one square, and each square has only one target, define the event is whether there is a target in the grid, Indicates that there is a target in the current square. Indicates that there is no target in the current square. In the unknown area, the probability of the preset target existing in each square is 0.5;

[0016] The threat zone is defined as a circle with the center of a square as the center, expressed as:

[0017] ,

[0018] In the formula, is the center of the threat zone, Number the threat zone, is the radius of the threat zone, R + represents a positive real number;

[0019] The drone needs to maintain a certain distance from the threat area during operation, which can be expressed as:

[0020] ,

[0021] In the formula, For drones The coordinates of To preset a safe distance between the drone and the danger zone;

[0022] The method for constructing the UAV model is as follows: the UAV performs mass point motion, and the kinematics of a single UAV is expressed as:

[0023] ,

[0024] In the formula, For speed, is the heading angle, is the angular velocity;

[0025] A single UAV forming a multi-UAV system is expressed as:

[0026] ,

[0027] In the formula, is the total number of drones;

[0028] Assuming that the drone moves only one square at a time, the mathematical relationship between the anti-collision positions of multiple drones is expressed as:

[0029] ,

[0030] In the formula, To preset a safe distance between multiple drones to prevent collisions, , For drones , the coordinates of j.

[0031] The design method of the drone search target detection rule in step 2 is:

[0032] Set event Indicates the time t drone In the grid The target was found in Indicates the drone at time t In the grid If no target is detected, then the drone In the grid The probability of detecting the target in is expressed as:

[0033] ,

[0034] In the formula, is the detection rate, is the false alarm rate, is the probability of missed detection, is the true missed detection probability, Represents a square There is a goal in Represents a square There is no target in

[0035] Grid The updating method of the target existence probability is as follows:

[0036] ,

[0037] In the formula, The drone at time t In the grid The probability of detecting the existence of the target;

[0038] Drones Determine the event based on the probability of the target existing in the current grid Values:

[0039] ,

[0040] When the square The probability of the target existing in the target exceeds the threshold When the drone Judgment Grid There is a target in .

[0041] The reward function in step 2 includes target reward, information collection reward, obstacle avoidance penalty, and time step penalty;

[0042] Among them, the target reward represents the reward given when the drone searches for a new target, and each target is rewarded to the drone only when it is discovered for the first time. The target reward function is expressed as:

[0043] ,

[0044] in, To adjust the coefficient of the target reward size, is the total number of drones, Different numbers for drones, The drone at time t In the grid The probability of detecting the existence of the target; is the threshold of the probability of target existence;

[0045] The information collection reward means that the drone should fly to cover as many squares as possible within a limited time. The information collection reward function is expressed as:

[0046] ,

[0047] In the formula, To adjust the information collection reward size coefficient, is the square at time t Number of searches by drones;

[0048] The obstacle avoidance penalty means avoiding collisions between drones during the mission. The obstacle avoidance penalty function is expressed as:

[0049] ,

[0050] In the formula, To adjust the penalty coefficient of the drone collision danger zone, To adjust the penalty coefficient for drone collisions; represents the threat zone, s is the threat zone number, , For drones , the coordinates of j, To preset a safe distance between the drone and the danger zone, To preset a safe distance between multiple drones to prevent collision;

[0051] The time step penalty means that the UAV flight time is fixed, and the time step penalty function is:

[0052] ,

[0053] In the formula, A coefficient to adjust the penalty size for exceeding the time step;

[0054] The reward obtained by all drones is the total reward, and the total reward function is expressed as:

[0055] .

[0056] The method for establishing the regional information trust map in step 3 is as follows: establishing a mapping area Regional infographic , where each cell Represents the corresponding square The regional information is credible, and the regional credible graph is expressed as:

[0057] ,

[0058] In the formula, is the regional credible map at time t, if the square Threat zone If , the regional credible map is set to 0;

[0059] is the change of the regional credible map after the drone information is collected at time t, expressed as:

[0060] ,

[0061] is the square at time t The number of searches, For square Initial regional information credibility;

[0062] is the change of the regional credible map after the drone searches for the target at time t, expressed as:

[0063] ,

[0064] In the formula, event The drone at time t In the grid The target was found in Indicates the drone at time t In the grid No target was detected. The drone at time t In the grid The probability of detecting the existence of the target.

[0065] The optimization problem in step 3 is expressed as:

[0066] ,

[0067] In the formula, is the regional credible graph at time t, , Indicates drone The horizontal and vertical coordinate range of Indicates drone In the area Flying inside.

[0068] The data processing process of the improved NRO-QMIX algorithm in step 4 is as follows:

[0069] Drone Current observations And the previous moment action Input the policy network of each agent network for processing and output the individual action value function , that is, local ;in, Indicates drone Its own historical actions and observations are expressed as: , Represents the drone predicted by the policy network Perform actions at the current moment;

[0070] Each drone Part of And the global observation s is input into the hybrid network to generate a joint action value function, that is, the global , expressed as:

[0071] ;

[0072] Establishing a global Maximum and local The mapping relationship between the maximum values ​​is expressed as:

[0073] ,

[0074] The constraint relationship to satisfy the mapping relationship is expressed as:

[0075] ,

[0076] In the formula, is the hybrid network parameter, is the total number of drones.

[0077] Among them, the NROWAN noise layer network structure described in step 4 is expressed as:

[0078] ,

[0079] In the formula, are learnable parameters, For noise.

[0080] The process of solving the optimization problem by the improved NRO-QMIX algorithm is as follows:

[0081] (1) At each discrete moment, the drone All interact with the environment of the exploration area. At time t, the drone Generate the current observation value of itself , each drone Self-observation And the previous moment action And global observation Input NRO-QMIX neural network, output each drone Corresponding to the global With local The action with the largest value , drones Execute the action , and obtain the total reward of multiple drones at the current time t , thus entering the next moment and completing a complete interaction cycle;

[0082] (2) The drone starts a new cycle, continues to interact with the environment, and continues to accumulate rewards. After a limited time, all cycles end and the overall reward of the multi-drone is obtained. ,During several rounds of training, the NRO-QMIX neural network is continuously updated based on ,previous training experience and the designed loss function until the ,training is completed;

[0083] (3) Using the trained NRO-QMIX neural network, obtain the drone The optimal action to perform given the current observations.

[0084] Wherein, the loss function is expressed as:

[0085] ,

[0086] In the formula, are the parameters of the target network, Responding to noise level, is the proportionality coefficient, is the discount factor, is the total reward of multiple drones at time t, Target global The noise value function, For the global Noise value function;

[0087] in, , ,

[0088] In the formula, is the input dimension of the last layer of the NROWAN noise network, is the number of output actions, for The final expected value of is the reward at the current time t, For the maximum reward, is the minimum reward, is the noise variance of the weight, is the noise variance of the deviation.

[0089] Beneficial effects: The present invention has the following advantages: 1. The method of the present invention transforms the combinatorial optimization problem into a deterministic task that only needs to optimize the regional information credible map by introducing a regional information credible map that integrates the target search task and the information collection task, thereby solving the problem that it is difficult to simultaneously perform the combinatorial optimization tasks of target search and information collection in a complex urban environment;

[0090] 2. The method described in the present invention is improved based on the existing QMIX algorithm, integrated into the NROWAN noise network, and optimized the reward function based on task requirements and environmental characteristics. It can effectively enhance the environmental exploration capability of the UAV and accelerate the convergence speed of the algorithm, and solve the problem that the existing reinforcement learning method uses a greedy strategy to select actions, resulting in low efficiency of intelligent agent and environmental exploration. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 This is a flow chart of the multi-UAV cooperative target search method of the present invention;

[0092] Figure 2 Schematic diagram of environmental modeling;

[0093] Figure 3 This is the architecture diagram of the improved NRO-QMIX algorithm;

[0094] Figure 4 Schematic diagram of the improved NRO-QMIX algorithm data processing code. DETAILED DESCRIPTION

[0095] The technical solution of the present invention is described in detail below in conjunction with the embodiments and drawings.

[0096] like Figure 1 As shown, the method for multi-UAV collaborative target search in an urban environment based on NRO-QMIX of the present invention comprises the following steps:

[0097] 1. Construct a multi-UAV search environment architecture, including the environment model, target and threat area model, and UAV model, such as Figure 2 shown.

[0098] (1) Building an environmental model

[0099] The area to be searched After rasterization, it is divided into multiple square grids. Represent the number of rows and columns of the grid respectively; define the grid as , and the grid center is represented as ,in, , .

[0100] (2) Constructing target and threat zone models

[0101] Assume that each target will only occupy one square, and there is only one target in each square; the description of whether there is a target in the unknown area adopts the Bernoulli distribution: define the event is whether there is a target in the grid, Indicates that there is a target in the current square. Indicates that there is no target in the current square. In the unknown area, the probability of the preset target existing in each square is 0.5.

[0102] The threat zone is defined as a circle with the center of a square as the center, expressed as:

[0103] (1),

[0104] in, is the center of the threat zone, Number the threat zone, is the radius of the threat zone, R + Represents a positive real number.

[0105] The drone needs to maintain a certain distance from the threat area during operation, which can be expressed as:

[0106] (2),

[0107] in, For drones The coordinates of To preset a safe distance between the drone and the danger zone.

[0108] (3) Build a drone model

[0109] The drone performs point motion, and the kinematics of a single drone is expressed as:

[0110] (3),

[0111] in For speed, is the heading angle, is the angular velocity.

[0112] A single UAV forming a multi-UAV system is expressed as:

[0113] (4),

[0114] in, is the total number of drones.

[0115] Assuming that the drone moves only one square at each moment, since the drone exists alone in one square at each moment t, the collaborative search of multiple drones needs to consider avoiding collisions with each other. The mathematical relationship between the anti-collision positions of multiple drones is expressed as:

[0116] (5),

[0117] in, To preset a safe distance between multiple drones to prevent collisions, , For drones , the coordinates of j.

[0118] 2. Design the detection rules for drone search targets.

[0119] The sensors carried by the drone may miss detections or give false alarms. Indicates the time t drone In the grid The target was found in Indicates the drone at time t In the grid If no target is detected, then the drone In the grid No target was detected. Combined with the above event B, different event combinations may result in the following four possibilities:

[0120] A. Drone The sensor is in the grid at time t The target is detected and the square There are goals in the memory;

[0121] B. Drone The sensor is in the grid at time t No target was detected, but the square There are goals in the memory;

[0122] C. Drone The sensor is in the grid at time t The target was detected, but the square There is no goal within;

[0123] D. Drone The sensor is in the grid at time t No target is detected, and the square There is no goal within;

[0124] Drones At time t in the grid The probability of detecting the target at is expressed mathematically as shown in formula (6):

[0125] (6),

[0126] in, is the detection rate, is the false alarm rate, is the probability of missed detection, is the true missed detection probability.

[0127] At each time t, according to formula (6) combined with the Bayesian criterion, The updating method of the target existence probability at a certain moment is as follows:

[0128] (7),

[0129] in, The drone at time t In the grid The probability of detecting the existence of the target.

[0130] Drones Determine the event based on the probability of the target existing in the current grid The value of is used to reduce missed detection, false alarms, etc. The judgment formula is:

[0131] (8),

[0132] When the probability of the target existing in the grid exceeds the threshold When the drone Judgment Grid There is a target in .

[0133] 3. Establish a credible regional information map based on drone search target detection rules.

[0134] Create a mapping area Regional infographic , where each cell Represents the corresponding square The credibility of regional information.

[0135] The regional information trust map will change as the drone searches for the target and collects information. The process of the drone searching for the target and collecting information is continuously updated and progresses as time t changes, and the regional information map also changes accordingly. The changes of the regional trust map after the drone searches for the target are as follows:

[0136] (9),

[0137] in, is the change of the regional credible map after the drone searches for the target at time t, For square Initial regional information credibility.

[0138] The changes in the regional trust map after drone information collection are as follows:

[0139] (10),

[0140] in, is the change of the regional credible map after the drone information is collected at time t, is the square at time t The number of times it has been searched.

[0141] (11),

[0142] in, is the regional credible graph at time t. If it is in the threat zone, the regional credibility map is set to 0.

[0143] Fourth, in the multi-UAV search environment architecture, an optimization problem is established to ensure that UAVs can search for more targets, collect more information, and maximize the update of the regional credible map at each moment.

[0144] This optimization problem means that within a limited area, multiple UAVs should complete the collaborative search task as well as possible under the constraints of avoiding collisions with each other and avoiding threat areas.

[0145] (12),

[0146] in, , Indicates drone The horizontal and vertical coordinate range of Indicates drone In the area Flying inside.

[0147] 5. Design a centrally trained and distributedly executed NRO-QMIX algorithm to coordinate the search behavior among multiple UAVs.

[0148] Among them, centralized training refers to training using global information. During the training phase, the algorithm considers the information of all drones to learn the global optimal strategy; distributed execution refers to relying on local information during execution. During the execution phase, each drone makes decisions based only on the local information it observes, thereby achieving distributed decision-making.

[0149] (1) Design a random noise mixed Q value algorithm (NRO-QMIX), such as Figure 3 shown.

[0150] In experiments, decomposition noise is usually used to generate noise in the input and output layers. The NRO-QMIX algorithm of the present invention is an improvement on the QMIX algorithm. Specifically, the MLP output layer of the agent network in the QMIX algorithm is replaced with the NROWAN (restricted random noise) noise layer for output, aiming to more effectively process the noise in the input and output layers, so as to better adapt to the high uncertainty and complexity in urban environments and improve the efficiency and effect of multi-UAV collaborative search.

[0151] The NROWAN noise layer has high stability, which enhances the agent's exploration power and makes network training more stable. Dimensional input and The fully connected layer with dimension output can be written as , the NROWAN noise layer network structure can be written as:

[0152] (13),

[0153] in, are learnable parameters, For noise.

[0154] The NRO-QMIX algorithm uses global information for training, but only relies on local information during execution, thereby achieving efficient distributed decision-making in complex environments. The specific implementation method is as follows:

[0155] The NRO-QMIX algorithm provides a Keep a separate policy network for handling drones Observed value The independent policy network outputs the individual action value function through the argmax operation after passing through the NROWAN noise layer. , also known as local The policy network data processing process is as follows:

[0156] Drone Current observations And the previous moment action Input the policy network of each agent network i for processing, mine useful information hidden in historical observations, and input the information into the NROWAN noise network for argmax operation, and finally output the individual action value function , also known as local .in, Indicates drone Its own historical actions and observations, , Represents the drone predicted by the policy network Perform an action at the current moment.

[0157] At the same time, the NRO-QMIX algorithm maintains a centralized hybrid network that connects each drone Part of Mix and add the global observation s (the global observation state is constructed by the global credible map and the overall probability map) to generate a joint action value function, also known as the global Q tot Each drone will Part of and global observations are input into the hybrid network for mixing to generate a joint action value function, also known as the global Q tot .

[0158] (14)

[0159] in, are the hybrid network parameters.

[0160] By constructing a global Q tot With local The constraint relationship between them establishes the global Q tot Maximum and local The mapping relationship between the maximum values ​​ensures that each drone The local optimal action is also the global optimal action. The specific implementation process is as follows:

[0161] Explore the joint action value function in formula (14) With local value function By extracting the decentralization strategy from the maximum joint value function, because taking argmax for the joint action value function is equivalent to taking argmax for each local action value function, Q must be satisfied while ensuring the same monotonicity tot Take the maximum value and Just take the maximum value, the mapping relationship is as follows:

[0162] (15)

[0163] To satisfy formula (14), the following constraints must be met:

[0164] (16)

[0165] in, is the total number of drones.

[0166] Formula (15) constraint relationship represents the global Qtot For each drone Part of The partial derivatives must be non-negative. If the drone Part of Increase, global Q tot This constraint relationship ensures that each drone The local optimal action can also lead to the global optimal action, thus simplifying the multi-agent coordination problem.

[0167] (2) Design the loss function of the NRO-QMIX algorithm to achieve network updating.

[0168] The NRO-QMIX algorithm uses a target network to maintain the stability of the training process and avoid oscillation divergence. At the same time, the NRO-QMIX algorithm uses a noise value function and Represents the global and target global , the NRO-QMIX loss function is as follows:

[0169] (17)

[0170] in, are the parameters of the target network, Responding to noise level, is the proportionality coefficient, is the discount factor, is the total reward of multiple drones at time t, Target global The noise value function, For the global Noise value function.

[0171] in: (18)

[0172] (19)

[0173] In formula (18), is the input dimension of the last layer of the NROWAN noise network, is the number of output actions.

[0174] In formula (19), for The final expected value of is the reward at the current time t, For the maximum reward, is the minimum reward, is the noise variance of the weight, is the noise variance of the deviation.

[0175] 6. Design the reward function for multi-UAV collaborative target search and information collection tasks.

[0176] The purpose of designing the reward function is to guide multiple drones to search for as many targets as possible, collect as much regional information as possible, and avoid potential risk areas in the area. The present invention designs the following four rewards in combination with actual goals:

[0177] (1) Designing target rewards based on human-machine search target detection rules

[0178] When the drone searches for a new target, it will be given a large reward. The drone will only be rewarded when each target is discovered for the first time. The target reward function is:

[0179] (20),

[0180] in, To adjust the coefficient of the target reward size, is the total number of drones, Different numbers for drones, .

[0181] (2) Information Collection Rewards

[0182] The information collection reward is designed to allow the drone to fly and cover as many squares as possible within a limited time. The information collection reward function is:

[0183] (twenty one),

[0184] in, To adjust the information collection reward size coefficient, is the square at time t By drone Number of visits.

[0185] (3) Obstacle Avoidance Penalty

[0186] The obstacle avoidance penalty is designed to prevent collisions between drones during the mission. The obstacle avoidance penalty function is:

[0187] (twenty two),

[0188] in, To adjust the penalty coefficient of the drone collision danger zone, To adjust the penalty coefficient for drone collisions.

[0189] (4) Time step penalty

[0190] The time step penalty is designed because the UAV carries limited energy and has a fixed flight time. The time step penalty function is:

[0191] (twenty three),

[0192] in, A coefficient that adjusts the penalty size for exceeding the time step.

[0193] During the reinforcement learning task, the total reward is the sum of all the rewards received by the drones. The total reward is:

[0194] (twenty four).

[0195] 7. Use the NRO-QMIX algorithm to solve the optimization problem and obtain the optimal solution for the execution action of each drone, such as Figure 4 shown.

[0196] At each discrete moment, the drone All interact with the environment of the exploration area. At time t, the drone Generate the current observation value of itself ;

[0197] Each drone Self-observation And the previous moment action And global observation Input NRO-QMIX neural network, output each drone Corresponding to the global With local The action with the largest value , drones Execute the action , and the total reward of multiple drones at the current time t .

[0198] The environment changes with the actions of the drone, and the next moment is entered, completing a complete interaction cycle. Afterwards, the drone starts a new cycle and continues to interact with the environment. This cycle repeats over and over again. The drone keeps taking actions and interacting with the environment, and continues to accumulate rewards. After a limited amount of time, all cycles end, and the current round of the mission ends, and the overall multi-drone reward is obtained. .

[0199] During several rounds of training, the NRO-QMIX algorithm is continuously updated based on the experience of previous training and the designed loss function until the training is completed.

[0200] Using the trained NRO-QMIX neural network, we can obtain the drone The optimal action executed under the current observation value enables multiple UAVs to maximize the update of the regional credible graph in the least number of moments, so as to complete the multi-UAV collaborative target search in the complex urban environment, and also obtain the maximum cumulative reward.

Claims

1. A multi-UAV collaborative target search method in urban environment based on NRO-QMIX, characterized in that: The following steps are involved: Step 1: Based on the area to be searched Build a multi-UAV search environment architecture; Step 2: Design the drone search target detection rules and reward function; Step 3: Create a mapping area The regional information credible graph is constructed, and an optimization problem is proposed to integrate target search and information collection to achieve maximum update of the regional credible graph. Step 4: Design an improved NRO-QMIX algorithm to solve the optimization problem, obtain the optimal solution for the execution action of each UAV, and complete the multi-UAV collaborative target search and information collection in the urban environment; The improved NRO-QMIX algorithm includes processing each UAV observation information separately to generate a local The policy network and local Perform a hybrid to generate a global A hybrid network, by building a global With local to establish a global constraint relationship between Maximum and local The mapping relationship between the maximum values ​​ensures that the local optimal action of each drone is also the global optimal action; among them, the output layer of the policy network is the NROWAN noise network.

2. The method for multi-UAV collaborative target search in urban environment based on NRO-QMIX according to claim 1 is characterized in that: The multi-UAV search environment architecture described in step 1 includes an environment model, a target and threat area model, and a UAV model; The method for constructing the environment model is as follows: After rasterization, it is divided into multiple squares. Represent the number of rows and columns of the grid respectively; define the grid as , and the grid center is represented as ,in, , ; The target and threat zone model is constructed as follows: Assuming that each target occupies only one square, and each square has only one target, define the event is whether there is a target in the grid, Indicates that there is a target in the current square. Indicates that there is no target in the current square. In the unknown area, the probability of the preset target existing in each square is 0.5; The threat zone is defined as a circle with the center of a square as the center, expressed as: ， In the formula, is the center of the threat zone, Number the threat zone, is the radius of the threat zone, R + represents a positive real number; The drone needs to maintain a certain distance from the threat area during operation, which can be expressed as: ， In the formula, For drones The coordinates of To preset a safe distance between the drone and the danger zone; The method for constructing the UAV model is as follows: the UAV performs mass point motion, and the kinematics of a single UAV is expressed as: ， In the formula, For speed, is the heading angle, is the angular velocity; A single UAV forming a multi-UAV system is expressed as: ， In the formula, is the total number of drones; Assuming that the drone moves only one square at a time, the mathematical relationship between the anti-collision positions of multiple drones is expressed as: ， In the formula, To preset a safe distance between multiple drones to prevent collisions, , For drones , the coordinates of j.

3. The method for multi-UAV collaborative target search in urban environment based on NRO-QMIX according to claim 1 is characterized in that: The design method of the drone search target detection rule in step 2 is: Set event Indicates the time t drone In the grid The target was found in Indicates the drone at time t In the grid If no target is detected, then the drone In the grid The probability of detecting the target in is expressed as: ， In the formula, is the detection rate, is the false alarm rate, is the probability of missed detection, is the true missed detection probability, Represents a square There is a goal in Represents a square There is no target in Grid The update method of the target existence probability is as follows: ， In the formula, The drone at time t In the grid The probability of detecting the existence of the target; Drones Determine the event based on the probability of the target existing in the current grid Values: ， When the square The probability of the target existing in the target exceeds the threshold When the drone Judgment Grid There is a target in .

4. The method for multi-UAV collaborative target search in urban environment based on NRO-QMIX according to claim 1 is characterized in that: The reward function described in step 2 includes target reward, information collection reward, obstacle avoidance penalty, and time step penalty; Among them, the target reward represents the reward given when the drone searches for a new target, and each target is rewarded to the drone only when it is discovered for the first time. The target reward function is expressed as: ， in, To adjust the coefficient of the target reward size, is the total number of drones, Different numbers for drones, The drone at time t In the grid The probability of detecting the existence of the target; is the threshold of the probability of target existence; The information collection reward means that the drone should fly to cover as many squares as possible within a limited time. The information collection reward function is expressed as: ， In the formula, To adjust the information collection reward size coefficient, is the square at time t Number of searches by drones; The obstacle avoidance penalty means avoiding collisions between drones during the mission. The obstacle avoidance penalty function is expressed as: ， In the formula, To adjust the penalty coefficient of the drone collision danger zone, To adjust the penalty coefficient for drone collisions; represents the threat zone, s is the threat zone number, , For drones , the coordinates of j, To preset a safe distance between the drone and the danger zone, To preset a safe distance between multiple drones to prevent collision; The time step penalty means that the UAV flight time is fixed, and the time step penalty function is: ， In the formula, A coefficient to adjust the penalty size for exceeding the time step; The reward obtained by all drones is the total reward, and the total reward function is expressed as: 。 5. The method for multi-UAV collaborative target search in urban environment based on NRO-QMIX according to claim 1 is characterized in that: The method for establishing the regional information trust map in step 3 is: establishing a mapping area Regional infographic , where each cell Represents the corresponding square The regional information is credible, and the regional credible graph is expressed as: ， In the formula, is the regional credible map at time t, if the square Threat zone If , the regional credible map is set to 0; is the change of the regional credible map after the drone information is collected at time t, expressed as: ， is the square at time t The number of searches, For square Initial regional information credibility; is the change of the regional credible map after the drone searches for the target at time t, expressed as: ， In the formula, event The drone at time t In the grid The target was found in Indicates the drone at time t In the grid No target was detected. The drone at time t In the grid The probability of detecting the existence of the target.

6. The method for multi-UAV collaborative target search in urban environment based on NRO-QMIX according to claim 1 is characterized in that: The optimization problem in step 3 is expressed as: ， In the formula, is the regional credible graph at time t, , Indicates drone The horizontal and vertical coordinate range of Indicates drone In the area Flying inside.

7. The method for multi-UAV collaborative target search in urban environment based on NRO-QMIX according to claim 1 is characterized in that: The data processing process of the improved NRO-QMIX algorithm described in step 4 is as follows: Drone Current observations And the previous moment action Input the policy network of each agent network for processing and output the individual action value function , that is, local ;in, Indicates drone Its own historical actions and observations are expressed as: , Represents the drone predicted by the policy network Perform actions at the current moment; Each drone Part of And the global observation s is input into the hybrid network to generate a joint action value function, that is, the global , expressed as: ； Establishing a global Maximum and local The mapping relationship between the maximum values ​​is expressed as: ， The constraint relationship to satisfy the mapping relationship is expressed as: ， In the formula, is the hybrid network parameter, is the total number of drones.

8. The method for multi-UAV collaborative target search in urban environment based on NRO-QMIX according to claim 1 is characterized in that: The NROWAN noise layer network structure described in step 4 is expressed as: ， In the formula, are learnable parameters, For noise.

9. The method for multi-UAV collaborative target search in urban environment based on NRO-QMIX according to claim 1 is characterized in that: The process of solving the optimization problem by the improved NRO-QMIX algorithm is as follows: (1) At each discrete moment, the drone interacts with the environment in the exploration area. At time t, the drone Generate the current observation value of itself , each drone Self-observation And the previous moment action And global observation Input NRO-QMIX neural network, output each drone Corresponding to the global With local The action with the largest value , drones Execute the action , and obtain the total reward of multiple drones at the current time t , thus entering the next moment and completing a complete interaction cycle; (2) The drone starts a new cycle, continues to interact with the environment, and continues to accumulate rewards. After a limited time, all cycles end and the overall reward of the multi-drone is obtained. ,During several rounds of training, the NRO-QMIX neural network is continuously updated based on ,previous training experience and the designed loss function until the ,training is completed; (3) Using the trained NRO-QMIX neural network, obtain the drone The optimal action to perform given the current observations.

10. The method for multi-UAV collaborative target search in urban environment based on NRO-QMIX according to claim 9 is characterized in that: The loss function is expressed as: ， In the formula, are the parameters of the target network, Responding to noise level, is the proportionality coefficient, is the discount factor, is the total reward of multiple drones at time t, Target global The noise value function, For the global Noise value function; in, , , In the formula, is the input dimension of the last layer of the NROWAN noise network, is the number of output actions, for The final expected value of is the reward at the current time t, For the maximum reward, For the minimum reward, is the noise variance of the weight, is the noise variance of the deviation.

Images