Unknown environment target searching method and system based on spatial clustering and deep reinforcement learning

Abstract

The invention discloses an unknown environment target searching method and system based on spatial clustering and deep reinforcement learning, and relates to the technical field of multi-target searching. Dividing the three-dimensional dynamic environment into task areas with the same number as the unmanned aerial vehicles, and distributing the task areas to the search unmanned aerial vehicles; simulating the movement of the target and the dynamic obstacle, constructing a reward function based on environment feedback, and constructing a mathematical model based on the search task; based on a neural network approximation method, obtaining a search strategy for searching the unmanned aerial vehicle; and on the basis of a Markov decision process and a gradient ascending method, an optimal search strategy for searching the unmanned aerial vehicle is obtained by searching the expected accumulated rewards of the unmanned aerial vehicle to the maximum. According to the invention, all searching unmanned aerial vehicles can safely search all targets in a three-dimensional environment within the shortest time and at the lowest cost, and the target searching capability of the searching unmanned aerial vehicles in a large-range airspace is improved.

Description

Technical Field

[0001] The present invention relates to the field of multi-target search technology, and more specifically to a method and system for searching targets in unknown environments based on spatial clustering and deep reinforcement learning. Background Art

[0002] At present, more flexible and lower-cost drones are gradually replacing traditional ground or fixed equipment in the fields of rescue, exploration, security, etc. Especially in the security field, aerial drones equipped with cameras, lidar and other detection equipment can conduct real-time and accurate monitoring of a wide range of places.

[0003] Traditional target search methods focus on identifying and capturing ground targets, using ground unmanned vehicles and their own detection equipment to perceive the environment, and efficiently capturing mobile targets under the guidance of aerial drones. However, this decision based on the knowledge of global environmental information requires a lot of resources to monitor the environment in real time.

[0004] Existing research on the problem of target search in unknown areas mainly focuses on two-dimensional space. Due to the reduction of the dimensionality of the search space, observations based on the basic information of the search drone, such as position and posture, can still meet the needs of the algorithm in the search task. However, in the face of a large range of three-dimensional space, the small state space may cause the algorithm to be difficult to converge, or even fall into serious problems such as an infinite loop. In addition, it is still difficult to deal with the problem of unstable convergence caused by a large-scale search space by relying on optimization algorithms, so the search space needs to be further optimized. However, only a small number of studies have considered the division of a large-scale search area, and these studies have integrated vision and radar to improve the detection range and detection accuracy of the search drone.

[0005] Therefore, with regard to the problem of multi-UAV multi-target search in three-dimensional dynamic space, how to improve its search capability and search efficiency is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0006] In view of this, the present invention provides a method and system for searching targets in unknown environments based on spatial clustering and deep reinforcement learning.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A method for searching targets in unknown environments based on spatial clustering and deep reinforcement learning is applied to multiple search drones for searching multiple targets in a three-dimensional dynamic environment. Each search drone is equipped with an airborne radar. The search method includes the following steps:

[0009] Step 1: Based on the spatial clustering method of the Voronoi diagram, the three-dimensional dynamic environment to be searched is divided into task areas equal to the number of drones according to the initial position of the search drone and the regional safety level, and the task areas are allocated to each search drone;

[0010] Step 2: In the actual search process, since the movement of the target and obstacles in the environment is usually random, the Gaussian distribution and random distribution methods are used to simulate the movement of the target and dynamic obstacles in the environment, the reward function is constructed based on the environmental feedback, and the mathematical model is built based on the search task;

[0011] Step 3: Based on the neural network approximation method, the state obtained by the search drone's airborne radar in each time step is used as input, and the probability of the search drone selecting a specific action in a specific state is obtained through a recurrent neural network, that is, the search strategy of the search drone;

[0012] Step 4: Based on the Markov decision process and the gradient ascent method, the optimal search strategy of the search drone is obtained by maximizing the expected cumulative reward of the search drone: the gradient is obtained by calculating the differential of the expected cumulative reward of the search drone, and the expected cumulative reward of the search drone is optimized along the direction of the gradient update according to the gradient ascent method. When the variance of the cumulative expected reward is less than the given value, the search drone obtains the optimal search strategy, and finally avoids all obstacles and detects all targets in the shortest time and at the lowest cost.

[0013] Furthermore, in step 1, the specific steps of dividing the task area are:

[0014] Step 1.1: Consider the existence of n in a three-dimensional dynamic environment s search drones, whose initial position is Calculate the initial position of any point p = (x, y, z) in the three-dimensional dynamic environment and the search drone The Euclidean distance That is: any point in three-dimensional space and the search drone The shortest distance;

[0015] Step 1.2: According to the clustering method of weighted Voronoi diagram, the three-dimensional dynamic environment is divided into n s Search drone mission area; set the initial position of all search drones As the centroid, combined with the safety levels of different areas, the safety level of each search drone’s mission area is determined, that is, the area weight w i (i=1,2,...,n S ); In the method, the security level and the weight value w i (i=1,2,...,n S) is inversely proportional to;

[0016] Step 1.3: Combine the security level w i (i=1,2,...,n S ) and Euclidean distance Spatial Clustering Method Based on Three-Dimensional Voronoi Diagram Find the mission area of ​​each search drone, that is, any point in space will be assigned to the search drone that is closest to it and has the highest regional safety level.

[0017] Furthermore, in step 2, the specific steps of constructing the reward function and the task mathematical model are:

[0018] Step 2.1: Consider the existence of n in a dynamic three-dimensional environment T targets, which follow a Gaussian distribution at each time step where μ T is the expectation of the target prior’s three-dimensional coordinates, σ T is the variance of the target prior’s three-dimensional coordinates;

[0019] Step 2.2: Consider the existence of n in a three-dimensional dynamic environment O obstacles, the obstacles are divided into static obstacles With dynamic obstacles Where l+m=n O ; Static obstacle P f Follow random distribution in dynamic 3D environment Dynamic obstacle P d It follows Brownian motion at each time step t(t∈T) in a dynamic 3D environment

[0020] Step 2.3: Based on the environmental feedback and the goal of the search task, a reward and punishment function for the interaction between the search drone and the environment is constructed. The reward r obtained by each search drone in one time step is i (t) is obtained by the cumulative sum of all reward and penalty values;

[0021] Step 2.4: Based on the Markov decision process, the search drone detects all targets with the shortest search time and the lowest cost while avoiding all obstacles, which is defined as maximizing the expected cumulative reward Where γ = [0.9, 0.99] is the reward discount factor.

[0022] Furthermore, in step 2.3, the reward and penalty values ​​include: the reward for finding the target r f , Exploration Area Rewards e , Flight Penalty c , cross-border punishment b, exploration time penalty r s .

[0023] Furthermore, in step 3, the specific steps of obtaining the search strategy are:

[0024] Step 3.1: According to the distributed deep reinforcement learning algorithm, first set the state space S of the search drone to be {s0,s1,...,s t}(t∈T), where the state of each search drone Including the three-dimensional coordinates of the search drone, the scale factors of the search drone's speed in the X, Y, and Z axes, and the regional safety level; secondly, set the action space A = {a1, a2, a3, a4, a5}, where action a i =[ω i ,ψ i ](i=1,2,...,5) represents the specific action that the search drone chooses to perform, including the heading angle ω of the search drone i With the azimuth ψ i ; The action space contains 5 actions, which means that the search drone will select a specific action from the action space to execute in each time step;

[0025] Step 3.2: In order to obtain the search strategy of the search drone, a recurrent neural network is designed. First, the state of each search drone is The input is sent to the input layer; secondly, the hidden layer extracts features and processes time-dependent input data through the network parameters θ, i.e., weights and biases, and activation functions; finally, the output layer obtains the probability P(a) of the search drone selecting the action to be performed in the current time step according to the state of the previous time step in each time step under the current parameters θ. t |s t-1 ,θ), that is: search strategy π θ (a t |s t-1 ); the probability P(a t |s t-1 ,θ) means that when the network parameter is θ, the state s is input to the recurrent neural network. t-1 After that, the search drone performs action a t The probability of

[0026] Step 3.3: Based on the search strategy π θ (a t |s t-1), the search drone executes the action with the highest probability value at a given exploration rate ε, and executes other actions except the action with the highest probability value at an exploration rate (1-ε). The exploration rate enables the search drone to solve the problem of low spatial exploration rate caused by only following the action selection with high probability value to a certain extent, thereby ensuring that the search drone makes action selection after weighing the regional exploration and experience utilization.

[0027] Furthermore, in step 4, the specific steps of obtaining the optimal search strategy are:

[0028] Step 4.1, based on the search time step limit T and the search UAV detecting all targets, set the termination exploration condition of the search UAV;

[0029] Step 4.2: Based on the Markov decision process, the search drone is assigned to the search strategy π θ All actions selected, all states experienced, and all rewards obtained when exploring a three-dimensional dynamic environment are taken as the exploration trajectory τ of the search drone. k ={s0,a1,r1,...,s t-1 ,a t ,r t}(t≤T); the trajectory represents the complete sequence of states, actions, and rewards generated by the search drone after exploring the environment in one iteration;

[0030] Step 4.3: Calculate the exploration trajectory τ of each search drone during the exploration process by weighted summation k Cumulative rewards for exploration tracks obtained in Where n e is the total number of exploration iterations; the cumulative reward of the exploration track is an exploration trajectory τ of a search drone in step 4.2 k ={s0,a1,r1,...,s t-1 ,a t ,r t The weighted sum of all rewards within}(t≤T);

[0031] Step 4.4: Based on the cumulative reward of each search drone’s exploration trajectory, calculate the reward of all search drones according to the search strategy π in one iteration. θ (a t |s t-1 ) Accumulated rewards for environmental detection after exploring a 3D dynamic environment The environmental detection cumulative reward is the value of n in step 4.3 S Cumulative reward for exploration tracks generated by search drones The sum of

[0032] Step 4.5: Based on the sequence sampling method, each search drone is θ (a t |s t-1 ) The trajectory τ generated in one iteration k ={s0,a1,r1,...,s t-1 ,a t ,r t}(t≤T) to perform sequence sampling and calculate the search drone’s trajectory τ k The probability of exploring the environment p(τ k |θ);

[0033] Step 4.6: The goal of the search task under the Markov decision process is to maximize the expected cumulative reward Based on mathematical approximation, the expected cumulative reward is approximately equal to the weighted sum of the trajectory exploration rewards of all search drones

[0034] Step 4.7: Construct the loss function of the algorithm based on the goal of the search task The loss function is a negative expected cumulative reward. When the minimum loss value is obtained based on the algorithm, the expected cumulative reward is the maximum;

[0035] Step 4.8: Based on the loss function, calculate the differential of the expected cumulative reward Based on the idea of ​​gradient ascent, θ←θ+η·G updates the parameters θ of the recurrent neural network, thereby updating the probability P(a) of the search drone selecting an action. t |s t-1 ,θ), where η is the learning rate; when the differential of the expected cumulative reward of a certain iteration is zero, the parameter θ stops updating, and the search strategy converges;

[0036] Step 4.9: Repeat steps 4.1 to 4.8 and obtain multiple cumulative rewards corresponding to the iteration. When the variance of the expected cumulative reward is less than the given value σ 2 When , the search strategy of the search drone converges. At this time, the search drone obtains the optimal search strategy and can find all targets in the three-dimensional dynamic environment in the shortest time and with the lowest search cost while avoiding all obstacles.

[0037] Furthermore, in step 4.1, the termination exploration conditions of the search drone include: the total exploration time step t≥T of the search drone in the three-dimensional dynamic environment; the search drone detects all targets in the three-dimensional dynamic environment; if any one of the termination exploration conditions of the search drone is met, the exploration mission is terminated.

[0038] A target search system in an unknown environment based on spatial clustering and deep reinforcement learning, using any of the target search methods described above for search, comprising a plurality of search drones, an airborne radar mounted on each search drone, and a processor module, wherein the processor module specifically comprises:

[0039] The task area division unit, based on the spatial clustering method of the Voronoi diagram, divides the three-dimensional dynamic environment to be searched into task areas equal to the number of drones according to the initial position of the search drone and the regional safety level, and allocates them to each search drone as a task area;

[0040] The search task modeling unit uses Gaussian distribution and random distribution methods to simulate the movement of targets and dynamic obstacles in the environment, constructs a reward function based on environmental feedback, and builds a mathematical model based on the search task;

[0041] The search strategy building unit is based on the neural network approximation method. It takes the state obtained by the airborne radar of the search UAV in each time step as input, and obtains the probability of the search UAV selecting a specific action in a specific state through a recurrent neural network, that is, the search strategy of the search UAV.

[0042] The search strategy optimization unit, based on the Markov decision process and the gradient ascent method, obtains the optimal search strategy for the search UAV by maximizing the expected cumulative reward of the search UAV.

[0043] It can be seen from the above technical solutions that the present invention provides a method and system for searching unknown environment targets based on spatial clustering and deep reinforcement learning, which has the following beneficial effects compared with the prior art:

[0044] (1) The present invention uses the spatial clustering method of the Voronoi diagram to rationally divide a large-scale three-dimensional dynamic environment into multiple sub-areas and assign them to different search drones, effectively reducing the search range of a single drone and making the search task more targeted, thereby improving the search efficiency and accuracy.

[0045] (2) The present invention uses Gaussian distribution and random distribution to simulate the movement of targets and dynamic obstacles, and constructs a mathematical model of search tasks that is closer to the actual environment. It can better cope with dynamically changing environments and improve the search capabilities of drones in complex environments.

[0046] (3) The present invention combines deep reinforcement learning technology, processes the state information obtained by the drone's airborne radar through a recurrent neural network, and optimizes the search strategy based on the Markov decision process, which can automatically learn and adapt to different environmental conditions.

[0047] (4) The present invention maximizes the expected cumulative reward of the search drone and detects all targets in the shortest time and at the lowest cost while avoiding all obstacles, which not only improves the search efficiency but also significantly reduces the search cost.

[0048] In summary, the solution of the present invention enables all search drones to safely search for all targets in a three-dimensional environment in the shortest time and at the lowest cost, further improving the target search capability of the search drones in a large airspace. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0050] Figure 1 is a working environment diagram of the search drone in the present invention;

[0051] Figure 2 It is a task area map allocated by the present invention according to the security level and the initial position of the search drone;

[0052] Figure 3 is a process diagram of updating the search strategy for searching drones in the present invention;

[0053] Figure 4 is a schematic diagram of the obstacle avoidance of the search drone in the present invention;

[0054] Figure 5 It is a schematic diagram of the steps of a method for searching targets in unknown environments based on spatial clustering and deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The embodiment of the present invention discloses a method for searching unknown environment targets based on spatial clustering and deep reinforcement learning. Figure 5, including the following steps: Step 1, based on the spatial clustering method of the Voronoi diagram, according to the initial position of the search drone and the regional safety level, the three-dimensional dynamic environment to be searched is divided into task areas equal to the number of drones, and the task areas are allocated to each search drone. Step 2, the Gaussian distribution and random distribution methods are used to simulate the movement of targets and dynamic obstacles in the environment, a reward function is constructed based on environmental feedback, and a mathematical model is constructed based on the search task. Step 3, based on the neural network approximation method, the state obtained by the airborne radar of the search drone in each time step is used as input, and the probability of the search drone selecting a specific action in a specific state is obtained through a recurrent neural network, that is, the search strategy of the search drone. Step 4, based on the Markov decision process and the gradient ascent method, the optimal search strategy of the search drone is obtained by maximizing the expected cumulative reward of the search drone.

[0057] The working environment in the embodiment of the present invention is as follows Figure 1 shown, including:

[0058] The radar and camera carried by the search drone can obtain the location information of targets, obstacles and other search drones within the sensing range. In the three-dimensional dynamic environment, the position of each search drone is p Si =(x Si ,y Si ,z Si )(i=1,2,...,n S ). Among them, the search drone can sense targets and obstacles within the detection range R. When obstacles or other search drones are within its detection range, the search drone will take obstacle avoidance measures; when the target appears within its perception range, the algorithm will update the state indicator variable S of the target in the environment f , where S f =1 indicates that the target has been discovered.

[0059] Dynamic targets are used as detection targets. During the exploration of the environment by the search drone, the target is Gaussian distributed in each time step in the environment. When the target is within the detection range of the search drone, its position will no longer change in the remaining time steps of the current exploration process.

[0060] Obstacles are divided into static obstacles and dynamic obstacles. Static obstacles always follow a random distribution in dynamic three-dimensional space. f ~F, and its position information does not change with the passage of time. In order to simulate the dynamic three-dimensional airspace environment, before the exploration mission begins, the dynamic obstacles follow the Gaussian distribution in space. When searching the drone detection environment, the dynamic obstacles follow the Brownian motion p in the three-dimensional airspace environment. d ~N[0,t].

[0061] A group of search drones detect dynamic targets in a large airspace environment with obstacles. Without communication constraints, the search drones can obtain the observation information of other search drones in the environment in real time. A suitable reward function is constructed to measure the quality of the trajectory generated by each search drone when detecting a target, where obstacles and targets change in real time.

[0062] In the present invention, the search for drone observation environment information includes the existence of targets within the detection radius, the current area security level w i (i=1,2,...,n S ), search for the drone's own location And posture [ω i ,ψ i ](i=1,..,5). When the search drone explores the environment, it will obtain the action to be performed from the action space A={a1,a2,a3,a4,a5} according to the observation information in each time step. The action space includes five types: tilt up a1=[π / 2,0], tilt down a2=[π / 2,π], left move a3=[0,π / 2], right move a4=[π,π / 2], and forward a5=[π / 2,π / 2].

[0063] In the embodiment of the present invention, the mission area of ​​the search drone is as follows: Figure 2 shown, including:

[0064] In the dynamic three-dimensional environment at the initial moment, the target exists in the environment in the form of Gaussian distribution. In order to approximate the possibility of the target appearing in different areas of the real environment, the target is distributed according to the regional security level w. i (i=1,2,...,n S ) Divide the three-dimensional dynamic environment into search drone mission areas of different sizes, wherein the search drone mission area is determined by the initial position of the search drone and the regional safety level. Specifically, the following steps are included:

[0065] Based on the known initial position information of the search drone, calculate the Euclidean distance between any point p in the three-dimensional airspace and all search drones

[0066] Based on the regional security level w i (i=1,2,...,n S ) and weighted K-means clustering method All points in the space are assigned to the search drone that is closest to them and has the highest regional safety level, forming n s The safety level w i (i=1,2,...,nS ) is used to measure the possibility of the existence of a target in a mission area. The higher the security level, the higher the probability of the existence of a target. i (i=1,2,...,n S ) is larger, the smaller the mission area assigned to the search drone is.

[0067] In the embodiment of the present invention, the update method of the search strategy of the search drone is as follows: Figure 3 As shown, the specific steps include:

[0068] In the first step, before the search drone explores the environment, the algorithm initializes the policy network used to obtain the search drone's actions. The policy network of the search drone is a recurrent neural network. The search drone is based on the initialized recurrent neural network by inputting the state information of the search drone. Thus, the probability P(a1,a2,a3,a4,a5) of each action being selected and executed in the action space A = {a1,a2,a3,a4,a5} is obtained at one time step. t |s t-1 ,θ), that is, the search strategy π θ (a t |s t-1 ).

[0069] In the second step, based on the initialized policy network, the search drone enters the state information of the current time step in the first iteration of exploration. Get the action a that needs to be performed at the current time step i =[ω i ,ψ i ](i=1,2,...,5). When the action is completed, the search drone will obtain an immediate reward r according to the reward function i Repeat this process until all search drones complete the exploration in the first iteration and obtain their corresponding exploration trajectory τ k ={s0,a1,r1,...,s t-1 ,a t ,r t}(t≤T). The exploration trajectory is the sequence of all states, actions and rewards generated by a search drone during an exploration process.

[0070] In the third step, based on the exploration trajectory, the algorithm calculates the cumulative exploration reward of each search drone. And store it in the experience pool, that is: a search drone based on the exploration trajectory τ k ={s0,a1,r1,...,s t-1 ,a t ,r tBased on mathematical approximation, the expected cumulative reward is approximately equal to the weighted sum of the trajectory exploration rewards of all search drones And construct the loss function based on the negative expected cumulative reward

[0071] The fourth step is based on the gradient ascent idea θ←θ+η·G, that is, the recurrent neural network parameter θ of the current iteration is composed of the parameter θ of the previous iteration and the gradient G. The present invention constructs the gradient by differentially expecting the cumulative reward The updated parameters θ are the weights and biases of the recurrent neural network. Based on the new weights and biases, when the state information of the search drone is input, the probability of each action being executed in the action space will change.

[0072] In the fifth step, when the variance of the expected cumulative reward is less than the given threshold in multiple preset iterations, the search strategy of the search drone converges. Based on this search strategy, the search drone will obtain the maximum expected cumulative reward, that is, the search drone detects all targets at the lowest cost while avoiding all obstacles.

[0073] In the embodiment of the present invention, the search drone avoids obstacles such as Figure 4 As shown, the following steps are included:

[0074] The search drone can sense the surrounding environment based on the onboard radar and can distinguish objects within its detection range through the camera. When a static obstacle appears within the detection range of the search drone, the search drone will update the instant reward and global state of the current time step. When other search drones or dynamic obstacles appear within the detection range of the search drone, the search drone avoids the drone by adjusting its flight attitude, that is: a i =[π-ω i ,π-ψ i ] and updates the immediate reward and global state of the current time step.

[0075] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0076] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for searching unknown environment targets based on spatial clustering and deep reinforcement learning, characterized in that: It is applied to multiple search drones to search for multiple targets in a three-dimensional dynamic environment. Each search drone is equipped with an airborne radar. The search method includes the following steps: Step 1: Based on the spatial clustering method of the Voronoi diagram, the three-dimensional dynamic environment to be searched is divided into task areas equal to the number of drones according to the initial position of the search drone and the regional safety level, and the task areas are allocated to each search drone; Step 2: Use Gaussian distribution and random distribution methods to simulate the movement of targets and dynamic obstacles in the environment, build a reward function based on environmental feedback, and build a mathematical model based on the search task; Step 3: Based on the neural network approximation method, the state obtained by the airborne radar of the search drone in each time step is used as input, and the probability of the search drone selecting a specific action in a specific state is obtained through a recurrent neural network, that is, the search strategy of the search drone; Step 4: Based on the Markov decision process and gradient ascent method, the optimal search strategy for the search drone is obtained by maximizing the expected cumulative reward of the search drone.

2. According to claim 1, a method for searching unknown environment targets based on spatial clustering and deep reinforcement learning is characterized in that: In step 1, the specific steps of dividing the task area are: Step 1.1: Consider the existence of n in a three-dimensional dynamic environment s search drones, whose initial position is Calculate the initial position of any point p = (x, y, z) in the three-dimensional dynamic environment and the search drone The Euclidean distance That is: any point in three-dimensional space and the search drone The shortest distance; Step 1.2: According to the clustering method of weighted Voronoi diagram, the three-dimensional dynamic environment is divided into n s Search drone mission area; set the initial position of all search drones As the centroid, combined with the safety levels of different areas, the safety level of each search drone’s mission area is determined, that is, the area weight w i (i=1,2,...,n S ); Step 1.3: Combine the security level w i (i=1,2,...,n S ) and Euclidean distance Spatial Clustering Method Based on Three-Dimensional Voronoi Diagram Find the mission area of ​​each search drone, that is, any point in space will be assigned to the search drone that is closest to it and has the highest regional safety level.

3. The method for searching unknown environment targets based on spatial clustering and deep reinforcement learning according to claim 1, characterized in that: In step 2, the specific steps of constructing the reward function and the task mathematical model are: Step 2.1: Consider the existence of n in a dynamic three-dimensional environment T targets, which follow a Gaussian distribution at each time step where μ T is the expectation of the target prior’s three-dimensional coordinates, σ T is the variance of the target prior’s three-dimensional coordinates; Step 2.2: Consider the existence of n in a three-dimensional dynamic environment O obstacles, the obstacles are divided into static obstacles With dynamic obstacles Where l+m=n O ; Static obstacle P f Follow random distribution in dynamic 3D environment Dynamic obstacle P d It follows Brownian motion at each time step t(t∈T) in a dynamic 3D environment Step 2.3: Based on the environmental feedback and the goal of the search task, a reward and punishment function for the interaction between the search drone and the environment is constructed. The reward r obtained by each search drone in one time step is i (t) is obtained by the cumulative sum of all reward and penalty values; Step 2.4: Based on the Markov decision process, the search drone detects all targets with the shortest search time and the lowest cost while avoiding all obstacles, which is defined as maximizing the expected cumulative reward Where γ = [0.9, 0.99] is the reward discount factor.

4. The method for searching unknown environment targets based on spatial clustering and deep reinforcement learning according to claim 3 is characterized in that: In step 2.3, the reward and penalty values ​​include: the reward for finding the target r f , Exploration Area Rewards e , Flight Penalty c , cross-border punishment b , exploration time penalty r s .

5. The method for searching unknown environment targets based on spatial clustering and deep reinforcement learning according to claim 1, characterized in that: In step 3, the specific steps of obtaining the search strategy are: Step 3.1: According to the distributed deep reinforcement learning algorithm, first set the state space S of the search drone to be {s0,s1,...,s t }(t∈T), where the state of each search drone Including the three-dimensional coordinates of the search drone, the scale factors of the search drone's speed in the X, Y, and Z axes, and the regional safety level; secondly, set the action space A = {a1, a2, a3, a4, a5}, where action a i =[ω i ,ψ i ](i=1,2,...,5) represents the specific action that the search drone chooses to perform, including the heading angle ω of the search drone i With the azimuth ψ i ; The action space contains 5 actions, which means that the search drone will select a specific action from the action space to execute in each time step; Step 3.2: In order to obtain the search strategy of the search drone, a recurrent neural network is designed. First, the state of each search drone is The input is sent to the input layer; secondly, the hidden layer extracts features and processes time-dependent input data through the network parameters θ, i.e., weights and biases, and activation functions; finally, the output layer obtains the probability P(a) of the search drone selecting the action to be performed in the current time step according to the state of the previous time step in each time step under the current parameters θ. t |s t-1 ,θ), that is: search strategy π θ (a t |s t-1 ); Step 3.3: Based on the search strategy π θ (a t |s t-1 ), the search drone performs the action with the largest probability value at a given exploration rate ε, and performs other actions except the action with the largest probability value at an exploration rate (1-ε).

6. The method for searching unknown environment targets based on spatial clustering and deep reinforcement learning according to claim 3, characterized in that: In step 4, the specific steps of obtaining the optimal search strategy are: Step 4.1, based on the search time step limit T and the search UAV detecting all targets, set the termination exploration condition of the search UAV; Step 4.2: Based on the Markov decision process, the search drone is assigned to the search strategy π θ All actions selected, all states experienced, and all rewards obtained when exploring a three-dimensional dynamic environment are taken as the exploration trajectory τ of the search drone. k ={s0,a1,r1,...,s t-1 ,a t ,r t }(t≤T); Step 4.3: Calculate the exploration trajectory τ of each search drone during the exploration process by weighted summation k Cumulative rewards for exploration tracks obtained in Where n e is the total number of iterations of exploration; Step 4.4: Based on the cumulative reward of each search drone’s exploration trajectory, calculate the reward of all search drones according to the search strategy π in one iteration. θ (a t |s t-1 ) Accumulated rewards for environmental detection after exploring a 3D dynamic environment Step 4.5: Based on the sequence sampling method, each search drone is θ (a t |s t-1 ) The trajectory τ generated in one iteration k ={s0,a1,r1,...,s t-1 ,a t ,r t }(t≤T) to perform sequence sampling and calculate the search drone’s trajectory τ k The probability of exploring the environment p(τ k |θ); Step 4.6: The goal of the search task under the Markov decision process is to maximize the expected cumulative reward Based on mathematical approximation, the expected cumulative reward is approximately equal to the weighted sum of the trajectory exploration rewards of all search drones Step 4.7: Construct the loss function of the algorithm based on the goal of the search task Step 4.8: Based on the loss function, calculate the differential of the expected cumulative reward Based on the idea of ​​gradient ascent, θ←θ+η·G updates the parameters θ of the recurrent neural network, thereby updating the probability P(a) of the search drone selecting an action. t |s t-1 ,θ), where η is the learning rate; when the differential of the expected cumulative reward of a certain iteration is zero, the parameter θ stops updating, and the search strategy converges; Step 4.9: Repeat steps 4.1 to 4.8 and obtain multiple cumulative rewards corresponding to the iteration. When the variance of the expected cumulative reward is less than the given value σ 2 When , the search strategy of the search drone converges, and at this time, the search drone obtains the optimal search strategy.

7. The method for searching unknown environment targets based on spatial clustering and deep reinforcement learning according to claim 6, characterized in that: In step 4.1, the termination exploration conditions of the search drone include: the total exploration time step t≥T of the search drone in the three-dimensional dynamic environment; the search drone detects all targets in the three-dimensional dynamic environment; if any one of the termination exploration conditions of the search drone is met, the exploration mission is terminated.

8. A target search system in an unknown environment based on spatial clustering and deep reinforcement learning, using the target search method according to any one of claims 1 to 7 for search, characterized in that: It includes multiple search drones, an airborne radar carried on each search drone, and a processor module, wherein the processor module specifically includes: The task area division unit, based on the spatial clustering method of the Voronoi diagram, divides the three-dimensional dynamic environment to be searched into task areas equal to the number of drones according to the initial position of the search drone and the regional safety level, and allocates them to each search drone as a task area; The search task modeling unit uses Gaussian distribution and random distribution methods to simulate the movement of targets and dynamic obstacles in the environment, constructs a reward function based on environmental feedback, and builds a mathematical model based on the search task; The search strategy building unit is based on the neural network approximation method. It takes the state obtained by the airborne radar of the search UAV in each time step as input, and obtains the probability of the search UAV selecting a specific action in a specific state through a recurrent neural network, that is, the search strategy of the search UAV. The search strategy optimization unit, based on the Markov decision process and the gradient ascent method, obtains the optimal search strategy for the search UAV by maximizing the expected cumulative reward of the search UAV.

Images