Multi-unmanned aerial vehicle unknown region target searching method based on hierarchical reinforcement learning

Abstract

The invention discloses a multi-unmanned aerial vehicle unknown area target search method based on hierarchical reinforcement learning, which comprises the following steps: dividing a search area comprising a plurality of targets and a plurality of obstacles into a plurality of search grids, distributing a search grid for each of a plurality of unmanned aerial vehicles, and searching the search area by using the unmanned aerial vehicles to obtain a plurality of target targets; acquiring search data and transmitting the search data to a ground station; at a ground station, based on centralized training, distributed execution and a hierarchical reinforcement learning framework, training an area coverage coordination strategy network as an upper layer decision network; training a target tracking identification strategy network as a lower layer decision network; and integrating the area coverage coordination strategy network and the target tracking identification strategy network to each unmanned aerial vehicle so as to guide the unmanned aerial vehicle to execute a target search task. According to the method, efficient search area coverage, continuous tracking and dynamic obstacle avoidance capabilities are realized in a multi-unmanned aerial vehicle cooperative target search task, and the robustness of the algorithm is improved through centralized training and distributed execution.

Description

Technical Field

[0001] The present invention belongs to the technical field of target search, and in particular relates to a multi-UAV unknown area target search method based on hierarchical reinforcement learning. Background Art

[0002] With the rapid development of drone technology and artificial intelligence, intelligent, autonomous drones have found widespread application in many fields. Due to their high maneuverability, low energy consumption, and flexibility in complex environments, drones are also being used for target search in unknown areas. Therefore, the collaborative operation of multiple drones to perform target search in unknown areas has become a hot topic in current research.

[0003] In practical applications, many missions require drones to search for targets in complex and unknown areas. These areas may include unpredictable terrain, constantly changing obstacles, and environmental uncertainty, and the target objects may be widely distributed. Limited by the hardware and technical limitations of the target recognition module, drones may need to track suspected targets and identify them multiple times to confirm whether they are the target being searched, which poses a significant challenge to drone search missions. As the number of targets increases, how to efficiently and reasonably allocate drone coverage areas and ensure that drones can adjust their search paths based on the real three-dimensional scene becomes a key factor in mission success.

[0004] Furthermore, the target area isn't just static; the environment constantly changes over time. For example, new obstacles may appear in the drone's path, or the target's position may shift. These changes significantly impact the drone's search strategy and execution efficiency. To cope with this dynamic environment, the drone needs to make real-time decisions and optimize its search path based on real-time feedback.

[0005] Drones are tasked with highly dynamic and complex multi-target search missions in unknown three-dimensional environments, requiring them to make autonomous decisions in rapidly changing environments. Using intelligent algorithms to improve search efficiency, reduce mission execution time, and ensure mission completion is a key research issue.

[0006] The collaborative target search problem involves multiple drones efficiently covering an unknown search area while tracking and identifying targets. Numerous methods and solutions have been proposed, primarily relying on centralized and distributed control of multiple drones, combined with sensors for target identification, and training multiple drones through reinforcement learning algorithms to address task allocation and path planning in target search.

[0007] Existing multi-UAV cooperative target search methods mainly fall into the following categories: the first category is the control method for multi-UAV target search, including multi-UAV target search methods based on centralized control and multi-UAV target search methods based on distributed control; the second category is multi-UAV cooperative target search based on reinforcement learning algorithms; and the third category is target recognition and tracking methods based on vision and sensor fusion. However, the above methods still have the following drawbacks:

[0008] 1. Limitations of centralized control and distributed control

[0009] Existing centralized control methods rely on real-time communication with the ground control center and are susceptible to communication delays, single point failures, etc.; distributed control lacks a global perspective, which may lead to the inability of drones to collaborate efficiently, affecting the search effect.

[0010] 2. Limitations of Reinforcement Learning Algorithms

[0011] Existing deep reinforcement learning methods typically require extensive computing resources and time for training, making the training process particularly slow in dynamic, complex three-dimensional environments. Furthermore, the sparse reward problem prevents drones from receiving timely feedback during execution, resulting in inefficient training and impacting search efficiency.

[0012] 3. Technical limitations of target recognition and tracking

[0013] Existing target recognition methods mainly target static targets and have poor processing capabilities for multiple targets moving in dynamic environments. Summary of the Invention

[0014] To address at least one of the above-mentioned problems in the prior art, the present invention provides a multi-UAV target search method in an unknown area based on hierarchical reinforcement learning. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0015] In the first aspect, the present invention proposes a multi-UAV unknown area target search method based on hierarchical reinforcement learning, comprising:

[0016] Step 1: Divide the search area, which includes multiple targets and multiple obstacles, into several search grids. Each of the multiple drones is assigned a search grid. The drones are used to search the search area, obtain search data, and transmit it to the ground station. Each target occupies only one search grid, and each obstacle occupies at least one search grid.

[0017] Step 2: At the ground station, a regional coverage coordination strategy network is trained as the upper-level decision network based on a centralized training, distributed execution, and hierarchical reinforcement learning framework. Simultaneously, a target tracking and identification strategy network is trained as the lower-level decision network. In the hierarchical reinforcement learning framework, the state space is defined as the set of search data, the action space is defined as the set of actions that the drone can perform, and the reward function is the sum of the reward value obtained after the drone successfully identifies a target and the penalty value imposed after the drone is damaged.

[0018] Step 3: Integrate the area coverage coordination strategy network and the target tracking and identification strategy network into each UAV to guide the UAV to perform target search tasks; among them, the area coverage coordination strategy network is used to select a flight path for each UAV; the target tracking and identification strategy network is used to adjust the UAV's flight attitude based on the flight path to continuously track and identify the target.

[0019] In a second aspect, the present invention proposes a multi-UAV unknown area target search device based on hierarchical reinforcement learning, which is used to implement the method provided in the first aspect of the present invention. The device includes:

[0020] An initialization module is used to divide the search area, which includes multiple targets and multiple obstacles, into several search grids, assign a search grid to each of the multiple drones, use the drones to search the search area, obtain search data, and transmit it to the ground station; each target occupies only one search grid, and each obstacle occupies at least one search grid;

[0021] The training module, set up at the ground station, includes a regional coverage coordination strategy network training unit and a target tracking and identification strategy network training unit, which are respectively used to train a regional coverage coordination strategy network as the upper-layer decision network based on centralized training, distributed execution, and a hierarchical reinforcement learning framework; and simultaneously train a target tracking and identification strategy network as the lower-layer decision network. In the hierarchical reinforcement learning framework, the state space is defined as the set of search data, the action space is defined as the set of actions that the drone can perform, and the reward function is the sum of the reward value obtained after the drone successfully identifies a target and the penalty value imposed after the drone is damaged.

[0022] The search execution module is used to integrate the area coverage coordination strategy network and the target tracking and identification strategy network into each UAV to guide the UAV to perform target search tasks; among them, the area coverage coordination strategy network is used to select a flight path for each UAV; the target tracking and identification strategy network is used to adjust the UAV's flight attitude based on the flight path to continuously track and identify the target.

[0023] In a third aspect, the present invention provides an electronic device comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0024] Memory for storing computer programs;

[0025] The processor is used to execute the program stored in the memory to implement the method provided by the first aspect of the present invention.

[0026] In a fourth aspect, the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method provided in the first aspect of the present invention is implemented.

[0027] Beneficial effects of the present invention:

[0028] The present invention provides a multi-UAV target search method in an unknown area based on hierarchical reinforcement learning. First, the search area is divided into multiple search grids, and each UAV is assigned a search grid for data search. Then, a hierarchical reinforcement learning framework is constructed at the ground station. A regional coverage coordination strategy network and a target tracking and identification strategy network are trained using the search data of multiple UAVs. Finally, the two obtained networks are integrated into each UAV to guide each UAV to perform the search task in a distributed manner. By combining hierarchical reinforcement learning, this method achieves efficient search area coverage, continuous tracking, and dynamic obstacle avoidance capabilities in the multi-UAV collaborative target search task, and improves the robustness of the algorithm through centralized training and distributed execution. The details are as follows:

[0029] 1. Improved search efficiency and coverage

[0030] By dividing the search area into multiple search grids and combining multiple drones, the present invention can effectively guide drones to efficiently cover the target area in a complex environment. It adopts a hierarchical reinforcement learning architecture, combines the advantages of centralized control and distributed control, and centrally allocates coverage areas and avoids obstacle areas through the upper layer, while the lower layer performs distributed target identification and tracking. This enables multiple drones to accurately select flight paths based on the target position information and obstacle information of the radar module, and autonomously adjust their flight posture in a three-dimensional environment. Through the collaborative work of the upper and lower network layers, each drone can collaborate with multiple drones to avoid repeated searches or collisions, thereby significantly improving search efficiency and area coverage.

[0031] 2. Achieved accurate target recognition and tracking capabilities

[0032] By incorporating a hierarchical reinforcement learning framework, this paper designs an algorithmic structure that adapts to sparse rewards. This allows for the rapid generation of feasible target recognition and tracking postures, enabling each drone to develop an effective search strategy within a limited training time, even without sufficient feedback. This allows the drone to precisely adjust its flight posture to continuously track and identify the target after detection. Furthermore, the flight control strategy optimized using the reinforcement learning algorithm ensures that the drone remains aligned with the target in three-dimensional space, improving both the accuracy and efficiency of target recognition.

[0033] 3. Enhanced system robustness and adaptability

[0034] This invention utilizes a hierarchical reinforcement learning framework, combining centralized training with distributed execution. This approach enables drone swarms to maintain high mission execution stability and system robustness in the event of emergencies (such as drone damage or communication interruption). Real-time feedback mechanisms and distributed communication ensure that even in the event of environmental changes or when individual drone capabilities are reduced, other drones can continue to execute and complete their missions. Furthermore, drones can continuously track suspicious dynamic targets, thereby improving their accuracy in identifying search targets.

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a flow chart of a multi-UAV unknown area target search method based on hierarchical reinforcement learning provided by an embodiment of the present invention;

[0037] Figure 2 A framework diagram of a multi-UAV unknown area target search method based on hierarchical reinforcement learning provided by an embodiment of the present invention;

[0038] Figure 3 Another flowchart of the multi-UAV unknown area target search method based on hierarchical reinforcement learning provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of a multi-UAV target collaborative search environment configuration according to an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of the skill space of a drone provided in an embodiment of the present invention;

[0041] Figure 6 This is a structural block diagram of a multi-UAV unknown area target search device based on hierarchical reinforcement learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] The first aspect of the present invention provides a method for target search in an unknown area by multiple UAVs based on hierarchical reinforcement learning. The main implementation scenario of this method is any unknown area to be explored in the actual terrain, which includes multiple targets and multiple static obstacles. Multiple UAVs take off from a well-established ground station to perform target search tasks, perceive the environment through the sensors and communication devices they carry, and exchange data with the ground station and other nearby UAVs. In this task, multiple UAVs need to work together in a dynamic environment to search and track targets while avoiding obstacles. The real environment can be regarded as a three-dimensional area representing the flight airspace, in which there are multiple targets and obstacles. Among them, the target is dynamically changing, and the position of the obstacle is unknown. The position and size of the obstacle will affect the flight path of the UAV.

[0044] Please see the joint Figure 1-3 , Figure 1 A flowchart of a multi-UAV unknown area target search method based on hierarchical reinforcement learning provided by an embodiment of the present invention is provided. Figure 2 A framework diagram of a multi-UAV unknown area target search method based on hierarchical reinforcement learning provided by an embodiment of the present invention. Figure 3 Another flowchart of a multi-UAV unknown area target search method based on hierarchical reinforcement learning provided by an embodiment of the present invention. The method mainly includes the following steps 1 to 3.

[0045] Step 1: Divide the search area, which includes multiple targets and multiple obstacles, into several search grids. Assign a search grid to each of the multiple drones. Use the drones to search the search area, obtain search data, and transmit it to the ground station. Each target occupies only one search grid, and each obstacle occupies at least one search grid.

[0046] It can be understood that the search grid division and drone allocation operations in step 1 can be understood as the modeling and initialization settings of the environment and drones in the initial stage.

[0047] Specifically, for environmental modeling, it can be implemented by following the steps A1-A3:

[0048] A1. The search area including multiple targets and multiple obstacles is limited to a rectangular area Ω with a length of L and a width of W.

[0049] A2. Based on mission requirements and aircraft performance, divide the rectangular area into m × n search grids of equal size. The side length s of each search grid is calculated as follows:

[0050]

[0051] In the formula, min(,) means taking the minimum value.

[0052] Optionally, this embodiment uses two-dimensional Cartesian coordinates to describe the location of the search grid, and sets the location of the ground station as the coordinate origin (0,0). Therefore, each search grid can be identified by its center, and the set of all search grids is represented by C m,n , the search grid is represented as:

[0053] c x,y =(x,y),x∈{1,2,…,m},y∈{1,2,…,n}.

[0054] A3. Assign a confidence value P to each search grid x,y , used to express the probability that the target exists in the search grid, and let P x,y = -1 means there are obstacles in the search grid (x, y).

[0055] See Figure 4 , Figure 4 Schematic diagram of the multi-UAV target collaborative search environment provided by the embodiment of the present invention. Assume that there are N randomly distributed in the search area Ω. t targets, each target occupies only one search grid, and each search grid is associated with the probability of the target existing in the grid. The confidence value of the target to be searched in the search grid (x, y) is expressed as P x,y . Randomly distribute N in the search area z Static obstacles, one obstacle can occupy multiple search grids. In addition, this embodiment also sets P x,y =-1 means there are obstacles in the search grid (x, y). In this case, the drone cannot fly into the grid.

[0056] After completing the environment modeling, you also need to model the drone, which includes the following steps B1-B3.

[0057] B1. Multiple drones are dispatched from a ground station and flown along an initially preset path until each drone occupies a search grid. Each drone is equipped with a radar module and a target recognition module.

[0058] Specifically, let all N uThe drones start from the ground station and fly straight and at a constant speed along the initially set flight direction. Each drone occupies a search grid. Since the actual scene is three-dimensional, each drone can be at a height not lower than the safe height h safe When there is no altitude adjustment command, each drone will maintain the initial altitude setting. i (i=1,2,…N u ) is equipped with a radar module and a target recognition module. The radar module is used to detect potential targets and obstacles. Its illumination range is large and can obtain information on the probability of the presence of targets or obstacles within multiple search grids. The target recognition module is used to confirm whether suspicious targets detected by the radar are the target to be searched. Its illumination range is small and requires continuous aiming at suspicious targets for identification.

[0059] Alternatively, the present invention employs a fixed-wing drone, using six-degree-of-freedom dynamic modeling to control the aircraft's pitch, roll, and yaw angles to adjust flight attitude. Because the three Euler angles are coupled, the yaw angle can be indirectly controlled through the pitch and roll angles. After the drone's target recognition module determines the relative position of a suspicious target, this information is input into the control strategy, which then outputs linear and angular velocity commands to directly control the aircraft. The specific model parameters are shown in Table 1.

[0060] Table 1 Flight state parameters of six-degree-of-freedom UAV

[0061]

[0062] B2. Each UAV uses its own radar module to search a square area with a range of f×f search grids centered on its own search grid, and exchanges information with neighboring UAVs during the search process to obtain radar recognition information. At the same time, each UAV uses its own target recognition module to search a circular area with a certain radius centered on the UAV to obtain target recognition information.

[0063] Specifically, the radar module detection range is a square area with the grid where the drone is located as the center and a range of f×f search grids. It can obtain the probability information of the target or obstacle in each search grid. Therefore, the radar module data (also known as radar cognitive information) obtained by each drone is an f×f matrix, and the value of each element of the matrix is the confidence value P of the existence of a suspicious target or obstacle in its corresponding search grid. x,y When P x,y When it is close to 1, it indicates that there is a high probability that a suspicious target exists in the search grid (x, y); when P x,y When it is close to 0, it indicates that there is a small probability of a suspicious target in the search grid (x, y); when P x,yWhen it is close to -1, it indicates that there is a high probability that there are obstacles in the search grid (x, y). Before the UAV has no observation information on the search grid (x, y), let P of the grid be x,y =0.

[0064] The target recognition module's illumination range is a circular area centered on the drone with a radius of e. Therefore, the target recognition module data (also known as target recognition information) is a scalar quantity, denoted by σ, representing the area of the target recognition module's illumination of the suspicious target. A larger σ value indicates a larger area of the target recognition module's illumination of the suspicious target, and thus a higher recognition accuracy.

[0065] It should be noted that, in this embodiment, target identification can only be performed when the UAV and the suspicious target are in the same search grid.

[0066] B3. Radar recognition information, target identification information, and the drone’s own state perception information are transmitted to the ground station as search data.

[0067] In this embodiment, the drone's self-state perception information is mainly the drone's self-state perception data, including: relative position with suspicious targets and roll angle, pitch angle and yaw angle. The drone can change its flight attitude by controlling linear velocity and angular velocity.

[0068] Each drone sends the collected data to the ground station, where it is combined to obtain global information data of the search area for subsequent reinforcement learning training.

[0069] Step 2: At the ground station, a regional coverage coordination strategy network is trained as the upper-level decision network based on a centralized training, distributed execution, and hierarchical reinforcement learning framework. At the same time, a target tracking and recognition strategy network is trained as the lower-level decision network. In the hierarchical reinforcement learning framework, the state space is defined as the set of search data, the action space is defined as the set of actions that the drone can perform, and the reward function is the sum of the reward value obtained after the drone successfully identifies a target and the penalty value imposed after the drone is damaged.

[0070] Specifically, through the hierarchical reinforcement learning framework, the regional coverage coordination strategy network is trained and recorded as π uppper , the target tracking and recognition strategy network is recorded as π lower The state space S is defined as the set containing all the sensor data of the drone (including the radar module and target recognition module) and the drone's state information; the action space A is the set of actions that the drone can perform, including changing the linear velocity u, v, w and the angular velocity p, q, r; and the reward function R is the sum of the reward value obtained by the drone after successfully identifying a target and the penalty value imposed after the drone is damaged.

[0071] This embodiment employs a centralized training and distributed execution approach to develop a regional coverage coordination strategy network, which is integrated into each drone as a top-level strategy to guide drones toward targets and away from obstacles, thereby coordinating drones to reduce repeated searches and collisions. Simultaneously, a target tracking and identification strategy network is trained through a hierarchical reinforcement learning framework and integrated into each drone as a bottom-level strategy to guide drones in tracking and identifying suspicious targets with appropriate flight attitudes after searching for them. Training data is collected by the drone's radar module and target recognition module, combined with the drone's own state perception data and location information transmitted through communication between adjacent drones. Each drone also transmits radar data to a ground station, combining them to generate global information about the search area for training.

[0072] The following is a detailed introduction to the training process of the area coverage coordination strategy network and the target tracking and recognition strategy network.

[0073] 1. Regional Coverage Coordination Strategy Network

[0074] In this embodiment, the QMIX (Q-value Mixing) algorithm is primarily used to train each drone's regional coverage strategy within the search area. Specifically, the algorithm guides the drone to select a search grid close to the suspected target and avoid obstacles based on the target's possible location information and obstacle location information received by the radar module. Using a centralized training and distributed execution approach, the training process involves the exchange of location data between adjacent drones and data between multiple drones and ground stations. Within a limited timeframe, an action-value network is generated as the regional coverage coordination network.

[0075] Specifically, the training process of the regional coverage coordination strategy network mainly includes the following steps C1-C3.

[0076] C1. Extract the target obstacle confidence map from the radar cognition information of each drone in the search data. At the same time, combine the radar cognition information of each drone to extract the environment search state map and drone position map. The environment search state map, drone position map and target obstacle confidence map of each drone are combined into global information as distributed interactive training data.

[0077] In this embodiment, each drone uses its radar module to obtain cognitive information about its search area (f × f), guiding its autonomous flight decisions within a fully distributed architecture. Three separate cognitive information maps can be extracted from each drone's radar cognitive information in the search data: a target obstacle confidence map, an environmental search status map, and a drone position map.

[0078] First, for the target obstacle confidence map, each UAV should have the ability to autonomously modify the target obstacle confidence map based solely on its own observations, without the need to exchange information with neighboring UAVs. This will ensure that the multi-UAV system remains robust to any interruptions in communication. In the absence of cooperation, each UAV uses the Bayesian rule to update the confidence value of the target obstacle, which is expressed as:

[0079]

[0080] Where, and They are the confidence level of a UAV regarding the existence of a suspicious target in the search grid (x, y) at time t-1 and time t, respectively. is the radar illumination result of the UAV on the search grid (x, y) at time t, Indicates that the radar has detected a suspicious target within the grid. Indicates that the radar has not detected any suspicious target in the grid, otherwise indicates that the radar has not illuminated the grid. Indicates that there is a suspicious target in the grid (x, y) in the real scene. Indicates that there is no suspicious target in the grid (x, y) in the real scene; P D Indicates the probability of a suspicious target existing in the grid (x, y) in the real scene, P F This represents the probability that no suspicious object exists within the grid (x, y) in the real scene. The calculation formula for the confidence level of obstacles is the same, but the opposite is used to distinguish it from the confidence level of suspicious objects.

[0081] Optionally, as an implementation method, the present invention uses a linear update method to reduce the computational complexity of updating the target obstacle confidence map.

[0082]

[0083] but Can be replaced by

[0084]

[0085] Therefore, the update formula of the target obstacle confidence map can be replaced by

[0086]

[0087] Where,

[0088]

[0089] When multiple drones are deployed at the same time, each drone can transmit cognitive information to adjacent drones by broadcasting, thus achieving information fusion and collaboration. After receiving the target obstacle confidence value of the same search grid (x, y) from other drones, Going through the following updates:

[0090]

[0091] Where η∈(0,1) is the information attenuation factor, w i as follows:

[0092]

[0093] The search grids and locations visited by each drone are sent to the ground station, which integrates this information as global state input into the super network of the QMIX training network.

[0094] Furthermore, for the environment search state diagram, the environment search state diagram maintained by the ground station is an m×n matrix. The matrix elements are initialized to 0. Each element records the latest detection time of the corresponding search grid. The element value is updated as follows:

[0095]

[0096] Where μ x,y Indicates the time when the grid (x, y) was last visited by the drone, Indicates that the grid (x, y) is occupied by the drone u at time t i access, Indicates that the grid (x, y) is visited by another UAV at time t-1. This formula indicates that when a UAV visits the grid (x, y) at time t, the corresponding element of the environment search state graph is updated to the current access time.

[0097] In addition, for the UAV location map, the UAV location map maintained by the ground station is a 2×N u The matrix elements are initialized to 0, and each row records the search grid where the corresponding drone is located.

[0098] At this point, three separate cognitive information maps are obtained: target obstacle confidence map, environment search status map, and drone position map.

[0099] C2. Perform reinforcement learning modeling on the upper-layer network decision-making problem.

[0100] Specifically, the goal of the present invention is to find as many targets as possible and cover as much mission area as possible within a limited time. Assuming that the search task requires T steps, the above goal is equal to maximizing:

[0101]

[0102] in, is the global information of the probability that the suspicious target exists in the search grid (x, y), is the number of times the search grid (x, y) is visited by the drone, and ε is a pre-set threshold. When the probability exceeds ε, it is considered that there is a suspicious target.

[0103] Secondly, the UAV is required to avoid collision with obstacles during the search mission. Assume that the safe distance between the UAV and the obstacle is d safe , then the drone u i Position at time t Must meet:

[0104]

[0105] Among them, c z Indicates the position occupied by the obstacle.

[0106] Then the upper-layer network decision problem can be expressed as:

[0107] max g=[g1,g2];

[0108]

[0109] Due to the limitations of drone observation and communication capabilities, in order to cope with the uncertainty brought by other drone behaviors, this paper uses a temporal abstraction method to expand the Markov decision process to a semi-Markov decision process, and obtains a distributed partially observable Markov decision process, which is defined as:

[0110] (S,{A i},T s ,{O i},T o ,R,N u ,γ);

[0111] Where, the state space S represents all possible states of the environment. The action space A of drone i is i Represents the set of actions that the drone can take, and the action spaces of all drones jointly constitute A; the state transition probability function T s Indicates that the environment is in state s t When the UAV takes joint action a t After that, the environment moves to the next state s t+1 The probability of UAV i’s observation space O i Represents the set of environmental information that the drone can observe. The observation space of all drones jointly constitutes O; the observation probability function T o Indicates that the drone is in a given action ai,t and the current environment state s t Observation o is obtained when i,t The reward function R represents the probability of all drones according to the current state s t and make a joint action a t After that, the immediate reward is obtained; the number of drones N u Represents the total number of drones participating in the collaborative task. The discount factor γ of the cumulative reward is used to measure the importance of future rewards to the current decision.

[0112] According to the modeling of the real search environment in the previous article, the observation space O of UAV i i Defined as containing the drone position [x i ,y i ], i∈U, speed v i , target and obstacle probability information detected by radar [P x,y ], x, y∈f and other UAV location information A collection of .

[0113] The present invention adopts the temporal abstraction method to extend the Markov decision process to the semi-Markov decision process, and uses skill k to represent a basic action sequence {a1, a2, ... a k In the training phase of the regional coverage coordination decision network, the skill space of k is defined as They respectively represent flying to the adjacent grid behind the left, flying to the adjacent grid on the left, flying to the adjacent grid in front of the left, flying to the adjacent grid in front of the front, flying to the adjacent grid in front of the right, flying to the adjacent grid on the right, flying to the adjacent grid behind the right, and flying to the adjacent grid behind. Figure 5 Demonstrates the drone's skill space.

[0114] The probability of a drone's skill transitioning from state s to state s' is described by the transfer function p(s'|s,k). Assuming the drone executes skill k with a step length of K, its cumulative reward r(s,k) is calculated as follows:

[0115] r(s,k)=E[r t+1 +γr t+2 +…+γ K-1 r t+K |s,k];

[0116] Note that each drone performs skills at time t Combination skills k t , then the immediate reward value r(s,k t ) is calculated as:

[0117]

[0118] r(s,k t )=w1r1+w2r2;

[0119] Among them, w1 and w2 are weighted parameters of reward value and penalty value.

[0120] C3. Based on distributed interactive training data, the QMIX algorithm is used to solve the upper-layer network decision problem, and a regional coverage coordination strategy network is trained as the upper-layer decision network.

[0121] Optionally, as an implementation method, this embodiment uses the QMIX algorithm to solve the above decision problem. First, based on the value decomposition network, the joint action value function Q tot (s,k) is represented by the individual action value function Q upper,i (o i ,k i )’s accumulation:

[0122]

[0123] Among them, θ i are the neural network parameters of the i-th UAV.

[0124] Then, a monotonic function is used to represent the joint action-value function, which relaxes the strict accumulation constraint of the value decomposition network, that is, satisfies:

[0125]

[0126] Each drone will observe the current moment and skills As input, the gated recurrent unit network is used to mine useful information from historical observations, and the drone network outputs the value of individual actions. The action value of each drone is input into the hybrid network, and the weight of the hybrid network is output by a separate super network and is non-negative. The input of the super network is the global state s t , the final hybrid network output Q tot (s t ,a t ) value.

[0127] The QMIX algorithm trains the network by minimizing the loss function L(θ). The loss function calculation formula is as follows:

[0128] L(θ)=(r(s,k)+γmax k′ Q tot (o′,k′,s′|θ′)-Q tot (o,k,s|θ)) 2 ;

[0129] Among them, γ is the discount factor, θ is the network parameter, and θ′ is the target network parameter. During the training process, the drone selects a search grid close to the suspected target and avoids the obstacle area based on the target possible location information and obstacle location information received by the radar module, and finally obtains the regional coverage coordination strategy network π uppper .

[0130] 2. Target Tracking and Recognition Strategy Network

[0131] In this embodiment, when a drone detects a suspicious target within a search grid using data received by its radar module, it uses a target tracking and recognition strategy network trained using the SAC algorithm to guide the drone in continuously tracking and identifying the suspicious target. This strategy controls the drone's flight attitude in three-dimensional space, enabling the target recognition module to accurately align with the target for further identification. The training process uses the drone's attitude, speed, relative position to the suspicious target, and the recognition module's illumination angle of the suspicious target as input data, optimizing the target tracking flight attitude so that the target recognition module can continuously focus on the suspicious target.

[0132] Specifically, the training process of the target tracking and recognition strategy network mainly includes the following steps D1-D3.

[0133] D1. Perform reinforcement learning modeling on the underlying network decision problem.

[0134] The upper-layer area coverage coordination decision model determines the general direction of target tracking based on global information (the environment search state map and UAV position map aggregated by the ground station, as well as the target obstacle confidence map for each UAV). Its output is the skill k for exploring each UAV grid. The lower-layer target tracking and recognition decision model uses the upper-layer skills and combines the UAV's own state information to perform fine trajectory planning in the real 3D scene. The state space is as follows:

[0135] S lower ={(x t ,y t ,h t ),(θ,φ,ψ),σ};

[0136] Where (x t ,y t ,h t ) represents the relative position of the suspicious target with the drone as the origin. (θ, φ, ψ) are the drone's flight attitude, which are the roll angle, pitch angle, and yaw angle, respectively. σ is the illumination area of the suspicious target by the target recognition module.

[0137] Action space A lower The set of actions that can be adjusted for the flight attitude of the drone is modeled as follows:

[0138] A lower ={(u,v,w),(p,q,r)};

[0139] Among them, (u, v, w) controls the linear velocity of the UAV in the x-, y-, and height directions, and the relative position of the suspicious target. (p, q, r) controls the flight attitude of the UAV, which are the roll angular velocity, pitch angular velocity, and yaw angular velocity, respectively.

[0140] Reward function R lower It depends on the tracking status of the drone on the suspicious target. If the target recognition module can continuously and stably aim at the suspicious target, it will be rewarded; if the target is lost or the recognition is interrupted, it will be punished.

[0141] The decision-making process of the lower network can be modeled as a Markov decision model, which is defined as:

[0142] (S lower ,A lower ,P lower ,R lower ,γ lower );

[0143] Among them, S lower is the state space of a single UAV, A lower is the action space of a single UAV. State transition probability P lower =P(s lower,t+1 |s lower,t ,a lower,t ) describes the current state s lower,t Next, perform action a lower,t Then, transfer to the next state s lower,t+1 The probability of this probability is determined by the UAV's dynamic model, environmental factors, and the target's motion characteristics. lower For evaluation in state s lower,t Next, perform action a lower,t The pros and cons of , which is directly related to state transfer. γ lower is the attenuation factor.

[0144] Through iterative training, minimize the loss function L lower To optimize the lower-level trajectory planning model:

[0145]

[0146] in,

[0147]

[0148] r lower,t is the reward at time t, γ loweris the discount factor, θ lower are the lower layer network parameters, are the lower layer target network parameters.

[0149] D2. Using the target recognition information of each drone in the search data and the drone's own state perception information as training data, the SAC (Soft Actor-Critic) algorithm is used to iteratively train a target tracking and recognition strategy network as the lower-level decision network.

[0150] In the target tracking and recognition layer, when entering the target tracking and recognition stage, the SAC algorithm is used for training. The SAC algorithm aims to maximize the objective function J(π):

[0151]

[0152] Among them, π is the policy network, a is the temperature parameter used to adjust the entropy of the policy H(π(·|s)), ρ π represents the state distribution under policy π.

[0153] During training, state-action pairs (s lower ,a lower ), and generate a new action a' based on the current strategy network lower , calculate the target value y lower :

[0154]

[0155] Then, the action value network and the policy network are updated by minimizing the mean square error loss function, and finally the target tracking and recognition network Q is obtained. lower .

[0156] Step 3: Integrate the area coverage coordination strategy network and the target tracking and identification strategy network into each UAV to guide the UAV to perform target search tasks; among them, the area coverage coordination strategy network is used to select a flight path for each UAV; the target tracking and identification strategy network is used to adjust the UAV's flight attitude based on the flight path to continuously track and identify the target.

[0157] Specifically, after training, the regional coverage coordination network is integrated into each drone's control system as the upper-layer network. While the parameters of the regional coverage coordination network integrated into different drones are identical, it guides the drone's selection of the next search grid to fly to based on information received by the radar module in the real environment. The target tracking and recognition network is integrated into each drone's control system as the lower-layer network. While the parameters of the target tracking and recognition network integrated into different drones are identical, it determines the appropriate flight attitude control strategy based on information received by the target recognition module in the real environment and the drone's own information.

[0158] The present invention integrates the trained target tracking and recognition network into the control system of each drone, enabling the drone to have the ability to autonomously control its flight attitude, aiming to achieve efficient tracking and identification of suspicious targets within the search area. During the actual mission execution process, each drone first receives a mission instruction k issued by the regional coverage coordination network, which is used to indicate the target flight area for the current flight phase. Subsequently, the target tracking and recognition network obtains the environmental information sensed by the target recognition module on board the drone and the drone's state information, which together constitute the current environmental state data s lower Based on this state lower , the target tracking and recognition network performs forward reasoning and outputs the corresponding flight attitude control strategy a lower This strategy includes action commands such as heading adjustment, speed control, and flight altitude adjustment, which guide the drone to continuously adjust its motion trajectory to achieve continuous tracking and dynamic identification of the target, improving the overall target locking accuracy and mission completion efficiency.

[0159] The present invention uses the real-time feedback mechanism of the reinforcement learning algorithm, and each drone updates its status in real time according to environmental changes. When there is an environmental change (such as an obstacle is detected, the target moves, etc.), the state The relevant information in will change accordingly.

[0160] Through the advantages of centralized training and distributed execution of the hierarchical reinforcement learning algorithm, when a small number of drones are damaged or communication is interrupted, assuming that drone u k Damage, ground stations and other drones j (j∈N u ,j≠k) will not receive any information about the drone u k The system can still use the regional coverage coordination network Q according to the updated status upper Guide the UAV to fly to the next search grid, while the target tracking and recognition network Q lower The input state of does not depend on other drones.

[0161] The proposed solution can adjust mission execution strategies based on environmental changes, ensuring that the entire drone swarm can continue to efficiently and stably execute search missions in complex environments. If a drone loses communication due to a malfunction, other drones, through a real-time feedback mechanism, reallocate the search area during subsequent area coverage coordination and target tracking and identification processes, avoiding search gaps caused by the faulty drone. Simultaneously, the distributed execution structure maintains the continuous tracking and identification of targets by individual drones, ensuring that the search mission is not significantly impacted.

[0162] In summary, the multi-UAV unknown area target search method based on hierarchical reinforcement learning provided by the present invention first divides the search area into multiple search grids and assigns a search grid to each UAV for data search; then, a hierarchical reinforcement learning framework is constructed at the ground station, and a regional coverage coordination strategy network and a target tracking and identification strategy network are trained using the search data of multiple UAVs; finally, the two obtained networks are integrated into each UAV to guide each UAV to perform the search task in a distributed manner; by combining hierarchical reinforcement learning, this method achieves efficient search area coverage, continuous tracking, and dynamic obstacle avoidance capabilities in the multi-UAV collaborative target search task, and improves the robustness of the algorithm through centralized training and distributed execution, as follows:

[0163] 1. Improved search efficiency and coverage

[0164] By dividing the search area into multiple search grids and combining multiple drones, the present invention can effectively guide drones to efficiently cover the target area in a complex environment. It adopts a hierarchical reinforcement learning architecture, combines the advantages of centralized control and distributed control, and centrally allocates coverage areas and avoids obstacle areas through the upper layer, while the lower layer performs distributed target identification and tracking. This enables multiple drones to accurately select flight paths based on the target position information and obstacle information of the radar module, and autonomously adjust their flight posture in a three-dimensional environment. Through the collaborative work of the upper and lower network layers, each drone can collaborate with multiple drones to avoid repeated searches or collisions, thereby significantly improving search efficiency and area coverage.

[0165] 2. Achieved accurate target recognition and tracking capabilities

[0166] By incorporating a hierarchical reinforcement learning framework, this paper designs an algorithmic structure that adapts to sparse rewards. This allows for the rapid generation of feasible target recognition and tracking postures, enabling each drone to develop an effective search strategy within a limited training time, even without sufficient feedback. This allows the drone to precisely adjust its flight posture to continuously track and identify the target after detection. Furthermore, the flight control strategy optimized using the reinforcement learning algorithm ensures that the drone remains aligned with the target in three-dimensional space, improving both the accuracy and efficiency of target recognition.

[0167] 3. Enhanced system robustness and adaptability

[0168] This invention utilizes a hierarchical reinforcement learning framework, combining centralized training with distributed execution. This approach enables drone swarms to maintain high mission execution stability and system robustness in the event of emergencies (such as drone damage or communication interruptions). Real-time feedback mechanisms and distributed communication ensure that even in complex and uncertain environments, such as changing conditions or reduced capabilities of individual drones, other drones can continue to execute their missions, ensuring mission completion. Furthermore, drones can continuously track suspicious dynamic targets, improving their accuracy in identifying search targets.

[0169] Based on the same inventive concept, the second aspect of the present invention also provides a multi-UAV unknown area target search device based on hierarchical reinforcement learning. Figure 6 , Figure 6 A structural block diagram of a multi-UAV unknown area target search device based on hierarchical reinforcement learning provided by an embodiment of the present invention, the device comprising:

[0170] An initialization module is used to divide the search area, which includes multiple targets and multiple obstacles, into several search grids, assign a search grid to each of the multiple drones, use the drones to search the search area, obtain search data, and transmit it to the ground station; each target occupies only one search grid, and each obstacle occupies at least one search grid;

[0171] A training module, provided at the ground station, includes a regional coverage coordination strategy network training unit and a target tracking and identification strategy network training unit, each used to train a regional coverage coordination strategy network as an upper-layer decision network and a target tracking and identification strategy network as a lower-layer decision network based on a hierarchical reinforcement learning framework. In the hierarchical reinforcement learning framework, the state space is defined as the set of search data, the action space is defined as the set of actions that the drone can perform, and the reward function is the sum of the reward value obtained after the drone successfully identifies a target and the penalty value imposed after the drone is damaged.

[0172] The search execution module is used to integrate the area coverage coordination strategy network and the target tracking and identification strategy network into each UAV to guide the UAV to perform target search tasks; among them, the area coverage coordination strategy network is used to select a flight path for each UAV; the target tracking and identification strategy network is used to adjust the UAV's flight attitude based on the flight path to continuously track and identify the target.

[0173] Based on the same inventive concept, the third aspect of the present invention further provides an electronic device, which includes a processor, a communication interface, a memory, and a communication bus; wherein the processor, the communication interface, and the memory communicate with each other via the communication bus.

[0174] Memory is used to store computer programs;

[0175] When the processor is used to execute the program stored in the memory, the method steps provided in the first aspect of the present invention are implemented.

[0176] Based on the same inventive concept, the fourth aspect of the present invention further proposes a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method steps provided in the first aspect of the present invention are implemented.

[0177] It should be noted that, for the device, electronic device and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0178] The device, electronic device and storage medium of the embodiments of the present invention are respectively the device, electronic device and storage medium for applying the above-mentioned multi-UAV unknown area target search method based on hierarchical reinforcement learning. All embodiments of the above-mentioned multi-UAV unknown area target search method based on hierarchical reinforcement learning are applicable to the device, electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0179] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A multi-UAV unknown area target search method based on hierarchical reinforcement learning, characterized in that: include: Step 1: Divide the search area, which includes multiple targets and multiple obstacles, into several search grids. Each of the multiple drones is assigned a search grid. The drones are used to search the search area, obtain search data, and transmit it to the ground station. Each target occupies only one search grid, and each obstacle occupies at least one search grid. Step 2: At the ground station, based on a centralized training, distributed execution, and hierarchical reinforcement learning framework, a regional coverage coordination strategy network is trained as the upper-level decision network; simultaneously, a target tracking and identification strategy network is trained as the lower-level decision network. In the hierarchical reinforcement learning framework, the state space is defined as the set of search data, the action space is defined as the set of actions that the drone can perform, and the reward function is defined as the sum of the reward value obtained after the drone successfully identifies a target and the penalty value imposed after the drone is damaged. Step 3: Integrate the area coverage coordination strategy network and the target tracking and identification strategy network into each UAV to guide the UAV to perform the target search task; wherein the area coverage coordination strategy network is used to select a flight path for each UAV; and the target tracking and identification strategy network is used to adjust the UAV's flight attitude based on the flight path to continuously track and identify the target.

2. The multi-UAV unknown area target search method based on hierarchical reinforcement learning according to claim 1 is characterized in that: In step 1, the search area including multiple targets and multiple obstacles is divided into several search grids, including: The search area including multiple targets and multiple obstacles is limited to a rectangular area with a length of L and a width of W; Based on the mission requirements and aircraft performance, the rectangular area is divided into m×n search grids of equal size. The side length s of each search grid is calculated as follows: In the formula, min(,) means taking the minimum value; Assign a confidence value P to each search grid (x,y) x,y , used to express the probability that the target exists in the search grid, and let P x,y = -1 means there are obstacles in the search grid (x, y).

3. The multi-UAV unknown area target search method based on hierarchical reinforcement learning according to claim 2 is characterized in that: In step 1, a search grid is assigned to each of the multiple drones. The drones are used to search the search area, obtain search data, and transmit it to the ground station, including: Multiple drones are ordered to depart from a ground station and fly along an initially preset path until each drone occupies a search grid; each drone is equipped with a radar module and a target recognition module; Each UAV uses its own radar module to search a square area with a range of f×f search grids centered on its own search grid, and exchanges information with neighboring UAVs during the search process to obtain radar cognition information; At the same time, each drone uses its own target recognition module to search a circular area with a certain radius centered on the drone to obtain target recognition information; The radar recognition information, the target identification information and the UAV's own state perception information are transmitted to the ground station as search data.

4. The multi-UAV unknown area target search method based on hierarchical reinforcement learning according to claim 3 is characterized in that: In step 2, a regional coverage coordination strategy network is trained as the upper-layer decision network, including: Extracting a target obstacle confidence map from the radar cognition information of each drone in the search data; combining the radar cognition information of each drone to extract an environment search state map and a drone position map; combining the environment search state map, the drone position map, and the target obstacle confidence map of each drone to form global information as distributed interactive training data; Reinforcement learning modeling of upper-layer network decision-making problems; Based on the distributed interactive training data, the QMIX algorithm is used to solve the upper-layer network decision problem, and a regional coverage coordination strategy network is trained to serve as the upper-layer decision network.

5. The multi-UAV unknown area target search method based on hierarchical reinforcement learning according to claim 4 is characterized in that: Reinforcement learning modeling of upper-layer network decision-making problems, including: The upper-layer network decision problem is expressed as: max g=[g1,g2]; In the formula, max means taking the maximum value; T is the number of steps required for the search task, is the global information of the probability that the suspicious target exists in the search grid (x, y); ε is a pre-set threshold; is the number of times the search grid (x, y) is visited by the drone; is the position of UAV i at time t, u i Indicates drone i, c z is the position occupied by obstacle z; d safe is the safe distance between the drone and obstacles, N z is the number of obstacles; The temporal abstraction method is used to extend the Markov decision process of the upper-layer network decision problem to a semi-Markov decision process, and a distributed partially observable Markov decision process is obtained, which can be expressed as: (S,{A i },T s ,{O i },T o ,R,N u ,γ); Where S is the state space, representing all possible states of the environment; A i is the action space, which represents the set of actions that drone i can take; T s is the state transition probability function, which means that when the environment is in state s t When the UAV takes joint action a t After that, the environment moves to the next state s t+1 The probability of i is the observation space, which represents the set of environmental information that UAV i can observe. The observation spaces of all UAVs jointly constitute O; T o is the observation probability function, which indicates that the drone takes a given action a i,t and the current environment state s t Observation o is obtained when i,t The probability of R is the reward function, which means that all drones have different rewards according to their current state s. t and make a joint action a t After that, the immediate reward is obtained; N u represents the total number of drones participating in the collaborative mission; γ is the discount factor of the cumulative reward, which is used to measure the importance of future rewards to the current decision.

6. The multi-UAV unknown area target search method based on hierarchical reinforcement learning according to claim 3 is characterized in that: In step 2, a target tracking and recognition strategy network is trained as the lower-level decision network, including: Reinforcement learning modeling of the underlying network decision-making problem; The target recognition information of each drone in the search data and the drone's own state perception information are used as training data, and a target tracking and recognition strategy network is iteratively trained using the SAC algorithm as the lower-level decision network.

7. The multi-UAV unknown area target search method based on hierarchical reinforcement learning according to claim 6 is characterized in that: Reinforcement learning modeling of the underlying network decision-making problem, including: For the state space S lower , modeled as follows: S lower ={(x t ,y t ,h t ),(θ,φ,ψ),σ}; In the formula, (x t ,y t ,h t ) represents the relative position of the suspicious target with the UAV as the origin; (θ, φ, ψ) represents the flight attitude of the UAV, which are the roll angle, pitch angle and yaw angle respectively; σ represents the illumination area of the target recognition module on the suspicious target; For the action space A lower , modeled as follows: A lower ={(u,v,w),(p,q,r)}; Where (u, v, w) is the linear velocity of the drone in the x-direction, y-direction, and height direction; (p, q, r) is the flight attitude of the drone, which are the roll angular velocity, pitch angular velocity, and yaw angular velocity respectively; For the reward function R lower ,It is defined as: if the target recognition module can continuously and stably aim at the suspicious target, a reward will be given; if the target is lost or the recognition is interrupted, a penalty will be given; Based on the above state space S lower , action space A lower and the reward function R lower , the decision process of the lower network is modeled as a Markov decision model, which is defined as: (S lower ,A lower ,P lower ,R lower ,c lower ); Where, P lower =P(s lower,t+1 |s lower,t ,a lower,t ) represents the state transition probability, which describes the state in the current state s lower,t Next, perform action a lower,t Then, transfer to the next state s lower,t+1 The possibility of reward function R lower For evaluation in state s lower,t Next, perform action a lower,t The advantages and disadvantages of γ lower is the attenuation factor.

8. A multi-UAV unknown area target search device based on hierarchical reinforcement learning, used to implement the method according to any one of claims 1 to 7, characterized in that: The device includes: An initialization module is used to divide the search area, which includes multiple targets and multiple obstacles, into several search grids, assign a search grid to each of the multiple drones, use the drones to search the search area, obtain search data, and transmit it to the ground station; each target occupies only one search grid, and each obstacle occupies at least one search grid; A training module, provided at a ground station, includes a regional coverage coordination strategy network training unit and a target tracking and identification strategy network training unit, each of which is used to train a regional coverage coordination strategy network as an upper-layer decision network based on centralized training, distributed execution, and a hierarchical reinforcement learning framework; and simultaneously train a target tracking and identification strategy network as a lower-layer decision network. In the hierarchical reinforcement learning framework, the state space is defined as a set of search data, the action space is defined as a set of actions that can be performed by the drone, and the reward function is defined as the sum of a reward value obtained after the drone successfully identifies a target and a penalty value imposed after the drone is damaged. The search execution module is used to integrate the area coverage coordination strategy network and the target tracking and identification strategy network into each UAV to guide the UAV to perform target search tasks; wherein the area coverage coordination strategy network is used to select a flight path for each UAV; and the target tracking and identification strategy network is used to adjust the UAV's flight attitude based on the flight path to continuously track and identify the target.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to execute a program stored in a memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 can be implemented.

Images