Task-driven unmanned aerial vehicle autonomous search method

Abstract

The invention relates to a task-driven unmanned aerial vehicle autonomous search method. The method comprises the following steps: acquiring an unmanned aerial vehicle speedometer by using a binocular RGB image and IMU data acquired by an unmanned aerial vehicle; obtaining three-dimensional point cloud data under a world coordinate system by using a depth map obtained by the unmanned aerial vehicle and an unmanned aerial vehicle speedometer, and establishing a probability voxel map through voxelization processing; performing target detection on the binocular RGB image by using a target detection network, and obtaining a target pose; establishing and updating a task scene map by using the target with the detection confidence exceeding the confidence upper limit and the probability voxel map; establishing a space boundary and a potential target boundary; establishing a search strategy of the unmanned aerial vehicle by using the space boundary and the potential target boundary, and determining a search viewpoint; and performing autonomous safety navigation of the unmanned aerial vehicle by using an unmanned aerial vehicle speedometer and a search viewpoint. According to the method, the problems of completely exploring a task space and simultaneously detecting a task target at high precision in a communication denial and unfamiliar and unknown scene are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous search of unmanned aerial vehicles, and in particular to a task-driven autonomous search method of unmanned aerial vehicles. Background Art

[0002] Drones are gradually being used in complex mission scenarios such as field rescue, tunnel inspections, and mine exploration. However, due to the lack of prior maps of the scene and the inability to use GPS positioning, it is extremely difficult to manually operate drones to complete tasks. With the rapid development of sensor manufacturing and artificial intelligence algorithms, using drones to autonomously complete search tasks has become a future development trend. Vision-based positioning and mapping technologies play an important role in autonomous drone search tasks, providing the required posture and scene map information for drone navigation; using target detection technology can search for targets of interest and achieve target detection and positioning.

[0003] In the UAV autonomous search mission, the UAV is not only required to quickly explore the unknown space, but also to accurately detect the mission target. In the published patents / literatures of autonomous exploration and autonomous search, the area between the known and unknown space is usually established as a common boundary, and the target of interest is established as a semantic boundary. Based on the boundary, greedy strategies or hierarchical planning strategies are used for exploration, focusing on exploration efficiency and coverage, without considering target detection accuracy, which can easily lead to misdetection or missed detection of mission targets, which limits the practical application of UAV autonomous search. Summary of the invention

[0004] The present invention provides a task-driven UAV autonomous search method to solve the difficult problem of fully exploring the task space and detecting the task target with high precision in a communication-denied, unfamiliar and unknown scenario.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a task-driven UAV autonomous search method, comprising the following steps:

[0006] S1, obtains the binocular RGB image, depth image and IMU data obtained by the drone during flight;

[0007] S2, using the binocular RGB image and IMU data to obtain the drone odometer;

[0008] S3, using the depth map and the drone odometer to obtain three-dimensional point cloud data in a world coordinate system, and establishing a probabilistic voxel map through voxelization processing;

[0009] S4, using a target detection network to perform target detection on the binocular RGB image, and calculating a target pixel coordinate set and a position and posture of the target in a world coordinate system for the detected target, so as to obtain the target position and posture;

[0010] S5, using the target whose detection confidence exceeds the upper confidence limit to fuse target information with the probabilistic voxel map, and updating the voxel occupancy probability through Bayesian probability estimation, to complete the establishment and update of the task scene map;

[0011] S6, clustering the regions between the explored free space and its neighboring unknown space in the task scene map using a region growing algorithm to establish a spatial boundary;

[0012] S7, taking the target whose detection confidence is between the lower confidence limit and the upper confidence limit as a potential target, and establishing the potential target boundary;

[0013] S8, establishing a search strategy for the drone using the space boundary and the potential target boundary, and determining a search viewpoint;

[0014] S9, using the drone odometer and the search viewpoint to perform autonomous and safe navigation of the drone.

[0015] The target pixel coordinate set calculation in step S4 specifically includes:

[0016] (1) Set the lower and upper confidence limits;

[0017] (2) obtaining a target detection bounding box whose detection confidence exceeds the lower limit of the confidence, and calculating a grayscale histogram of pixels within the target detection bounding box to obtain a pixel intensity distribution;

[0018] (3) calculating the center of gravity of the target in the target detection bounding box using a weighted average method based on the pixel intensity distribution;

[0019] (4) The center of gravity is set as a threshold for distinguishing target pixels and background pixels in the target detection bounding box, and pixel coordinates greater than the threshold are established as a target pixel coordinate set.

[0020] The calculation of the position and posture of the target in the world coordinate system in step S4 specifically includes:

[0021] (1) obtaining a depth map corresponding to the binocular RGB image and performing scale alignment;

[0022] (2) extracting a depth value set corresponding to the target pixel coordinate set on the aligned depth map, and removing noise by median filtering to obtain a target point cloud;

[0023] (3) obtaining the centroid of the point cloud by calculating the average value of the coordinates of all points in the target point cloud, and using the centroid of the point cloud as the target position in the camera coordinate system;

[0024] (4) Finding the neighborhood points for each point in the target point cloud, performing plane fitting based on the neighborhood points, obtaining the normal vector of the point cloud plane, and converting the normal vector of the point cloud plane from the camera coordinate system to the world coordinate system.

[0025] The target information fusion of the target whose detection confidence exceeds the upper confidence limit and the probabilistic voxel map in step S5 is specifically as follows: the target point cloud of the target whose detection confidence exceeds the upper confidence limit is mapped to the probabilistic voxel map, and the category and confidence information of the target are added to the corresponding voxel, and different voxel colors are used to represent targets of different categories; the position information of the target is stored in the probabilistic voxel map, and the position is marked with a number at the center of the target voxel, and the posture is marked with an arrow.

[0026] The step S5 of updating the voxel occupancy probability by Bayesian probability estimation is specifically as follows: using the Bayesian probability estimation method to fuse the detection confidence and the prior occupancy probability of the probability voxel map to obtain a posterior occupancy probability.

[0027] The search strategy of the drone in step S8 includes a global coverage planning part and a local viewpoint optimization part, wherein the global coverage planning part takes the center of each space boundary and the center of the potential target boundary as the path points to guide the exploration, and roughly plans a path covering each space boundary and the potential target boundary; the local viewpoint optimization part randomly samples the space boundaries within the optimization range in its sampling space, and uses the utility function to sort the sampling viewpoints of each space boundary and the sampling viewpoints with the highest utility value, establishes the retained sampling viewpoints as graph nodes, and constructs a graph search algorithm through a graph search method to determine the optimal local path, and each viewpoint in the local optimization path is a search viewpoint.

[0028] The global coverage planning part is specifically as follows: using the A* algorithm to calculate the traversable path from the current position of the drone to the position of the center of each spatial boundary and the position of the center of each potential target boundary, taking the time cost and flight continuity cost of reaching the center of each spatial boundary and the center of each potential target boundary as the search cost, converting the traversal problem of each spatial boundary and potential target boundary into an asymmetric traveling salesman problem, and solving it to obtain the traversal order of each spatial boundary and potential target boundary.

[0029] The utility function used in the local viewpoint optimization part is a function that is compatible with exploration coverage and target detection confidence, wherein the exploration coverage is represented by the ratio of the coverable volume to the boundary volume; and the target detection confidence is represented by the detection distance and the detection direction.

[0030] The technical solution adopted by the present invention to solve its technical problem is: to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned task-driven UAV autonomous search method when executing the computer program.

[0031] The technical solution adopted by the present invention to solve its technical problem is: providing a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned task-driven drone autonomous search method are implemented.

[0032] Beneficial Effects

[0033] Due to the adoption of the above technical scheme, the present invention has the following advantages and positive effects compared with the prior art: the present invention first uses a vision-based positioning and mapping algorithm to realize the positioning of the drone and the construction of a local scene map. The vision-based positioning method can solve the positioning problem under GPS denial conditions, expand the application scenarios of drones, especially the scenarios with terrain obstruction, building obstruction and base stations too far away; then, based on the probabilistic voxel map, lightweight target detection and efficient posture calculation process, a mission scene map is established, and the establishment of spatial boundaries and potential target boundaries is realized; and then the spatial boundaries and potential target boundaries are used to design the drone search strategy, including guiding the drone to completely cover the mission scene through global coverage planning, and determining the search destination that takes into account coverage, detection confidence and search efficiency through local viewpoint optimization; finally, the drone is searched safely through an autonomous navigation algorithm based on the drone odometer and the search destination. Repeat the above process until there is no search boundary within the preset range. The present invention is particularly suitable for mission scenarios such as communication denial, unfamiliar and unknown field rescue, tunnel inspection, mine exploration, etc., and meets the requirements of complete exploration of mission space and high-precision detection of mission targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a system block diagram of a task-driven autonomous search method for a UAV according to a first embodiment of the present invention;

[0035] Figure 2 is a flow chart of a task-driven autonomous search method for a UAV according to a first embodiment of the present invention;

[0036] Figure 3 is a flowchart of obtaining a target posture in the first embodiment of the present invention;

[0037] Figure 4 Schematic diagram of the drone search strategy in the first embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0039] The first embodiment of the present invention relates to a task-driven autonomous search method for a drone, which is based on Figure 1 The system architecture shown mainly includes sensor data acquisition and preprocessing part, odometer calculation part, mapping and search boundary establishment part, search strategy establishment part, autonomous navigation planning part and flight control part. The sensor data acquisition and preprocessing part mainly collects binocular RGB images, depth maps and IMU data, as well as performs denoising and downsampling processing; the odometer part uses binocular RGB images and IMU data to realize odometer calculation; the mapping and search boundary establishment part first obtains point cloud data with the odometer and preprocessed depth map as input and establishes a probabilistic voxel map through voxelization, and then establishes the area with known probability and unknown probability around it as the spatial boundary. At the same time, the target detection algorithm is used to detect the target of interest required by the task and calculate the pose, and the target that meets the detection confidence requirement is directly marked in the probabilistic voxel map, otherwise it is established as a potential target boundary and needs to be further processed. One-step fine detection; the search strategy establishment part includes global coverage planning and local viewpoint optimization. In the global coverage planning, it is considered to guide the drone to cover all boundaries completely, and a coarse-grained path is planned to obtain the traversal order of all boundaries; in the local viewpoint optimization, the coverage rate and detection confidence are taken into account to sample the viewpoints within the optimization range, and then the optimal local path is determined by the graph search algorithm with the goal of improving the search efficiency. The local path point is the search viewpoint; the autonomous navigation part uses the determined search viewpoint as the navigation destination and obtains the control instruction through the path search and trajectory optimization algorithm; the flight control part uses the control instruction as input and obtains the control signal of each rotor motor through the controller solution. During the flight of the drone, the sensor data, map, search boundary and search viewpoint are dynamically updated, and the above process is repeated until there is no search boundary within the preset range.

[0040] like Figure 2 As shown, the task-driven autonomous search method of the UAV in this embodiment includes:

[0041] S1: Use the Intel RealSense D435i camera to collect binocular RGB images, depth images, and IMU data, and transmit them to the drone host in real time for data preprocessing, including converting the binocular RGB images into grayscale images, filtering out depth and IMU noise, etc.

[0042] S2: Using the binocular RGB image and IMU data obtained by preprocessing in S1, this implementation method obtains the drone odometer through the VINS-Fusion algorithm. At the same time, in order to simplify the calculation, the origin of the default odometer coordinate system is consistent with the origin of the world coordinate system, so the drone odometer can be used to represent the drone's position in the world coordinate system.

[0043] S3: The depth map preprocessed in S1 is used to obtain the three-dimensional scene point cloud in the coordinate system of the drone camera through back projection calculation, and the three-dimensional point cloud in the world coordinate system is obtained through coordinate system pose transformation combined with the drone odometer in S2. To improve the quality of the point cloud, the radius filtering algorithm can be further used to remove the point cloud noise, and the downsampling algorithm can be used to reduce the number of point clouds to improve the real-time mapping. The space is divided into a series of voxel units through the voxelization process, and each voxel unit is used to represent a three-dimensional space block of a fixed size. The point cloud data is mapped to the voxel grid. For each voxel, the point cloud point is counted to see whether it falls within the voxel. The final occupancy state of the voxel is calculated through the point occupancy probability after multiple observations, including occupied, free, and unknown, thereby establishing a probabilistic voxel map.

[0044] S4: In order to reduce power consumption and computing power, this implementation uses a lightweight Yolo-based improved target detection network to perform target detection on the grayscale image preprocessed in S1, and obtains the target pose through efficient target pixel coordinate set calculation and target pose calculation in the world coordinate system.

[0045] like Figure 3 As shown, in the target pixel coordinate set calculation process, the confidence interval [conf l ,conf h ](To avoid missed detection, set the confidence lower limit conf l is a smaller value, which is set to 0.3 in this implementation); then, the detection confidence exceeds the confidence lower limit conf l The target detection bounding box is obtained by using OpenCV to calculate the pixel grayscale histogram of the target detection bounding box to obtain the pixel intensity distribution. In order to determine the target area relatively accurately, the weighted average method is used to calculate the center of gravity of the target in the target detection bounding box, that is, the main grayscale value C, according to the pixel intensity distribution obtained from the pixel grayscale histogram. The calculation method is:

[0046]

[0047] Where i is the grayscale value, ranging from 0 to 255; H(i) is the number of pixels corresponding to the grayscale value, and N is the number of grayscale levels, which is 256.

[0048] The main gray value C is set as the threshold to distinguish the target pixels and background pixels in the bounding box, and the pixel coordinates greater than the threshold are established as the target pixel coordinate set S, which is expressed as:

[0049] S = {(x,y)|I(x,y)>C};

[0050] Where I(x,y) is the pixel in the target detection bounding box that meets the threshold requirement, and (x,y) is the corresponding pixel coordinate.

[0051] like Figure 3 As shown in the figure, the target pose calculation process in the world coordinate system includes extracting the depth set corresponding to the target pixel and calculating the target point cloud, calculating the target position and orientation in the camera coordinate system, and calculating the target position and orientation in the world coordinate system, and finally obtaining the target pose in the world coordinate system. The details are as follows:

[0052] (1) obtaining a depth map corresponding to the binocular RGB image and performing scale alignment;

[0053] (2) Extract the depth value set corresponding to the target pixel coordinate set from the aligned depth map, remove the noise through median filtering, and then calculate the target point cloud. The point cloud position calculation process in the camera coordinate system is:

[0054] Assume that the number of target pixels after filtering is n, and the corresponding depth set is {z1, z2, z3, ..., z n}, with a certain depth z i For example, using the camera internal parameters including focal length f x and f y And the offset c of the camera optical axis in the image coordinate system x and c y The position P in the camera coordinate system is obtained by back-projection calculation i , its coordinate value (X i ,Y i ,Z i ) is:

[0055]

[0056] Using the above formula, the point cloud of the target in the camera coordinate system can be obtained as {P1, P2, P3, ..., P i , ..., P n};

[0057] (3) The point cloud center of gravity is obtained by calculating the average value of all point coordinates in the target point cloud and taking it as the target position in the camera coordinate system. c ,Y c ,Z c) is calculated as:

[0058]

[0059] Where m is the total number of point clouds. Further, according to the rigid body transformation between the camera and the drone And drone odometer The position P of the target in the world coordinate system can be calculated w , the expression is:

[0060]

[0061] The first matrix on the right side of the equation converts a four-dimensional vector into a three-dimensional position vector. The homogeneous transformation matrix representing the position and orientation of the drone in the world coordinate system, Represents the fixed transformation matrix between the drone body and the camera;

[0062] (4) Find the neighboring points for each point in the target point cloud, and perform plane fitting based on these points to obtain the normal vector of the point cloud plane, that is, the orientation n of the target in the camera coordinate system c =(n x ,n y ,n z ). To transform the target orientation from the camera coordinate system to the world coordinate system, first convert the orientation vector into a homogeneous direction vector V c =[n c ,0] T , and then according to the transformation matrix from the camera to the drone body And the transformation matrix from the drone body to the world coordinate system Calculate the homogeneous direction vector V of the target in the world coordinate system w , the expression is:

[0063]

[0064] According to the above formula, the direction of the target in the world coordinate system can be calculated as n w =(V w [0],V w [1],V w [2]).

[0065] S5: Using the probability voxel map in S3 and the detection confidence in S4 exceeding the confidence limit conf h The goal is to establish and update the task scene map, including target information fusion and occupancy probability update, that is, the probability voxel map in S3 and the detection confidence in S4 exceed the confidence limit conf h (To improve detection accuracy, set the confidence limit confh The target with a value of 0.9 is fused and the voxel occupancy probability is updated through Bayesian probability estimation. At the same time, as the sensor data on the drone changes, the mission scene map is updated using the latest probabilistic voxel map and target detection results.

[0066] In this step, during the process of target information fusion, the detection confidence exceeds the confidence upper limit conf h The target point cloud of the target is mapped to the probabilistic voxel map in S3, the target category and confidence information are added to the corresponding voxels, and different voxel colors are used to represent targets of different categories. At the same time, the target's pose information is stored in the voxel map, and the position is marked with a number and the pose is marked with an arrow at the center of the target voxel.

[0067] In this step, in the process of updating the voxel occupancy probability through Bayesian probability estimation, the detection confidence and the prior occupancy probability of the voxel map are fused using the Bayesian probability estimation method to obtain a new and more accurate posterior occupancy probability. The expression of the updated occupancy probability P(Occ|Det) is:

[0068]

[0069] Among them, P(Det|Occ) is the probability of detecting the target under the premise that the voxel is occupied, which is given by the target detection confidence; P(Occ) is the prior probability that the voxel is in an occupied state, that is, the initial occupancy probability of the voxel before the target is detected; P(Det) is the total probability of the detection result, which indicates the probability of detecting the target in all cases.

[0070] S6: Use the region growing algorithm to cluster the area (non-target area) between the explored free space and its neighboring unknown space in the mission scene map to establish the space boundary. At the same time, when the mission scene map is updated with the latest sensor data in S5, the space boundary also needs to be updated, and the space boundary with changed occupancy status (unknown space is determined to be free or occupied) is removed, and new space boundaries are searched at the same time. For large-scale boundary clustering, the principal component analysis algorithm is used to divide it into two uniform boundaries to ensure that the drone can fully cover the space boundary during the search process.

[0071] S7: Set the detection confidence level in [conf l ,conf h] are regarded as potential targets and require further fine detection, so they are established as potential target boundaries. Boundary information includes clustering of target point clouds after voxelization (the clustering process is similar to the process used in S6, but only the target voxels are clustered), target pose, target category and confidence, and the confidence of the potential target boundary is updated during the autonomous search process of the drone. Targets that meet the detection confidence requirements (the average confidence value of multiple detections is ≥0.8) during the search process are moved out of the potential target boundary and merged into the mission scene map in S5; targets that do not meet the confidence requirements are removed from the potential target boundary set, and the related point clouds are ordinary occupied voxels. At the same time, the objects newly detected by the onboard camera continue to be processed by S4 to supplement the potential target boundary set.

[0072] S8: Using the space boundary in S6 and the potential target boundary in S7 to establish a drone search strategy, the drone search strategy in this implementation mainly includes a global coverage planning part and a local viewpoint optimization part, such as Figure 4 shown.

[0073] In the global coverage planning part, the global path traversing each boundary is planned to guide the UAV to search unknown areas and avoid the UAV from falling into the fine detection of potential targets. In this process, in order to efficiently respond to scene changes, the global coverage path is coarse-grained planned to guide the UAV to the area near the boundary. The traversal coverage problem of each boundary is transformed into an asymmetric traveling salesman problem, and the flight time and flight continuity of the UAV's current posture to reach each boundary posture are established as the search cost. The search order of each boundary is obtained by solving the asymmetric traveling salesman problem.

[0074] In this implementation, the global coverage planning part uses the center of each boundary as a path point to guide exploration, and roughly plans a path covering each space boundary and potential target boundary (see Figure 4 ). Different boundaries are not treated differently in the global coverage path planning, which is conducive to efficiently guiding the drone to cover all boundaries. Specifically, the A* algorithm is used to calculate the position p0 from the current position of the drone to the center of each boundary (including the center of the space boundary and the center of the potential target boundary). k The traversable path between the two boundaries takes the time cost to reach the center of each boundary and the flight continuity cost as its search cost, which is expressed as:

[0075] M tsp (0,k)=t(x0,x k )+w c ·c c (x k,1 );

[0076] where t(x0,x k) is the time cost between boundary x0 and boundary xk, which is determined by respectively calculating the ratio of the path length between the two boundaries to the maximum linear velocity, and the ratio of the rotation angle to the maximum angular velocity, and taking the maximum value; k is the boundary number, satisfying k∈{1,2,L,N cls}; The expression of flight continuity cost cc is:

[0077]

[0078] Then, the traversal problem of each boundary is transformed into an asymmetric traveling salesman problem and solved using the Lin-Hernighanheuristic algorithm to obtain the traversal order of each boundary.

[0079] In local viewpoint optimization, the spatial boundaries within the optimization range are randomly sampled in their sampling space. The sampled viewpoints of each boundary are sorted using a utility function that takes into account both coverage and detection confidence, and the top three sampled viewpoints are retained. For the potential target boundary, sampling is performed in half of the sampling space area to achieve fine detection, and the sampled viewpoints are mirrored relative to the normal vector of the potential target boundary, thereby increasing the detection viewpoints to guide the drone to perform fine detection. The retained sampling viewpoints are established as graph nodes, and each graph node is connected to the graph node of the next boundary through a directed edge (the target detection viewpoint is directly connected to its mirror point) to form a directed acyclic graph. The cost between the graph nodes is the same as the search cost in the global coverage planning, and the Dijkstra algorithm is used to search for the local optimization path with the minimum cost. Each viewpoint in the local optimization path is the actual search destination (i.e., the search viewpoint) in the autonomous navigation of the drone.

[0080] The utility function used in the local viewpoint optimization part is compatible with the exploration coverage and target detection confidence. At the boundary k, viewpoint j k The utility function expression is:

[0081] U(k,j k )=λ1I(k,j k )+λ2D(k,j k );

[0082] Where λ1 and λ2 are the exploration space coverage I(k,j k ) and target detection confidence D(k,j k ) is the weight factor. I(k,j k ) through the coverable volume V cover (k,j k ) and the boundary volume V frontier (k) is expressed as:

[0083]

[0084] Among them, V cover (k,j k ) is calculated as follows: In the sampling direction within the camera field of view, from viewpoint j k Extend to the maximum sensing depth of the sensor to form a truncated ray, and count the number of unknown voxels intersecting with the truncated ray to calculate V cover (k,j k ); V can be calculated by counting the number of unknown voxels in the process of boundary establishment frontier (k).

[0085] D(k,j k ) is related to the detection distance and detection direction. The farther the camera is from the target, the weaker the target features that can be captured, and the lower the detection confidence. When the target faces the camera, the features are most obvious and the confidence is the highest. As the target direction deviates from the camera, the confidence will continue to decrease. The joint model of the impact of detection distance and detection direction on confidence can be established as:

[0086] D(k,j k )=D0·e -αd(jk) ·cos(θ jk );

[0087] Where D0 is the initial confidence of the target at close range; α is the coefficient that controls the speed at which the confidence decreases; d is the viewpoint j k The Euclidean distance between the detected target and the jk To detect the target and viewpoint j k The angle between the camera optical axes can be calculated based on the orientation of the target in the camera coordinate system.

[0088] S9: Using the drone odometer in S2 and the search viewpoint determined in S8, the drone performs autonomous and safe navigation, which mainly includes path search and trajectory optimization. * The path search algorithm plans a safe and collision-free path from the current position of the UAV to the search viewpoint; then, considering the flight smoothness and motion constraints, the trajectory optimization algorithm based on B-spline curves is used to smoothly optimize the planned path.

[0089] It is not difficult to find that the present invention first uses a vision-based positioning and mapping algorithm to realize the positioning of the drone and the construction of a local scene map. The vision-based positioning method can solve the positioning problem under GPS denial conditions, expand the application scenarios of drones, especially the scenarios where there are terrain obstructions, building obstructions, and base stations are too far away; then, based on the probabilistic voxel map, lightweight target detection, and efficient posture calculation process, a mission scene map is established, and the establishment of spatial boundaries and potential target boundaries is realized; and then the spatial boundaries and potential target boundaries are used to design the drone search strategy, including guiding the drone to completely cover the mission scene through global coverage planning, and determining the search destination that takes into account coverage, detection confidence, and search efficiency through local viewpoint optimization; finally, the drone's safe search is realized through an autonomous navigation algorithm based on the drone's odometer and search destination, and the above process is repeated until there is no search boundary within a preset range. The present invention is particularly suitable for mission scenarios such as communication denial, unfamiliar and unknown field rescue, tunnel inspection, and mine exploration, and meets the requirements of complete exploration of mission space and high-precision detection of mission objectives.

[0090] A second embodiment of the present invention relates to an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the task-driven drone autonomous search method of the first embodiment when executing the computer program.

[0091] A third embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the task-driven autonomous search method for unmanned aerial vehicles of the first embodiment are implemented.

[0092] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0093] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction method, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0096] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A task-driven autonomous search method for unmanned aerial vehicles, characterized in that: The following steps are involved: S1, obtains the binocular RGB image, depth image and IMU data obtained by the drone during flight; S2, using the binocular RGB image and IMU data to obtain the drone odometer; S3, using the depth map and the drone odometer to obtain three-dimensional point cloud data in a world coordinate system, and establishing a probabilistic voxel map through voxelization processing; S4, using a target detection network to perform target detection on the binocular RGB image, and calculating a target pixel coordinate set and a position and posture of the target in a world coordinate system for the detected target, so as to obtain the target position and posture; S5, using the target whose detection confidence exceeds the upper confidence limit to fuse target information with the probabilistic voxel map, and updating the voxel occupancy probability through Bayesian probability estimation, to complete the establishment and update of the task scene map; S6, clustering the regions between the explored free space and its neighboring unknown space in the task scene map using a region growing algorithm to establish a spatial boundary; S7, taking the target whose detection confidence is between the lower confidence limit and the upper confidence limit as a potential target, and establishing the potential target boundary; S8, establishing a search strategy for the drone using the space boundary and the potential target boundary, and determining a search viewpoint; S9, using the drone odometer and the search viewpoint to perform autonomous and safe navigation of the drone.

2. The task-driven autonomous search method for unmanned aerial vehicles according to claim 1, characterized in that: The target pixel coordinate set calculation in step S4 specifically includes: (1) Set the lower and upper confidence limits; (2) obtaining a target detection bounding box whose detection confidence exceeds the lower limit of the confidence, and calculating a grayscale histogram of pixels within the target detection bounding box to obtain a pixel intensity distribution; (3) calculating the center of gravity of the target in the target detection bounding box using a weighted average method based on the pixel intensity distribution; (4) The center of gravity is set as a threshold for distinguishing target pixels and background pixels in the target detection bounding box, and pixel coordinates greater than the threshold are established as a target pixel coordinate set.

3. The task-driven autonomous search method for unmanned aerial vehicles according to claim 1, characterized in that: The calculation of the position and posture of the target in the world coordinate system in step S4 specifically includes: (1) obtaining a depth map corresponding to the binocular RGB image and performing scale alignment; (2) extracting a depth value set corresponding to the target pixel coordinate set on the aligned depth map, and removing noise by median filtering to obtain a target point cloud; (3) obtaining the centroid of the point cloud by calculating the average value of the coordinates of all points in the target point cloud, and using the centroid of the point cloud as the target position in the camera coordinate system; (4) Finding the neighborhood points for each point in the target point cloud, performing plane fitting based on the neighborhood points, obtaining the normal vector of the point cloud plane, and converting the normal vector of the point cloud plane from the camera coordinate system to the world coordinate system.

4. The task-driven autonomous search method for unmanned aerial vehicles according to claim 1, characterized in that: The target information fusion of the target whose detection confidence exceeds the upper confidence limit and the probabilistic voxel map in step S5 is specifically as follows: the target point cloud of the target whose detection confidence exceeds the upper confidence limit is mapped to the probabilistic voxel map, and the category and confidence information of the target are added to the corresponding voxel, and different voxel colors are used to represent targets of different categories; the position information of the target is stored in the probabilistic voxel map, and the position is marked with a number at the center of the target voxel, and the posture is marked with an arrow.

5. The task-driven autonomous search method for unmanned aerial vehicles according to claim 1, characterized in that: The step S5 of updating the voxel occupancy probability by Bayesian probability estimation is specifically as follows: using the Bayesian probability estimation method to fuse the detection confidence and the prior occupancy probability of the probability voxel map to obtain a posterior occupancy probability.

6. The task-driven autonomous search method for unmanned aerial vehicles according to claim 1, characterized in that: The search strategy of the drone in step S8 includes a global coverage planning part and a local viewpoint optimization part, wherein the global coverage planning part takes the center of each space boundary and the center of the potential target boundary as the path points to guide the exploration, and roughly plans a path covering each space boundary and the potential target boundary; the local viewpoint optimization part randomly samples the space boundaries within the optimization range in its sampling space, and uses the utility function to sort the sampling viewpoints of each space boundary and the sampling viewpoints with the highest utility value, establishes the retained sampling viewpoints as graph nodes, and constructs a graph search algorithm through a graph search method to determine the optimal local path, and each viewpoint in the local optimization path is a search viewpoint.

7. The task-driven autonomous search method for unmanned aerial vehicles according to claim 6, characterized in that: The global coverage planning part is specifically as follows: using the A* algorithm to calculate the traversable path from the current position of the drone to the position of the center of each spatial boundary and the position of the center of each potential target boundary, taking the time cost and flight continuity cost of reaching the center of each spatial boundary and the center of each potential target boundary as the search cost, converting the traversal problem of each spatial boundary and potential target boundary into an asymmetric traveling salesman problem, and solving it to obtain the traversal order of each spatial boundary and potential target boundary.

8. The task-driven autonomous search method for unmanned aerial vehicles according to claim 6, characterized in that: The utility function used in the local viewpoint optimization part is a function that is compatible with exploration coverage and target detection confidence, wherein the exploration coverage is represented by the ratio of the coverable volume to the boundary volume; and the target detection confidence is represented by the detection distance and the detection direction.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the task-driven autonomous search method for unmanned aerial vehicles as described in any one of claims 1-8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the task-driven autonomous search method for unmanned aerial vehicles as described in any one of claims 1 to 8 are implemented.

Images