Unmanned aerial vehicle autonomous exploration and multi-modal target detection method and system in unknown environment

Abstract

The invention provides an unmanned aerial vehicle autonomous exploration and multi-mode target detection method and system in an unknown environment, and the method comprises the steps: S1, collecting and fusing the information of an unmanned aerial vehicle sensor, and obtaining the position information and attitude information of an unmanned aerial vehicle; s2, based on the fusion result of the unmanned aerial vehicle sensor information, establishing a map and extracting map boundary increment information, and then clustering to generate a boundary cluster; and S3, based on the boundary cluster, calculating to obtain an optimal exploration sequence and a global path, and further generating a track and navigating. According to the invention, full coverage and efficient autonomous exploration of the unmanned aerial vehicle in an unknown environment and target detection capability of an unstructured ground scene under different illumination conditions are realized.

Description

Technical Field

[0001] This invention belongs to the field of autonomous exploration and target detection technology for unmanned aerial vehicles (UAVs), specifically, it relates to a method and system for autonomous exploration and multimodal target detection of UAVs in unknown environments. Background Technology

[0002] In recent years, unmanned aerial vehicles (UAVs) have demonstrated enormous application potential in fields such as battlefield reconnaissance, disaster relief, and environmental monitoring due to their mobility, flexibility, and low-cost deployment. In complex and unknown environments, UAVs autonomously exploring and completing target detection and search tasks has become a development trend. However, in GNSS-denied scenarios, such as indoors, underground, areas with electromagnetic interference, or environments with dense obstruction, and under extreme sensing conditions, such as drastic changes in lighting or partial or complete target obstruction, UAVs face difficulties and challenges such as low positioning accuracy, poor detection performance, and low exploration efficiency.

[0003] With the development of simultaneous localization and mapping (SLAM) technologies, UAVs can make autonomous navigation decisions and achieve autonomous flight exploration based on SLAM-based maps and real-time positioning information. Simultaneously, combining intelligent target detection technology allows for target search within a wide aerial field of view, enabling target detection and localization. However, existing autonomous UAV exploration technologies often suffer from greedy strategies that easily get trapped in local optima, leading to path redundancy and low exploration efficiency. Furthermore, single-modal detection technologies, such as visible light detection, experience reduced perception capabilities and target detection accuracy under drastic lighting changes or target occlusion. These limitations restrict the application of autonomous UAV exploration and target detection.

[0004] In summary, the problems with existing technologies are as follows: First, under GNSS denial conditions, UAVs cannot acquire their own pose. Second, existing exploration algorithms based on greedy strategies are prone to getting trapped in local optima, leading to path redundancy and low exploration efficiency. Third, traditional single-modal detection technologies, such as visible light detection, experience a significant decrease in perception capability under drastic lighting changes or target occlusion.

[0005] Patent document CN119832176A discloses an autonomous exploration method for unmanned systems based on multi-view vision scene understanding. This scheme achieves pixel-level semantic segmentation through multimodal image fusion, lightweight target detection, and depth-threshold-based background filtering, accurately estimating the pose of the target of interest. It constructs exploration boundaries using voxel maps and target detection results, and based on this, implements exploration decisions that consider both exploration space and scene understanding, guiding the unmanned system to perform autonomous exploration tasks. However, this scheme cannot improve the hierarchy, efficiency, and trajectory quality of path planning, and it cannot achieve deep fusion and enhancement of multimodal image features, thus failing to substantially enhance the perception and computing capabilities of the unmanned aerial vehicle (UAV).

[0006] This problem urgently needs to be solved. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for autonomous exploration and multimodal target detection by unmanned aerial vehicles (UAVs) in unknown environments.

[0008] The present invention provides a method for autonomous exploration and multimodal target detection of unmanned aerial vehicles (UAVs) in unknown environments, comprising: step S1: collecting and fusing UAV sensor information to obtain the UAV's position and attitude information; Step S2: Based on the fusion results of UAV sensor information, build a map and extract map boundary incremental information, and then cluster to generate boundary clusters; Step S3: Based on the boundary cluster, calculate the optimal exploration order and global path, and then generate a trajectory and navigate.

[0009] Preferably, the method further includes step S4: during navigation, visible light images and infrared images are acquired through the visible light sensor and infrared sensor of the UAV, and then fused to obtain and detect the dual-light image to generate the target point position.

[0010] Preferably, in step S1, the sensor information includes: RGB image and IMU data; Step S1 includes: Step S1.1: Collect sensor information using the UAV's binocular depth camera and IMU sensor; Step S1.2: Based on the RGB image and IMU data of the sensor information, construct a visual inertial odometry (VIO); then obtain the position and attitude information of the UAV through the visual inertial odometry (VIO).

[0011] Preferably, in step S2, an occupied grid map is established based on the location information, i.e., three-dimensional point cloud data in the world coordinate system; it is determined whether the boundary clusters of the occupied grid map are outdated. If the result is yes, they are deleted, and a new boundary cluster is obtained by searching for new boundary clusters through a region growing algorithm; if the result is no, no processing is performed. Boundary viewpoints are generated based on the new boundary clusters.

[0012] Preferably, step S3 includes: Step S3.1: Set up the cost matrix to obtain the asymmetric traveling salesman problem; Step S3.2: Solve the asymmetric traveling salesman problem using the LKH algorithm to obtain the optimal exploration order; Step S3.3: Based on the optimal exploration order, generate a global path using the A* algorithm; Preferably, in step S3.1, the asymmetric traveling salesman problem is to find an open-loop path that passes through all boundary cluster viewpoints and has the minimum total cost, starting from the current viewpoint. The mathematical expression for the cost matrix is:

[0013]

[0014] in, express The connection cost, Indicates the first The boundary cluster and the first A boundary cluster; Representing the cost matrix, i.e., viewpoint and The lower limit of the time; This indicates the number of clusters obtained after clustering the frontier points; The current viewpoint of the cost matrix and Relationship, starting from the current viewpoint, the first A boundary cluster is evaluated by the following mathematical expression:

[0015]

[0016] in, Indicates the current viewpoint, that is with viewpoint The lower limit of the time interval; Indicates the consistency cost weight; Indicate viewpoint The consistency cost; the symbol · represents scalar multiplication; The mathematical expression for the consistency cost is:

[0017] in, express The cost of consistency Indicates the first The first cluster The three-dimensional position of each viewpoint; Indicates the current location of the drone; Indicates the current speed of the drone; This represents the L2 norm.

[0018] Preferably, step S3 includes: Step S3.1: Based on the global path, select a continuous boundary cluster whose distance from the current viewpoint is less than a preset threshold, and then establish a directed acyclic graph; Step S3.2: The optimal exploration order and global path are obtained in the directed acyclic graph using Dijkstra's algorithm, and then a trajectory is generated and navigation is performed.

[0019] Preferably, in step S4, fusing the visible light image and the infrared image includes: Step S4.1: Convolve and normalize the visible light image and the infrared image to obtain a dual-light image as a fused image; Step S4.2: Perform a pooling operation on the fused image and output the semantic segmentation result of the fused image; Step S4.3: Based on the semantic segmentation results, obtain the reflection component using the Gaussian central function. R Then, interpolation is performed layer by layer to obtain a fused image with reduced illumination; The reflection component is obtained based on the semantic segmentation result and through the Gaussian central function. R The mathematical expression is:

[0020] Where x and y represent the pixel coordinates in the image. The estimated reflection component, i.e., the reflection component R , The grayscale values ​​of the original image. The standard deviation is The Gaussian central function, i.e., the two-dimensional Gaussian function, This is a convolution operation; The mathematical expression is:

[0021] in, Let be the standard deviation of the Gaussian kernel, · denotes multiplication, and exp is the exponential function.

[0022] A system for autonomous exploration and multimodal target detection of unmanned aerial vehicles in unknown environments, provided by the present invention, includes: Module M1: Collects and fuses UAV sensor information to obtain the UAV's position and attitude information; Module M2: Based on the fusion results of UAV sensor information, a map is built and incremental information of map boundaries is extracted, and then clustered to generate boundary clusters; Module M3: Based on the boundary cluster, calculates the optimal exploration order and global path, and then generates a trajectory and provides navigation.

[0023] Preferably, it also includes module M4: during navigation, visible light images and infrared images are acquired through the visible light sensor and infrared sensor of the UAV, and then fused to obtain and detect the dual-light image to generate the target point position.

[0024] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention uses visual inertial odometry technology to provide continuous pose estimation when GNSS fails by tightly coupling visual and inertial data. Combined with semantic feature constraints and dynamic compensation, it significantly suppresses positioning drift and provides the UAV with its own position and attitude and map information of unknown scenes for autonomous navigation decision-making. In other words, this invention realizes the positioning of UAVs, solves the positioning problem under GNSS denial conditions, and expands the application scenarios of UAVs. Specifically, this invention is applicable to communication denial, unfamiliar and unknown field rescue, tunnel inspection, and mine exploration tasks.

[0025] 2. This invention proposes a hierarchical autonomous planning strategy, utilizes the occupied grid map to extract rich boundary incremental information, performs local viewpoint optimization based on graph search methods, and uses Dijkstra's algorithm to generate the optimal local path, thereby improving the efficiency of autonomous exploration.

[0026] 3. This invention obtains more comprehensive scene information by multimodal fusion of visible light and infrared, which includes important infrared targets and retains rich visible light details, thereby improving the target detection capability of unstructured ground scenes under different lighting conditions and enhancing environmental adaptability.

[0027] 4. This invention uses a dual-light image fusion target detection algorithm. It achieves feature-enhanced dual-light image fusion through a convolutional fusion network, a semantic segmentation network, and a visual enhancement module. It uses YOLO v8 target detection to perform target detection on the fused image, which significantly improves the target detection capability of unstructured ground scenes under different lighting conditions. It solves the problem of weak image information from a single light source in the current UAV target detection process and enhances the reliable and efficient perception computing capability of UAVs. Attached Figure Description

[0028] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a system block diagram of the standard detection method provided by the present invention; Figure 2 A flowchart of a visual inertial odometry system based on binocular RGB images and IMU data provided for this invention; Figure 3 This is a schematic diagram illustrating the principle of the dual-light image fusion target detection algorithm provided by the present invention; Figure 4This is a schematic diagram illustrating the target detection simulation effect provided by the present invention. Detailed Implementation

[0029] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0030] This invention employs a hierarchical exploration planning and local viewpoint optimization strategy. It first solves the global path, then introduces graph search and nonlinear optimization within a local scope to locally adjust and smooth the path. This improves the hierarchy, efficiency, and trajectory quality of path planning, avoiding problems such as unreasonable paths and sudden changes in flight direction, effectively enhancing the UAV's exploration capabilities. The invention also utilizes a dual-light image fusion target detection algorithm. Through a convolutional fusion network, a semantic segmentation network, and a visual enhancement module, it achieves deep fusion and enhancement of multimodal image features. Based on this, a high-performance YOLO v8 target detection algorithm is used to detect the fused image, effectively improving the system's robustness to target detection under complex lighting conditions and unstructured ground scenes. This overcomes the instability caused by weak image information from a single light source in existing technologies, significantly enhancing the UAV's perception and computing capabilities.

[0031] A method for autonomous exploration and multimodal target detection by unmanned aerial vehicles in unknown environments, provided by the present invention, includes: S1: Employing a fusion of binocular depth camera and inertial measurement unit (IMU), it perceives external environmental information in GNSS-denied environments, constructs visual odometry based on binocular RGB images and IMU data, and obtains the position and attitude of the UAV; In other words, data from multiple sensors, such as binocular depth cameras and IMUs, is collected, and real-time fusion of information from each sensor is achieved through filtering, time synchronization, and spatial alignment techniques.

[0032] S2: Use depth map and visual inertial odometry information to obtain 3D point cloud data in the world coordinate system, build an occupied grid map, use the occupied grid map to extract rich boundary incremental information, perform incremental boundary detection and clustering, and perform viewpoint generation and cost update based on the boundary clusters generated by clustering. S3: The LKH algorithm is used to solve the exploration problem of the boundary viewpoints to obtain the optimal exploration order, and the global path is generated by the A* algorithm; S4: Local viewpoint optimization is performed through graph search methods. Dijkstra's algorithm is used to generate the optimal local path, and nonlinear optimization is employed to generate a smooth, collision-free, and shortest-time B-spline optimized trajectory to complete autonomous navigation. S5: Target detection is performed using a dual-light fusion target detection algorithm by fusing visible light and infrared images, and the position of the detected target is estimated.

[0033] Fusion of visible light and infrared images, including: The visible light image and the infrared image are convolved and normalized to obtain a two-light image as a fused image; The pooling operation is performed on the fused image to output the semantic segmentation result of the fused image; Based on the semantic segmentation results, the reflection component is obtained through the Gaussian central function. R Then, interpolation is performed layer by layer to obtain a fused image with reduced illumination; The reflection component is obtained based on the semantic segmentation result and through the Gaussian central function. R The mathematical expression is:

[0034] Where x and y represent the pixel coordinates in the image. The estimated reflection component, i.e., the reflection component R , The grayscale values ​​of the original image. The standard deviation is The Gaussian central function, i.e., the two-dimensional Gaussian function, This is a convolution operation; The mathematical expression is:

[0035] in, Let be the standard deviation of the Gaussian kernel, · denotes multiplication, and exp is the exponential function.

[0036] Specifically, the positioning module employs a simultaneous localization and mapping algorithm, fusing data from the inertial measurement unit (IMU) and binocular depth camera sensors to achieve its own pose estimation. The visual inertial odometry (VIO) functional module can include five parts: front-end data preprocessing, initialization, back-end nonlinear optimization, loop closure detection, and loop closure optimization. Upon system startup, the first few frames are initialized to obtain high-quality map points for pose estimation using the PnP method. For each image frame read, front-end data preprocessing is performed, including feature point identification and matching. Based on the matched feature points, the PnP method is used to solve for the relative pose. The back-end uses visual reprojection to continuously optimize the keyframes and map points selected from the front-end, making the results more accurate. Loop closure detection checks for loops to reduce global errors. The positioning module, through a tightly coupled fusion strategy, jointly optimizes the high-speed dynamic information provided by the IMU and the image features acquired by the visual sensor, enabling continuous and high-precision pose estimation of the UAV even in GNSS-denied environments, ensuring the system's autonomous navigation capability.

[0037] Specifically, the depth images from the depth camera are denoised and converted into point clouds. The current pose of the UAV is obtained based on visual inertial odometry information, and then 3D point cloud data in the world coordinate system is obtained. The point cloud data is voxelized and filtered to establish an occupied grid map. Each time the map is updated by sensor measurement, the updated area is recorded. Outdated boundary clusters are deleted within this area, and new boundary clusters are searched. After removal, a region growing algorithm is used to search for new boundaries and cluster them into groups. Among these groups, groups with a small number of noise sensor observations are usually ignored. However, the remaining groups may contain large clusters, which are not conducive to distinguishing unique unknown areas and making complex decisions. Therefore, if the maximum eigenvalue exceeds a threshold, the present invention performs PCA principal component analysis on each cluster and divides it into two uniform clusters along the first principal axis. The segmentation is recursive, so all large clusters are divided into smaller clusters. Finally, viewpoint generation and cost updates are performed based on the boundary clusters generated by clustering.

[0038] Specifically, the LKH algorithm is used to solve the exploration problem of the boundary viewpoints to obtain the optimal exploration order, and the global path is generated by the A* algorithm.

[0039] The exploration plan of this invention begins with finding a global path that can effectively cover the existing boundary clusters. This invention formulates this as a variant of the Traveling Salesman Problem (TSP), which computes an open-loop path starting from the current viewpoint and passing through the viewpoints of all clusters. By rationally designing the participation cost matrix Mtsp, this invention simplifies this variant to the standard Asymmetric Traveling Salesman Problem (ATSP), which can be solved quickly using the LKH algorithm.

[0040] Specifically, although global path planning finds an efficient order that can access all clusters, it only involves one viewpoint per cluster, which is not necessarily the optimal combination of all viewpoints. Local viewpoint optimization is performed using a graph search method to provide a richer set of viewpoints at the truncated portions of the global path, further improving the exploration rate. Then, Dijkstra's algorithm is used to search for the optimal local path. Subsequently, a trajectory optimization algorithm based on a nonlinear least squares optimization framework, namely Ceres Solver, is introduced to smooth the discrete path generated by Dijkstra's algorithm, generating a smooth, safe, and dynamically feasible B-spline trajectory. This ensures that the generated flight path effectively avoids obstacles and meets dynamic feasibility requirements, improving flight efficiency. The flight control module, based on the flight path output by the path planning module and combined with the UAV dynamics model, adjusts the UAV's attitude and speed in real time through a trajectory tracking controller and a gain-scheduled PID controller to achieve stable and efficient autonomous flight.

[0041] Specifically, in step S3.1, the asymmetric traveling salesman problem involves finding an open-loop path that minimizes the total cost, starting from the current viewpoint and passing through all boundary cluster viewpoints. This path sequentially visits at least one representative viewpoint in each boundary cluster in the environment to maximize the exploration coverage of the unknown environment. This problem constructs nodes from the UAV's current viewpoint and all candidate viewpoints in the boundary clusters to be explored, and builds a path cost matrix Mtsp. The matrix elements represent the flight costs between different viewpoints, including a lower bound on path time and motion consistency costs.

[0042] Specifically, the target detection module utilizes visible light and infrared sensors mounted on the UAV, employing a dual-light image fusion target detection algorithm to improve target detection accuracy under different lighting conditions. Infrared images, based on thermal radiation, have advantages in representing salient targets, but suffer from poor texture and low resolution. In contrast, visible light images offer high resolution and rich texture details, but are easily affected by environmental factors; for example, their target depiction capability significantly decreases in low-light or hazy conditions. The module achieves feature-enhanced dual-light image fusion through a designed convolutional fusion network, semantic segmentation network, and visual enhancement module. YOLO v8 target detection is then used to perform target detection on the fused image, significantly improving target detection capabilities in unstructured ground scenes under different lighting conditions. The module calculates the target's position from pixel coordinates to the world coordinate system, thus addressing the current issues of weak image information from a single light source in UAV target detection and enhancing the UAV's reliable and efficient perception and computing capabilities.

[0043] Example 1: This embodiment provides a method for autonomous exploration and multimodal target detection by unmanned aerial vehicles (UAVs) in unknown environments. The system block diagram of this method is as follows: Figure 1As shown, it mainly includes a multi-sensor fusion module, a localization module, a path planning module, a trajectory planning module, a flight control module, and a target detection and pose estimation module. The steps are as follows: Step 1: After acquiring data from multiple sensors such as the binocular depth camera Intel Realsense D435i and IMU, noise suppression and outlier removal are performed on the IMU measurements and camera images through filtering. The measurement times of each sensor are aligned using time synchronization technology, and the coordinate system is unified through spatial alignment method to achieve data fusion at the measurement level of each sensor.

[0044] The sensor information includes: RGB images and IMU data; Step 2: Using a fusion method of a binocular depth camera D435i and an inertial measurement unit (IMU), external environmental information is perceived in GNSS-denied environments. A visual inertial odometry (VIO) is constructed based on the binocular RGB images and IMU data, such as... Figure 2 As shown, the position and attitude of the drone are obtained; The Visual Inertial Odometry (VIO) system comprises five functional modules: front-end data preprocessing, initialization, back-end nonlinear optimization, loop closure detection, and loop closure optimization. Upon system startup, the first few frames of images are initialized to obtain high-quality map points for subsequent pose calculation using the PnP method. For each image frame read, front-end data preprocessing is performed, including feature point recognition and matching. Based on the matched feature points, the relative pose is solved using the PnP method. The back-end uses visual reprojection to continuously optimize the keyframes and map points selected by the front-end, making the results more accurate. Loop closure detection checks for the presence of loops to reduce global errors. The positioning module uses a tightly coupled fusion strategy to jointly optimize the high-speed dynamic information provided by the IMU and the image features acquired by the visual sensor. This enables continuous and high-precision estimation of the UAV's pose in environments without GNSS signals or where GNSS is denied, ensuring the system's autonomous navigation capability.

[0045] Step 3: Denoise and convert the depth image from the depth camera into a point cloud. Based on the visual inertial odometry information, obtain the current pose of the UAV and then calculate the 3D point cloud data in the world coordinate system. Perform voxelization filtering on the point cloud data to build an occupied grid map. Each time the map is updated via sensor measurements, record the updated area, delete outdated boundary clusters within that area, and search for new boundary clusters. After removal, use a region growing algorithm to search for new boundaries and cluster them into groups. In these groups, the smaller groups, typically caused by noisy sensor observations, are ignored; however, the remaining groups may contain large clusters, which are not conducive to distinguishing unique unknown regions and making complex decisions; therefore, if the maximum eigenvalue exceeds a threshold, the present invention performs PCA principal component analysis on each cluster and divides it into two uniform clusters along the first principal axis; the partitioning is performed recursively, so all large clusters are divided into smaller clusters; finally, viewpoint generation and cost updates are performed based on the boundary clusters generated by the clusters.

[0046] Step 4: Use the LKH algorithm to solve the exploration problem of the boundary viewpoint, obtain the optimal exploration order, and generate the global path using the A* algorithm.

[0047] The exploration planning of this invention begins with finding a global path that effectively covers the existing boundary clusters. This invention formulates it as a variant of the Traveling Salesman Problem (TSP), which calculates an open-loop path starting from the current viewpoint, passing through viewpoints in all clusters, and sequentially visiting at least one representative viewpoint in each boundary cluster of the environment to maximize the exploration coverage of the unknown environment. To meet practical flight constraints and mission requirements, this invention further simplifies the above path planning problem and models it as a standard asymmetric Traveling Salesman Problem (ATSP) by rationally designing the path cost matrix Mtsp. The standard TSP is closed-loop, requiring a return to the starting point, and the route is symmetrical.

[0048] However, this invention requires finding the optimal open-loop path that passes through all boundary clusters, without requiring a return to the starting point. Moreover, the path has a directional penalty term, which is intended to punish unreasonable heading changes, resulting in an asymmetric path cost. Therefore, it is modeled as an asymmetric traveling salesman problem.

[0049] For this asymmetric path planning model, this invention uses the heuristic LKH algorithm to solve it, which can quickly obtain the near-optimal access order and path planning results.

[0050] Specifically, the optimal exploration order is based on the aforementioned asymmetric traveling salesman problem, and is solved using the LKH heuristic algorithm to quickly obtain an approximately optimal access order and determine the optimal sequence for the UAV to visit candidate viewpoints of each boundary cluster.

[0051] Specifically, the global path is a feasible path planned by the A* algorithm in the occupied grid map based on the optimal access order, so as to allow the UAV to visit each cluster of representative viewpoints in sequence from the starting point, thereby forming a global path that satisfies environmental obstacle constraints.

[0052] Assuming there are a total of Clusters, Corresponding to A square matrix of dimension 1. The main part consists of the connection cost between each pair of boundary clusters. The block, mathematically expressed as:

[0053]

[0054] in, Indicates the first and the A boundary cluster, This indicates the connection cost between the two. viewpoint and The lower bound of the time is used to represent the cost matrix. This indicates the number of clusters obtained after clustering the frontier points; Cost Matrix The first row and first column are related to the current viewpoint and Clusters are associated. From Beginning, the first Each boundary cluster is evaluated using the following formula:

[0055]

[0056] in, Indicate viewpoint and The lower limit of the time interval; Indicates the consistency cost weight; Represents the consistency cost; the symbol · represents scalar multiplication; Consistency Cost As shown in the following formula:

[0057] in, This represents the 3D position of the j-th viewpoint within the k-th cluster; and These represent the current position and speed of the drone, respectively. Represents the magnitude of a vector.

[0058] in, This is the current speed. In some cases, the lower time limits for multiple trips are similar, which may lead to back-and-forth maneuvers in consecutive planning steps, thus slowing down progress. Using To eliminate this inconsistency, it penalizes large changes in flight direction.

[0059] Step 5: Local viewpoint optimization is performed using a graph search method to provide a richer set of viewpoints on the truncated portion of the global path, further improving the exploration rate. Then, Dijkstra's algorithm is used to search for the optimal local path. Subsequently, a nonlinear optimization algorithm is introduced to optimize and smooth the trajectory, generating a smooth, safe, and dynamically feasible B-spline trajectory. This ensures that the generated flight path effectively avoids obstacles while meeting dynamic feasibility requirements, improving flight safety and mission completion efficiency. The flight control module, based on the flight path output by the path planning module and combined with the UAV dynamics model, adjusts the UAV's attitude and speed in real time through a trajectory tracking controller and a gain-scheduled PID controller to achieve stable and efficient autonomous flight. In other words, optimizing the local path of the global path using Dijkstra's algorithm is specifically implemented as follows: on the planned global path, select a viewpoint whose distance from the current viewpoint is less than a given threshold. A continuous boundary cluster, denoted as The candidate viewpoint set and the current viewpoint for these clusters A directed acyclic graph (DAG) is constructed, where each node represents a candidate viewpoint, and each node is connected only to viewpoint nodes in the next cluster via directed edges, forming a directed graph structure that captures local path changes. Using the time lower bound and motion consistency cost as weights, Dijkstra's algorithm is employed to search for a local path with the minimum total cost within this directed graph, thus obtaining the optimal viewpoint access order within the current local scope.

[0060] Step 6: Using the visible light sensor and infrared sensor mounted on the drone, a dual-light image fusion target detection algorithm is employed, such as... Figure 3 As shown, this improves the accuracy of target detection under different lighting conditions.

[0061] Specifically, infrared images, based on thermal radiation, have an advantage in representing salient targets, but they have poor texture and low resolution. In contrast, visible light images have high resolution and rich texture details, but they are easily affected by the environment. For example, in low light or hazy weather, the ability to depict targets will be significantly reduced.

[0062] This invention first uses a convolutional fusion network to perform convolution operations with a kernel size of 3×3 on visible light and infrared images, batch normalization (BN), and ReLU function activation to achieve better extraction of shallow features; Then, a semantic segmentation network is used to pass the fused image through a convolutional normalization activation layer, i.e., Conv+BN+ReLU, followed by max and average pooling operations. The features obtained from the pooling operations are summed and then passed through a Conv+BN+Sigmoid layer to extract low-level spatial features and high-level semantic features, outputting the semantic segmentation result of the fused image. The visual perception enhancement module is based on an image pyramid. Starting from the smallest scale, it estimates the reflection components of a pixel by comparing it with its neighborhood using a Gaussian center function. R By interpolating layer by layer and passing the estimation results, an enhanced image with reduced illumination is finally obtained, which greatly eliminates the impact of poor lighting conditions on image quality and makes small targets that are difficult to detect visible. Finally, the YOLOv8 object detection network is used to detect objects in the fused image, achieving highly robust object recognition.

[0063] The target's pose is calculated from pixel coordinates to world coordinates to obtain its position; assuming image points... , , Let z represent the pixel coordinates of the image plane, and T represent the transpose of the vector; the depth value is z, and the camera intrinsic parameter matrix is... ,as follows:

[0064] in, , These represent the camera's focal length in the horizontal and vertical directions, respectively. , These represent the x and y coordinates of the camera's principal point, respectively. Using the inverse matrix of camera intrinsics Projecting the pixels back onto the camera coordinate system:

[0065] in, Represents the three-dimensional coordinates of a point in the camera coordinate system; the symbol · indicates matrix multiplication; The camera extrinsic rotation matrix is The translation vector is Transform it to the world coordinate system using the camera pose:

[0066] in, This represents the three-dimensional coordinates of a point in the world coordinate system. This method solves the problem of weak image information from a single light source in the current UAV target detection process, and can effectively provide the target point location, thereby improving the UAV's reliable and efficient perception computing capabilities.

[0067] Simulation Experiments: Based on the Gazebo and XTDrone simulation platforms, simulation experiments were conducted on the proposed method for autonomous exploration and multimodal target detection of unmanned aerial vehicles in unknown environments. Figure 4As shown. This invention acquires data from a stereo depth camera, RealSense. Data from multiple sensors, including the D435i and IMU, is fused in real-time through filtering, time synchronization, and spatial alignment techniques. A fusion method using a binocular depth camera and an inertial measurement unit (IMU) is employed to perceive external environmental information in GNSS-denied environments. Visual odometry is constructed based on binocular RGB images and IMU data to obtain the UAV's position and attitude. Three-dimensional point cloud data in the world coordinate system is obtained using depth maps and visual odometry information to build an occupancy grid map. Rich boundary increment information is extracted from the occupancy grid map for incremental boundary detection and clustering. Viewpoint generation and cost updates are performed based on the boundary clusters generated by clustering. The LKH algorithm is used to solve the boundary viewpoint exploration problem to obtain the optimal exploration order, and the A* algorithm generates a global path. Local viewpoint optimization is performed using a graph search method, and the Dijkstra algorithm is used to generate the optimal local path. Nonlinear optimization is employed to generate a smooth, collision-free, and shortest-time B-spline optimized trajectory for autonomous navigation. Multimodal target detection is performed using a dual-light image fusion target detection algorithm through visible light and infrared image fusion, and the pose of the detected targets is estimated. Figure 4 As shown, the proposed method for autonomous UAV exploration and multimodal target detection in unknown environments demonstrates excellent performance in the constructed simulation scenario, effectively detecting targets such as pedestrians and vehicles. These results indicate that the proposed method has practical application value.

[0068] The present invention also provides an autonomous exploration and multimodal target detection system for unmanned aerial vehicles (UAVs) in unknown environments. The autonomous exploration and multimodal target detection system for UAVs in unknown environments can be implemented by executing the process steps of the autonomous exploration and multimodal target detection method for UAVs in unknown environments. That is, those skilled in the art can understand the autonomous exploration and multimodal target detection method for UAVs in unknown environments as a preferred embodiment of the autonomous exploration and multimodal target detection system for UAVs in unknown environments.

[0069] A system for autonomous exploration and multimodal target detection of unmanned aerial vehicles in unknown environments, provided by the present invention, includes: Module M1: Collects and fuses UAV sensor information to obtain the UAV's position and attitude information; Module M2: Based on the fusion results of UAV sensor information, a map is built and incremental information of map boundaries is extracted, and then clustered to generate boundary clusters; Module M3: Based on the boundary cluster, calculates the optimal exploration order and global path, and then generates a trajectory and provides navigation.

[0070] Specifically, it also includes module M4: in navigation, visible light and infrared images are collected by the visible light sensor and infrared sensor of the UAV, and then fused to obtain and detect the dual-light image to generate the target point position.

[0071] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0072] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for autonomous exploration and multimodal target detection by unmanned aerial vehicles (UAVs) in unknown environments, characterized in that, include: Step S1: Collect and fuse UAV sensor information to obtain the UAV's position and attitude information; Step S2: Based on the fusion results of UAV sensor information, build a map and extract map boundary incremental information, and then cluster to generate boundary clusters; Step S3: Based on the boundary cluster, calculate the optimal exploration order and global path, and then generate a trajectory and navigate.

2. The method for autonomous exploration and multimodal target detection of unmanned aerial vehicles in unknown environments according to claim 1, characterized in that, It also includes step S4: During navigation, visible light images and infrared images are acquired through the visible light sensor and infrared sensor of the UAV, and then fused to obtain and detect the dual-light image to generate the target point position.

3. The method for autonomous exploration and multimodal target detection of unmanned aerial vehicles in unknown environments according to claim 1, characterized in that, In step S1, the sensor information includes: RGB image and IMU data; Step S1 includes: Step S1.1: Collect sensor information using the UAV's binocular depth camera and IMU sensor; Step S1.2: Based on the RGB image and IMU data of the sensor information, construct a visual inertial odometry (VIO); then obtain the position and attitude information of the UAV through the visual inertial odometry (VIO).

4. The method for autonomous exploration and multimodal target detection of unmanned aerial vehicles in unknown environments according to claim 1, characterized in that, In step S2, based on the location information, i.e., the three-dimensional point cloud data in the world coordinate system, an occupied grid map is established; it is determined whether the boundary clusters of the occupied grid map are outdated. If the result is yes, they are deleted, and a new boundary cluster is searched for by a region growing algorithm to obtain a new boundary cluster; if the result is no, no processing is performed. Boundary viewpoints are generated based on the new boundary clusters.

5. The method for autonomous exploration and multimodal target detection of unmanned aerial vehicles in unknown environments according to claim 1, characterized in that, Step S3 includes: Step S3.1: Set up the cost matrix to obtain the asymmetric traveling salesman problem; Step S3.2: Solve the asymmetric traveling salesman problem using the LKH algorithm to obtain the optimal exploration order; Step S3.3: Based on the optimal exploration order, generate a global path using the A* algorithm.

6. The method for autonomous exploration and multimodal target detection of unmanned aerial vehicles in unknown environments according to claim 5, characterized in that, In step S3.1, the asymmetric traveling salesman problem is to find an open-loop path that passes through all boundary cluster viewpoints and has the minimum total cost, starting from the current viewpoint. The mathematical expression for the cost matrix is: in, express The connection cost, Indicates the first The boundary cluster and the first A boundary cluster; Representing the cost matrix, i.e., viewpoint and The lower limit of the time; This indicates the number of clusters obtained after clustering the frontier points; The current viewpoint of the cost matrix and Relationship, starting from the current viewpoint, the first A boundary cluster is evaluated using the following mathematical expression: in, Indicates the current viewpoint, that is with viewpoint The lower limit of the time interval; Indicates the consistency cost weight; Indicate viewpoint The consistency cost; the symbol · represents scalar multiplication; The mathematical expression for the consistency cost is: in, express The cost of consistency Indicates the first The first cluster The three-dimensional position of each viewpoint; Indicates the current location of the drone; Indicates the current speed of the drone; This represents the L2 norm.

7. The method for autonomous exploration and multimodal target detection of unmanned aerial vehicles in unknown environments according to claim 5, characterized in that, Step S3 includes: Step S3.1: Based on the global path, select a continuous boundary cluster whose distance from the current viewpoint is less than a preset threshold, and then establish a directed acyclic graph; Step S3.2: The optimal exploration order and global path are obtained in the directed acyclic graph using Dijkstra's algorithm, and then a trajectory is generated and navigation is performed.

8. The method for autonomous exploration and multimodal target detection of unmanned aerial vehicles in unknown environments according to claim 2, characterized in that, In step S4, fusing the visible light image and the infrared image includes: Step S4.1: Convolve and normalize the visible light image and the infrared image to obtain a dual-light image as a fused image; Step S4.2: Perform a pooling operation on the fused image and output the semantic segmentation result of the fused image; Step S4.3: Based on the semantic segmentation results, obtain the reflection component using the Gaussian central function. R Then, interpolation is performed layer by layer to obtain a fused image with reduced illumination; The reflection component is obtained based on the semantic segmentation result and through the Gaussian central function. R The mathematical expression is: Where x and y represent the pixel coordinates in the image. The estimated reflection component, i.e., the reflection component R ; The grayscale values ​​of the original image. The standard deviation is The Gaussian central function, i.e., the two-dimensional Gaussian function, This is a convolution operation; The mathematical expression is: in, Let be the standard deviation of the Gaussian kernel, · denotes multiplication, and exp is the exponential function.

9. A system for autonomous exploration and multimodal target detection by unmanned aerial vehicles in unknown environments, characterized in that: include: Module M1: Collects and fuses UAV sensor information to obtain the UAV's position and attitude information; Module M2: Based on the fusion results of UAV sensor information, a map is built and incremental information of map boundaries is extracted, and then clustered to generate boundary clusters; Module M3: Based on the boundary cluster, calculates the optimal exploration order and global path, and then generates a trajectory and provides navigation.

10. The unmanned aerial vehicle (UAV) autonomous exploration and multimodal target detection system in unknown environments according to claim 9, characterized in that, It also includes module M4: In navigation, visible light and infrared images are collected by the UAV's visible light sensor and infrared sensor, then fused to obtain and detect the dual-light image to generate the target point position.

Images