A mine unmanned aerial vehicle intelligent surveying and mapping method and system fusing multiple sensors

Abstract

The application provides a mine unmanned aerial vehicle intelligent surveying and mapping method and system fusing multiple sensors, and belongs to the technical field of unmanned aerial vehicle surveying and mapping. The method comprises the following steps: obtaining three-dimensional point cloud data, visible light image data and IMU attitude data of a target mining area, constructing a slope gradient field of a mining area surface based on the three-dimensional point cloud data and extracting a steep slope area; determining a set of anchor primitives with spatial invariance from the three-dimensional point cloud in the steep slope area; determining spatio-temporal registration constraints of multiple source data according to the set of anchor primitives in combination with the visible light image of the steep slope area; taking the spatio-temporal registration constraints as prior constraint terms, and integrating into a tight coupling pose optimization solving process based on the IMU attitude data, the three-dimensional point cloud data and the visible light image data to obtain optimal pose estimation; and constructing a three-dimensional surveying and mapping model of the steep slope area based on the optimal pose estimation. The technical scheme provided by the application can improve spatio-temporal registration of multiple source data in the surveying and mapping process of a mine unmanned aerial vehicle.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) surveying technology, and more specifically, to a mining UAV intelligent surveying method and system that integrates multiple sensors. Background Technology

[0002] In recent years, UAV mapping has been widely used in various production processes in the mining industry due to its advantages such as mobility, efficiency, and low cost. These include mining area topographic mapping, mining design, and slope stability monitoring. Existing multi-sensor UAV mapping systems can acquire large-scale, high-density three-dimensional spatial information of mining areas in a short time, and have become an important technical support for the intelligentization of modern mines.

[0003] In existing UAV mapping, the main approach is based on photogrammetry and remote sensing principles. UAVs, equipped with high-precision digital cameras and LiDAR sensors, collect low-altitude, high-overlap imagery or point cloud data of the survey area via preset flight paths. In post-processing, algorithms such as corresponding image point matching and bundle adjustment are used to calculate the 3D coordinates of ground features, generating digital surface models, digital orthophotos, and true orthophotos to map the target area. However, mining scenarios have unique characteristics such as complex terrain, steep slopes, poor texture, and severe signal obstruction. Open-pit mine slopes commonly exhibit large areas of abrupt slope changes. Traditional multi-sensor mapping methods based on unified fusion weights suffer from insufficient spatiotemporal registration accuracy in such areas, leading to geometric distortion and texture misalignment. This affects the accuracy of slope deformation monitoring and mining design. Furthermore, the poor texture in mining areas renders conventional feature matching of visible light images ineffective, further exacerbating the difficulty of spatiotemporal registration of multi-source data. Therefore, improving the spatiotemporal registration of multi-source data during UAV mapping in mining has become a significant challenge for the industry. Summary of the Invention

[0004] This application provides a method and system for intelligent mapping of mining drones that integrates multiple sensors, which can improve the spatiotemporal registration of multi-source data during the mapping process of mining drones.

[0005] In a first aspect, this application provides an intelligent mapping method for mining drones that integrates multiple sensors, comprising the following steps:

[0006] The mining drone acquires 3D point cloud data, visible light image data and IMU attitude data of the target mining area, and constructs a slope gradient field of the mining area surface based on the 3D point cloud data, and extracts steep slope areas from the slope gradient field according to a preset slope gradient threshold.

[0007] Local geometric features are extracted from the three-dimensional point cloud within the steep slope transition area to obtain a set of anchored primitives with spatial invariance;

[0008] The anchoring primitive set is used to perform cross-modal feature matching on the visible light image of the steep slope area to obtain the spatiotemporal registration constraints of the multi-source data.

[0009] The spatiotemporal registration constraint is used as a priori constraint and integrated into the tightly coupled pose optimization solution process based on the IMU attitude data, the three-dimensional point cloud data and the visible light image data to obtain the optimal pose estimation of the steep slope region.

[0010] Based on the optimal pose estimation, the three-dimensional point cloud data and visible light image data of the steep slope area are fused to generate a three-dimensional mapping model of the steep slope area.

[0011] In some embodiments, the target mining area's three-dimensional point cloud data, visible light image data, and IMU attitude data are acquired using multiple types of sensors carried by the mining drone.

[0012] In some embodiments, constructing the slope gradient field of the mining area surface based on the three-dimensional point cloud data specifically includes:

[0013] Outlier removal is performed on the three-dimensional point cloud data to obtain a denoised point cloud, and an irregular triangular network is constructed based on the denoised point cloud to obtain a surface triangular network model of the mining area.

[0014] Based on the surface triangulation model of the mining area, the normal vector of the triangular facet is calculated using the triangular facet as the unit to obtain the normal vector field.

[0015] Using the normal vector field as input, calculate the angle between the normal vector and the vertical direction for each triangular facet to obtain the slope scalar field;

[0016] Spatial gradient calculation is performed on the slope scalar field to obtain the slope gradient field.

[0017] In some embodiments, extracting the steep slope transition region from the slope gradient field based on a preset slope gradient threshold specifically includes:

[0018] The gradient magnitude of the slope gradient field is calculated to obtain a gradient magnitude map;

[0019] The gradient magnitude map is binary segmented using the preset slope gradient threshold to obtain a steep transition candidate mask;

[0020] Connectivity analysis is performed on the candidate mask for steep slope changes to extract connected regions with areas greater than a preset minimum area threshold, thus obtaining the steep slope change region.

[0021] In some embodiments, extracting local geometric features from the three-dimensional point cloud within the steep slope transition area to obtain a spatially invariant set of anchored primitives specifically includes:

[0022] The three-dimensional point cloud in the steep slope area is downsampled using voxels to obtain a downsampled point cloud;

[0023] For each point in the downsampled point cloud, a neighborhood search is performed to construct a local neighborhood point set;

[0024] The covariance matrix of the local neighborhood point set is decomposed to obtain eigenvalues ​​and eigenvectors;

[0025] Using the aforementioned feature values ​​as input, calculate the local curvature and linearity, and filter out points whose local curvature is greater than a preset curvature threshold and whose linearity is greater than a preset linearity threshold to obtain a candidate corner point set;

[0026] For each candidate corner in the candidate corner set, the eigenvector corresponding to the minimum eigenvalue is used as the normal vector, and a local surface is fitted with the candidate corner as the center. The histogram features of the normal vector direction at the candidate corner are extracted to obtain a set of anchored primitives with spatial invariance.

[0027] In some embodiments, using the anchoring primitive set to perform cross-modal feature matching on the visible light image of the steep slope area to obtain spatiotemporal registration constraints for multi-source data specifically includes:

[0028] The three-dimensional coordinates of each anchoring primitive in the anchoring primitive set are projected onto the visible light image plane through the initial camera pose to obtain the projection point set;

[0029] An image search window is constructed with each projection point in the projection point set as the center, and SIFT feature points are extracted within the image search window to obtain a candidate matching point set.

[0030] Normalized cross-correlation calculation is performed on the normal vector direction histogram features of each anchor primitive in the anchor primitive set and the SIFT feature descriptors of each candidate matching point in the candidate matching point set. The candidate matching point with the largest cross-correlation value that exceeds the preset cross-correlation threshold is taken as the corresponding image point to obtain a two-dimensional-three-dimensional corresponding point pair set.

[0031] The spatiotemporal registration constraints of multi-source data are determined using the set of corresponding two-dimensional and three-dimensional point pairs as observations.

[0032] In some embodiments, the multi-type sensors include a lidar scanner, a high-resolution optical camera, and an inertial measurement unit.

[0033] Secondly, this application provides a multi-sensor integrated intelligent mapping system for mining drones, used to execute a multi-sensor integrated intelligent mapping method for mining drones, the system comprising:

[0034] The acquisition module is used to acquire three-dimensional point cloud data, visible light image data and IMU attitude data of the target mining area through mining drones, and to construct the slope gradient field of the mining area surface based on the three-dimensional point cloud data, and extract the steep slope area from the slope gradient field according to the preset slope gradient threshold.

[0035] The processing module is used to extract local geometric features from the three-dimensional point cloud in the steep slope area to obtain a set of anchored primitives with spatial invariance.

[0036] The processing module is also used to perform cross-modal feature matching on the visible light image of the steep slope area using the anchoring primitive set, so as to obtain the spatiotemporal registration constraints of the multi-source data;

[0037] The processing module is also used to incorporate the spatiotemporal registration constraint as a priori constraint into the tightly coupled pose optimization solution process based on the IMU attitude data, the three-dimensional point cloud data and the visible light image data, so as to obtain the optimal pose estimate of the steep slope region.

[0038] The execution module is used to fuse the three-dimensional point cloud data and visible light image data of the steep slope area based on the optimal pose estimation to generate a three-dimensional mapping model of the steep slope area.

[0039] Thirdly, this application provides a computer device, the computer device including a memory and a processor, the memory storing code, and the processor being configured to acquire the code and execute the above-described intelligent mapping method for mining drones that integrates multiple sensors.

[0040] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described intelligent mapping method for mining drones that integrates multiple sensors.

[0041] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects:

[0042] The intelligent mapping method and system for mining drones that integrates multiple sensors provided in this application first acquires 3D point cloud data, visible light image data, and IMU attitude data of the target mining area using a mining drone. Based on the 3D point cloud data, a slope gradient field of the mining area's surface is constructed, and steep slope transition regions are extracted from the slope gradient field according to a preset slope gradient threshold. Second, local geometric features are extracted from the 3D point cloud within the steep slope transition regions to obtain a spatially invariant set of anchored primitives. Further, the anchored primitives set is used to perform cross-modal feature matching on the visible light image of the steep slope transition regions to obtain spatiotemporal registration constraints for the multi-source data. Then, the spatiotemporal registration constraints are used as prior constraints and integrated into a tightly coupled pose optimization solution process based on the IMU attitude data, the 3D point cloud data, and the visible light image data to obtain the optimal pose estimate for the steep slope transition regions. Finally, based on the optimal pose estimate, the 3D point cloud data and visible light image data of the steep slope transition regions are fused to generate a 3D mapping model of the steep slope transition regions.

[0043] Therefore, this application can improve the spatiotemporal registration of multi-source data in the mapping process of mining UAVs. First, by constructing a slope gradient field of the mining area surface and automatically extracting steep slope areas based on a preset slope gradient threshold, the traditional terrain analysis method relying on manual interpretation or single slope threshold segmentation is transformed into automatic slope boundary detection on the gradient domain, providing a clear spatial focus range for subsequent processing and avoiding indiscriminate redundant calculations on flat areas. Second, by jointly screening and extracting normal vector direction histogram features through dual geometric indices of local curvature and linearity, a spatially invariant anchoring system is constructed. The primitive set replaces traditional point cloud features dependent on a local reference coordinate system with rotation- and translation-invariant statistical feature descriptors, eliminating the descriptor mismatch problem caused by estimation bias of the point cloud's local reference system. This provides stable 3D primitives that can be repeatedly identified under different scanning views and positions for cross-modal matching. Furthermore, by projecting anchored primitives onto visible light images and performing normalized cross-correlation matching with SIFT feature points within a search window to obtain 2D-3D corresponding point pairs, explicit cross-modal association between the 3D point cloud geometry and 2D image texture information is achieved at the feature level. The constructed spatiotemporal registration constraint is used for re- The projection error form provides an absolute spatial reference independent of inertial pre-integration and point cloud registration for subsequent pose optimization, avoiding the problem that single sensor internal constraints cannot eliminate geometric misalignment between 3D point clouds and images. Then, the spatiotemporal registration constraint is used as a priori constraint term and jointly constructed with IMU pre-integration constraints, visual reprojection constraints, and laser odometry constraints to form a global cost function for tightly coupled nonlinear optimization. This allows cross-modal corresponding point pair constraints to directly participate in the iterative update of pose state variables. When the single sensor constraint degrades due to geometric repetition or sparse point clouds in steep slope regions, the cross-modal constraints provide compensation. The positioning reference effectively suppresses pose drift and obtains the globally optimal pose estimation under multi-sensor constraints. Finally, using the optimal pose estimation as a unified coordinate reference, the laser point cloud and the sparse point cloud of the image are registered and fused, and texture color is mapped. This eliminates coordinate deviations and scale differences between multi-source data, so that the final generated 3D mapping model has both the high-precision geometric structure information of the lidar and the real visual texture details of the visible light image, achieving effective spatiotemporal registration of multi-source data. In summary, the technical solution provided in this application can improve the spatiotemporal registration of multi-source data in the mapping process of mining UAVs. Attached Figure Description

[0044] Figure 1 This is an exemplary flowchart of a multi-sensor intelligent mapping method for mining drones, according to some embodiments of this application;

[0045] Figure 2 This is a flowchart illustrating the multi-sensor data acquisition process of a mining drone according to some embodiments of this application;

[0046] Figure 3This is an exemplary flowchart illustrating the determination of the anchoring primitive set according to some embodiments of this application;

[0047] Figure 4 This is a schematic diagram of the structure of a mining UAV intelligent mapping system that integrates multiple sensors, as shown in some embodiments of this application;

[0048] Figure 5 This is a schematic diagram of the structure of a computer device for implementing a multi-sensor intelligent mapping method for mining drones, according to some embodiments of this application. Detailed Implementation

[0049] To better understand the technical solution of this application, the technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] refer to Figure 1 This figure is an exemplary flowchart of a multi-sensor intelligent mapping method for mining drones, according to some embodiments of this application. The figure mainly includes the following steps:

[0051] In step S101, three-dimensional point cloud data, visible light image data and IMU attitude data of the target mining area are acquired by a mining drone, and a slope gradient field of the mining area surface is constructed based on the three-dimensional point cloud data. The steep slope area is extracted from the slope gradient field according to a preset slope gradient threshold.

[0052] Specifically, the mining drone acquires 3D point cloud data, visible light image data, and IMU attitude data of the target mining area using multiple types of sensors, including a lidar scanner, a high-resolution optical camera, and an inertial measurement unit (IMU). During mission execution, the mining drone autonomously flies and scans the target mining area. The lidar scanner acquires the 3D spatial coordinate information of the target mining area by emitting laser beams and receiving echoes, thereby generating 3D point cloud data. The high-resolution optical camera simultaneously captures high-definition images of the target mining area, obtaining visible light image data. The inertial measurement unit measures the inertial data of the mining drone itself, including acceleration and angular velocity, in real time, obtaining IMU attitude data. These three types of data are typically synchronized and recorded by an onboard computer, ensuring that the three types of data are alignable in the time dimension.

[0053] refer to Figure 2This figure is a flowchart of a multi-sensor data acquisition process for a mining drone according to some embodiments of this application. The figure shows the process by which a mining drone equipped with a lidar scanner, a high-resolution optical camera and an inertial measurement unit (IMU) outputs three-dimensional point cloud data, visible light image data and IMU attitude data respectively when performing surveying operations in a target mining area. It clearly shows the correspondence and flow between the carrier, sensors, the work object and the collected data.

[0054] It should be noted that, in this application, the target mining area refers to the open-pit mine slope area that requires intelligent mapping, including but not limited to the mining face, stockpile, spoil heap, and transportation roads; 3D point cloud data refers to the set of 3D spatial coordinate points on the surface of the target mining area. Each point typically contains X, Y, and Z coordinate information and may be supplemented with attributes such as reflection intensity or echo count, used to characterize the geometric shape and spatial location of surface objects; visible light image data refers to red, green, and blue three-channel color digital images of the mining area surface taken by a high-resolution optical camera, which can provide rich texture, color, and semantic information for identifying ground object categories and assisting in feature matching; IMU attitude data refers to the UAV's own three-axis acceleration, angular velocity, and calculated pitch, roll, and heading angles measured in real time by the inertial measurement unit, used to describe the UAV's spatial motion state and orientation at each moment.

[0055] In some embodiments, the slope gradient field of the mining area surface is constructed based on the three-dimensional point cloud data using the following steps:

[0056] Outlier removal is performed on the three-dimensional point cloud data to obtain a denoised point cloud, and an irregular triangular network is constructed based on the denoised point cloud to obtain a surface triangular network model of the mining area.

[0057] Based on the surface triangulation model of the mining area, the normal vector of the triangular facet is calculated using the triangular facet as the unit to obtain the normal vector field.

[0058] Using the normal vector field as input, calculate the angle between the normal vector and the vertical direction for each triangular facet to obtain the slope scalar field;

[0059] Spatial gradient calculation is performed on the slope scalar field to obtain the slope gradient field.

[0060] In specific implementation, firstly, statistical outlier removal is performed on the 3D point cloud data. This involves constructing a Gaussian distribution model based on the average distance between each 3D point cloud and its K nearest neighbors. 3D point clouds exceeding the mean ± 3 standard deviations are considered noise and removed to obtain a denoised point cloud. Here, K nearest neighbors refer to the K sampling points with the smallest Euclidean distance to the corresponding 3D point cloud in 3D space. In this embodiment, K is set to 20, but this is not a limitation. Subsequently, an irregular triangular mesh is constructed on the denoised point cloud using the Delaunay triangulation algorithm to obtain a surface triangular mesh model of the mining area. The triangulation algorithm is a spatial partitioning method that satisfies the empty circle property and maximizes the sum of the minimum interior angles of all triangular facets. The mining area surface triangulation model refers to a continuous surface representation of the undulating surface of the mining area, approximating the topography of the mining area with discrete point clouds as vertices and a set of non-overlapping triangular facets. Secondly, based on the mining area surface triangulation model, using triangular facets as computational units, the cross product of the two edge vectors of each triangular facet starting from a fixed point is taken and normalized to obtain the normal vector of the corresponding triangular facet. The set of all facet normal vectors constitutes a normal vector field, where the normal vector refers to the unit angle perpendicular to the plane containing the triangular facet. The normal vector field refers to the spatial distribution of normal vectors defined on each facet of the triangular mesh model, representing the spatial orientation of local micro-elements on the surface of the mining area. Then, for each triangular facet, the angle between its normal vector and the vertical unit vector is calculated; this angle is the slope angle at that facet. The scalar distribution of slope angles across all facets constitutes the slope scalar field. The vertical unit vector is a spatial vector with a magnitude of 1, pointing in the opposite direction to gravity. The slope scalar field refers to the spatial distribution of slope angles defined on each facet of the triangular mesh model, representing the steepness or gentleness of the surface at various points in the mining area. Finally, the slope scalar... The field performs spatial gradient calculations, which involves using the central difference scheme to numerically discretize the slope change rate between adjacent triangular faces. Specifically, the difference between the slope value of the current triangular face and its one-ring neighboring triangular faces is divided by the spatial distance between the centers of the faces. After taking the modulus, the slope change intensity at that triangular face is obtained. The slope change intensities at all faces together constitute the slope gradient field. The central difference scheme is a numerical differentiation method that uses symmetrically distributed adjacent sampling points to approximate the difference quotient to reduce truncation error. The one-ring neighboring triangular faces refer to all adjacent triangular faces that share at least one edge with the current triangular face.

[0061] It should be noted that, in this application, the slope gradient field refers to the spatial distribution of slope change intensity defined on each facet of the triangular mesh model. It characterizes the spatial location and magnitude of drastic changes in the surface slope of the mining area. Due to the discontinuous terrain formed by mining disturbances, such as stepped steep slopes, loose deposits, and fault slip surfaces, a single slope value can only reflect the local degree of inclination and cannot distinguish the boundary location of spatial abrupt changes in slope. The slope gradient field, by performing spatial first-order differentiation on the slope scalar field, quantifies the drastic degree of slope change between adjacent locations with the gradient magnitude. It can numerically separate gradually changing terrain from steeply changing terrain, providing a quantifiable basis for the automatic extraction of steep slope areas.

[0062] In some embodiments, the extraction of steep slope transition regions from the slope gradient field based on a preset slope gradient threshold is achieved through the following steps:

[0063] The gradient magnitude of the slope gradient field is calculated to obtain a gradient magnitude map;

[0064] The gradient magnitude map is binary segmented using the preset slope gradient threshold to obtain a steep transition candidate mask;

[0065] Connectivity analysis is performed on the candidate mask for steep slope changes to extract connected regions with areas greater than a preset minimum area threshold, thus obtaining the steep slope change region.

[0066] In specific implementation, firstly, the gradient vector at each triangular facet in the slope gradient field is moduloed, i.e., the sum of the squares of the three components of the vector is taken and then the square root is obtained to obtain a scalar value representing the intensity of slope change at that facet. All scalar values ​​corresponding to the triangular facests are arranged according to their triangular facet indices to form a gradient modulus map. The gradient modulus map is a two-dimensional scalar array that uses the triangular mesh model facests as spatial carriers to record the intensity of slope change at each facet. Then, the gradient modulus map is binary segmented using a preset slope gradient threshold, i.e., each triangular facet in the gradient modulus map is traversed, and faces with gradient modulus values ​​greater than the preset slope gradient threshold are marked as 1, while faces with gradient modulus values ​​less than or equal to the threshold are marked as 0. This yields a steep change candidate mask composed of 0 and 1 values. The preset slope gradient threshold is a gradient modulus critical value set based on geological survey experience of mine slopes to distinguish the boundary between gradual and steep slope changes. The steep change candidate mask refers to the gradient modulus critical value marked in the triangular mesh model. A binary identifier map of the set of faces belonging to the area of ​​drastic slope change; finally, connected component analysis is performed on the candidate mask for steep slope, that is, the Two-Pass algorithm is used to mark the connected components of the faces marked as 1 in the mask. The Two-Pass algorithm refers to a two-scan connected component marking method that establishes an equivalent label table of face adjacency relationship through the first scan and unifies the equivalent labels into the final connected component number through the second scan. After traversal, the projected area of ​​the faces contained in each connected component is counted. Connected components with an area greater than a preset minimum area threshold are retained, and connected components with an area less than or equal to the threshold are removed as noise. The surface space range corresponding to the finally retained connected components is the extracted steep slope area. The area is calculated by summing the projected areas of each triangular face in the connected component on the horizontal projection plane. The preset minimum area threshold is 50 square meters to filter out small pseudo steep slope areas caused by local undulations of the point cloud, which is not limited here.

[0067] It should be noted that, in this application, the steep slope transition region refers to the spatial location in the mining area topography where the slope changes abruptly. Specifically, it refers to continuous surface units in the mining area where the slope gradient modulus exceeds a preset slope gradient threshold and the area is greater than a preset minimum area threshold. The steep slope transition region includes the slope shoulder and toe lines of mining benches, the contact transition zone between loose debris and bedrock in spoil heaps, and steep slopes formed by exposed rock mass structural surfaces, etc., which are significant discontinuous interfaces generated by the combined effects of geological and engineering activities. Identifying the steep slope transition region can separate the local abrupt changes that are of key geomechanical significance for slope stability analysis from the continuous topography in the 3D point cloud data. This provides a clear spatial processing range for subsequent cross-modal feature matching and tightly coupled pose optimization, so that local geometric feature extraction only applies to the subset of point clouds with significant structural information, avoiding the waste of computational resources caused by extracting redundant features in gentle and homogeneous areas.

[0068] In step S102, local geometric features are extracted from the three-dimensional point cloud in the steep slope area to obtain a set of anchored primitives with spatial invariance.

[0069] In some embodiments, reference Figure 3 As shown in the figure, this is an exemplary flowchart of determining the anchoring primitive set according to some embodiments of this application. In this embodiment, the local geometric features of the three-dimensional point cloud in the steep slope area are extracted to obtain the spatially invariant anchoring primitive set by the following steps:

[0070] In step S1021, the three-dimensional point cloud in the steep slope area is downsampled using voxels to obtain a downsampled point cloud;

[0071] In step S1022, a neighborhood search is performed on each point in the downsampled point cloud to construct a local neighborhood point set;

[0072] In step S1023, the covariance matrix is ​​decomposed on the local neighborhood point set to obtain eigenvalues ​​and eigenvectors;

[0073] In step S1024, the local curvature and linearity are calculated using the feature values ​​as input, and points with local curvature greater than a preset curvature threshold and linearity greater than a preset linearity threshold are selected to obtain a candidate corner point set;

[0074] In step S1025, for each candidate corner point in the candidate corner point set, the eigenvector corresponding to the minimum eigenvalue is used as the normal vector, a local surface is fitted with the candidate corner point as the center, and the histogram features of the normal vector direction at the candidate corner point are extracted to obtain a set of anchored primitives with spatial invariance.

[0075] In specific implementation, firstly, voxel downsampling is performed on the 3D point cloud in the steep slope transition area. This involves setting the voxel grid side length to 3 to 5 times the average point spacing of the point cloud, and replacing all points falling within the same voxel grid with their centroids. This results in a downsampled point cloud with uniform density while maintaining the macroscopic geometric shape. A voxel grid refers to a 3D space uniformly divided into cubic mesh units with equal side lengths. A downsampled point cloud refers to a sparse point cloud with a reduced number of points but preserved overall geometric features. Secondly, a neighborhood search is performed on each point in the downsampled point cloud. This involves constructing a local spherical neighborhood centered on the current point and within a preset search radius. A KD-Tree data structure is used to accelerate radius lookup and to identify points falling within the same voxel grid. Points within the neighborhood are sorted in ascending order of Euclidean distance. The top Q points are taken to form the local neighborhood set of each point. KD-Tree refers to a tree-like index structure constructed by performing a dimensional spatial bisection on the 3D point cloud to achieve fast nearest neighbor retrieval. The local neighborhood set refers to the set of neighboring points used to describe the local spatial morphology of the corresponding point in the downsampled point cloud. The search radius is twice the side length of the voxel grid, and the Q value is set to 30 to balance the discriminative power of local features and the stability of the covariance matrix solution. This is not a limitation. Furthermore, covariance matrix decomposition is performed on each local neighborhood set, that is, the covariance matrix of the difference between the coordinates of each point in the neighborhood and the coordinates of the neighborhood center is calculated. Eigenvalue decomposition is performed on the covariance matrix to obtain three eigenvalues ​​sorted in descending order. Covariance matrix decomposition, which involves converting the covariance matrix describing the spatial distribution of neighborhood points into eigenvalues ​​and eigenvectors, is a principal component analysis method. Then, using the eigenvalues ​​as input, local curvature and linearity are calculated, with local curvature defined as the minimum eigenvalue. The ratio of the sum of the three eigenvalues Linearity is used to measure the degree to which a neighborhood point set deviates from a planar distribution and exhibits surface curvature; it is defined as follows: This is used to measure the degree of linear distribution of a neighborhood point set extending in a single direction. Points with local curvature greater than a preset curvature threshold and linearity greater than a preset linearity threshold are selected to form a candidate corner point set. The preset curvature threshold is set to 0.05 and the preset linearity threshold is set to 0.6 to select geometric abrupt change points that are both located in significant curved parts of the surface and have linear structural characteristics. There are no restrictions on this. The candidate corner point set refers to the set of geometrically significant points in the downsampled point cloud that simultaneously satisfy the curvature and linearity conditions. Finally, for each candidate corner point in the candidate corner point set, the minimum eigenvalue obtained by covariance matrix decomposition is used. The corresponding feature vector serves as the normal vector of the local surface at that point. A quadratic surface is fitted with the candidate corner point as the center and the search radius as the neighborhood range. Within this neighborhood, the distribution of the angle between the normal vector and the normal vector of the neighboring points is statistically constructed to form a normal vector direction histogram feature. The normal vector direction histogram feature refers to the multi-dimensional vector formed by dividing the normal vector direction of each point in the neighborhood into multiple direction intervals according to spherical coordinates and counting the number of points falling into each interval. Since this feature is invariant under point cloud rotation and translation, all candidate corner points and their histogram features together constitute a spatially invariant set of anchored primitives.

[0076] It should be noted that the spatially invariant anchoring primitive set in this application refers to the set of spatially salient points with rotation-translation invariant geometric descriptors, used to achieve repeatable cross-modal feature matching and localization benchmarks in multi-source data. Existing technologies directly use single geometric saliency detection methods such as point cloud ISS keypoints or Harris corner points to extract feature points. This scheme constructs a hierarchical, progressive anchoring primitive extraction paradigm of "joint screening of local curvature and linearity dual thresholds—minimum eigenvector normal constraint—normal vector direction histogram feature encoding." This paradigm first derives two complementary geometric indices, local curvature and linearity, through eigenvalue decomposition for dual discrimination, ensuring that the selected candidate corner points are not only... Where the surface curvature is significant, redundant corner points without discriminative power are avoided in the gentle areas inside the slope patch. Then, the local surface is fitted with the eigenvector corresponding to the minimum eigenvalue as the normal vector and the histogram feature of the normal vector direction is extracted. This descriptor encodes spatial orientation information into spherical partition statistics and has rotation and translation invariance. This allows the same geological structure boundary to obtain a consistent descriptive vector under different scanning views and positions. Unlike conventional descriptors that rely on local reference coordinate systems and are sensitive to coordinate axis definitions, this effectively eliminates the descriptor mismatch problem caused by the estimation bias of the local reference system of the point cloud. As a result, it significantly improves the accuracy and robustness of anchored primitive re-identification from 3D point cloud to 2D visible light image in subsequent cross-modal matching.

[0077] In step S103, the anchoring primitive set is used to perform cross-modal feature matching on the visible light image of the steep slope area to obtain the spatiotemporal registration constraints of the multi-source data.

[0078] In some embodiments, the spatiotemporal registration constraints of multi-source data are obtained by performing cross-modal feature matching on the visible light images of the steep slope area using the anchoring primitive set, and the results are achieved through the following steps:

[0079] The three-dimensional coordinates of each anchoring primitive in the anchoring primitive set are projected onto the visible light image plane through the initial camera pose to obtain the projection point set;

[0080] An image search window is constructed with each projection point in the projection point set as the center, and SIFT feature points are extracted within the image search window to obtain a candidate matching point set.

[0081] Normalized cross-correlation calculation is performed on the normal vector direction histogram features of each anchor primitive in the anchor primitive set and the SIFT feature descriptors of each candidate matching point in the candidate matching point set. The candidate matching point with the largest cross-correlation value that exceeds the preset cross-correlation threshold is taken as the corresponding image point to obtain a two-dimensional-three-dimensional corresponding point pair set.

[0082] The spatiotemporal registration constraints of multi-source data are determined using the set of corresponding two-dimensional and three-dimensional point pairs as observations.

[0083] In specific implementation, firstly, the 3D world coordinates of each anchored primitive in the anchored primitive set are transformed to the camera coordinate system through the initial camera pose, and then projected onto the visible light image plane through the camera intrinsic parameter matrix to obtain the corresponding 2D pixel coordinates of each anchored primitive. The pixel coordinates of all projected points constitute the projection point set. The initial camera pose refers to the spatial position and attitude parameters of the UAV when capturing the image frame provided by the airborne GNSS / IMU system. The projection point set refers to the set of pixel positions of the 3D anchored primitives mapped onto the 2D image according to the perspective projection relationship. Secondly, an image search window is constructed with each projection point in the projection point set as the center. The side length of the search window is selected according to actual needs; for example, this application selects 81 pixels. SIFT feature points are extracted from the visible light image within the search window. This involves constructing a Gaussian difference pyramid for the image within the search window and locating key points at scale-space extrema. After removing low-contrast points and edge response points, a principal direction is assigned to each key point, and a 128-dimensional SIFT feature descriptor is generated. All SIFT feature points within the window constitute a candidate matching point set. SIFT feature points refer to local image feature points that are invariant to image scaling and rotation and stable to changes in illumination and viewing angle. The candidate matching point set refers to the set of SIFT feature points within the search window that are adjacent to the projected position of the anchor primitive. Then, the histogram of the normal vector direction of each anchor primitive in the anchor primitive set is calculated. Normalized cross-correlation is performed on the SIFT feature descriptors of each candidate matching point in the image feature set. Normalized cross-correlation calculation involves taking the mean of each set of feature vectors, calculating their inner product, and dividing by their respective moduli. The value falls within a certain range; a value closer to 1 indicates higher feature similarity. An empirical cross-correlation threshold is set. For the same anchor primitive, the candidate matching point with the largest cross-correlation value exceeding the threshold is selected as the corresponding image point of that anchor primitive. If the largest cross-correlation value is lower than the threshold, the anchor primitive is considered to have no reliable corresponding image point in the current image and is discarded. All anchor primitives and their corresponding corresponding image points constitute a 2D-3D corresponding point pair set. A corresponding image point refers to a 3D anchor primitive. In the visible light image, the corresponding two-dimensional pixel points verified by feature similarity are used. The two-dimensional-three-dimensional corresponding point pair set refers to the set of matching pairs formed by a three-dimensional spatial point and its corresponding image point in the two-dimensional image. Finally, the two-dimensional-three-dimensional corresponding point pair set is used as the observation value to construct the reprojection error constraint. The reprojection error constraint refers to reprojecting the three-dimensional coordinates of each anchoring primitive onto the image plane according to the camera pose to be optimized, and calculating the Euclidean distance between the projected point and the measured pixel coordinates of the corresponding image point as the residual term. The sum of squared residuals of all corresponding point pairs constitutes the spatiotemporal registration constraint of the multi-source data. The residual term is a two-dimensional vector composed of the deviation components of the reprojected position and the measured image point position along the row and column directions in the pixel coordinate system.

[0084] It should be noted that the spatiotemporal registration constraint of multi-source data in this application refers to the mathematical constraint that unifies the association between the 3D point cloud and the visible light image in both the temporal and spatial dimensions. The spatiotemporal registration constraint of multi-source data is determined because the lidar and visible light camera mounted on the mining UAV have physical spatial offsets in their sensor coordinate system origins and temporal asynchrony in their sampling trigger times. Furthermore, the initial pose provided by the airborne GNSS / IMU system is limited by positioning accuracy and attitude drift. Relying solely on the internal constraints of a single sensor for pose estimation will lead to geometric misalignment between the 3D point cloud and the visible light image due to accumulated errors. This constraint is determined by... The spatiotemporal registration constraints of the source data can transform the cross-modal correspondence between the anchored primitives from the 3D point cloud and the corresponding image points from the visible light image into an explicit reprojection error constraint equation. This equation is then injected into the subsequent tightly coupled pose optimization solution process in the form of observations. In turn, it provides an external absolute scale benchmark for pose estimation that is independent of IMU pre-integration and point cloud registration during multi-sensor data fusion. This constrains the translational and rotational degrees of freedom of the camera pose in space, eliminates texture mapping deviations when the 3D point cloud is projected onto the image, and ensures that the geometry and texture colors in the final generated 3D mapping model are accurately aligned at the pixel level.

[0085] In step S104, the spatiotemporal registration constraint is used as a priori constraint and integrated into the tightly coupled pose optimization solution process based on the IMU attitude data, the three-dimensional point cloud data and the visible light image data to obtain the optimal pose estimate of the steep slope region.

[0086] In some embodiments, the spatiotemporal registration constraint is used as a priori constraint term and integrated into a tightly coupled pose optimization solution process based on the IMU attitude data, the 3D point cloud data, and the visible light image data. The optimal pose estimation for the steep slope region is obtained through the following steps:

[0087] The IMU attitude data is pre-integrated to obtain the IMU pre-integration constraint;

[0088] Visual reprojection constraints are constructed by matching consecutive inter-frame features of the visible light image data;

[0089] Iterative nearest-point registration is performed on adjacent frame point clouds corresponding to the steep slope change region in the three-dimensional point cloud data to obtain laser odometry constraints.

[0090] The spatiotemporal registration constraint, the IMU pre-integration constraint, and the visual reprojection constraint are jointly used to construct a global cost function;

[0091] The optimal pose estimate for the steep slope region is obtained by performing nonlinear optimization with the goal of minimizing the global cost function.

[0092] In specific implementation, firstly, the IMU attitude data undergoes pre-integration processing. Specifically, within the time interval between two visible light image frames, the angular velocity measured by the gyroscope and the relative force measured by the accelerometer are continuously integrated over time, transforming them into a relative pose increment independent of the previous frame's state. This constructs IMU pre-integration constraints that constrain position, velocity, and rotation changes between adjacent frames. The IMU pre-integration constraints refer to using the pose increment accumulated by the inertial measurement unit between frames as the relative pose observation equation between adjacent state nodes. The integration employs the fourth-order Runge-Kutta method, which is a Runge-Kutta numerical integration method using a weighted average of four intermediate slopes. Secondly, visual reprojection constraints are constructed using continuous inter-frame feature matching of visible light image data. Specifically, ORB feature points are extracted from two adjacent visible light image frames and a brute-force matching method is used to establish... The inter-frame correspondence involves projecting the 3D coordinates of feature points from the previous frame onto the image plane of the next frame according to the camera motion model. A visual reprojection constraint is constructed using the Euclidean distance between the projected point and the matching point pixel coordinates as the residual term. The RB feature points refer to fast image feature points based on FAST corner detection and supplemented with BRIEF binary descriptors. The visual reprojection constraint refers to the observation equation constraining the relative motion pose of the camera based on the correspondence between continuous inter-frame image features. Furthermore, iterative nearest-neighbor registration is performed on adjacent frame point clouds corresponding to steep slope changes in the 3D point cloud data to obtain laser odometry constraints. That is, the previous frame point cloud is used as the reference, and the next frame point cloud is used as the source. The nearest neighbor point in the reference point cloud is searched point-by-point in the source point cloud as the corresponding point pair. An error function is constructed by minimizing the point-to-plane distance between corresponding point pairs, and the inter-frame rigid body transformation is iteratively solved until the pose change between two adjacent iterations is less than [a certain value]. The change in rotation is less than 1 meter The convergence occurs in radians, and the resulting inter-frame transformation matrix constitutes the laser odometry constraint. Iterative nearest-neighbor registration refers to a point cloud registration method that gradually approximates the optimal aligned pose by alternately performing nearest-neighbor search and rigid body transformation estimation. The laser odometry constraint refers to the relative pose transformation constraint equation between adjacent frames obtained based on continuous frame point cloud registration. Then, the spatiotemporal registration constraint, IMU pre-integration constraint, visual reprojection constraint, and laser odometry constraint are jointly constructed to form a global cost function. A factor graph optimization framework is used to add each constraint term as a factor to the corresponding state node in the graph. Each constraint term is based on... The noise characteristics of each sensor are assigned as information matrices as weights, which will not be elaborated here. The global cost function is a scalar objective function that combines the multi-sensor observation constraints in a weighted nonlinear least squares manner to determine all the pose states to be estimated. The factor graph is a graph optimization model in which variable nodes represent the pose states to be estimated and factor nodes represent the constraint equations. Finally, a nonlinear optimization solution is performed with the goal of minimizing the global cost function. The Levenberg-Marquardt algorithm is used to iteratively update all state variables along the negative gradient direction of the cost function from the current pose estimate. When the infinity norm of the gradient decreases to... The iteration stops when the state variable value is obtained, which is the optimal pose estimate of the steep slope region. The Levenberg-Marquardt algorithm is a damped least squares nonlinear optimization algorithm that adaptively interpolates between the steepest descent method and the Gauss-Newton method.

[0093] It should be noted that tightly coupled pose optimization refers to the joint state estimation of observation constraints from different sensors within the same optimization framework, rather than a pose estimation method that solves them independently and then aligns and fuses them. In this scheme, tightly coupled pose optimization specifically involves incorporating four heterogeneous observation equations—spatiotemporal registration constraints, IMU pre-integration constraints, visual reprojection constraints, and laser odometry constraints—into a single global cost function. A factor graph is constructed using the six-DOF poses of the UAV in all frames as the state variables to be estimated. Each constraint, weighted by its respective information matrix, jointly acts on the iterative update process of the state variables. This ensures that the pose solution for each frame is simultaneously constrained by four types of information: high-frequency motion recursion of the inertial measurement unit, inter-frame texture tracking of visible light images, geometric registration of laser point clouds, and projection consistency of cross-modal anchoring primitives. Unlike loosely coupled architectures where laser odometry and visual odometry output pose trajectories independently and are then fused by external timestamp alignment or trajectory matching, tightly coupled architectures utilize the raw observation information of all sensors simultaneously in each iteration of pose estimation. This allows for compensation from other sensor constraints when a single sensor constraint degrades, effectively suppressing error accumulation and improving the accuracy and robustness of pose estimation in geometrically complex areas such as steep slopes.

[0094] It should also be noted that, in this application, optimal pose estimation refers to the trajectory and orientation of the UAV in the global coordinate system representing the motion of multiple sensors when the UAV traverses a steep slope terrain change zone. Specifically, it is the set of spatial position and attitude parameters of the UAV at each sampling time in the steep slope area that minimizes the weighted sum of squared residuals of all sensor observation constraints. This scheme injects the spatiotemporal registration constraints generated by cross-modal feature matching as independent and equally weighted prior constraints into the tightly coupled pose optimization factor graph, so that the explicit two-dimensional-three-dimensional correspondence between the 3D point cloud anchoring primitives and the SIFT feature points of the 2D visible light image directly participates in the nonlinearity of the global cost function. Through iterative optimization, this design transforms the cross-modal matching results, which were originally only used as external verification or post-processing texture mapping basis, into a first-level observation constraint within the pose estimation process. This allows the translational and rotational degrees of freedom of the camera pose to be jointly constrained by the laser geometry, visual texture tracking, and the cross-modal projection consistency of the anchoring primitives. Consequently, in steep slope areas where the laser odometry constraint degrades or the visual reprojection constraint is insufficient due to geometric repetition or sparse point cloud, the absolute spatial positioning reference provided by the cross-modal corresponding point pairs can still effectively constrain pose drift, significantly improving the robustness of pose estimation at the slope boundary and the geometric-texture registration accuracy of the 3D mapping model.

[0095] In step S105, the three-dimensional point cloud data and visible light image data of the steep slope area are fused based on the optimal pose estimation to generate a three-dimensional mapping model of the steep slope area.

[0096] In some embodiments, the fusion of 3D point cloud data and visible light image data of the steep slope transition area based on the optimal pose estimation to generate a 3D mapping model of the steep slope transition area is achieved through the following steps:

[0097] The three-dimensional point cloud data of the steep slope region is transformed to the global coordinate system using the optimal pose estimation to obtain a globally consistent point cloud;

[0098] The visible light image data of the steep slope area is reconstructed into a sparse point cloud using the optimal pose estimation to obtain the image sparse point cloud.

[0099] The image sparse point cloud and the globally consistent point cloud are iteratively registered with the nearest point to obtain the point cloud fusion transformation parameters.

[0100] The globally consistent point cloud and the sparse point cloud of the image are fused using the point cloud fusion transformation parameters to obtain a dense fused point cloud;

[0101] The texture information of the visible light image data is mapped to the dense fused point cloud using the optimal pose estimation to generate a three-dimensional mapping model of the steep slope area.

[0102] In specific implementation, firstly, using the transformation matrix of each frame of the LiDAR coordinate system relative to the global coordinate system in the optimal pose estimation, the point coordinates of each frame of the 3D point cloud data in the steep slope area are transformed point by point to the global coordinate system. Then, the transformed point clouds from multiple frames are superimposed and merged to obtain a globally consistent point cloud. The global coordinate system refers to a spatial rectangular coordinate system with the origin at the ground point directly below the first UAV takeoff point in the mining area and the east-north-sky direction as the coordinate axes. The globally consistent point cloud refers to the overall point cloud continuously covering the steep slope area formed after unifying the point clouds of each frame to the same spatial reference. Secondly, using the camera poses of each frame in the optimal pose estimation as input, the structure-of-motion reconstruction method is used to reconstruct sparse point clouds from the visible light image data of the steep slope area. This involves SIFT feature extraction and inter-frame feature matching for each frame of imagery, utilizing the known camera poses through triangulation... The method calculates the 3D spatial coordinates of matching feature points to obtain the sparse point cloud of the image. The motion reconstruction structure method refers to the reconstruction method of the 3D sparse geometric structure of the scene by back-calculating the feature correspondence between multi-view images and camera pose parameters. The triangulation method refers to the geometric back-calculation method of solving the 3D coordinates of the same spatial point by intersecting the projection rays of the same spatial point in two or more frames of images. The sparse point cloud of the image refers to the set of spatial 3D points reconstructed from the visible light image sequence through multi-view stereo matching. Further, the sparse point cloud of the image and the globally consistent point cloud are iteratively registered with the nearest neighbor. The globally consistent point cloud is used as the reference point cloud and the sparse point cloud of the image is used as the source point cloud. The nearest neighbor pair is searched by KD-Tree and outlier corresponding points with a distance greater than 0.3 meters are removed. The rigid body transformation parameters between the two sets of point clouds are iteratively solved with the point-to-plane residual as the error metric. When the translation increment is less than Meters and rotation increment less than The convergence is achieved in radians, yielding point cloud fusion transformation parameters. These parameters are 3×4 rigid body transformation matrices describing the coordinate alignment between the sparse point cloud in the image and the globally consistent point cloud, comprising a 3×3 rotation matrix and a 3×1 translation vector. Then, the sparse point cloud is transformed to the coordinate system of the globally consistent point cloud using these parameters. The transformed sparse point cloud and the globally consistent point cloud are merged into a dense fused point cloud, which is a three-dimensional point set that fuses the high-density geometric structure directly measured by the lidar with the texture edge feature points obtained from image reconstruction. Finally, the texture information from the visible light image data is... The optimal pose estimation is used to map the information onto the dense fused point cloud. That is, for each point in the dense fused point cloud, the optimal pose estimation is used to project the point coordinates onto all visible image frames of that point. The image with the smallest angle between the normal vector of each visible image frame at that point and the image viewpoint is selected as the best texture source. The pixel position of the projection of this point onto the best texture image is sampled for texture color and assigned an RGB color value to the point. After point-by-point mapping is completed, a three-dimensional mapping model of the steep slope area is generated. Texture mapping refers to the method of assigning the pixel color values ​​of two-dimensional images to the surface of a three-dimensional point cloud according to the projection relationship to generate a model rendering method with realistic color appearance.

[0103] It should be noted that the three-dimensional mapping model in this application refers to the three-dimensional digital mapping results that include the geometric shape and real texture and color information of the steep slope area, which can be used for topographic profile measurement and geological structure interpretation.

[0104] Furthermore, in another aspect of this application, in some embodiments, this application provides a mining drone intelligent mapping system that integrates multiple sensors, with reference to... Figure 4 The figure is a schematic diagram of the structure of a mining UAV intelligent mapping system integrating multiple sensors, according to some embodiments of this application. This system includes an acquisition module 201, a processing module 202, and an execution module 203, which are described below:

[0105] The acquisition module 201 in this application is mainly used to acquire three-dimensional point cloud data, visible light image data and IMU attitude data of the target mining area through mining drones, and construct the slope gradient field of the mining area surface based on the three-dimensional point cloud data, and extract the steep slope area from the slope gradient field according to the preset slope gradient threshold.

[0106] Processing module 202, in this application, is mainly used to extract local geometric features from the three-dimensional point cloud in the steep slope area to obtain a set of anchored primitives with spatial invariance;

[0107] The processing module 202 is also used to perform cross-modal feature matching on the visible light image of the steep slope area using the anchoring primitive set to obtain the spatiotemporal registration constraints of the multi-source data;

[0108] In addition, the processing module 202 is also used to incorporate the spatiotemporal registration constraint as a priori constraint into the tightly coupled pose optimization solution process based on the IMU attitude data, the three-dimensional point cloud data and the visible light image data, so as to obtain the optimal pose estimate of the steep slope area.

[0109] The execution module 203 in this application is mainly used to fuse the three-dimensional point cloud data and visible light image data of the steep slope area based on the optimal pose estimation, and generate a three-dimensional mapping model of the steep slope area.

[0110] In addition, this application also provides a computer device, the computer device including a memory and a processor, the memory storing code, and the processor being configured to acquire the code and execute the above-described intelligent mapping method for mining UAVs that integrates multiple sensors.

[0111] In some embodiments, reference Figure 5 The figure is a schematic diagram of the structure of a computer device for implementing a multi-sensor fusion intelligent mapping method for mining drones, according to some embodiments of this application. The multi-sensor fusion intelligent mapping method for mining drones in the above embodiments can be implemented through... Figure 5 The computer device shown is used to implement this, and the computer device includes at least one processor 301, a communication bus 302, a memory 303, and at least one communication interface 304.

[0112] The processor 301 can be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more devices used to control the execution of the intelligent mapping method for mining UAVs that integrates multiple sensors in this application.

[0113] The communication bus 302 can be used to transmit information between the aforementioned components.

[0114] The memory 303 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 303 may exist independently and be connected to the processor 301 via the communication bus 302. The memory 303 may also be integrated with the processor 301.

[0115] The memory 303 stores program code for executing the scheme of this application, and its execution is controlled by the processor 301. The processor 301 executes the program code stored in the memory 303. The program code may include one or more software modules. In the above embodiments, the determination of the intelligent mapping method for mining UAVs that integrates multiple sensors can be achieved by the processor 301 and one or more software modules in the program code in the memory 303.

[0116] Communication interface 304 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0117] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0118] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0119] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described intelligent mapping method for mining drones that integrates multiple sensors.

[0120] Although preferred embodiments of this application have been described, those skilled in the art, once they have learned the basic inventive concept, can make other changes and modifications to these embodiments.

[0121] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application.

Claims

1. A mining UAV intelligent mapping method integrating multiple sensors, characterized in that, Includes the following steps: The mining drone acquires 3D point cloud data, visible light image data and IMU attitude data of the target mining area, and constructs a slope gradient field of the mining area surface based on the 3D point cloud data, and extracts steep slope areas from the slope gradient field according to a preset slope gradient threshold. Local geometric features are extracted from the three-dimensional point cloud within the steep slope transition area to obtain a set of anchored primitives with spatial invariance; The anchoring primitive set is used to perform cross-modal feature matching on the visible light image of the steep slope area to obtain the spatiotemporal registration constraints of the multi-source data. The spatiotemporal registration constraint is used as a priori constraint and integrated into the tightly coupled pose optimization solution process based on the IMU attitude data, the three-dimensional point cloud data and the visible light image data to obtain the optimal pose estimation of the steep slope region. Based on the optimal pose estimation, the three-dimensional point cloud data and visible light image data of the steep slope area are fused to generate a three-dimensional mapping model of the steep slope area.

2. The method as described in claim 1, characterized in that, The mining drone acquires 3D point cloud data, visible light image data, and IMU attitude data of the target mining area using multiple types of sensors.

3. The method as described in claim 1, characterized in that, Constructing the slope gradient field of the mining area surface based on the aforementioned 3D point cloud data specifically includes: Outlier removal is performed on the three-dimensional point cloud data to obtain a denoised point cloud, and an irregular triangular network is constructed based on the denoised point cloud to obtain a surface triangular network model of the mining area. Based on the surface triangulation model of the mining area, the normal vector of the triangular facet is calculated using the triangular facet as the unit to obtain the normal vector field. Using the normal vector field as input, calculate the angle between the normal vector and the vertical direction for each triangular facet to obtain the slope scalar field; Spatial gradient calculation is performed on the slope scalar field to obtain the slope gradient field.

4. The method as described in claim 1, characterized in that, Extracting steep slope transition regions from the slope gradient field based on a preset slope gradient threshold specifically includes: The gradient magnitude of the slope gradient field is calculated to obtain a gradient magnitude map; The gradient magnitude map is binary segmented using the preset slope gradient threshold to obtain a steep transition candidate mask; Connectivity analysis is performed on the candidate mask for steep slope changes to extract connected regions with areas greater than a preset minimum area threshold, thus obtaining the steep slope change region.

5. The method as described in claim 1, characterized in that, Local geometric feature extraction is performed on the 3D point cloud within the steep slope transition area to obtain a spatially invariant set of anchored primitives, specifically including: The three-dimensional point cloud in the steep slope area is downsampled using voxels to obtain a downsampled point cloud; For each point in the downsampled point cloud, a neighborhood search is performed to construct a local neighborhood point set; The covariance matrix of the local neighborhood point set is decomposed to obtain eigenvalues ​​and eigenvectors; Using the aforementioned feature values ​​as input, calculate the local curvature and linearity, and filter out points whose local curvature is greater than a preset curvature threshold and whose linearity is greater than a preset linearity threshold to obtain a candidate corner point set; For each candidate corner in the candidate corner set, the eigenvector corresponding to the minimum eigenvalue is used as the normal vector, and a local surface is fitted with the candidate corner as the center. The histogram features of the normal vector direction at the candidate corner are extracted to obtain a set of anchored primitives with spatial invariance.

6. The method as described in claim 1, characterized in that, Using the anchoring primitive set to perform cross-modal feature matching on the visible light images of the steep slope area, the spatiotemporal registration constraints of the multi-source data specifically include: The three-dimensional coordinates of each anchoring primitive in the anchoring primitive set are projected onto the visible light image plane through the initial camera pose to obtain the projection point set; An image search window is constructed with each projection point in the projection point set as the center, and SIFT feature points are extracted within the image search window to obtain a candidate matching point set. Normalized cross-correlation calculation is performed on the normal vector direction histogram features of each anchor primitive in the anchor primitive set and the SIFT feature descriptors of each candidate matching point in the candidate matching point set. The candidate matching point with the largest cross-correlation value that exceeds the preset cross-correlation threshold is taken as the corresponding image point to obtain a two-dimensional-three-dimensional corresponding point pair set. The spatiotemporal registration constraints of multi-source data are determined using the set of corresponding two-dimensional and three-dimensional point pairs as observations.

7. The method as described in claim 2, characterized in that, The various types of sensors include lidar scanners, high-resolution optical cameras, and inertial measurement units.

8. A multi-sensor integrated intelligent mapping system for mining drones, used to execute the multi-sensor integrated intelligent mapping method for mining drones as described in any one of claims 1 to 7, characterized in that, The system includes: The acquisition module is used to acquire three-dimensional point cloud data, visible light image data and IMU attitude data of the target mining area through mining drones, and to construct the slope gradient field of the mining area surface based on the three-dimensional point cloud data, and extract the steep slope area from the slope gradient field according to the preset slope gradient threshold. The processing module is used to extract local geometric features from the three-dimensional point cloud in the steep slope area to obtain a set of anchored primitives with spatial invariance. The processing module is also used to perform cross-modal feature matching on the visible light image of the steep slope area using the anchoring primitive set, so as to obtain the spatiotemporal registration constraints of the multi-source data; The processing module is also used to incorporate the spatiotemporal registration constraint as a priori constraint into the tightly coupled pose optimization solution process based on the IMU attitude data, the three-dimensional point cloud data and the visible light image data, so as to obtain the optimal pose estimate of the steep slope region. The execution module is used to fuse the three-dimensional point cloud data and visible light image data of the steep slope area based on the optimal pose estimation to generate a three-dimensional mapping model of the steep slope area.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing code, and the processor being configured to acquire the code and execute the intelligent mapping method for mining drones that integrates multiple sensors as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the intelligent mapping method for mining drones that integrates multiple sensors as described in any one of claims 1 to 7.

Images