High-precision laser radar mapping method and system based on multi-source data fusion

Abstract

The present invention relates to the field of surveying and mapping technology, and discloses a high-precision laser radar surveying and mapping method and system based on multi-source data fusion. The method comprises acquiring point cloud data and impact data of a target area, and recording the posture and position information during the data acquisition process through an inertial navigation device. The method performs denoising and downsampling processing on the point cloud data, geometric correction and enhancement processing on the image data, error correction on the inertial navigation data, and registering the point cloud data with the image data and the inertial navigation data respectively. The method also constructs a data fusion model, fuses the registered point cloud data, image data, and inertial navigation data through the data fusion model, outputs a fusion result, and performs terrain surveying and analysis based on the fusion result. The method improves the accuracy and reliability of surveying and mapping.

Description

Technical Field

[0001] The present invention relates to the field of surveying and mapping technology, and in particular to a high-precision laser radar surveying and mapping method and system based on multi-source data fusion. Background Art

[0002] As an efficient means of spatial data collection, LiDAR mapping technology has been widely used in fields such as terrain mapping, urban modeling, and intelligent transportation. However, single LiDAR data has certain limitations. For example, in complex environments, LiDAR may be affected by factors such as occlusion and reflection, resulting in missing or inaccurate data. Furthermore, LiDAR data itself has a weak ability to express information such as texture and color. While some existing mapping methods attempt to incorporate additional data sources for assistance, they still lack the methods and accuracy of data fusion, unable to fully utilize the advantages of multi-source data and struggling to meet the needs of high-precision mapping. Summary of the Invention

[0003] The purpose of the present invention is to solve the above problems and design a high-precision laser radar mapping method and system based on multi-source data fusion.

[0004] A first aspect of the present invention provides a high-precision laser radar mapping method based on multi-source data fusion, the method comprising the following steps:

[0005] Obtain point cloud data and impact data of the target area, and record the attitude and position information during the data acquisition process through inertial navigation equipment;

[0006] De-noise and downsample the point cloud data, perform geometric correction and enhancement on the image data, perform error correction on the inertial navigation data, and align the point cloud data with the image data and inertial navigation data respectively;

[0007] A data fusion model is constructed, and the registered point cloud data, image data and inertial navigation data are fused through the data fusion model to output the fusion results, and terrain mapping and analysis are performed based on the fusion results.

[0008] Optionally, in a first implementation of the first aspect of the present invention, the denoising and downsampling of the point cloud data, the geometric correction and enhancement of the image data, and the error correction of the inertial navigation data include:

[0009] Determine the neighborhood range of each point in the point cloud data, count the distances from all points in the neighborhood of each point in the point cloud data to the point, and obtain the point cloud data after preliminary denoising;

[0010] The voxel grid size is determined based on the spatial distribution range of the point cloud data after preliminary denoising and the desired degree of downsampling. The original points in each voxel grid unit are removed and the center of gravity is retained to complete the downsampling process.

[0011] Obtain control points on the image data, establish a transformation model, substitute the image coordinates and actual geographic coordinates of the control points into the transformation model and calculate the transformation parameters, transform the coordinates of all points on the image data according to the transformation parameters, and complete the geometric correction process;

[0012] Obtain the grayscale histogram of the image data, calculate the cumulative distribution function, and remap the grayscale value according to the cumulative distribution function to complete the histogram equalization enhancement;

[0013] According to the statistical characteristics of inertial navigation system noise and observation noise, the Kalman filter algorithm is used to filter the inertial navigation data in real time to remove noise and drift errors and complete the error correction processing.

[0014] Optionally, in a second implementation of the first aspect of the present invention, registering the point cloud data with the image data and the inertial navigation data respectively includes:

[0015] Project the point cloud data onto the image plane and construct a Gaussian difference pyramid within the projection area;

[0016] Find extreme points in each layer of the Gaussian difference pyramid and detect their respective feature points in the image data and the projected point cloud data;

[0017] Taking the feature point as the center, the gradient direction histogram of the pixels in the neighborhood is counted, and the main direction is assigned to the feature point according to the direction of the histogram peak;

[0018] A fixed-size neighborhood window is taken with each feature point as the center, and the gradient amplitude and direction of the pixels in the neighborhood are calculated to form a feature descriptor;

[0019] Calculate the Euclidean distance between the feature descriptor of a feature point in the image data and the descriptors of all feature points in the point cloud projection area, and find the two point cloud feature points closest to each other, which are recorded as the nearest neighbor and the next nearest neighbor respectively;

[0020] Calculate the distance ratio between the nearest neighbor point and the next nearest neighbor point. If the ratio is less than the set threshold, it is considered that the feature point in the image is successfully matched with the nearest neighbor point in the point cloud projection area. The threshold is 0.8.

[0021] The image and point cloud feature point pairs obtained through feature matching are summarized to form a set of matching point pairs, and the registration of point cloud data and image data is achieved based on the least squares method.

[0022] Optionally, in a third implementation of the first aspect of the present invention, registering the point cloud data with the image data and the inertial navigation data, respectively, further includes:

[0023] Convert point cloud data from the LiDAR device coordinate system to the geographic coordinate system;

[0024] Select the initial corresponding point pairs, calculate the centroid of the point cloud point set and the centroid of the inertial navigation position point set in the corresponding point pairs, and obtain the centroid coordinates;

[0025] The covariance matrix is ​​calculated based on the centroid coordinates, and the covariance matrix is ​​decomposed by the singular value decomposition method to obtain the optimal rotation moment and translation vector;

[0026] The points in the point cloud data are rotated and translated according to the optimal rotation torque and translation vector, and the iterative process is repeated. After each iteration, the average distance between the corresponding point pairs is calculated to complete the registration of the point cloud data and the inertial navigation data.

[0027] Optionally, in a fourth implementation of the first aspect of the present invention, building a data fusion model includes:

[0028] Acquire historical multi-source data from different sensors, and determine basic architectural parameters of VisionTransformer based on the historical multi-source data;

[0029] Construct a 3D sparse convolution layer and determine the number of input and output channels of the sparse convolution based on the feature dimensions of the point cloud in historical multi-source data;

[0030] Taking the points in historical multi-source data as nodes, the edge connection relationship is determined according to the spatial distance between the points, the historical multi-source data is constructed into a graph structure, and the graph attention network is applied to the constructed graph;

[0031] Integrate Vision Transformer, 3D sparse convolution, and graph attention network to obtain a multimodal search space;

[0032] The mean square error after multi-source data fusion is used to measure the deviation between the fused data and the true value to determine the objective function;

[0033] Input historical multi-source data and perform forward propagation according to the current parameterized network architecture to calculate the objective function value. Based on the objective function value, the network structure parameter vector is updated by backpropagation using a gradient-based optimization algorithm.

[0034] After multiple rounds of iterative search, the network structure parameters that make the objective function value optimal are found, the optimal network structure is generated, and the fusion network is obtained;

[0035] The hyperparameters adjusted by the two-layer optimization framework are applied to the fusion network to obtain the final data fusion model.

[0036] Optionally, in a fifth implementation of the first aspect of the present invention, fusing the registered point cloud data, image data, and inertial navigation data using a data fusion model, outputting a fusion result, and performing terrain mapping and analysis based on the fusion result includes:

[0037] Collect the registered point cloud data, image data and inertial navigation data, convert the data from different data sources into a unified data format, and input it into the data fusion model;

[0038] The image is divided into patches of fixed size and each patch is linearly embedded to obtain an embedding vector. After passing through a multi-layer multi-head self-attention mechanism and a feedforward neural network, the global and local features of the image are extracted.

[0039] Through multiple layers of 3D sparse convolutional layers, the geometric and spatial features of point cloud data are extracted;

[0040] Using the graph attention network, the feature representation of each node is calculated through the attention mechanism to extract the dynamic features and correlation features of the inertial navigation data;

[0041] The image features, point cloud features and inertial navigation features are fused, and the final fusion result is obtained through the nonlinear transformation and feature mapping of the data fusion model;

[0042] The terrain-related data is extracted from the fusion results to construct a digital elevation model, and terrain parameter calculation and terrain analysis are performed based on the digital elevation model.

[0043] A second aspect of the present invention provides a high-precision laser radar mapping system based on multi-source data fusion, the system comprising:

[0044] The data acquisition module is used to obtain point cloud data and impact data of the target area, and record the attitude and position information during the data acquisition process through the inertial navigation device;

[0045] The data processing module is used to perform denoising and downsampling on the point cloud data, geometric correction and enhancement on the image data, error correction on the inertial navigation data, and registration of the point cloud data with the image data and inertial navigation data respectively;

[0046] The terrain mapping module is used to build a data fusion model, fuse the registered point cloud data, image data and inertial navigation data through the data fusion model, output the fusion result, and perform terrain mapping and analysis based on the fusion result.

[0047] Optionally, in a first implementation of the second aspect of the present invention, the data processing module includes:

[0048] The statistical submodule is used to determine the neighborhood range of each point in the point cloud data, count the distances of all points in the neighborhood of each point in the point cloud data to the point, and obtain the point cloud data after preliminary denoising;

[0049] The partitioning submodule is used to determine the voxel grid size according to the spatial distribution range of the point cloud data after preliminary denoising and the desired downsampling degree, remove the original points in each voxel grid unit and retain the center of gravity to complete the downsampling process;

[0050] Establish a submodule for obtaining control points on the image data, establishing a transformation model, substituting the image coordinates and actual geographic coordinates of the control points into the transformation model and calculating transformation parameters, transforming the coordinates of all points on the image data according to the transformation parameters, and completing the geometric correction process;

[0051] The calculation submodule is used to obtain the grayscale histogram of the image data, calculate the cumulative distribution function, and remap the grayscale value according to the cumulative distribution function to complete the histogram equalization enhancement;

[0052] The filtering submodule is used to perform real-time filtering on the inertial navigation data using the Kalman filtering algorithm based on the statistical characteristics of the inertial navigation system noise and observation noise, thereby removing noise and drift errors and completing error correction processing.

[0053] Optionally, in a second implementation of the second aspect of the present invention, the terrain mapping module includes:

[0054] A determination submodule is configured to obtain historical multi-source data from different sensors and determine basic architectural parameters of the Vision Transformer based on the historical multi-source data;

[0055] Set up submodules for constructing 3D sparse convolution layers and determine the number of input and output channels of sparse convolution based on the feature dimensions of point clouds in historical multi-source data;

[0056] The adjustment submodule is used to construct the historical multi-source data into a graph structure using points in the historical multi-source data as nodes, determine the edge connection relationship based on the spatial distance between the points, apply the graph attention network to the constructed graph;

[0057] The integration submodule is used to integrate the Vision Transformer, 3D sparse convolution, and graph attention network to obtain a multimodal search space;

[0058] The allocation submodule is used to determine the objective function by using the mean square error after multi-source data fusion to measure the deviation between the fused data and the true value;

[0059] The propagation submodule is used to input historical multi-source data and perform forward propagation according to the current parameterized network architecture, calculate the objective function value, and update the network structure parameter vector based on the objective function value using the gradient-based optimization algorithm.

[0060] The generation submodule is used to find the network structure parameters that optimize the objective function value through multiple rounds of iterative search, generate the optimal network structure, and obtain the fusion network;

[0061] The application submodule is used to apply the hyperparameters adjusted by the two-layer optimization framework to the fusion network to obtain the final data fusion model.

[0062] A second aspect of the present invention provides a high-precision laser radar mapping system based on multi-source data fusion, the system comprising:

[0063] The data acquisition module is used to obtain point cloud data and impact data of the target area, and record the attitude and position information during the data acquisition process through the inertial navigation device;

[0064] The data processing module is used to perform denoising and downsampling on the point cloud data, geometric correction and enhancement on the image data, error correction on the inertial navigation data, and registration of the point cloud data with the image data and inertial navigation data respectively;

[0065] The terrain mapping module is used to build a data fusion model, fuse the registered point cloud data, image data and inertial navigation data through the data fusion model, output the fusion result, and perform terrain mapping and analysis based on the fusion result.

[0066] Optionally, in a first implementation of the second aspect of the present invention, the data processing module includes:

[0067] The statistical submodule is used to determine the neighborhood range of each point in the point cloud data, count the distances of all points in the neighborhood of each point in the point cloud data to the point, and obtain the point cloud data after preliminary denoising;

[0068] The partitioning submodule is used to determine the voxel grid size according to the spatial distribution range of the point cloud data after preliminary denoising and the desired downsampling degree, remove the original points in each voxel grid unit and retain the center of gravity to complete the downsampling process;

[0069] Establish a submodule for obtaining control points on the image data, establishing a transformation model, substituting the image coordinates and actual geographic coordinates of the control points into the transformation model and calculating the transformation parameters, transforming the coordinates of all points on the image data according to the transformation parameters, and completing the geometric correction process;

[0070] The calculation submodule is used to obtain the grayscale histogram of the image data, calculate the cumulative distribution function, and remap the grayscale value according to the cumulative distribution function to complete the histogram equalization enhancement;

[0071] The filtering submodule is used to perform real-time filtering on the inertial navigation data using the Kalman filtering algorithm based on the statistical characteristics of the inertial navigation system noise and observation noise, thereby removing noise and drift errors and completing error correction processing.

[0072] Optionally, in a second implementation of the second aspect of the present invention, the terrain mapping module includes:

[0073] A determination submodule is configured to obtain historical multi-source data from different sensors and determine basic architectural parameters of the Vision Transformer based on the historical multi-source data;

[0074] Set up submodules for constructing 3D sparse convolution layers and determine the number of input and output channels of sparse convolution based on the feature dimensions of point clouds in historical multi-source data;

[0075] The adjustment submodule is used to construct the historical multi-source data into a graph structure using points in the historical multi-source data as nodes, determine the edge connection relationship based on the spatial distance between the points, apply the graph attention network to the constructed graph;

[0076] The integration submodule is used to integrate the Vision Transformer, 3D sparse convolution, and graph attention network to obtain a multimodal search space;

[0077] The allocation submodule is used to determine the objective function by using the mean square error after multi-source data fusion to measure the deviation between the fused data and the true value;

[0078] The propagation submodule is used to input historical multi-source data and perform forward propagation according to the current parameterized network architecture, calculate the objective function value, and update the network structure parameter vector based on the objective function value using the gradient-based optimization algorithm.

[0079] The generation submodule is used to find the network structure parameters that optimize the objective function value through multiple rounds of iterative search, generate the optimal network structure, and obtain the fusion network;

[0080] The application submodule is used to apply the hyperparameters adjusted by the two-layer optimization framework to the fusion network to obtain the final data fusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Various other advantages and benefits will become apparent to those skilled in the art by reading the following detailed description of the preferred embodiment.The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present invention.

[0082] Figure 1 A schematic diagram of a first embodiment of a high-precision laser radar mapping method based on multi-source data fusion provided by an embodiment of the present invention;

[0083] Figure 2 A schematic diagram of a second embodiment of a high-precision laser radar mapping method based on multi-source data fusion provided by an embodiment of the present invention;

[0084] Figure 3 A schematic diagram of a third embodiment of a high-precision laser radar mapping method based on multi-source data fusion provided by an embodiment of the present invention;

[0085] Figure 4 A schematic structural diagram of a high-precision lidar mapping system based on multi-source data fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0086] The terms "first," "second," "third," "fourth," and so forth (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar items and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that shown or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, product, or device.

[0087] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 A schematic diagram of a first embodiment of a high-precision laser radar mapping method based on multi-source data fusion provided by an embodiment of the present invention, wherein the method specifically comprises the following steps:

[0088] Step 101: Acquire point cloud data and impact data of the target area, and record the attitude and position information during the data acquisition process through the inertial navigation device;

[0089] In this embodiment, a laser radar device is used to obtain point cloud data of the target area, an image acquisition device is used to acquire image data of the target area, and an inertial navigation device is used to record the posture and position information of the laser radar and the image acquisition device during the data acquisition process.

[0090] Step 102: De-noise and downsample the point cloud data, perform geometric correction and enhancement on the image data, perform error correction on the inertial navigation data, and register the point cloud data with the image data and the inertial navigation data respectively.

[0091] In this embodiment, the image data is convolved with Gaussian kernel functions of different scales to generate a series of images at different scales. Adjacent scale images are subtracted to obtain Gaussian difference images to construct a Gaussian difference pyramid. The point cloud data is projected onto the image plane, and a Gaussian difference pyramid is constructed within the projection area. At each layer of the Gaussian difference pyramid, extreme points are found by comparing the grayscale value changes of the pixel point with those of its neighboring pixels. At each layer of the Gaussian difference pyramid, respective feature points are detected in the image data and the projected point cloud data. With the feature point as the center, the gradient direction histogram of the pixels in the neighborhood is statistically calculated, and the main direction is assigned to the feature point according to the direction of the histogram peak. For each feature point in the image data and the point cloud projection area, the extreme point is taken with it as the center. A fixed-size neighborhood window is taken with each feature point as the center, and the gradient amplitude and direction of the pixel points in the neighborhood are calculated to form a feature descriptor; the nearest neighbor distance ratio algorithm is used to calculate the Euclidean distance between the feature descriptor of a feature point in the image data and the descriptors of all feature points in the point cloud projection area, and the two closest point cloud feature points are found, which are recorded as the nearest neighbor point and the second nearest neighbor point respectively; the distance ratio between the nearest neighbor point and the second nearest neighbor point is calculated. If the ratio is less than the set threshold, it is considered that the feature point in the image is successfully matched with the nearest neighbor point in the point cloud projection area, where the threshold is 0.8; the image and point cloud feature point pairs obtained by feature matching are summarized to form a set of matching point pairs, and the point cloud data and image data are aligned based on the least squares method.

[0092] In this embodiment, a rotation matrix is ​​constructed based on the laser radar device posture information recorded by the inertial navigation data, and the point cloud data is converted from the laser radar device coordinate system to the geographic coordinate system in combination with the laser radar device position information recorded by the inertial navigation data. Between the point cloud data after conversion to the geographic coordinate system and the position information recorded by the inertial navigation data, point pairs within a distance threshold range are selected as initial corresponding point pairs; the centroid of the point cloud point set and the centroid of the inertial navigation position point set in the corresponding point pairs are calculated respectively, and the point cloud points and inertial navigation position points in the corresponding point pairs are respectively subtracted from their respective centroid coordinates to obtain de-centroided coordinates; the covariance matrix is ​​calculated based on the de-centroided coordinates, and the covariance matrix is ​​decomposed by the singular value decomposition method to obtain the optimal rotation moment and translation vector; the points in the point cloud data are rotated and translated according to the optimal rotation moment and translation vector, and the iterative process is repeated. After each iteration, the average distance between the corresponding point pairs is calculated. The iterative process is repeated. After each iteration, the average distance between the corresponding point pairs is calculated. When the maximum number of iterations is reached, the iteration is stopped to complete the alignment of the point cloud data and the inertial navigation data.

[0093] Step 103: construct a data fusion model, fuse the registered point cloud data, image data, and inertial navigation data through the data fusion model, output a fusion result, and perform terrain mapping and analysis based on the fusion result;

[0094] In this embodiment, historical multi-source data from different sensors are obtained, and the basic architectural parameters of VisionTransformer are determined based on the historical multi-source data, including at least the patch size, embedding dimension, number of heads and number of layers of the multi-head attention mechanism; the size of the sparse convolution kernel and the step size are set according to the spatial distribution range of the point cloud in the historical multi-source data to determine the convolution operation mode in the three-dimensional space to construct a 3D sparse convolution layer, and the number of input and output channels of the sparse convolution is determined according to the feature dimension of the point cloud in the historical multi-source data; the points in the historical multi-source data are used as nodes, the connection relationship of the edges is determined according to the spatial distance between the points, the historical multi-source data is constructed into a graph structure, and the graph attention network is applied to the constructed graph to enable the nodes to adaptively adjust their own feature weights according to the information of the neighboring nodes; the Vision Transformer, 3D sparse convolution, and graph attention network are integrated to obtain a multimodal search space; a set of learnable parameter vectors is assigned to each possible network architecture in the search space, and the mean square error after multi-source data fusion is used to measure the deviation between the fused data and the true value to determine the objective function; historical multi-source data is input and forward propagated according to the network architecture represented by the current parameterization, and the objective function value is calculated. According to the objective function value, the network structure parameter vector is updated through backpropagation using a gradient-based optimization algorithm; after multiple rounds of iterative search, the network structure parameters that optimize the objective function value are found, thereby automatically generating the optimal network structure and obtaining a fused network.

[0095] In this embodiment, the upper layer: constructs a multidimensional value space containing all global hyperparameters to be optimized in the fusion network, randomly selects 20 groups of hyperparameter combinations in the multidimensional value space through Latin hypercube sampling, uses these hyperparameter combinations to train the fusion network, and records the corresponding performance index values ​​as initial data for constructing the Gaussian process model. In each round of iteration, the Gaussian process model is used to predict the objective function values ​​of different points in the hyperparameter space. Based on the prediction results, the acquisition function is used to determine the combination that can maximize the improvement of the current best performance. The selected hyperparameter combination is used to retrain the fusion network, update the performance index value, and feed back the new hyperparameter combination and the corresponding performance index value to the Gaussian process model to update the parameters of the Gaussian process model and find the optimal global hyperparameter combination; the lower layer: parameters for different sensors A sensor-level parameter space is constructed. Each individual represents a set of sensor parameter value combinations in the sensor-level parameter space. These individuals constitute the initial population. At the same time, the covariance matrix of the initial population is set. The sensor parameters corresponding to each individual in the population are applied to the optimal global hyperparameter combination generated by the upper layer, and the performance index of the model is calculated. This performance index is used as the fitness value of the individual. Based on the fitness value of the individual, the CMA-ES evolutionary algorithm is used for selection, recombination and mutation operations. According to the fitness changes of the individuals during the evolution process and the parameter distribution of the newly generated individuals, the covariance matrix is ​​updated. After multiple rounds of evolution, the optimal sensor-level parameter combination is found. The hyperparameters adjusted by the two-layer optimization framework are applied to the fusion network to obtain the final data fusion model.

[0096] See also Figure 2 , a schematic diagram of a second embodiment of a high-precision laser radar mapping method based on multi-source data fusion provided by an embodiment of the present invention, the method comprising:

[0097] Step 201: determine the neighborhood range of each point in the point cloud data, count the distances from all points in the neighborhood of each point in the point cloud data to the point, and obtain the point cloud data after preliminary denoising;

[0098] In this embodiment, if the distance from the point to the neighborhood mean point is greater than the distance threshold, it is determined to be a noise point and removed from the point cloud data to obtain point cloud data after preliminary denoising.

[0099] Step 202: Determine the voxel grid size based on the spatial distribution range of the point cloud data after preliminary denoising and the desired downsampling degree, remove the original points in each voxel grid unit and retain the center of gravity point to complete the downsampling process;

[0100] In this embodiment, the three-dimensional space is divided into voxel grids, each point in the point cloud after preliminary denoising is assigned to the corresponding voxel grid unit, the centroid coordinates of all points in each voxel grid unit are calculated, and the centroid coordinates are used to represent all points in the voxel grid unit. The original points in each voxel grid unit are removed and the centroid points are retained to complete the downsampling process.

[0101] Step 203: Acquire control points on the image data, establish a transformation model, substitute the image coordinates and actual geographic coordinates of the control points into the transformation model and calculate the transformation parameters. Then, transform the coordinates of all points on the image data according to the transformation parameters to complete the geometric correction process.

[0102] Step 204: Obtain a grayscale histogram of the image data, calculate a cumulative distribution function, and remap the grayscale values ​​according to the cumulative distribution function;

[0103] In this embodiment, each pixel in the image data is traversed, and its original grayscale value is replaced with a new grayscale value according to a mapping relationship to complete histogram equalization enhancement.

[0104] Step 205: Based on the statistical characteristics of the inertial navigation system noise and the observation noise, a Kalman filter algorithm is used to perform real-time filtering on the inertial navigation data to remove noise and drift errors and complete error correction processing.

[0105] See also Figure 3 , a schematic diagram of a third embodiment of a high-precision laser radar mapping method based on multi-source data fusion provided by an embodiment of the present invention, the method comprising:

[0106] Step 301: Collect the registered point cloud data, image data, and inertial navigation data, convert the data from different data sources into a unified data format, and input them into the data fusion model;

[0107] Step 302: Segment the image into fixed-size patches and perform linear embedding on each patch to obtain an embedding vector. Then, use a multi-layer self-attention mechanism and a feedforward neural network to extract global and local features of the image.

[0108] In this embodiment, the registered image data is input into the Vision Transformer module.

[0109] Step 303: Extract the geometric features and spatial features of the point cloud data through multiple layers of 3D sparse convolutional layers;

[0110] In this embodiment, a 3D sparse convolution module is used to process the registered point cloud data, and a sparse convolution kernel is constructed according to the spatial distribution of the point cloud data.

[0111] Step 304: Utilize the graph attention network to calculate the feature representation of each node through the attention mechanism, and extract the dynamic features and correlation features of the inertial navigation data;

[0112] In this embodiment, the inertial navigation data is constructed into a graph structure, with different parameters in the inertial navigation data as nodes and the relationships between the nodes as edges.

[0113] Step 305: Fusing the image features, point cloud features, and inertial navigation features, and obtaining the final fusion result through nonlinear transformation and feature mapping of the data fusion model;

[0114] Step 306: Extract terrain-related data from the fusion result to construct a digital elevation model, and perform terrain parameter calculation and terrain analysis based on the digital elevation model.

[0115] See also Figure 4 , a schematic structural diagram of a high-precision laser radar mapping system based on multi-source data fusion provided by an embodiment of the present invention, the system comprising:

[0116] The data acquisition module is used to obtain point cloud data and impact data of the target area, and record the attitude and position information during the data acquisition process through the inertial navigation device;

[0117] The data processing module is used to perform denoising and downsampling on the point cloud data, geometric correction and enhancement on the image data, error correction on the inertial navigation data, and registration of the point cloud data with the image data and inertial navigation data respectively;

[0118] The terrain mapping module is used to build a data fusion model, fuse the registered point cloud data, image data and inertial navigation data through the data fusion model, output the fusion result, and perform terrain mapping and analysis based on the fusion result.

[0119] In this embodiment, the data processing module includes:

[0120] The statistical submodule is used to determine the neighborhood range of each point in the point cloud data, count the distances of all points in the neighborhood of each point in the point cloud data to the point, and obtain the point cloud data after preliminary denoising;

[0121] The partitioning submodule is used to determine the voxel grid size according to the spatial distribution range of the point cloud data after preliminary denoising and the desired downsampling degree, remove the original points in each voxel grid unit and retain the center of gravity to complete the downsampling process;

[0122] Establish a submodule for obtaining control points on the image data, establishing a transformation model, substituting the image coordinates and actual geographic coordinates of the control points into the transformation model and calculating transformation parameters, transforming the coordinates of all points on the image data according to the transformation parameters, and completing the geometric correction process;

[0123] The calculation submodule is used to obtain the grayscale histogram of the image data, calculate the cumulative distribution function, and remap the grayscale value according to the cumulative distribution function to complete the histogram equalization enhancement;

[0124] The filtering submodule is used to perform real-time filtering on the inertial navigation data using the Kalman filtering algorithm based on the statistical characteristics of the inertial navigation system noise and observation noise, thereby removing noise and drift errors and completing error correction processing.

[0125] In this embodiment, the terrain surveying and mapping module includes:

[0126] A determination submodule is configured to obtain historical multi-source data from different sensors and determine basic architectural parameters of the Vision Transformer based on the historical multi-source data;

[0127] Set up submodules for constructing 3D sparse convolution layers and determine the number of input and output channels of sparse convolution based on the feature dimensions of point clouds in historical multi-source data;

[0128] The adjustment submodule is used to construct the historical multi-source data into a graph structure using points in the historical multi-source data as nodes, determine the edge connection relationship based on the spatial distance between the points, apply the graph attention network to the constructed graph;

[0129] The integration submodule is used to integrate the Vision Transformer, 3D sparse convolution, and graph attention network to obtain a multimodal search space;

[0130] The allocation submodule is used to determine the objective function by using the mean square error after multi-source data fusion to measure the deviation between the fused data and the true value;

[0131] The propagation submodule is used to input historical multi-source data and perform forward propagation according to the current parameterized network architecture, calculate the objective function value, and update the network structure parameter vector based on the objective function value using the gradient-based optimization algorithm.

[0132] The generation submodule is used to find the network structure parameters that optimize the objective function value through multiple rounds of iterative search, generate the optimal network structure, and obtain the fusion network;

[0133] The application submodule is used to apply the hyperparameters adjusted by the two-layer optimization framework to the fusion network to obtain the final data fusion model.

[0134] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A high-precision laser radar mapping method based on multi-source data fusion, characterized in that: The method comprises the following steps: Obtain point cloud data and impact data of the target area, and record the attitude and position information during the data acquisition process through inertial navigation equipment; De-noise and downsample the point cloud data, perform geometric correction and enhancement on the image data, perform error correction on the inertial navigation data, and align the point cloud data with the image data and inertial navigation data respectively; Constructing a data fusion model, fusing the registered point cloud data, image data, and inertial navigation data through the data fusion model, outputting the fusion results, and performing terrain mapping and analysis based on the fusion results; The constructing of the data fusion model includes: Obtain historical multi-source data from different sensors and determine the basic architectural parameters of the VisionTransformer based on this historical multi-source data, including patch size, embedding dimension, number of heads in the multi-head attention mechanism, and number of layers. The size of the sparse convolution kernel and the step size are set according to the spatial distribution range of the point cloud in the historical multi-source data to determine the convolution operation mode in the three-dimensional space to construct a 3D sparse convolution layer. The number of input and output channels of the sparse convolution is determined according to the feature dimension of the point cloud in the historical multi-source data. Taking points in historical multi-source data as nodes, the edge connection relationship is determined according to the spatial distance between points, and the historical multi-source data is constructed into a graph structure. The graph attention network is applied to the constructed graph to enable nodes to adaptively adjust their own feature weights based on the information of neighboring nodes; Integrate Vision Transformer, 3D sparse convolution, and graph attention network to obtain a multimodal search space; A set of learnable parameter vectors is assigned to each possible network architecture in the search space, and the mean square error after multi-source data fusion is used to measure the deviation between the fused data and the true value to determine the objective function; Input historical multi-source data and perform forward propagation according to the current parameterized network architecture to calculate the objective function value. Based on the objective function value, the network structure parameter vector is updated by backpropagation using a gradient-based optimization algorithm. After multiple rounds of iterative search, the network structure parameters that make the objective function value optimal are found, thereby automatically generating the optimal network structure and obtaining a fusion network; The hyperparameters adjusted by the two-layer optimization framework are applied to the fusion network to obtain the final data fusion model.

2. The high-precision laser radar mapping method based on multi-source data fusion according to claim 1, characterized in that: The denoising and downsampling processing of the point cloud data, the geometric correction and enhancement processing of the image data, and the error correction of the inertial navigation data include: Determine the neighborhood range of each point in the point cloud data, count the distances from all points in the neighborhood of each point in the point cloud data to the point, and obtain the point cloud data after preliminary denoising; The voxel grid size is determined based on the spatial distribution range of the point cloud data after preliminary denoising and the desired degree of downsampling. The original points in each voxel grid unit are removed and the center of gravity is retained to complete the downsampling process. Obtain control points on the image data, establish a transformation model, substitute the image coordinates and actual geographic coordinates of the control points into the transformation model and calculate the transformation parameters, transform the coordinates of all points on the image data according to the transformation parameters, and complete the geometric correction process; Obtain the grayscale histogram of the image data, calculate the cumulative distribution function, and remap the grayscale value according to the cumulative distribution function to complete the histogram equalization enhancement; According to the statistical characteristics of inertial navigation system noise and observation noise, the Kalman filter algorithm is used to filter the inertial navigation data in real time to remove noise and drift errors and complete the error correction processing.

3. The high-precision laser radar mapping method based on multi-source data fusion according to claim 1, characterized in that: The registering of the point cloud data with the image data and the inertial navigation data respectively includes: Project the point cloud data onto the image plane and construct a Gaussian difference pyramid within the projection area; Find extreme points in each layer of the Gaussian difference pyramid and detect their respective feature points in the image data and the projected point cloud data; Taking the feature point as the center, the gradient direction histogram of the pixels in the neighborhood is counted, and the main direction is assigned to the feature point according to the direction of the histogram peak; A fixed-size neighborhood window is taken with each feature point as the center, and the gradient amplitude and direction of the pixels in the neighborhood are calculated to form a feature descriptor; Calculate the Euclidean distance between the feature descriptor of a feature point in the image data and the descriptors of all feature points in the point cloud projection area, and find the two point cloud feature points closest to each other, which are recorded as the nearest neighbor and the next nearest neighbor respectively; Calculate the distance ratio between the nearest neighbor point and the next nearest neighbor point. If the ratio is less than the set threshold, it is considered that the feature point in the image is successfully matched with the nearest neighbor point in the point cloud projection area. The threshold is 0.

8. The image and point cloud feature point pairs obtained through feature matching are summarized to form a set of matching point pairs, and the registration of point cloud data and image data is achieved based on the least squares method.

4. The high-precision laser radar mapping method based on multi-source data fusion according to claim 1, characterized in that: The registering of the point cloud data with the image data and the inertial navigation data respectively further includes: Convert point cloud data from the LiDAR device coordinate system to the geographic coordinate system; Select the initial corresponding point pairs, calculate the centroid of the point cloud point set and the centroid of the inertial navigation position point set in the corresponding point pairs, and obtain the centroid coordinates; The covariance matrix is ​​calculated based on the centroid coordinates, and the covariance matrix is ​​decomposed by the singular value decomposition method to obtain the optimal rotation moment and translation vector; The points in the point cloud data are rotated and translated according to the optimal rotation torque and translation vector, and the iterative process is repeated. After each iteration, the average distance between the corresponding point pairs is calculated to complete the registration of the point cloud data and the inertial navigation data.

5. The high-precision laser radar mapping method based on multi-source data fusion according to claim 1, characterized in that: The method of fusing the registered point cloud data, image data and inertial navigation data through the data fusion model, outputting a fusion result, and performing terrain mapping and analysis based on the fusion result includes: Collect the registered point cloud data, image data and inertial navigation data, convert the data from different data sources into a unified data format, and input it into the data fusion model; The image is divided into patches of fixed size and each patch is linearly embedded to obtain an embedding vector. After passing through a multi-layer multi-head self-attention mechanism and a feedforward neural network, the global and local features of the image are extracted. Through multiple layers of 3D sparse convolutional layers, the geometric and spatial features of point cloud data are extracted; Using the graph attention network, the feature representation of each node is calculated through the attention mechanism to extract the dynamic features and correlation features of the inertial navigation data; The image features, point cloud features and inertial navigation features are fused, and the final fusion result is obtained through the nonlinear transformation and feature mapping of the data fusion model; The terrain-related data is extracted from the fusion results to construct a digital elevation model, and terrain parameter calculation and terrain analysis are performed based on the digital elevation model.

6. The high-precision laser radar mapping system based on multi-source data fusion is characterized by: The system includes: The data acquisition module is used to obtain point cloud data and impact data of the target area, and record the attitude and position information during the data acquisition process through the inertial navigation device; The data processing module is used to perform denoising and downsampling on the point cloud data, geometric correction and enhancement on the image data, error correction on the inertial navigation data, and registration of the point cloud data with the image data and inertial navigation data respectively; A terrain mapping module is used to build a data fusion model, fuse the registered point cloud data, image data and inertial navigation data through the data fusion model, output the fusion results, and perform terrain mapping and analysis based on the fusion results; The constructing of the data fusion model includes: Obtain historical multi-source data from different sensors and determine the basic architectural parameters of the VisionTransformer based on this historical multi-source data, including patch size, embedding dimension, number of heads in the multi-head attention mechanism, and number of layers. The size of the sparse convolution kernel and the step size are set according to the spatial distribution range of the point cloud in the historical multi-source data to determine the convolution operation mode in the three-dimensional space to construct a 3D sparse convolution layer. The number of input and output channels of the sparse convolution is determined according to the feature dimension of the point cloud in the historical multi-source data. Taking points in historical multi-source data as nodes, the edge connection relationship is determined according to the spatial distance between points, and the historical multi-source data is constructed into a graph structure. The graph attention network is applied to the constructed graph to enable nodes to adaptively adjust their own feature weights based on the information of neighboring nodes; Integrate Vision Transformer, 3D sparse convolution, and graph attention network to obtain a multimodal search space; A set of learnable parameter vectors is assigned to each possible network architecture in the search space, and the mean square error after multi-source data fusion is used to measure the deviation between the fused data and the true value to determine the objective function; Input historical multi-source data and perform forward propagation according to the current parameterized network architecture to calculate the objective function value. Based on the objective function value, the network structure parameter vector is updated by backpropagation using a gradient-based optimization algorithm. After multiple rounds of iterative search, the network structure parameters that make the objective function value optimal are found, thereby automatically generating the optimal network structure and obtaining a fusion network; The hyperparameters adjusted by the two-layer optimization framework are applied to the fusion network to obtain the final data fusion model.

7. The high-precision laser radar mapping system based on multi-source data fusion according to claim 6, characterized in that: The data processing module includes: The statistical submodule is used to determine the neighborhood range of each point in the point cloud data, count the distances of all points in the neighborhood of each point in the point cloud data to the point, and obtain the point cloud data after preliminary denoising; The partitioning submodule is used to determine the voxel grid size according to the spatial distribution range of the point cloud data after preliminary denoising and the desired downsampling degree, remove the original points in each voxel grid unit and retain the center of gravity to complete the downsampling process; Establish a submodule for obtaining control points on the image data, establishing a transformation model, substituting the image coordinates and actual geographic coordinates of the control points into the transformation model and calculating the transformation parameters, transforming the coordinates of all points on the image data according to the transformation parameters, and completing the geometric correction process; The calculation submodule is used to obtain the grayscale histogram of the image data, calculate the cumulative distribution function, and remap the grayscale value according to the cumulative distribution function to complete the histogram equalization enhancement; The filtering submodule is used to perform real-time filtering on the inertial navigation data using the Kalman filtering algorithm based on the statistical characteristics of the inertial navigation system noise and observation noise, thereby removing noise and drift errors and completing error correction processing.

Images