A method for fusing three-dimensional laser point cloud of space targets with point cloud reconstruction using visible light images

By improving the multi-scale geometric element fusion feature extraction algorithm and multi-modal weak paired feature fusion algorithm, the problem of incomplete feature extraction in point cloud and image data fusion is solved, and the efficient fusion of three-dimensional laser point cloud and visible image is achieved. Accurate reconstruction.

CN118134787BActive Publication Date: 2024-11-15ZHONGJIAO ROAD & BRIDGE (HEBEI) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410471516.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-19
Publication Date
2024-11-15
Estimated Expiration
2044-04-19

AI Technical Summary

Technical Problem

In the fusion of point clouds and image data, it is difficult to effectively extract feature information of three-dimensional laser point clouds and visible light images, resulting in the incomplete and accurate three-dimensional scene information.

Method used

It proposes an improved multi-scale geometric fusion feature extraction algorithm and multi-modal weak paired feature fusion algorithm to extract local features of three-dimensional laser point cloud data through geometric primitives and nonlinear functions, and features are combined with the image scale and neighborhood information. Extraction and fusion, data fusion is performed using alternating maximization and Frobenius norm regularization optimization formulas.

Benefits of technology

Effective feature extraction and fusion of three-dimensional laser point clouds and visible image data is achieved, which improves the accuracy and comprehensiveness of three-dimensional target reconstruction and avoids overfitting of data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118134787B_ABST
    Figure CN118134787B_ABST
Patent Text Reader

Abstract

A method for fusing three-dimensional laser point cloud and visible light image reconstruction point cloud of a space target includes a data acquisition end, a data preprocessing end, a feature extraction and matching end, a point cloud and image fusion reconstruction end, and a GUI interface end. The data acquisition end is used to acquire three-dimensional laser point cloud data and visible light image data, the data preprocessing end is used to preprocess the acquired point cloud data and image data, the feature extraction and matching end is used to extract and match features of the point cloud data and image data, the point cloud and image fusion reconstruction end is used to fuse the point cloud and image, and the GUI interface end is used to visualize the fused data. The present invention proposes an improved multi-scale geometric primitive fusion feature extraction algorithm to extract features from point cloud data and image data, and proposes an improved multi-modal weak pairing feature fusion algorithm to fuse laser point cloud and visible light image, providing a better solution for a method for fusing three-dimensional laser point cloud and visible light image reconstruction point cloud of a space target.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of point cloud and image feature extraction and point cloud and image data fusion, and specifically to a method for fusing three-dimensional laser point cloud of a space target with a point cloud reconstructed by a visible light image. Background Art

[0002] The feature extraction technology of point cloud and image is a technology that extracts rich feature information from three-dimensional laser point cloud data and visible light images. It involves extracting key features from laser point cloud, such as surface normal, curvature and point cloud density, to describe the shape and structure information in the point cloud, extracting visual features from the image, such as color, texture and edge, to describe the information in the image, projecting the point cloud to the image plane to analyze the point cloud information on the image, and processing the point cloud using deep learning networks, such as PointNet and PointNet++, to extract the semantic information of the point cloud. The mutual cooperation of these technologies jointly constructs the feature extraction technology of point cloud and image, which can more effectively learn and express complex features.

[0003] Point cloud and image data fusion technology is a technology that fuses 3D laser point cloud data and visible light image data from different sensors to obtain more comprehensive and accurate 3D scene information. It involves accurately registering data from different sensors (cloud registration, image registration, and registration between point cloud and image), ensuring that point cloud and image data are aligned in the same coordinate system, and fusing data from different sensors or modalities to obtain more comprehensive information. Point clouds and images are jointly processed through deep learning. Deep learning structures such as convolutional neural networks (CNN) and recurrent neural networks (RNN) have also been widely used in point cloud and image fusion. These technologies together constitute the technical basis of the point cloud and visible light image data fusion method, which enables more comprehensive acquisition of scene information in spatial target reconstruction. Summary of the invention

[0004] In view of the above problems, the present invention aims to provide a method for fusing three-dimensional laser point cloud of space targets with point cloud reconstructed from visible light images.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a method for fusing a three-dimensional laser point cloud of a space target with a point cloud reconstructed by a visible light image, comprising a data acquisition end, a data preprocessing end, a feature extraction and matching end, a point cloud and image fusion reconstruction end and a GUI interface end, the data acquisition end comprising a three-dimensional laser point cloud data acquisition module and a visible light image data acquisition module, the three-dimensional laser point cloud data acquisition module is used to acquire three-dimensional laser point cloud data, the visible light image data acquisition module is used to acquire visible light image data, the data preprocessing end comprises a three-dimensional laser point cloud data preprocessing module and a visible light image data preprocessing module, the three-dimensional laser point cloud data preprocessing module is used to preprocess the acquired three-dimensional laser point cloud data, and the visible light image data preprocessing module is used to preprocess the acquired visible light image data Processing, the feature extraction and matching end includes a feature extraction module and a feature matching module. The feature extraction module proposes an improved multi-scale geometric primitive fusion feature extraction algorithm to extract features from three-dimensional laser point cloud data and visible light image data. The feature matching module is used for feature matching of three-dimensional laser point cloud data and visible light image data. The point cloud and image fusion reconstruction end includes a coordinate conversion module, a point cloud and image fusion module and a point cloud reconstruction and optimization module. The coordinate conversion module is used to unify the coordinate systems of the laser point cloud and the visible light image. The point cloud and image fusion module proposes an improved multi-modal weak pairing feature fusion algorithm to fuse the laser point cloud and the visible light image. The point cloud reconstruction and optimization module is used to reconstruct and optimize the fused data for three-dimensional targets. The GUI interface end is used to visualize the fused data.

[0006] Furthermore, the three-dimensional laser point cloud data acquisition module acquires three-dimensional laser point cloud data through a laser radar device.

[0007] Furthermore, the visible light image data acquisition module acquires visible light image data through a visible light camera.

[0008] Furthermore, the 3D laser point cloud data preprocessing module preprocesses the collected 3D laser point cloud data through denoising, filtering and coordinate transformation to improve the point cloud data quality.

[0009] Furthermore, the visible light image data preprocessing module preprocesses the collected visible light image data through camera calibration and dedistortion to improve the visible light image quality.

[0010] Furthermore, the feature extraction module proposes an improved multi-scale geometric primitive fusion feature extraction algorithm to perform feature extraction on three-dimensional laser point cloud data and visible light image data.

[0011] Furthermore, the improved multi-scale geometric primitive fusion feature extraction algorithm is as follows: In order to better adapt to the local characteristics of 3D laser point cloud data, weight coefficients and nonlinear functions are proposed to improve the distance function from the plane sampling point to the plane model, that is, in, For any plane sampling point s to the plane model θ s The distance function between p is the set of plane sampling points, s is the set S p Plane sampling points in , w s is the weight coefficient for adjusting the projection distance, f(·) is the nonlinear function for correcting the projection distance, [a s ,b s ,c s ] is the normal vector of the plane, a s 、b s 、c s is the component of the plane normal vector on the coordinate axis, [a s ,b s ,c s ] ′ is the transpose of the normal vector of the plane, h s is the vertical offset of any plane sampling point s in the direction of the plane normal vector, g(s,θ s ) is the local feature weighting item of the plane model. The distance function from the cylindrical sampling point to the cylindrical model is improved in the same way. Then, the local features of the three-dimensional laser point cloud data are extracted according to the distance function calculation results of different geometric primitives. In order to obtain richer image features, more scale and neighborhood information are proposed to improve the segmentation process. Then, based on the visible light image edge extraction result, the visible light image is divided into sub-bands on three layers with different scales, different directions and different neighborhoods. In order to better reflect the image features in a specific direction, a weight coefficient is proposed to improve the average value calculation of the four symbiotic quantities of image texture uniformity HOM, directionality AMS, complexity ENT and contrast DIS, so as to obtain the symbiotic features of the midpoints of the visible light image, so as to complete the feature extraction of the visible light image.

[0012] Furthermore, the feature matching module establishes a correspondence between the point cloud and the image through the features extracted by the feature extraction module, and performs feature matching between the point cloud and the image.

[0013] Furthermore, the coordinate conversion module is used to align the three-dimensional laser point cloud data and the visible light image data in the same coordinate system.

[0014] Furthermore, the point cloud and image fusion module proposes an improved multimodal weak pairing feature fusion algorithm to fuse three-dimensional laser point cloud data with visible light images.

[0015] Furthermore, the improved multimodal weak pairing feature fusion algorithm is as follows: In order to better express the features in the fused representation space, a weak pairing multimodal fusion is proposed to fuse the two modal data, that is, through nonlinear representation Map the 3D laser point cloud data, where F CP (CP) is the high-level feature nonlinear mapping function of 3D laser point cloud data, R is the real number domain, N cp is the dimension of the modal matrix CP, L l The dimension of the laser point cloud data after nonlinear representation is mapped in the same way as the visible light image data, so as to fuse the data of two different modes according to the two nonlinear representations. In order to prevent data overfitting, alternating maximization and Frobenius norm regularization are proposed to improve the solution formula of the nonlinear weak pairing multi-mode fusion optimization formula. After solving, reasonable source domain feature transformation matrix and target domain feature transformation matrix are obtained. By effectively executing the nonlinear representation of each mode, the obtained multi-modal matrices are combined to perform multi-modal data fusion of three-dimensional laser point cloud data and visible light image data.

[0016] Furthermore, the point cloud reconstruction and optimization module reconstructs and optimizes the three-dimensional target by using the fused data.

[0017] Furthermore, the visualization module visualizes the fused three-dimensional data through a provided user interface so as to intuitively observe the reconstruction effect.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] 1. An improved multi-scale geometric primitive fusion feature extraction algorithm is proposed to perform feature extraction on three-dimensional laser point cloud data and visible light image data. The innovation of the present invention lies in that the improved multi-scale geometric primitive fusion feature extraction algorithm first performs feature extraction on three-dimensional laser point cloud data through geometric primitives to better adapt to the shape characteristics and local structure of point cloud data, and then proposes weight coefficients and nonlinear functions to improve the distance function from plane sampling points to plane models and the distance function from cylinder sampling points to cylinder models to better adapt to the local characteristics of three-dimensional laser point cloud data, and then proposes more scale and neighborhood information to improve the non-subsampled contourlet transform image segmentation process to obtain richer image features, and finally proposes weight coefficients to improve the average value calculation of four symbiotic quantities to better reflect the image characteristics in a specific direction, so as to realize the feature extraction of three-dimensional laser point cloud data and visible light image data.

[0020] 1. An improved multimodal weak pairing feature fusion algorithm is proposed to fuse three-dimensional laser point cloud data with visible light images. The innovation of the present invention lies in that the improved multimodal weak pairing feature fusion algorithm first proposes weak pairing multi-mode fusion to fuse the two modal data so as to perform better feature expression in the fused representation space, and then proposes alternating maximization to find the local optimal solution of the optimization target, so as to focus more on the local optimization of the current variable and optimize each variable separately. Finally, Frobenius norm regularization is proposed to improve the solution formula to prevent data overfitting, and then a combination of multimodal matrices is obtained to realize the data fusion of three-dimensional laser point cloud data and visible light images. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The invention is further illustrated by the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0022] Figure 1 It is a schematic diagram of the structure of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The present invention provides a method for fusing a three-dimensional laser point cloud of a space target with a point cloud reconstructed by a visible light image, comprising a data acquisition end, a data preprocessing end, a feature extraction and matching end, a point cloud and image fusion reconstruction end and a GUI interface end, wherein the data acquisition end comprises a three-dimensional laser point cloud data acquisition module and a visible light image data acquisition module, the three-dimensional laser point cloud data acquisition module is used to acquire three-dimensional laser point cloud data, the visible light image data acquisition module is used to acquire visible light image data, the data preprocessing end comprises a three-dimensional laser point cloud data preprocessing module and a visible light image data preprocessing module, the three-dimensional laser point cloud data preprocessing module is used to preprocess the acquired three-dimensional laser point cloud data, the visible light image data preprocessing module is used to preprocess the acquired visible light image data, the feature extraction The matching end includes a feature extraction module and a feature matching module. The feature extraction module proposes an improved multi-scale geometric primitive fusion feature extraction algorithm to extract features from three-dimensional laser point cloud data and visible light image data. The feature matching module is used to perform feature matching on three-dimensional laser point cloud data and visible light image data. The point cloud and image fusion and reconstruction end includes a coordinate conversion module, a point cloud and image fusion module, and a point cloud reconstruction and optimization module. The coordinate conversion module is used to unify the coordinate systems of the laser point cloud and the visible light image. The point cloud and image fusion module proposes an improved multi-modal weak pairing feature fusion algorithm to fuse the laser point cloud and the visible light image. The point cloud reconstruction and optimization module is used to reconstruct and optimize the three-dimensional target of the fused data. The GUI interface end is used to visualize the fused data.

[0025] Preferably, the three-dimensional laser point cloud data acquisition module acquires three-dimensional laser point cloud data through a laser radar device.

[0026] Preferably, the visible light image data acquisition module acquires visible light image data through a visible light camera.

[0027] Preferably, the 3D laser point cloud data preprocessing module preprocesses the collected 3D laser point cloud data through denoising, filtering and coordinate transformation to improve the point cloud data quality.

[0028] Preferably, the visible light image data preprocessing module preprocesses the collected visible light image data through camera calibration and dedistortion to improve the visible light image quality.

[0029] Preferably, the feature extraction module proposes an improved multi-scale geometric primitive fusion feature extraction algorithm to perform feature extraction on three-dimensional laser point cloud data and visible light image data.

[0030] Specifically, the improved multi-scale geometric primitive fusion feature extraction algorithm is as follows: Assume that n three-dimensional laser point cloud data are D = {d1, d2, ..., d n}, where D is the set of 3D laser point cloud data, d1 is the first 3D laser point cloud data, d2 is the second 3D laser point cloud data, and d n is the nth 3D laser point cloud data, each 3D laser point cloud data contains a set of multiple 3D points (x, y, z), where (x, y, z) is the position of the laser point in space, x, y, z are the coordinates of the laser point on the three coordinate axes, and the 3D laser point cloud data is feature extracted through geometric primitives to better adapt to the shape characteristics and local structure of the point cloud data. It is assumed that the geometric primitives in the 3D laser point cloud are G = {S, M, θ, Dis}, where G is the set of geometric primitives in the 3D laser point cloud, S is the minimum support set of geometric primitives, M is the model generation function, θ is the model parameter corresponding to M, Dis is a function that measures the distance from the laser point to the model parameter θ, and for the point geometric primitive model, the sampling point is P m ={x m ,y m ,z m}, P n ={x n ,y n ,z n}, where P m is the mth sampling point, x m ,y m 、z m P m The coordinates on the three axes, P n is the nth sampling point, x n ,y n 、z n P n The coordinates on the three coordinate axes, the distance function between points is Among them, Dis P (P m ,P n ) is point P m With point P n The point distance function between them is: for the plane geometric primitive model, the plane sampling point is S p ={s1,s2,s3}, where S p is a set of plane sampling points, s1 is the first plane sampling point, s2 is the second plane sampling point, and s3 is the third plane sampling point. In order to better adapt to the local characteristics of 3D laser point cloud data, weight coefficients and nonlinear functions are proposed to improve the distance function from the plane sampling point to the plane model, that is, in, For any plane sampling point s to the plane model θ s The distance function between pis the set of plane sampling points, s is the set S p Plane sampling points in , w s is the weight coefficient for adjusting the projection distance, f(·) is the nonlinear function for correcting the projection distance, [a s ,b s ,c s ] is the normal vector of the plane, a s 、b s 、c s is the component of the plane normal vector on the coordinate axis, [a s ,b s ,c s ] ′ is the transpose of the normal vector of the plane, h s is the vertical offset of any plane sampling point s in the direction of the plane normal vector, g(s,θ s ) is the local feature weighting item of the plane model. For the cylindrical geometric primitive model, the cylindrical sampling point is C p ={c1,c2}, and the distance function from the cylinder sampling point to the cylinder model is also improved, that is, Among them, Dis C (c,θ c ) is the distance from any cylindrical sampling point c to the cylindrical model θ c The distance function between c is the weight coefficient for adjusting the distance, c0 is the axis point of the cylinder, v c is the axis vector of the cylinder, r c is the radius of the cylinder, l(c,θ c ) is the local feature weighting item of the cylinder model. Other types of geometric primitive models can be expanded on this basis, and then the local features of the 3D laser point cloud data are extracted according to the distance function calculation results of different geometric primitives;

[0031] Assuming that the visible light image data is u(x,y), the visible light image is segmented by non-subsampled contourlet transform, and more scale and neighborhood information is proposed to improve the segmentation process to obtain richer image features. The segmentation process is: Among them, j is the scale, k is the direction, q is the neighborhood, J is the number of feature components of the image at different scales, K is the number of feature components of the image in different directions, Q is the number of neighborhoods, and lf j,k,q is the high-frequency subband of the visible light image at j scale, k direction and q neighborhood, hf j,qis the low-frequency subband of the visible light image with j scale and q neighborhood. Based on the edge extraction result of the visible light image, the image is transformed into three layers. The visible light image is divided into subbands on the three layers with different scales, different directions and different neighborhoods. The total number of subbands with the same size as the original visible light image is obtained. Each directional subband represents the visible light image texture in one direction. Then, the low-frequency subband of the visible light image is uniformly quantized with a quantization level of T. The uniform quantization process is: Among them, G is the quantization result, Z is the low-frequency subband of the visible light image, and Z min is the minimum value of the low-frequency subband value of the visible light image, Z max is the maximum value of the low-frequency subband value of the visible light image. The subbands that affect the details of visible light are non-uniformly quantized with a quantization level of T. The variance σ of each subband at the same scale is estimated. l , ±3σ is used as the limit value during quantization, so the non-uniform quantization process is Among them, NG is the result of non-uniform quantization, α, β, γ and δ are all adjustment parameters of non-uniform quantization, Z(j,k,q) is the influencing detail subband value of the visible light image at j scale, k direction and q neighborhood, representing the texture information of the visible light image at j scale, k direction and q neighborhood, and then the window traversal mode and window size are selected. The window traversal image mode uses overlapping windows, and uses windows of different sizes at different segmentation scales. The first scale subband image of the visible light image uses N=3×3, and the second scale subband image and the low-frequency subband use windows of size N=5×5, and then proceed Row distance parameter setting,The setting of segmentation parameters changes the number of segmentation directions, and the difference between the proportions will change with the change of the direction parameters. Therefore, when the classification sub-band is used for grayscale symbiosis, different distance intervals are used for different scales. Finally, the direction selection is performed: the four symbiosis quantities of image texture uniformity HOM, directionality AMS, complexity ENT and contrast DIS are calculated for each sub-band at four angles of 0°, 45°, 90°, and 135°. In order to better reflect the image features in a specific direction, a weight coefficient is proposed to improve the average calculation of the four symbiosis quantities, namely Among them, MV is the average value of the symbiosis amount calculated in four different directions, w 0 and f 0 is the weight coefficient and symbiotic value in the 0° direction, w 45 and f 45 is the weight coefficient and symbiotic value in the 45° direction, w 90 and f 90 is the weight coefficient and symbiotic value in the 90° direction, w 135 and f 135is the weight coefficient and symbiosis value in the 135° direction, and then the symbiosis feature NG(j,kk,q) of the point (j,k,q) in the visible light image is obtained, that is, NG(j,k,q)=[HOM i ,AMS i ,ENT i ,DIS i ], among which, HOM i is the uniformity in the ith direction, AMS i is the directionality in the i-th direction, ENT i is the complexity in the i-th direction, DIS i is the contrast in the i-th direction, so as to complete the feature extraction of visible light images. The improved multi-scale geometric primitive fusion feature extraction algorithm first extracts features of 3D laser point cloud data through geometric primitives to better adapt to the shape characteristics and local structure of point cloud data, and then proposes weight coefficients and nonlinear functions to improve the distance function from plane sampling points to plane models and the distance function from cylinder sampling points to cylinder models to better adapt to the local characteristics of 3D laser point cloud data, and then proposes more scale and neighborhood information to improve the non-subsampled contourlet transform image segmentation process to obtain richer image features, and finally proposes weight coefficients to improve the average value calculation of four symbiotic quantities to better reflect the image characteristics in a specific direction, so as to realize the feature extraction of 3D laser point cloud data and visible light image data.

[0032] Preferably, the feature matching module establishes a correspondence between the point cloud and the image through the features extracted by the feature extraction module, and performs feature matching between the point cloud and the image.

[0033] Preferably, the coordinate conversion module converts the coordinates of the three-dimensional laser point cloud data and the visible light image data from their local coordinate systems to the global coordinate system so as to align the two different data in the same coordinate system.

[0034] Preferably, the point cloud and image fusion module proposes an improved multimodal weak pairing feature fusion algorithm to fuse three-dimensional laser point cloud data with visible light images.

[0035] Specifically, the improved multimodal weak pairing feature fusion algorithm is as follows: 3D laser point cloud data and visible light image data belong to different modalities. Assume that the modality of 3D laser point cloud data is The modality of visible light image data is Among them, CP is the modal matrix of 3D laser point cloud data, VP is the modal matrix of visible light image data, R is the real number domain, N cp is the dimension of the modal matrix CP, N vpis the dimension of the modality matrix VP. In order to better express the features in the fused representation space, a weak pairing multi-modal fusion is proposed to fuse the two modal data, that is, through nonlinear representation Map the 3D laser point cloud data, where F CP (CP) is the advanced feature nonlinear mapping function of 3D laser point cloud data, L l The dimension after nonlinear representation of laser point cloud data. Map the visible light image data, where F VP (VP) is the high-level feature nonlinear mapping function of visible light image data, which fuses the data of two different modalities through these two nonlinear representations. The nonlinear weak pairing multi-modal fusion optimization formula is: Among them, QT is the optimization target, min(·) is the minimum value function, F VP (VP) is the transpose, tr is the trace of the matrix, A, B and Π are all optimization parameter matrices, A is the source domain feature transformation matrix, A T is the transposed matrix of the source domain feature transformation matrix A, B is the target domain feature transformation matrix, Π is the transfer matrix of the source domain feature mapping process to the target domain, and there is no single closed form solution that can exist simultaneously in the optimization parameter matrices A, B and Π. Therefore, in order to focus more on the local optimization of the current variable and optimize each variable separately, alternating maximization is proposed to find the local optimal solution of the optimization target. At the same time, in order to prevent data overfitting, Frobenius norm regularization is proposed to improve the solution formula, that is: when Π is known, the solution formulas of A and B are Among them, argmax(·) is The values ​​of A and B corresponding to the maximum value, is the data feature fusion process, that is, the modal CP of the 3D laser point cloud data and the modal VP of the visible light image data are mapped to the same high-dimensional feature space through their respective nonlinear mapping functions, and then fused through the feature transformation matrices A and B to obtain a final fusion feature. λ is the regularization parameter. is the Frobenius norm of matrix A, is the Frobenius norm of matrix B. For known A and B, the solution formula of Π is By solving the above formula, we can obtain a reasonable source domain feature transformation matrix A and target domain feature transformation matrix B. Since the target value of formula QT will increase monotonically during the iteration process, a natural stopping criterion is provided for the target value to stabilize. By effectively executing the nonlinear representation of each modality, the obtained multimodal matrices are combined to perform multimodal data fusion of three-dimensional laser point cloud data and visible light image data. The improved multimodal weak pairing feature fusion algorithm first proposes weak pairing multimodal fusion to fuse the two modal data for better feature expression in the fused representation space, and then proposes alternating maximization to find the local optimal solution of the optimization target, so as to focus more on the local optimization of the current variable and optimize each variable separately. Finally, Frobenius norm regularization is proposed to improve the solution formula to prevent data overfitting, and then a combination of multimodal matrices is obtained to realize data fusion of three-dimensional laser point cloud data and visible light image.

[0036] Preferably, the point cloud reconstruction and optimization module reconstructs and optimizes the three-dimensional target by using the fused data.

[0037] Preferably, the visualization module visualizes the fused three-dimensional data through a provided user interface so as to intuitively observe the reconstruction effect.

[0038] A method for fusing a three-dimensional laser point cloud of a space target with a point cloud reconstructed from a visible light image is proposed, which is used to fuse three-dimensional laser point cloud data and visible light images. A method for fusing a three-dimensional laser point cloud of a space target with a point cloud reconstructed from a visible light image is provided through the fusion of a data acquisition end, a data preprocessing end, a feature extraction and matching end, a point cloud and image fusion reconstruction end, and a GUI interface end. An improved multi-scale geometric primitive fusion feature extraction algorithm is proposed to extract features from three-dimensional laser point cloud data and visible light image data. The innovation of the present invention lies in that the improved multi-scale geometric primitive fusion feature extraction algorithm first extracts features from three-dimensional laser point cloud data through geometric primitives to better adapt to the shape features and local structures of point cloud data, and then proposes weight coefficients and nonlinear functions to improve the distance function from plane sampling points to plane models and the distance function from cylinder sampling points to cylinder models to better adapt to the local features of three-dimensional laser point cloud data, and then proposes more scale and neighborhood information to improve the non-subsampled contourlet transform image segmentation process to obtain richer image features, and finally proposes weight coefficients to improve the average value calculation of four symbiotic quantities to better reflect the image features in a specific direction, so as to achieve three The invention proposes a method for extracting features of three-dimensional laser point cloud data and visible light image data, and proposes an improved multimodal weak pairing feature fusion algorithm to fuse three-dimensional laser point cloud data with visible light images. The innovation of the invention lies in that the improved multimodal weak pairing feature fusion algorithm first proposes weak pairing multi-mode fusion to fuse the two modal data to perform better feature expression in the fused representation space, and then proposes alternating maximization to find the local optimal solution of the optimization target, so as to focus more on the local optimization of the current variable and optimize each variable separately. Finally, the Frobenius norm regularization is proposed to improve the solution formula to prevent data overfitting, and then the combination of multimodal matrices is obtained to realize the data fusion of three-dimensional laser point cloud data and visible light images, effectively improve the working effect of a method for reconstructing point cloud of three-dimensional laser point cloud and visible light image of space target, provide more comprehensive and accurate technical support for the fusion of three-dimensional laser point cloud and visible light image, and provide better decision support for the scientific and efficient fusion of three-dimensional laser point cloud and visible light image. At the same time, the invention relates to point cloud and image feature extraction and point cloud and image data fusion technology, provides people with an efficient and convenient fusion method, and contributes important application value to the field of data fusion.

[0039] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for fusing three-dimensional laser point cloud of space target with point cloud reconstruction of visible light image, characterized in that: The invention comprises a data acquisition end, a data preprocessing end, a feature extraction and matching end, a point cloud and image fusion reconstruction end and a GUI interface end. The data acquisition end comprises a 3D laser point cloud data acquisition module and a visible light image data acquisition module. The 3D laser point cloud data acquisition module is used to acquire 3D laser point cloud data, and the visible light image data acquisition module is used to acquire visible light image data. The data preprocessing end comprises a 3D laser point cloud data preprocessing module and a visible light image data preprocessing module. The 3D laser point cloud data preprocessing module is used to preprocess the acquired 3D laser point cloud data, and the visible light image data preprocessing module is used to preprocess the acquired visible light image data. The feature extraction and matching end comprises a feature extraction module and a feature matching module. The feature extraction module proposes an improved multi-scale geometric primitive fusion feature extraction algorithm to extract features from three-dimensional laser point cloud data and visible light image data. The feature matching module is used for feature matching of three-dimensional laser point cloud data and visible light image data. The point cloud and image fusion reconstruction end includes a coordinate conversion module, a point cloud and image fusion module, and a point cloud reconstruction and optimization module. The coordinate conversion module is used to unify the coordinate systems of the laser point cloud and the visible light image. The point cloud and image fusion module proposes an improved multi-modal weak pairing feature fusion algorithm to fuse the laser point cloud and the visible light image. The point cloud reconstruction and optimization module is used to reconstruct and optimize the fused data for three-dimensional targets. The GUI interface end is used to visualize the fused data. The feature extraction module proposes an improved multi-scale geometric primitive fusion feature extraction algorithm to extract features from 3D laser point cloud data and visible light image data; The improved multi-scale geometric primitive fusion feature extraction algorithm is as follows: In order to better adapt to the local characteristics of 3D laser point cloud data, weight coefficients and nonlinear functions are proposed to improve the distance function from the plane sampling point to the plane model, that is, ,in, Any plane sampling point To plane model The distance function between is the set of plane sampling points, For collection The plane sampling points in To adjust the weight coefficient of projection distance, is the nonlinear function to correct the projection distance, is the normal vector of the plane, , , is the component of the plane’s normal vector on the coordinate axis, is the transpose of the plane normal vector, Any plane sampling point The vertical offset in the direction of the plane's normal vector, The local feature weighting term of the plane model is used. The distance function from the cylindrical sampling point to the cylindrical model is improved in the same way. Then, the local features of the 3D laser point cloud data are extracted according to the distance function calculation results of different geometric primitives. In order to obtain richer image features, more scale and neighborhood information is proposed to improve the segmentation process. Then, based on the visible light image edge extraction results, the visible light image is divided into sub-bands on three layers with different scales, different directions and different neighborhoods. In order to better reflect the image features in a specific direction, the weight coefficient is proposed to improve the uniformity of the image texture. Directionality , Complexity and contrast The average value calculation of these four symbiotic quantities is improved to obtain the symbiotic features of the midpoints of the visible light image, so as to complete the feature extraction of the visible light image; The coordinate conversion module is used to align the 3D laser point cloud data and the visible light image data in the same coordinate system; The point cloud and image fusion module proposes an improved multimodal weak pairing feature fusion algorithm to fuse 3D laser point cloud data with visible light images; The improved multimodal weak pairing feature fusion algorithm is specifically as follows: In order to perform better feature expression in the fused representation space, a weak pairing multimodal fusion is proposed to fuse the two modal data, that is, through nonlinear representation Map the 3D laser point cloud data, where: It is a high-level feature nonlinear mapping function for 3D laser point cloud data. is the field of real numbers, is the modal matrix The dimension of The dimension of the laser point cloud data after nonlinear representation is mapped in the same way to the visible light image data, so as to fuse the data of two different modes according to the two nonlinear representations. In order to prevent data overfitting, alternating maximization and Frobenius norm regularization are proposed to improve the solution formula of the nonlinear weak pairing multi-mode fusion optimization formula. After solving, reasonable source domain feature transformation matrix and target domain feature transformation matrix are obtained. By effectively executing the nonlinear representation of each mode, the obtained multi-modal matrices are combined to perform multi-modal data fusion of three-dimensional laser point cloud data and visible light image data.

2. The method for fusing three-dimensional laser point cloud of space target with point cloud reconstructed from visible light image according to claim 1, characterized in that: The 3D laser point cloud data acquisition module collects 3D laser point cloud data through laser radar equipment; the point cloud reconstruction and optimization module reconstructs and optimizes the 3D target through the fused data; the visualization module visualizes the fused 3D data through the provided user interface.

3. The method for fusing three-dimensional laser point cloud of space target with point cloud reconstructed from visible light image according to claim 1, characterized in that: The visible light image data acquisition module acquires visible light image data through a visible light camera.

4. The method for fusing three-dimensional laser point cloud of space target with visible light image reconstruction point cloud according to claim 1, characterized in that: The 3D laser point cloud data preprocessing module preprocesses the collected 3D laser point cloud data through denoising, filtering and coordinate transformation to improve the point cloud data quality.

5. The method for fusing three-dimensional laser point cloud of space target with point cloud reconstructed from visible light image according to claim 1, characterized in that: The visible light image data preprocessing module preprocesses the collected visible light image data through camera calibration and dedistortion to improve the visible light image quality.

6. The method for fusing three-dimensional laser point cloud of space target with visible light image reconstruction point cloud according to claim 1, characterized in that: The feature matching module establishes the correspondence between the point cloud and the image through the features extracted by the feature extraction module, and performs feature matching between the point cloud and the image.

7. The method for fusing three-dimensional laser point cloud of space target with point cloud reconstructed from visible light image according to claim 1, characterized in that: The coordinate transformation module is used to align the 3D laser point cloud data and the visible light image data in the same coordinate system.