A multi-piece fine-grained dense matching method guided by laser point cloud

Abstract

The present invention discloses a multi-piece refined dense matching method guided by laser point cloud. The existing real-scene three-dimensional reconstruction process is prone to problems such as information loss and large noise fluctuations. The present invention includes obtaining the basic situation of the survey area, and designing a solution for collaboratively collecting oblique image data and laser point cloud data; collecting multi-source data to complete the coarse matching between heterogeneous data; through the precise matching of the coordinate system, making the final residual error of the heterogeneous perception data system below the sub-ground resolution; initializing the same-name points, anchoring the laser point cloud data in each image; depth map diffusion, depth map fusion, eliminating redundant observations; noise filtering and compensation, eliminating fluctuation points, and then back-projecting them into each depth map to fill the holes; and finally exporting the dense point cloud. The present invention solves the problems of unreliable and poor-quality data results generated by a single data source, as well as the superposition of redundant data and the submergence of high-precision geometric information.

Description

Technical Field

[0001] The present invention belongs to the field of surveying technology, and in particular relates to a multi-piece refined dense matching method guided by laser point cloud. Background Art

[0002] Real-world 3D spatial information serves as a crucial geographic information base for surveying and digital twins. It provides the data foundation for real-world intelligent applications and augmented reality services, and is a key technical enabler for the development of a real-world China and the digital twin ecosystem. However, current real-world 3D perception methods face the technical challenge of overcoming the contradiction between efficiency and accuracy, severely hindering their potential contributions and hindering the digitalization of surveying. Consequently, a compromise solution is urgently needed. Given this significant demand, efficient and accurate acquisition of real-world 3D information is a key research hotspot within the surveying industry, and a pressing issue in other interdisciplinary fields.

[0003] Existing methods for perceiving real-world information primarily include passive image remote sensing and active laser scanning technology. Passive image remote sensing technology utilizes overlapping multi-directional two-dimensional image acquisition to reconstruct three-dimensional real-world information from multiple two-dimensional information sources. This technology boasts high acquisition efficiency, low cost, and high resolution. However, it is limited by two-dimensional acquisition and environmental constraints, and the three-dimensional reconstruction of real-world images is prone to information loss, large noise fluctuations, and long reconstruction times. Active laser scanning technology fully addresses the shortcomings of all remote sensing image perception technologies. However, due to the high cost of fine scanning equipment, low acquisition efficiency, and a lack of real-world texture information, its widespread application in surveying has been severely hampered. The maturity of laser scanning equipment and technology in recent years has only just barely resolved the high equipment investment cost issue. However, compared to passive remote sensing technology, it still differs significantly in terms of accessibility, convenience, high resolution, and real-world perception.

[0004] To address this issue, the "LiDAR Point Cloud-Assisted Dense Matching Method and System for Stereo Images" [Publication No.: CN105160702A] proposes using laser point clouds to constrain the depth value range of stereo pairs and establish a depth value pyramid range to reduce the number of dense matching iterations. However, this method does not consider the initial error between the image and the laser point cloud. The forced parallax will cause interference in areas with sudden depth changes. In addition, these two stereo matching methods have observation biases, and there is an overall difference in effect compared to the multi-piece stereo matching method, which has limited improvement in the dense matching effect.

[0005] "Dense Matching Method for Stereo Images Based on Global Block Optimization" [Publication No.: CN105160702A] proposes a least squares global dense matching method based on superpixel segmentation constraints to solve the "parallax ladder" problem that is prevalent in traditional stereo image dense matching algorithms, thereby generating a high-precision dense point cloud. However, this method cannot resist the radiation differences caused by ambient light, resulting in superpixel constraint failure and instead producing an inverse noise effect.

[0006] "A Dense Matching Method for Remote Sensing Images Based on an Accurate Point Prediction Model" [Publication No.: CN104299228A] proposes a method based on hierarchical sparse matching points to predict the positions of points of the same name in different images step by step, thereby diffusing the depth map that fills all images and completing dense matching depth filling. However, this method still only relies on the information of the image itself and is subject to information environment interference problems. In addition, due to the sparse density of the point cloud generated by the initial aerial triangulation, the reliable acceleration effect of dense matching is limited. Summary of the Invention

[0007] In order to make up for the shortcomings of the existing technology, the present invention provides a laser point cloud guided multi-piece refined dense matching method, which solves the problems of unreliable and poor quality data results generated by a single data source and the superposition of redundant data and the submersion of high-precision geometric information.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A laser point cloud guided multi-piece refined dense matching method, characterized by comprising the following steps:

[0010] Step 1: Survey the survey area according to the project requirements and obtain basic information about the survey area;

[0011] Step 2: Design a solution for collaboratively collecting oblique image data and laser point cloud data based on site conditions and design requirements;

[0012] Step 3: Collect multi-source data to obtain remote sensing image data with appropriate resolution and high-precision laser point cloud data with density distribution;

[0013] Step 4: Perform aerial triangulation on the remote sensing image data to obtain the precise mapping relationship between any image data and the object point; pre-process the laser point cloud data to remove noise data and complete the coarse matching process between heterogeneous data;

[0014] Step 5: Through the coordinate system fine-tuning process, the residual errors between the heterogeneous perception data systems are corrected and eliminated, so that the final residual errors of the heterogeneous perception data systems are below sub-ground resolution.

[0015] Step 6: Initialize the points with the same name and anchor high-precision laser point cloud data in each image;

[0016] Step 7: Depth map diffusion, starting from the anchor point position of the same name in each image and diffusing in parallel to fill the entire image;

[0017] Step 8: Depth map fusion, clustering and fusion of the depth map corresponding to each image in the object space to eliminate redundant observations;

[0018] Step 9: Noise filtering and compensation: noise smoothing filtering is performed on the depth map object fusion result to remove fluctuation points, and then back-projected to each depth map to compensate for the holes caused by texture;

[0019] Step 10: Export dense point cloud.

[0020] Furthermore, the step five includes:

[0021] S5.1: Use the laser point cloud data to project the window grayscale value information in different images, accurately fine-tune the image space reference system and the laser point cloud reference system, and calculate the fine-tuning conversion parameters. The fine-tuning conversion parameters between remote sensing image data and laser point cloud data are:

[0022]

[0023]

[0024] Where f(*) is the grayscale value of the reference image, g(*) is the grayscale value of the image to be matched, (x, y) is the coordinate of the object point p in the window [w×w] in the image, which can be determined by the known projection matrix of the image and the camera distortion parameters, (a0, a1, a2, b0, b1, b2,) is the matching coefficient, and L(p) is the grayscale value of the object point p in the observable image set V. p Gray value observation error in ;

[0025]

[0026] T(p) is the coordinate transformation fine-tuning matrix for the object point p. When there is a deviation between the object point cloud coordinate system and the known image coordinate system, this prefabricated fine-tuning matrix can be used to correct it. E is the error energy function between the matching coefficient and the coordinate transformation fine-tuning matrix obtained based on the grayscale value residual of the same-name point.

[0027] S5.2: During the precise fine-tuning of the image space reference system and the laser point cloud reference system, a least-squares dynamic matching adjustment method is used to iteratively update the mutual mapping geometric transformation parameters of different same-name point windows in the visual image set until the optimal error is within the controllable range. At this time, the corresponding data space reference system is converted to the optimal precise matching parameters.

[0028] Further, the step seven includes:

[0029] 7.1 Diffusion of the image depth map: Initialize the neighboring pixels without depth values radially in the current pixel window; provide an initialized depth information and normal vector information for the current pixel value within the neighborhood range, calculate its evaluation value according to formula (4), and then optimize and solve the optimal depth value and normal vector value of the current pixel value.

[0030]

[0031]

[0032] Where σ y is the variance, reference image I r , the image set to be matched {I i}, N(p) is the neighborhood set of a certain object point p in the reference image; diff(x, y) is a smooth constraint function used to measure the difference between observations. is the cross-correlation function between the reference image and the image to be matched; d(p) and n(p) are the depth value and normal vector of the object point p in the reference image, and d(q) and n(q) are the depth value and normal vector of the object point q in the reference image; The depth score evaluation value of the object point p in the reference image;

[0033] 7.2 All pixels of non-anchor points with the same name need to be evaluated iteratively through the evaluation function, and the optimal depth value is optimized and selected as the current best solution.

[0034] Beneficial effects of the present invention:

[0035] 1) This invention fully utilizes the rich texture information of remote sensing images and the high-precision point cloud information of lidar, solving the problems of unreliable and poor quality data generated by a single data source;

[0036] 2) This paper proposes a dense matching method for deeply coupled image information and laser point clouds, which avoids the problems of redundant data superposition and the submergence of high-precision geometric information. It can also accelerate the global dense process and avoid the situation where the local minimum is regarded as the optimal solution.

[0037] 3) The present invention aims at efficient and accurate real-scene three-dimensional perception technology. On the basis of image perception, it collaboratively integrates and utilizes three-dimensional laser scanning technology, so that the two types of perception data constrain each other to make up for inherent limitations. Moreover, this fusion is a tightly coupled method, avoiding the problem of uneven scale and precision distribution brought about by the non-traditional loose coupling mode, which results in insufficient utilization of three-dimensional laser point cloud information, low signal-to-noise ratio of real-scene point cloud, and large amount of redundant data for subsequent network construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Generate a flowchart for the laser point cloud guided multi-piece refined dense matching method;

[0039] Figure 2 Schematic diagram of precise matching transformation between remote sensing image and laser point cloud;

[0040] Figure 3 It is the constraint graph of the anchor points with the same name in the laser point cloud;

[0041] Figure 4 Schematic diagram of depth value diffusion;

[0042] Figure 5 Schematic diagram of optimal depth value evaluation. DETAILED DESCRIPTION

[0043] The present invention will be described in detail below with reference to specific embodiments.

[0044] This invention incorporates the excellent features of remote sensing images and laser point clouds, making up for the problems of poor geometric accuracy, resolution, and texture realism in a single data source. At the same time, compared with the multi-piece stereo dense matching method, it produces more accurate point cloud effects and efficiency effects, achieving the goal of improving the quality and efficiency of three-dimensional information perception of image data.

[0045] like Figure 1 As shown, the present invention specifically includes the following steps:

[0046] Step 1: Conduct a detailed survey of the survey area according to the requirements of the project technical indicators to obtain basic information about the survey area, such as the scope, trend, and undulation of the survey area;

[0047] Step 2: Based on the project requirements and specific conditions of the survey area, the laser scanner and image sensor parameters are integrated to design a collaborative acquisition technology solution that meets at least a 4-degree overlap. This solution also provides high-precision platform-assisted positioning and target control information for the collaborative acquisition platform.

[0048] Step 3: Collect multi-source data to obtain remote sensing image data with appropriate resolution and high-precision laser point cloud data with density distribution;

[0049] Step 4: Perform aerial triangulation on the remote sensing image data to obtain the precise mapping relationship between any image data and the object point; this includes the internal and external orientation elements of the remote sensing image data and the camera distortion parameters;

[0050] Pre-process the laser point cloud data, remove noise data, and integrate the laser point cloud data into a unified spatial reference framework system for remote sensing image data to complete the coarse matching process between heterogeneous data; this allows the remote sensing image data to have spatial posture attributes, and the laser point cloud data to complete the fusion filtering and geolocation process;

[0051] Step 5: Through the coordinate system fine matching process, the high-precision laser point cloud data and remote sensing image data are precisely matched to correct and eliminate the residual errors between the heterogeneous perception data systems; the final residual error of the heterogeneous perception data system is made below the sub-ground resolution, and the deviations between the heterogeneous data systems are solved, such as Figure 2 As shown;

[0052] Step five includes:

[0053] S5.1: The coordinate system fine-tuning process is to use the laser point cloud data to project the window gray value information in different images, accurately adjust the image space reference system and the laser point cloud reference system, and calculate the fine-tuning conversion parameters. The fine-tuning conversion parameters between remote sensing image data and laser point cloud data are:

[0054]

[0055]

[0056] Where f(*) is the grayscale value of the reference image, g(*) is the grayscale value of the image to be matched, (x, y) is the coordinate of the object point p in the window [w×w] in the image, which can be determined by the known projection matrix of the image and the camera distortion parameters, (a0, a1, a2, b 0， b1, b2) are matching coefficients, L(p) is the object point p in the observable image set V p Gray value observation error in ;

[0057]

[0058] T(p) is the coordinate transformation fine-tuning matrix for the object point p. When there is a deviation between the object point cloud coordinate system and the known image coordinate system, this prefabricated fine-tuning matrix can be used to correct it. E is the error energy function between the matching coefficient and the coordinate transformation fine-tuning matrix obtained based on the grayscale value residual of the same-name point.

[0059] S5.2: During the precise fine-tuning of the image space reference system and the laser point cloud reference system, a least-squares dynamic matching adjustment method is used to iteratively update the mutual mapping geometric transformation parameters of different same-name point windows in the visual image set until the optimal error is within the controllable range. At this time, the corresponding data space reference system is converted to the optimal precise matching parameters.

[0060] The matching coefficients are iteratively updated to provide reliable multi-source observation window data information for fine-tuning the transformation parameters. The minimum residual constraint of Formula 1 is used to ensure that the shape of the window with the same name in each image is affine invariant. In order to prevent the influence of the selection of the reference image on the residual data, a reference value image rotation strategy is adopted, and the reference image corresponding to the minimum grayscale error is selected as the best reference image.

[0061] Step 6: Initialize the same-name points and anchor high-precision laser point cloud data in each image to provide depth constraints for subsequent use; Figure 3 As shown in the figure, all high-precision dense point clouds are projected onto each remote sensing image to provide quasi-dense homonymous point control information for the image;

[0062] Step 7: Depth map diffusion, starting from the anchor point position of the same name in each image and diffusing in parallel to fill the entire image;

[0063] Step seven includes:

[0064] 7.1 Diffusion of the image depth map: radially initialize neighboring pixels without depth values in the current pixel window; Figure 4 As shown, an initialized depth information and normal vector information is provided for the current pixel value within the neighborhood range, and its evaluation value is calculated according to formula (4), and then the optimal depth value and normal vector value of the current pixel value are optimized and solved.

[0065]

[0066]

[0067] Where, σ y is the variance, reference image I r , the image set to be matched {I i}, N(p) is the neighborhood set of a certain object point p in the reference image; diff(x, y) is a smooth constraint function used to measure the difference between observations. is the cross-correlation function between the reference image and the image to be matched; d(p) and n(p) are the depth value and normal vector of the object point p in the reference image, and d(q) and n(q) are the depth value and normal vector of the object point q in the reference image; The depth score evaluation value of the object point p in the reference image;

[0068] 7.2 All pixels of non-anchor points with the same name need to be evaluated iteratively through the evaluation function, and the optimal depth value is optimized to be the current best solution, such as Figure 5 As shown in the figure. By radially initializing the depth values, an initial depth map is provided for each image. However, due to low smoothness, the data suffers from significant jumps in local accuracy. Therefore, each pixel must undergo depth value residual diffusion optimization within its neighborhood until no pixel depth value changes in the image. The depth value residual diffusion optimization process also uses the depth value range within the neighborhood to re-estimate and optimize the new evaluation value of the current pixel. If it is smaller than the current old evaluation value, it indicates that the optimization is insufficient and the iteration continues.

[0069] Step 8: Depth map fusion: Cluster the depth maps corresponding to each image in the object space to eliminate redundant observations. Use meanshift cluster fusion to perform weighted fusion within the adaptive ground resolution range to form a true value data, avoiding the generation of redundant data.

[0070] Step 9: Noise filtering and compensation: noise smoothing filtering is performed on the depth map object fusion result to remove fluctuation points, and then back-projected to each depth map to compensate for the holes caused by texture;

[0071] Step 10: Dense point cloud export. Structural optimization and texture assignment are performed before high-precision dense export to further improve the quality of the point cloud data and its real-scene perception capabilities, providing rich and accurate input data for subsequent point cloud meshing.

[0072] Application example: Using oblique images and high-precision lidar data from a certain area, the patented laser point cloud-guided multi-piece refined dense matching method of this invention provides quasi-dense matching anchor point homonymous depth constraint information for image dense matching. Compared with sparse matching points based on aerial triangulation, this method can provide at least 10 times more constraint information, directly accelerating the dense matching efficiency by dozens of times. The generated point cloud data has a significant improvement in geometric flatness and edge sharpness.

[0073] The content of the present invention is not limited to the embodiments listed. Any equivalent transformation of the technical solution of the present invention made by ordinary technicians in this field after reading the description of the present invention is covered by the claims of the present invention.

Claims

1. A laser point cloud guided multi-piece refined dense matching method, characterized by: The steps include: Step 1: Survey the survey area according to project requirements and obtain survey area data; Step 2: Design a solution for collaboratively collecting oblique image data and laser point cloud data based on site conditions and design requirements; Step 3: Collect multi-source data to obtain remote sensing image data with appropriate resolution and high-precision laser point cloud data with density distribution; Step 4: Perform aerial triangulation on the remote sensing image data to obtain the precise mapping relationship between any image data and the object point; pre-process the laser point cloud data to remove noise data, and integrate the laser point cloud data into the unified spatial reference framework system of remote sensing image data to complete the coarse matching process between heterogeneous data; Step 5: Through the coordinate system fine-tuning process, the residual errors between the heterogeneous perception data systems are corrected and eliminated, so that the final residual errors of the heterogeneous perception data systems are below sub-ground resolution. Step 6: Initialize the points with the same name and anchor high-precision laser point cloud data in each image; Step 7: Depth map diffusion, starting from the anchor point position of the same name in each image and diffusing in parallel to fill the entire image; Step 8: Depth map fusion, clustering and fusion of the depth map corresponding to each image in the object space to eliminate redundant observations; Step 9: Noise filtering and compensation: noise smoothing filtering is performed on the depth map object fusion result to remove fluctuation points, and then back-projected to each depth map to compensate for the holes caused by texture; Step 10: Export dense point cloud.

2. The laser point cloud-guided multi-piece refined dense matching method according to claim 1, characterized in that: The step five includes: S5.1: Use the laser point cloud data to project the window grayscale value information in different images, accurately fine-tune the image space reference system and the laser point cloud reference system, and calculate the fine-tuning conversion parameters. The fine-tuning conversion parameters between remote sensing image data and laser point cloud data are: (Formula 1) (Formula 2) Where, (*) is the grayscale value of the reference image, g (*) is the grayscale value of the image to be matched, (x, y) is the object point p Window in Image The coordinates inside are determined by the known projection matrix of the image and the camera distortion parameters. is the matching coefficient, L (p) is the object point p In the observable image collection Gray value observation error in ; (Formula 3) For the object point p The coordinate conversion fine-tuning transformation matrix is used. When there is a deviation between the object point cloud coordinate system and the known image coordinate system, it is corrected by this pre-made fine-tuning transformation matrix. E The error energy function of solving the matching coefficient and fine-tuning the transformation matrix of coordinate transformation is obtained based on the residual gray value of the same-name points; S5.2: During the precise fine-tuning of the image space reference system and the laser point cloud reference system, a least-squares dynamic matching adjustment method is used to iteratively update the mutual mapping geometric transformation parameters of different same-name point windows in the visual image set until the optimal error is within the controllable range. At this time, the corresponding data space reference system is converted to the optimal precise matching parameters.

3. The laser point cloud-guided multi-piece refined dense matching method according to claim 1, characterized in that: The step seven comprises: 7.1 Diffusion of the image depth map: radially initialize neighboring pixels without depth values in the current pixel window; provide an initialized depth information and normal vector information for the current pixel value within the neighborhood range, and calculate its evaluation value according to formula (4), and then optimize and solve the optimal depth value and normal vector value of the current pixel value; (Formula 4) (Formula 5) Where, is the variance, reference image , image set to be matched , For a certain point p Neighborhood collection in the reference image; is a smooth constraint function used to measure the difference between observations. is the cross-correlation function between the reference image and the image to be matched; 、 Object point p Depth value and normal vector in the reference image, 、 Object point q Depth value and normal vector in the reference image; Object point p Depth value scoring evaluation value in the reference image; 7.2 All pixels of non-anchor points with the same name need to be evaluated iteratively through the evaluation function, and the optimal depth value is optimized and selected as the current best solution.

Images