Registration fusion method based on multispectral image and laser point cloud data

Abstract

The invention relates to a multi-spectral image and laser point cloud data-based registration fusion method, which comprises the following steps of: converting a registered data source, and recovering three-dimensional information from an image sequence by using a multi-view geometric principle, so that the registration of a three-dimensional laser point and a two-dimensional image is converted into a space registration problem of two three-dimensional point sets; the problem that the feature difference between a three-dimensional laser point cloud and a two-dimensional optical image is large is solved.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image processing, in particular to a method based on multispectral image and laser point cloud data registration and fusion. Background Art

[0002] LiDAR (Light Detection And Ranging) technology has made great progress in the past 20 years. As an active remote sensing technology, LiDAR directly determines the spatial position of the target by emitting laser pulses and receiving the target's reflected signal. It has the advantages of fast data acquisition speed and high geometric positioning accuracy. Traditional optical images can obtain rich spectral information and texture details of objects. Combining the two types of data can give full play to their respective advantages and is widely used in digital cities, disaster assessment, precision agriculture and forestry, etc., achieving huge social and economic benefits. However, the geometric reference frames of laser point clouds and optical images are different, and the two cannot often be directly and accurately aligned.

[0003] In order to achieve effective fusion and application of the two, the geometric registration problem between the two must be solved first. Compared with the traditional image registration problem, laser point cloud and optical image are two cross-modal heterogeneous data, and the huge difference between the two brings great difficulties to the registration problem. Unlike traditional optical image registration, the huge difference between point cloud and image makes it very difficult to determine the observation value of the same name. The data difference between the two can be summarized into three aspects: ① The physical properties of the two are different. The point cloud reflects the backscattering characteristics of the target to the laser beam and the three-dimensional geometric characteristics of the scene; while the image records the reflection of the target to the sunlight, reflecting the physical and material properties of the target. ② The geometric models of the two are different. The point cloud is three-dimensional data, which adopts a direct positioning model based on angle measurement and distance measurement; while the image is two-dimensional data, which adopts a pinhole imaging model based on collinear equations. ③ The sampling methods of the two are different. The point cloud is a typical discrete sampling, and its data distribution is restricted by the laser emission frequency and the system scanning frequency; while the image is generally a planar array or linear array imaging, which is a continuous sampling.

[0004] In order to overcome the differences between data and extract stable and reliable observations with the same name, a new registration and fusion method for multispectral images and laser point cloud data is urgently needed. Summary of the invention

[0005] The purpose of the present invention is to provide a method for the registration and fusion of multispectral images and laser point cloud data, which converts the data source of the registration and uses the principle of multi-view geometry to restore three-dimensional information from the image sequence, thereby converting the registration of three-dimensional laser points and two-dimensional images into a spatial registration problem of two three-dimensional point sets, thereby overcoming the problem of large feature differences between three-dimensional laser point clouds and two-dimensional optical images.

[0006] The technical solution to achieve the purpose of the present invention is: A method for fusion of multispectral image and laser point cloud data registration, the method comprising: Step 1: Preprocess the multispectral image and laser point cloud data respectively; Step 2: Convert the preprocessed multispectral image into multispectral image point cloud data; Step 3: extracting feature lines from the multispectral image point cloud data generated in step 2 and the laser point cloud data preprocessed in step 1 respectively; Step 4: Perform registration and fusion based on the feature lines extracted in step 3.

[0007] Further: the laser point cloud data is preprocessed in step 1, including: Calculate the average distance from each point in the laser point cloud data to its nearest K neighboring points. If the average distance is greater than the set distance threshold, it is filtered out. Otherwise, it is retained to obtain the laser point cloud data after the initial denoising. The laser point cloud data after the initial denoising is smoothed to obtain the final laser point cloud data.

[0008] Further: the multispectral image is preprocessed in step 1, including: Calibrate the camera, obtain accurate camera distortion parameters, and perform distortion correction on multispectral images.

[0009] Furthermore: multispectral image preprocessing, its radiation correction and geometric correction are implemented using the Post-Processing software module integrated with the sensor, and atmospheric correction is implemented using the FLAASH algorithm.

[0010] Further: the step 2 of generating multispectral image point cloud data specifically includes: Through the multi-view image exterior orientation elements provided by the auxiliary POS system, the same-name point matching and free network adjustment of the ground feature elements of each level of the pre-processed multispectral image are carried out, and the self-calibration regional network adjustment equation of the element connection points, connection lines, control point coordinates, and pan-tilt auxiliary data is established, and the aerial triangulation solution results are obtained through joint solution; Based on the results of aerial triangulation, the position and posture information of the camera when taking pictures are restored; through feature matching and pixel-by-pixel spatial dense matching, a high-precision and high-spatial resolution digital ground model is obtained, and the point cloud filtering and matching units are fused to form a unified full-factor point cloud, completing the point cloud conversion to generate point cloud data with natural texture and color.

[0011] Further: in step 3, feature line extraction is performed on the multispectral image point cloud data generated in step 2 and the laser point cloud data preprocessed in step 1, respectively, including: Construct a set of neighboring points of a point cloud; Calculate the main direction of the adjacent point set and construct a Householder transformation matrix to standardize the posture of the adjacent point set; Perform surface fitting on the adjacent point set to obtain the surface equation; Based on the surface equation, the two principal curvatures of the point cloud are calculated; Select the one with the larger absolute value of the principal curvature as the curvature estimate of the point cloud; The curvature estimation value of all point clouds is calculated, and the point clouds with a value greater than a given threshold are used as line feature points to realize feature line extraction.

[0012] The present invention also provides a system based on multispectral image and laser point cloud data registration and fusion, comprising: A preprocessing module is used to preprocess multispectral images and laser point cloud data respectively; Point cloud conversion module, used to convert the pre-processed multispectral image into multispectral image point cloud data; Feature line extraction module, used to extract feature lines from point cloud data; The registration and fusion module is used to perform point cloud registration and fusion based on feature lines.

[0013] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the steps of the method based on multispectral image and laser point cloud data registration and fusion are implemented as described above.

[0014] The present invention also provides a storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the steps of the method based on multispectral image and laser point cloud data registration and fusion as described above are implemented.

[0015] Compared with the prior art, the present invention has the following significant advantages: this project converts the data source for registration and uses the principle of multi-view geometry to recover three-dimensional information from the image sequence, thereby converting the registration of three-dimensional laser points and two-dimensional images into the spatial registration problem of two three-dimensional point sets, overcoming the problem of large feature differences between three-dimensional laser point clouds and two-dimensional optical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0019] In order to overcome the problem of large feature differences between three-dimensional laser point clouds and two-dimensional optical images, the present invention converts the data source for registration and uses the principle of multi-view geometry to recover three-dimensional information from the image sequence, thereby converting the registration of three-dimensional laser points and two-dimensional images into a spatial registration problem of two three-dimensional point sets.

[0020] A registration and fusion method based on multispectral image and laser point cloud data, such as Figure 1 As shown, the method includes: Step 1: Preprocess the multispectral image and laser point cloud data respectively.

[0021] Among them, the laser point cloud data is preprocessed, including: Calculate the average distance from each point in the laser point cloud data to its nearest K neighboring points. If the average distance is greater than the set distance threshold, it is filtered out. Otherwise, it is retained to obtain the laser point cloud data after the initial denoising. The laser point cloud data after the initial denoising is smoothed to obtain the final laser point cloud data.

[0022] The multispectral image is preprocessed, including: calibrating the camera, obtaining accurate camera distortion parameters, and correcting the distortion of the multispectral image.

[0023] Among them, the multispectral image preprocessing, its radiation correction and geometric correction are implemented using the Post-Processing software module integrated with the sensor, and the atmospheric correction is implemented using the FLAASH algorithm.

[0024] Step 2: Convert the preprocessed multispectral image into multispectral image point cloud data, including: Through the multi-view image exterior orientation elements provided by the auxiliary POS system, the same-name point matching and free network adjustment of the ground feature elements of each level of the pre-processed multispectral image are carried out, and the self-calibration regional network adjustment equation of the element connection points, connection lines, control point coordinates, and pan-tilt auxiliary data is established, and the aerial triangulation solution results are obtained through joint solution; Based on the results of aerial triangulation, the position and posture information of the camera when taking pictures are restored; through feature matching and pixel-by-pixel spatial dense matching, a high-precision and high-spatial resolution digital ground model is obtained, and the point cloud filtering and matching units are fused to form a unified full-factor point cloud, completing the point cloud conversion to generate point cloud data with natural texture and color.

[0025] Step 3: Extract feature lines from the multispectral image point cloud data generated in step 2 and the laser point cloud data preprocessed in step 1 respectively.

[0026] Specifically, feature line extraction includes: Construct a set of neighboring points of a point cloud; Calculate the main direction of the adjacent point set and construct a Householder transformation matrix to standardize the posture of the adjacent point set; Perform surface fitting on the adjacent point set to obtain the surface equation; Based on the surface equation, the two principal curvatures of the point cloud are calculated; Select the one with the larger absolute value of the principal curvature as the curvature estimate of the point cloud; The curvature estimation value of all point clouds is calculated, and the point clouds with a value greater than a given threshold are used as line feature points to realize feature line extraction.

[0027] Step 4: Perform registration and fusion based on the feature lines extracted in step 3.

[0028] The present invention also provides a system based on multispectral image and laser point cloud data registration and fusion, comprising: A preprocessing module is used to preprocess multispectral images and laser point cloud data respectively; Point cloud conversion module, used to convert the pre-processed multispectral image into multispectral image point cloud data; Feature line extraction module, used to extract feature lines from point cloud data; The registration and fusion module is used to perform point cloud registration and fusion based on feature lines.

[0029] The above-mentioned technical solution based on the multispectral image and laser point cloud data registration and fusion system is consistent with the above-mentioned technical solution based on the multispectral image and laser point cloud data registration and fusion method, which will not be repeated here.

[0030] Based on the same technical solution, the present invention also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the method based on multispectral image and laser point cloud data registration and fusion as described above are implemented.

[0031] Based on the same technical solution, a storage medium is provided, wherein computer executable instructions are stored in the storage medium. When the computer executable instructions are loaded and executed by a processor, the steps of the method for registration and fusion of multispectral images and laser point cloud data as described above are implemented.

[0032] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0033] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0034] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0035] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0036] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0037] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0038] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0039] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. Based on the multispectral image and laser point cloud data registration and fusion method, the characteristics are: The method comprises: Step 1: Preprocess the multispectral image and laser point cloud data respectively; Step 2: Convert the preprocessed multispectral image into multispectral image point cloud data; Step 3: extracting feature lines from the multispectral image point cloud data generated in step 2 and the laser point cloud data preprocessed in step 1 respectively; Step 4: Perform registration and fusion based on the feature lines extracted in step 3.

2. The method for registration and fusion of multispectral images and laser point cloud data according to claim 1 is characterized in that: In step 1, the laser point cloud data is preprocessed, including: Calculate the average distance from each point in the laser point cloud data to its nearest K neighboring points. If the average distance is greater than the set distance threshold, it is filtered out. Otherwise, it is retained to obtain the laser point cloud data after the initial denoising. The laser point cloud data after the initial denoising is smoothed to obtain the final laser point cloud data.

3. The method for registration and fusion of multispectral images and laser point cloud data according to claim 1, characterized in that: In step 1, the multispectral image is preprocessed, including: Calibrate the camera, obtain accurate camera distortion parameters, and perform distortion correction on multispectral images.

4. The method for registration and fusion of multispectral image and laser point cloud data according to claim 1, characterized in that: For multispectral image preprocessing, its radiation correction and geometric correction are implemented using the Post-Processing software module integrated with the sensor, and atmospheric correction is implemented using the FLAASH algorithm.

5. The method for registration and fusion of multispectral image and laser point cloud data according to claim 1, characterized in that: The step 2 of generating multispectral image point cloud data specifically includes: Through the multi-view image exterior orientation elements provided by the auxiliary POS system, the same-name point matching and free network adjustment of the ground feature elements of each level of the pre-processed multispectral image are carried out, and the self-calibration regional network adjustment equation of the element connection points, connection lines, control point coordinates, and pan-tilt auxiliary data is established, and the aerial triangulation solution results are obtained through joint solution; Based on the results of aerial triangulation, the position and posture information of the camera when taking pictures are restored; through feature matching and pixel-by-pixel spatial dense matching, a high-precision and high-spatial resolution digital ground model is obtained, and the point cloud filtering and matching units are fused to form a unified full-factor point cloud, completing the point cloud conversion to generate point cloud data with natural texture and color.

6. The method for registration and fusion of multispectral image and laser point cloud data according to claim 1, characterized in that: In step 3, feature line extraction is performed on the multispectral image point cloud data generated in step 2 and the laser point cloud data preprocessed in step 1, respectively, including: Construct a set of neighboring points of a point cloud; Calculate the main direction of the adjacent point set and construct a Householder transformation matrix to standardize the posture of the adjacent point set; Perform surface fitting on the adjacent point set to obtain the surface equation; Based on the surface equation, the two principal curvatures of the point cloud are calculated; Select the one with the larger absolute value of the principal curvature as the curvature estimate of the point cloud; The curvature estimation value of all point clouds is calculated, and the point clouds with a value greater than a given threshold are used as line feature points to realize feature line extraction.

7. Based on the multispectral image and laser point cloud data registration and fusion system, it is characterized by: include: A preprocessing module is used to preprocess multispectral images and laser point cloud data respectively; Point cloud conversion module, used to convert the pre-processed multispectral image into multispectral image point cloud data; Feature line extraction module, used to extract feature lines from point cloud data; The registration and fusion module is used to perform point cloud registration and fusion based on feature lines.

8. An electronic device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the method based on multispectral image and laser point cloud data registration and fusion as described in any one of claims 1 to 6 are implemented.

9. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the steps of the method based on multispectral image and laser point cloud data registration and fusion as described in any one of claims 1 to 6 are implemented.

Images