High dynamic range laser 3D scanning method based on single frame image

The parallel positioning of laser stripe images is solved through pixel-level filter array, and the detection problem of laser 3D scanning system under high dynamic range light intensity changes is achieved, and faster and more efficient laser 3D scanning is achieved without increasing equipment complexity or cost. .

CN113963115BActive Publication Date: 2024-11-15SHANDONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111260769.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-28
Publication Date
2024-11-15
Estimated Expiration
2041-10-28

AI Technical Summary

Technical Problem

When existing laser three-dimensional scanning systems face objects with different surface reflectivity or colorful textures, it is difficult to effectively deal with changes in high dynamic range light intensity, resulting in overexposed or underexposed, affecting the detection and positioning of laser stripes, and traditional multiple exposure methods Inefficient time and is not conducive to continuous scanning.

Method used

A pixel-level filter array is used to decompose a single-frame laser stripe image into multiple sub-images, and laser stripe positioning is performed in parallel. By analyzing the positioning parameters and accuracy correlation, selecting the optimal center coordinates, realizing high-dynamic range laser three-dimensional scanning based on a single-frame image .

Benefits of technology

It significantly reduces the laser stripe image acquisition time, improves the speed and efficiency of laser three-dimensional scanning, reduces the requirements for camera acquisition frame speed, and does not increase hardware cost, achieving more efficient high-dynamic range imaging.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113963115B_ABST
    Figure CN113963115B_ABST
Patent Text Reader

Abstract

A high dynamic range laser 3D scanning method based on a single frame image, each time scanning, an image sensor with a pixel-level filter array is used to collect a single frame laser stripe image and decompose it into multiple sub-images, and a laser stripe positioning algorithm is used in parallel to synthesize the optimal center coordinates of each sub-image, and a three-dimensional point cloud is reconstructed with the optimal center coordinates of the laser stripes to achieve high dynamic range laser 3D scanning. This method uses the pixel-level filter array in the image sensor to achieve different proportions of light intensity attenuation in parallel, and sacrifices spatial resolution in exchange for time efficiency to achieve high dynamic range laser 3D scanning based on a single frame image, which can significantly reduce the time required to collect laser stripe images, reduce the requirements for camera acquisition frame rate, and achieve faster laser 3D scanning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a high dynamic range imaging method for laser three-dimensional scanning, and belongs to the field of three-dimensional imaging and measurement. Background Art

[0002] Optical 3D measurement technology is widely used in many fields such as industrial manufacturing, surveying and mapping, medical treatment, cultural heritage, etc. due to its advantages of non-contact, high precision and easy operation. Laser 3D scanning, as a typical technology of optical 3D measurement, can provide moderate measurement accuracy through simple and low-cost optical configuration, and has been vigorously promoted in recent years.

[0003] The classic laser 3D scanning system consists of a camera, a laser source, and a mechanical device for scanning. The laser source projects a laser line onto the surface of an object. The plane where the laser line is located intersects with the surface of the object to produce laser stripes; the camera then captures the image of the laser stripes. The curve variation of the laser stripes is modulated by the surface contour of the object. Therefore, the image coordinates of the laser stripes can be extracted, and the 3D data of the surface shape of the object can be reconstructed through the triangular geometry between the laser source and the camera.

[0004] From the working principle of the laser 3D scanning system, it can be seen that the performance of 3D measurement depends largely on the detection and positioning of the center of the laser stripes. When designing or selecting the laser stripe center extraction algorithm, the positioning accuracy and robustness to noise are mainly considered. The default premise is that the image of the laser stripes should have good contrast and signal-to-noise ratio. As an optical 3D imaging technology based on active illumination, the brightness change of the laser stripes of laser 3D scanning depends on the intensity of the laser line emitted by the light source and the change of the reflectivity of the object surface; and because the laser is monochromatic, the surface texture and color of the object will also significantly change the brightness of the laser stripes. At this time, if the system uses the standard imaging method to collect the image of the laser stripes, it is easy to overexposure or underexposure, which affects the detection and positioning of the laser stripes. Compared with standard imaging, high dynamic range imaging can respond more accurately to the drastic changes in light intensity in the field of view. Therefore, when the object has different surface reflectivities or colorful surface textures, the laser 3D scanning system needs to use high dynamic range imaging technology to collect the image of the laser stripes.

[0005] The high dynamic range imaging technology commonly used in laser 3D scanning systems is a multiple exposure method inherited from computer vision. The principle and algorithm of this method are relatively simple, but each high dynamic range imaging requires the acquisition of multiple frames of images, so the time efficiency is very low. In addition, this method theoretically requires that the object remain still during the acquisition of multiple frames of images, which is not conducive to the continuous laser scanning. Summary of the invention

[0006] The purpose of the present invention is to provide a high dynamic range laser 3D scanning method based on a single frame image, which neither increases the image acquisition time nor the complexity of the equipment. The pixel-level filter array in the image sensor is used to parallelly realize light intensity attenuation of different proportions, and the high dynamic range laser 3D scanning based on a single frame image is realized by sacrificing spatial resolution in exchange for time efficiency.

[0007] The high dynamic range laser three-dimensional scanning method based on a single frame image of the present invention is:

[0008] During each scan, an image sensor with a pixel-level filter array is used to capture a single-frame laser stripe image and decompose it into multiple sub-images. The laser stripe positioning algorithm is used in parallel to synthesize the optimal center coordinates of each sub-image. The three-dimensional point cloud is reconstructed using the optimal center coordinates of the laser stripes to achieve high dynamic range laser three-dimensional scanning.

[0009] The pixel-level filter array is a periodic distribution in which adjacent pixels have a transmittance variation, and the specific distribution can be designed by oneself, including but not limited to 2×2 and 3×3 periodic structures.

[0010] The pixel level filter array may be a Bayer filter in a commercial color image sensor.

[0011] The decomposition into multiple sub-images is to decompose a single-frame laser stripe image into multiple sub-images with different brightness, directly perform laser stripe positioning on each sub-image and select the optimal coordinates from the positioning results, rather than synthesizing the sub-images into a frame of high dynamic range image and then performing laser stripe positioning.

[0012] The selection of the optimal center coordinates depends on the positioning parameters related to image brightness, contrast and signal-to-noise ratio output by the positioning algorithm; by analyzing the positive or negative correlation between the positioning parameters and the positioning accuracy, the optimal center coordinates can be determined based on the maximum or minimum parameters.

[0013] The laser stripe positioning algorithm is: first, according to the periodic distribution of the filter array, a single-frame input image is decomposed into sub-images with different average brightness; then, the same laser stripe positioning algorithm is used to perform center positioning on each sub-image; a known positioning algorithm (such as an extreme value method, a grayscale centroid method, a Hessian matrix structure analysis method, etc.) is selected according to the requirements of speed and accuracy, and the optimal laser stripe center coordinates are selected from the positioning results of multiple sub-images by analyzing the positive or negative correlation between the positioning parameters and the positioning accuracy; considering the coordinate offset introduced when decomposing the sub-images, the coordinate offset should be correspondingly compensated when synthesizing the optimal coordinates, and finally the high dynamic range optimal center coordinates of the entire laser stripe are output.

[0014] Since the starting point of laser stripe positioning is always the image grayscale value distribution in the neighborhood of the laser stripe image, most positioning algorithms can simultaneously output positioning parameters related to image brightness / contrast / signal-to-noise ratio. Reconstructing the 3D point cloud with the optimal center coordinates of the laser stripes essentially achieves high dynamic range laser 3D scanning.

[0015] The present invention has the following characteristics:

[0016] (1) Compared with the traditional multiple-exposure high dynamic range laser scanning method, it can significantly reduce the time required to acquire laser stripe images, reduce the requirements for camera acquisition frame rate, and achieve faster laser 3D scanning;

[0017] (2) The acquisition of high dynamic range images is based on a pixel-level filter array with periodic changes in transmittance. The periodic changes in the filter transmittance can be designed by ourselves, which can better balance the dynamic range and resolution.

[0018] (3) Since the laser itself has good monochromaticity, the color filter array in the commercial color camera can be directly used as a periodic filter array with three different transmittances (see the embodiment); directly using the commercial color camera does not increase any hardware cost;

[0019] (4) After the sub-image is decomposed, the center of the laser stripe is directly located instead of synthesizing the high dynamic range laser stripe image, and the algorithm has higher execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a conceptual schematic diagram of the pixel-level filter array in the present invention.

[0021] Figure 2 It is the overall flow chart of the high dynamic range laser stripe center positioning algorithm in the present invention.

[0022] Figure 3 Schematic diagram of a pixel-level filter array (laser wavelength is 520 nm) corresponding to the Bayer filter in an embodiment of the present invention.

[0023] Figure 4 : The original image captured by the color image sensor in the embodiment of the present invention and its decomposed sub-images (laser wavelength is 520nm); where: (a) a single frame original image; (b) a local magnified view, where the brightness attenuation effect of the Bayer filter can be seen; (c) a sub-image I of the red channel r ; (d) Green channel sub-image I g ; (e) Sub-image I of the blue channel b . DETAILED DESCRIPTION

[0024] The grayscale value of the pixel in the laser stripe image collected by the camera is proportional to the quantum efficiency q of the image sensor. e And the object surface reflectivity r(x, y):

[0025] I(x,y)∝q e r(x, y),

[0026] When the surface reflectivity of an object is relatively constant, a laser stripe image with appropriate grayscale can be obtained by properly setting the camera parameters. For objects with rich textures or reflective surfaces, their surface reflectivity may change dramatically in different areas or viewpoints, resulting in the brightness of the laser stripes in the image being too small or too large (oversaturated), thereby introducing more positioning errors.

[0027] The laser stripe image acquisition of the present invention uses an image sensor with a pixel-level filter array. Existing image sensors are all digital sensors, and the smallest unit of recording and acquiring images is a pixel, so it is possible to design an independent filter for each pixel. The pixel-level filter array with periodic transmittance is designed so that different pixels of the image sensor have different quantum efficiencies, so that laser stripe images with different brightness can be separated from a single frame image, and then used in the subsequent high dynamic range laser stripe positioning algorithm.

[0028] A typical pixel-level filter array is Figure 1 As shown, the period of transmittance change is 2×2 pixels. Dark colors represent filters with lower transmittance, and light colors represent filters with higher transmittance. The quantum efficiency of four pixels in one period has the following relationship:

[0029] q1<q2<q3<q4,

[0030] The corresponding image brightness must satisfy:

[0031] I1<I2<I3<I4,

[0032] This means that four sub-images with different average brightness can be separated from a single frame image.

[0033] The design concept of the present invention can be used to design any pixel-level filter array with periodic transmittance variation, and is not limited to Figure 1 Increasing the number of pixels in a period can separate more sub-images with different average brightness from a single frame image, at the expense of further reduction in the resolution of each sub-image.

[0034] In the laser 3D scanning system, high dynamic range imaging technology serves the accuracy and reliability of laser stripe positioning. In practice, it is not necessary to synthesize a high dynamic range image for the laser stripes, but only to locate them in each sub-image and then select the coordinates with the best accuracy and reliability. Based on the above considerations, the overall process of the high dynamic range laser stripe center positioning algorithm proposed in the present invention is as follows: Figure 2 shown.

[0035] First, according to the periodic distribution of the filter array, the single-frame input image is decomposed into sub-images with different average brightness. Then, the same laser stripe positioning algorithm is used to locate the center of each sub-image; at this time, the existing positioning algorithm can be flexibly selected according to the requirements of speed and accuracy. Since the starting point of laser stripe positioning is always the image gray value distribution of the laser stripe image neighborhood, most positioning algorithms can simultaneously output positioning parameters related to image brightness, contrast, and signal-to-noise ratio.

[0036] By analyzing the positive or negative correlation between positioning parameters and positioning accuracy, the optimal center coordinates of the laser stripes can be selected from the positioning results of multiple sub-images. Considering the coordinate offset introduced when decomposing the sub-images, the coordinate offset should be compensated accordingly when synthesizing the optimal coordinates, and finally the high dynamic range optimal center coordinates of the entire laser stripes are output. The three-dimensional point cloud is reconstructed with the optimal center coordinates of the laser stripes, which essentially realizes high dynamic range laser three-dimensional scanning.

[0037] Taking a commercial color camera as an example, the specific implementation of the present invention is described. The advantage of using a commercial color camera is that it does not increase any hardware cost, but the disadvantage is that the transmittance of the filter cannot be customized. According to the content of the invention, a pixel-level filter array with periodic transmittance changes can also be designed by oneself, and the corresponding camera can be processed and manufactured, which can better balance the dynamic range and resolution.

[0038] Typical image sensors such as complementary metal oxide semiconductor (CMOS) chips do not have color perception capabilities themselves. Therefore, ordinary CMOS chips can only be used as monochrome image sensors. In order to capture color images with a single chip, commercial color cameras generally use Bayer filters to separate different colors. The Bayer filter is a color filter array that inlays red, green, and blue filters on a square grid, and the size of each filter unit exactly matches the size of each pixel on the CMOS chip. Therefore, filters of different colors form different color channels, and the three colors of red, green, and blue will be recorded in different channels (pixels). Filters of different colors have different transmittances to wavelengths, resulting in different quantum efficiencies of different color channels for the same wavelength. For example, a typical color CMOS chip has a blue channel that is sensitive to wavelengths in the range of 400-500nm, a green channel that is sensitive to wavelengths in the range of 480-600nm, and a red channel that is sensitive to wavelengths in the range of 580-750nm.

[0039] Laser has good monochromaticity, so the transmittance of any wavelength of laser in the three color channels must be inconsistent. Take the green semiconductor laser commonly used in laser 3D scanning system as an example, its wavelength is 520nm. When using a color image sensor with a Bayer filter to collect laser stripe images, the transmittance of the green channel is the highest, the blue channel is second, and the red channel has the lowest transmittance. Therefore, for a laser with a wavelength of 520nm, the red, green, and blue filters of the Bayer filter are equivalent to three filters with different transmittances, such as Figure 3 As shown. At this time, 4 sub-images with 3 different average brightness can be separated from the single frame image, of which 2 sub-images corresponding to the green channel have the same average brightness, and there are:

[0040] I r <I b <I g

[0041] like Figure 4 shown.

[0042] The overall algorithm flow is as follows Figure 2 As shown in the figure. The laser stripe location algorithm can be used to independently calculate the coordinates of the laser stripe for each sub-image. Therefore, the four coordinates of any point on the laser stripe can be obtained from different sub-images. The four coordinates have different reliability and accuracy, depending on the brightness level of the laser stripe at that point. Since the dynamic range of the sub-image is relatively low, the sub-image I corresponding to the red channel is r In the image, the brightness level of the laser stripes may be appropriate in areas with high surface reflectivity, but too low in areas with low surface reflectivity; in the sub-image I corresponding to the green channel, gIn the example above, the brightness level of the laser stripes may be appropriate in areas with low surface reflectivity, while the brightness level is too high in areas with high surface reflectivity. In practice, it is difficult to develop a universal standard to judge whether the brightness level is appropriate. Therefore, we use the eigenvalues ​​corresponding to the positioning algorithm to evaluate the quality of the coordinates. In theory, the larger the absolute value of the eigenvalue, the higher the reliability and accuracy of the relevant coordinates. Therefore, the optimal coordinates can be selected by comparing the relevant 4 eigenvalues. Finally, we need to compensate for the coordinate offsets from different sub-images. According to the arrangement of the filter array in the Bayer filter, if we choose the sub-image of the red channel as the reference, the coordinate offsets in the sub-images of the two green and blue channels should be (0, 0.5), (0.5, 0), and (0.5, 0.5), respectively.

[0043] The 3D point cloud is reconstructed with the optimal center coordinates of the laser stripes to achieve high dynamic range laser 3D scanning. The 3D point cloud reconstruction adopts the common line-surface intersection model, and the relevant model parameters are obtained by system calibration based on the classic pinhole camera model.

Claims

1. A high dynamic range laser three-dimensional scanning method based on a single frame image, characterized by: During each scan, an image sensor with a pixel-level filter array is used to collect a single-frame laser stripe image and decompose it into multiple sub-images; a laser stripe positioning algorithm is used in parallel to synthesize the optimal center coordinates of each sub-image; a three-dimensional point cloud is reconstructed using the optimal center coordinates of the laser stripes to achieve high dynamic range laser three-dimensional scanning; The decomposition into multiple sub-images is to decompose a single-frame laser stripe image into multiple sub-images with different brightness, and directly perform laser stripe positioning on each sub-image and select the optimal coordinates from the positioning results, rather than synthesizing the sub-images into a frame of high dynamic range image and then performing laser stripe positioning; The selection of the optimal center coordinates depends on the positioning parameters output by the positioning algorithm and related to image brightness, contrast, and signal-to-noise ratio; the optimal center coordinates are determined by analyzing the positive or negative correlation between the positioning parameters and the positioning accuracy, that is, according to the maximum or minimum parameters; The laser stripe positioning algorithm is: first, according to the periodic distribution of the filter array, a single-frame input image is decomposed into sub-images with different average brightness; then, the same laser stripe positioning algorithm is used to perform center positioning on each sub-image; a known positioning algorithm is selected according to the requirements of speed and accuracy, and the optimal laser stripe center coordinates are selected from the positioning results of multiple sub-images by analyzing the positive or negative correlation between the positioning parameters and the positioning accuracy; the coordinate offset introduced when decomposing the sub-images is considered, and the coordinate offset should be compensated accordingly when synthesizing the optimal coordinates, and finally the high dynamic range optimal center coordinates of the entire laser stripe are output; The pixel-level filter array is a periodic distribution in which adjacent pixels have a change in transmittance, including 2×2 and 3×3 periodic structures; The pixel level filter array is a Bayer filter in a color image sensor.