A measurement method based on a combination of a linear array camera and a ground laser radar device

Abstract

This application discloses a measurement method based on a combination of a line scan camera and a ground-based lidar system. In this method, the acquisition by the line scan camera and the laser scanner are synchronized, with the scanning surface of the line scan camera parallel to the laser scanning surface. This device leverages the advantages of the line scan camera, such as faster acquisition speed, higher resolution, and less distortion. The line scan camera and laser scanner rotate together, and the line scan camera continuously acquires images triggered by the rotation angle, obtaining a complete 360° panoramic image. No image stitching is required at the same height, reducing data stacking and requiring less storage space. The cost is lower than devices with equivalent functionality. Furthermore, the fusion of point cloud data and image data is relatively convenient, enabling more efficient real-time measurement and color rendering of 3D scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of surveying and mapping, and in particular to a measurement method based on a combination of a linear array camera and a ground laser radar device. BACKGROUND

[0002] Laser scanning technology is one of the important means for quickly and efficiently obtaining spatial data. A laser scanner obtains laser point cloud data of surrounding target points. In general, the scanner is configured with at least a panoramic camera combined with a planar array imaging device, and the camera is used to shoot RGB image data of a scanned target region. The target point cloud data obtained by the laser scanning device is a series of coordinate points established in the local coordinate system of the scanning device, and the image coordinates of the target in the scanned region collected by the camera imaging device are established in the spatial coordinate system of the planar array imaging device. In order to realize the fusion of the target point cloud data and the image data and give the point cloud data RGB color information, registration needs to be performed on the target laser point cloud data and the image data. The registration process mainly calculates the image coordinates corresponding to each laser point through a coordinate transformation formula in the geometric space according to the laser point coordinates in the local coordinate system of the scanner and the pose parameters of the camera device, and gives the corresponding laser point the RGB color of the calculated image coordinates, so as to realize the color rendering effect of the laser point cloud data.

[0003] The combination of panoramic cameras and laser radars for observation improves the efficiency of modern surveying and mapping work. At present, panoramic cameras are developing rapidly, and various panoramic cameras with different technical parameters, different working modes and different combination modes appear on the market. Generally, they can be divided into three imaging types: camera cluster type, fisheye lens type, stitched image type and panoramic scanning type. At present, most panoramic cameras on the market are measurement systems combined with at least one planar array camera and a laser scanner. The system of this combination form has high requirements for the performance of the camera, and the corresponding cost is also high. In addition, the planar array camera and the three-dimensional laser radar cannot be measured synchronously and dynamically, the image acquisition speed is slow, a large-capacity storage device is needed, and there is a large amount of data overlap; the image data processing workload is large in the later stage, such as distortion correction, splicing and the like. SUMMARY

[0004] Therefore, the present application provides a measurement method based on a combination of a linear array camera and a ground laser radar device, which can realize image splicing without image splicing at the same height, reduce data stacking, require smaller storage space, and reduce the difficulty of fusion of point cloud data and image data.

[0005] The present application provides a measurement method based on a combination of a linear array camera and a ground laser radar device, comprising:

[0006] The image data is collected by a linear array camera, and the point cloud data is collected by a laser radar, the collection action of the linear array camera and the collection action of the laser radar are kept synchronous, and the scanning surface when the linear array camera collects data is parallel to the scanning surface when the laser collects data;

[0007] The image data is corrected;

[0008] The corrected image data is registered and fused with the point cloud data.

[0009] Optionally, the data collection action of the linear array camera and the data collection of the laser radar are periodic collection.

[0010] Optionally, the linear array camera is composed of a plurality of linear array camera units, and the principal optical axes of different linear array camera units are coplanar.

[0011] Optionally, the correction includes:

[0012] An ellipse is detected by using Hough transform, and a rough extraction is performed on the edge of the spherical target, a region of interest is determined from the pixel-level edge, and the region is enlarged to contain the fuzzy edge part;

[0013] A three-level gray edge model is established, and a sub-pixel edge positioning is performed on the edge model by using Zernike matrix;

[0014] The ellipse is repeatedly fitted, and the boundary pixels with residual error exceeding the threshold are removed to achieve the purpose of optimizing the boundary, and then the maximum and minimum values of the longitudinal coordinate and the transverse coordinate of the ellipse edge point are obtained;

[0015] A projection model of the spherical target is established to obtain the number of pixels of the ellipse in the horizontal axis direction and the actual number of pixels, and then an image scaling ratio is obtained;

[0016] According to the image scaling ratio, the to-be-corrected image is corrected by using a bilinear interpolation method.

[0017] Optionally, the edge model adopts the following formula,

[0018]

[0019] Wherein, is the background gray level, is the target gray level, is the boundary gray level, is the edge coordinate is the standard deviation of the edge gray distribution model at the edge coordinate, is the background gray level, is the gray difference between the foreground and the background.

[0020] Optionally, the Zernike moment is calculated by the following formula,

[0021] ;

[0022] wherein, represents the distance from the image origin to the lower boundary of the edge part, represents the distance from the image origin to the upper boundary. represents the mean value of the gray scale of the edge region.

[0023] Optionally, the projection model is calculated by the following formula,

[0024] ;

[0025] L 1 is the geometric length A'O' of the projection point A' of the point A on the spherical target on the image plane to the image plane center O', and similarly L 2 is C'O', L 3 is B'O', is the height angle of the center of the spherical target relative to the principal axis, f is the focal length of the linear camera.

[0026] Optionally, the image scaling ratio is obtained by the following formula,

[0027] ;

[0028] wherein, T is the scaling ratio, is the maximum value of the horizontal axis of the boundary set is the minimum value of the horizontal axis of the boundary set is the maximum value of the vertical axis of the boundary set is the minimum value of the vertical axis of the boundary set .

[0029] Optionally, the registration fusion is obtained by coordinate transformation of the coordinate system of the linear array camera and the point cloud coordinate system of the laser radar.

[0030] Optionally, the coordinate transformation is performed by the following formula,

[0031]

[0032] wherein, the pixel coordinates of the point P in the photograph are , the coordinates of the point P in the point cloud coordinate system are , is the rotation matrix parameter, and the angular resolution of the image vertical direction of the linear camera is ​​​, the angular resolution of the image horizontal direction of the linear camera is .

[0033] The application provides a measurement method based on a linear camera and a ground laser radar combination device. Multiple linear cameras and a laser scanner are synchronously observed. The scanning surface of the linear camera is parallel to the laser scanning surface, and the spatial positions of the two are relatively simple. The device takes advantage of the faster acquisition speed, higher resolution, and smaller distortion of the linear camera. The linear camera and the laser scanner rotate together. The linear camera continuously acquires images through rotation angle triggering, obtains a 360° complete panoramic image, does not need image stitching at the same height, reduces data stacking, requires smaller storage space, has lower cost than devices with the same function, and is more convenient and efficient for fusion of point cloud data and image data, and real-time measurement and color rendering of a three-dimensional scene. BRIEF DESCRIPTION OF DRAWINGS

[0034] The technical solutions and other beneficial effects of the application will be apparent through the following detailed description of the specific embodiments of the application in combination with the drawings.

[0035] Figure 1 A structural schematic diagram of the camera scanning surface and the scanning surface layout of the laser radar provided for the embodiments of the application.

[0036] Figure 2 A structural schematic diagram of an imaging model in the linear camera provided for the embodiments of the application.

[0037] Figure 3 A structural schematic diagram of the linear camera and the ground laser radar combination device provided for the embodiments of the application.

[0038] Figure 4 A structural schematic diagram of a panoramic camera composed of three linear cameras provided for the embodiments of the application.

[0039] Figure 5 A light path distribution diagram provided for the embodiments of the application.

[0040] Figure 6 An instrument scanning field of view range schematic diagram provided for the embodiments of the application.

[0041] Figure 7 A working control diagram of the combination device provided for the embodiments of the application.

[0042] Figure 8 A schematic diagram of the projection area of the linear camera provided for the embodiments of the application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0044] In the description of the present application, it should be understood that the terms "first", "second" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise explicitly specified and limited.

[0045] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connection" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integrally connected; it can be mechanical connection, or electrical connection or can communicate with each other; it can be directly connected, or indirectly connected through an intermediate medium, or the communication between two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0046] The following disclosure provides many different embodiments or examples for implementing different structures of the present application. In order to simplify the disclosure of the present application, the components and arrangements of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeatedly refer to the same reference numerals and / or reference letters in different examples. Such repetition is for the purpose of simplification and clarity, and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. In addition, the present application provides various specific examples of processes and materials, but those skilled in the art can realize the application of other processes and / or the use of other materials.

[0047] [Line array camera combined with ground laser radar device]

[0048] Reference Figure 1 The combined device includes a scanner of a laser radar, one or more line array cameras, and an upper computer, which mainly relates to the layout structure of the laser scanner and the camera and the sensor control module. The scanning surface of the line array camera is parallel to the scanning surface F of the three-dimensional laser radar, and the two are fixedly connected. The camera rotates with the three-dimensional laser radar to complete the 360° observation of the camera scanning range.

[0049] The host computer sends a rotation measurement command to the laser scanner. The motor drives the encoder and laser scanner to rotate coaxially. The synchronous control board records the rotation angle of the laser radar in the horizontal and vertical directions, and sends a frame trigger command to the camera when the horizontal angle rotates by 0.01°, continuously acquiring observation images.

[0050] refer to Figure 3 The scanning planes of the cameras and the laser scanner are parallel, with each camera's scanning plane parallel to the laser scanner's scanning plane. The line scan cameras rotate synchronously with the laser scanner to continuously acquire target image data. A single camera can generate a complete image of the target from all four sides, eliminating the need for horizontal stitching, making the process more convenient and saving storage space. The line scan cameras can be commercial cameras, industrial cameras, or camera modules, etc.

[0051] refer to Figure 4 Three line-scan cameras (Camera 1, Camera 2, and Camera 3) are used to form a line-scan panoramic camera. The three principal optical axes (z1, z2, z3) of the line-scan cameras are coplanar and arranged in an arc-shaped array at a certain angle. Using 4K color line-scan cameras, an approximately 10K arc-shaped panoramic camera is formed. A 25mm lens is used to achieve a 160° vertical observation angle. The optical path distribution is as follows. Figure 5 As shown. The camera rotates with the lidar laser scanner to complete a 360°*320° field of view, as... Figure 6 By calibrating the equipment, the pose relationship of each camera relative to the laser scanner is obtained.

[0052] [Host computer]

[0053] The host computer sends a rotation signal to the 3D laser scanner. The motor drives the encoder and laser radar to rotate coaxially. The encoder records the rotation angle and sends a pulse signal to the synchronization control board. The synchronization control board records the rotation angle of the laser radar in the horizontal and vertical directions. For every 0.01° rotation in the horizontal direction, the synchronization control board simultaneously sends a frame trigger image signal to the three line scan cameras and assigns a horizontal angle mark to each frame. The working control diagram is as follows: Figure 7 Three linear array cameras continuously acquire images as the lidar rotates, with each image being approximately 360 million pixels in size, enabling simultaneous acquisition of point cloud data and image data.

[0054] [Image correction of line array camera]

[0055] When a linear array camera rotates with a lidar sensor for measurement, the camera's sampling is triggered by a fixed rotation angle. Therefore, the observed object exhibits lateral stretching in the panoramic image, but no vertical stretching. Image stretching correction is a key aspect of this patent. Considering the diversity of object imaging angles, and given the rotational invariance of a sphere, a spherical target is chosen as the reference for calculating the stretching ratio. Specific steps...

[0056] 1) Detect ellipses using Hough transform and coarsely extract the edges of spherical targets. Determine the region of interest from the pixel-level edges and appropriately expand the region to include the blurred edge portion.

[0057] 2) Establish a three-level grayscale edge model and use Zernike moments and the proposed edge model to locate sub-pixel edges.

[0058] 3) Repeatedly fit the ellipse and remove boundary pixels whose residuals exceed the threshold to optimize the boundary. Then obtain the maximum and minimum values ​​of the ordinate and abscissa of the ellipse edge points.

[0059] 4) Establish a projection model of the spherical target to obtain the theoretical number of pixels and the actual number of pixels of the ellipse in the horizontal axis direction, and then obtain the image scaling ratio.

[0060] 5) Based on the image scaling ratio in step 4), perform image correction on the original image using bilinear interpolation.

[0061] To accelerate the localization of spherical targets in the image, step 1) uses Hough transform to detect ellipses, accurately locates the position of the spherical target, obtains the pixel-level boundary of the target, and appropriately expands the boundary area to include the blurred edge part.

[0062] Due to factors such as lighting and defocus, images exhibit different grayscale spatial distribution patterns in different directions. To improve boundary positioning accuracy, a three-level grayscale edge model is established, consisting of background level grayscale and background level grayscale. Target level grayscale Boundary level grayscale Establish the following boundary model:

[0063]

[0064] in, edge coordinates The standard deviation of the edge gray-scale distribution model at that location Background grayscale The grayscale difference between the foreground and background is given. Based on the proposed edge model, the Zernike moment formula (2) for the image can be recalculated.

[0065]

[0066] make After sorting, we can obtain the following:

[0067]

[0068] in, This represents the distance from the image origin to the lower boundary of the edge portion. This represents the distance from the image origin to the upper boundary. This represents the average gray level of the edge region. (From...) , , , Substituting into formula (4) yields the sub-pixel edge. The expression is,

[0069]

[0070] Due to factors such as uneven lighting and defocusing, and The nonlinear functional relationship can be obtained using the above model. The expression is ,in ,therefore

[0071]

[0072] The actual image edge constraint can be obtained. Therefore, numerical calculation can be used to solve it. .

[0073] In summary, the error of the sub-pixel edge points in the imaging of the spherical target can be compensated by formula (6).

[0074]

[0075] Given the complexity of the observation environment, to eliminate background interference pixels and improve the positioning accuracy of the spherical target image edge, step 3) uses a multiple-fit ellipse algorithm to filter the extracted boundary pixels and remove gross errors. The equation of the ellipse curve is...

[0076]

[0077] Substitute the extracted edge pixels into formula (7) and use the least squares principle to calculate the parameters. , , , , , The optimal solution is obtained, and the fitting residuals for each edge pixel are calculated. Edge points with large residual values ​​are removed. This process is repeated until all residuals are less than a certain empirical threshold, at which point the ellipse fitting is stopped, and the optimal set of boundary pixels is obtained. .

[0078] To obtain the accurate ellipticity of the spherical target image boundary, step 4) establishes an imaging model of the spherical target in the linear array camera, such as... Figure 2 .

[0079] From the above diagram, we can establish relation (8).

[0080]

[0081] The theoretical value of the major axis of the spherical target boundary is Similarly, the theoretical value of the minor axis can be derived. The ellipticity of the projected ellipse of the spherical target is,

[0082]

[0083] generally The value is very small and can be ignored when the camera and the spherical target are at a certain distance. Formula (9) can be expressed as follows:

[0084]

[0085] In addition, the target's direction angle can be solved using formula (8). ,

[0086]

[0087] It should be noted that when the scanning surface of the linear scan camera passes through the center of the sphere, the boundary of the spherical target is circular, that is... , .

[0088] Statistical boundary set The maximum and minimum values ​​of the horizontal and vertical axes are respectively , , , Images observed by a line-scan camera are stretched only along the horizontal axis, with no stretching along the vertical axis. Therefore, the image scaling ratio can be obtained from the above process. ,

[0089]

[0090] Get accurate image scaling Step 5) Scale the stretched image using bilinear interpolation. The image scaling steps are as follows:

[0091] ① Pixel coordinates in the target image This allows us to obtain the corresponding pixel coordinates in the original image. ,

[0092]

[0093] ② Order , The coordinates of the four surrounding pixels are obtained as follows: , , , .

[0094] ③ Using bilinear interpolation, let express Pixel value at that location, , Then the target point The pixel at that location is

[0095]

[0096] In summary, the problem of scene stretching in panoramic images acquired by line scan cameras has been effectively corrected.

[0097] [Registration and fusion of image and point cloud data]

[0098] The three cameras are identical in model, size, and lens, therefore they have the same angular resolution in both the horizontal and vertical directions. Furthermore, the image planes of all three cameras are parallel to the scanning plane of the laser scanner, and the scanning planes of the three cameras are coplanar.

[0099] For a point P on the point cloud Pixel projection is performed on each camera.

[0100] refer to Figure 8 Point cloud data can be broadly categorized into two types: one type corresponds to pixels in only a single photograph, such as region 1, region 3, and region 5; the other type corresponds to pixels in two photographs, such as region 2 and region 4. The regions of a photograph can be divided based on the elevation angle of the point cloud, a process that involves comparing data to determine the appropriate intervals.

[0101] For regions 1, 3, and 5:

[0102] The midpoint of the camera was obtained through calibration experiments. ( Main point, (Half the image length) and its horizontal and vertical angles in the local coordinate system of the lidar. This area point The coordinates are After normalizing it, the transformation relationship between the points on the photo and the coordinates of the point cloud can be established, as shown in formula (21).

[0103]

[0104] in For rotation matrix parameters and related.

[0105]

[0106] The corresponding pixel coordinates can then be calculated, as shown in formula (23).

[0107]

[0108] Finally, the RGB values ​​of the pixels in the image are assigned to the point cloud, and the point cloud is given color.

[0109] For regions 2 and 4:

[0110] By using the same processing procedure as in regions 1, 3, and 5, the coordinates of point W are found. The corresponding pixel coordinates in the two photos. , .

[0111]

[0112] Get the midpoint of the image , The RGB values ​​are respectively , The maximum height of the photo is The height at which the two photos overlap is It was calculated using a distance-weighted method. Give the point cloud color, as in formula (26).

[0113]

[0114] The measurement method described in this application has the following advantages:

[0115] 1) The imaging planes of multiple linear scan cameras are coplanar and parallel to the scanning plane of the laser scanner. The structure between the sensors is simple, the correspondence between the point cloud and the image is strict, and the point cloud coloring process is relatively simple, requiring only two coordinate rotations.

[0116] 2) The camera and laser scanner synchronously and dynamically acquire data about the surrounding environment, resulting in clear images and fast acquisition speed;

[0117] 3) Linear scan cameras provide continuous dynamic imaging with a large field of view, eliminating the need for excessively long exposure times, resulting in images without significant distortion, no need for image stitching, smaller image data volume, no wasted storage space, and lower equipment costs.

[0118] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A measurement method based on a combination of a linear array camera and a ground-based lidar, characterized in that, include: Image data is acquired by a line scan camera and point cloud data is acquired by a lidar. The acquisition actions of the line scan camera and the lidar are synchronized, and the scanning surface of the line scan camera when acquiring data is parallel to the scanning surface of the lidar when acquiring data. The image data is corrected; The corrected image data is then registered and fused with the point cloud data. The correction includes: Ellipses are detected using Hough transform, and the edges of spherical targets are coarsely extracted. The region of interest is determined from the pixel-level edges, and the region is expanded to include the blurred edge parts. A three-level grayscale edge model is established, and sub-pixel edge localization is performed using Zernike moments and the edge model. Repeatedly fit the ellipse and remove boundary pixels whose residuals exceed the threshold to optimize the boundary. Then obtain the maximum and minimum values ​​of the ordinate and abscissa of the edge points of the ellipse. Establish a projection model of a spherical target to obtain the theoretical number of pixels and the actual number of pixels in the horizontal axis of the ellipse, and then obtain the image scaling ratio; Based on the image scaling ratio, the image to be corrected is corrected using bilinear interpolation. The edge model uses the following formula. ； in, For background grayscale, For target level grayscale, For boundary level grayscale, edge coordinates The standard deviation of the edge gray-scale distribution model at that location, Background grayscale The difference in grayscale between the foreground and background; The Zernike moment is given by the following formula. ； in, This represents the distance from the image origin to the lower boundary of the edge portion. This represents the distance from the image origin to the upper boundary. This represents the average gray level of the edge region.

2. The measurement method according to claim 1, characterized in that, The data acquisition actions of the line array camera and the lidar are periodic.

3. The measurement method according to claim 1, characterized in that, The line scan camera is composed of multiple line scan camera units, and the principal optical axes of different line scan camera units are coplanar.

4. The measurement method according to claim 1, characterized in that, The projection model uses the following formula. ； L 1 represents the geometric length A'O' of the projection point A' of point A on the spherical target onto the image plane, from the center O' of the image plane. Similarly, L 2 represents C'O'. L 3 represents B'O', and θ is the elevation angle of the center of the spherical target relative to the principal optical axis. f This is the focal length of the linear camera.

5. The measurement method according to claim 1, characterized in that, The image scaling ratio can be obtained using the following formula. ； Where T is the scaling ratio. e Let be the ellipticity of the projected ellipse of the spherical target. Boundary set The maximum value of the horizontal axis Boundary set Minimum value of the horizontal axis Boundary set The maximum value of the vertical axis, Boundary set The minimum value of the vertical axis.

6. The measurement method according to claim 1, characterized in that, The registration and fusion method involves obtaining the coordinate system of the linear array camera and the point cloud coordinate system of the lidar through coordinate transformation.

7. The measurement method according to claim 6, characterized in that, The coordinate transformation is performed using the following formula. Where, the pixel coordinates of P in the photo are Point P in the point cloud coordinate system The coordinates in are , Here are the rotation matrix parameters, and the vertical angular resolution of the linear camera image is... The horizontal angular resolution of the linear camera image is .

Images