Dual-camera calibration method based on three-dimensional line-scan laser detection system

Abstract

The application provides a two-camera calibration method based on a three-dimensional line scanning laser detection system, a three-dimensional imaging platform is self-built, the height of the device is raised to expand the field of view, and in order to ensure the accuracy under the condition of improving the detection range, a two-camera oblique measurement method is adopted; the self-made sawtooth calibration block adopts a sawtooth calibration algorithm to calibrate the Z direction of the two cameras, and Halcon is used to calibrate the X and Y directions; finally, the two-camera calibration data is fused and rendered to obtain a picture with a wider field of view and higher accuracy, so that the system reliability and accuracy in the practical scene are met.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of road traffic detection, and particularly to a dual-camera calibration method based on a three-dimensional line-scan laser detection system. BACKGROUND

[0002] Road management departments need to conduct regular annual inspections of highway lanes, and need to obtain complete and detailed information about the entire four-meter lane in order to accurately identify diseases on the road.

[0003] Compared with traditional detection techniques, the pavement three-dimensional profile detection technique selects a high-energy line laser and a high-resolution high-frame-rate 3D camera as the hardware basis to realize automatic detection of pavement three-dimensional profiles. The scanning range can cover the entire lane width, and the three-dimensional line laser imaging technology automatically obtains the microscopic deformation of the pavement, and then extracts the damage information to complete the classification and automatic detection of pavement cracks, ruts, pits, and heaves. The collection of three-dimensional data greatly reduces the influence of interference factors such as road oil stains, repairs, shadows, and uneven lighting, and has a higher recognition rate and a lower misjudgment rate. Through data analysis, a comprehensive evaluation of the pavement diseases and damage can be made. SUMMARY

[0004] In order to meet the actual needs of the prior art, the present application proposes a calibration method for two oblique cameras based on a three-dimensional detection system to improve the effect of the collected pictures.

[0005] The hardware composition of the road inspection system (RIS) is the first step in the design of the present application. A 3D line array industrial camera, a laser emitter, an incremental rotary photoelectric encoder (DMI), a global positioning system module (GPS), and a device host are integrated in the RIS. The collection frequency design synchronizes these sensors and also matches the multi-source data.

[0006] The Z-direction correction of the 3D line array industrial camera is the second step of the present application. The production of the calibration block facilitates the extraction of feature points, and the sawtooth-shaped calibration block feature points are extracted by an algorithm to obtain the camera internal and external parameters and convert the Z-direction pixel coordinates into real coordinates.

[0007] The X and Y direction correction of the 3D line array industrial camera is the third step of the present application. The image is segmented according to the required measurement range, and then the calibration block correction parameters are used to fit the X and Y direction data using Halcon, and the real coordinates of the X and Y directions are obtained through coordinate mapping.

[0008] Three-dimensional reconstruction is the last step of the present application. The images collected by the calibrated camera are efficiently stitched, the real values of the images are first mapped to 16 bits to enhance the depth information, and then a complete and seamlessly connected image result is formed through differential correction.

[0009] Finally, the camera detection accuracy is studied by examples. On the other hand, four kinds of fast calibration methods are selected from the case study by different placement methods as the verification experiment method. Manual measurement is carried out, and the results are regarded as the "ground truth" to verify the accuracy of the proposed method.

[0010] The purpose of the application is to propose a dual-camera calibration method based on a three-dimensional detection system, which collects road multi-source data through a road detection system; a self-made sawtooth calibration block is used to calibrate the Z direction of the dual camera by using a sawtooth calibration algorithm, and the X and Y directions are fitted based on the calibration parameters and Halcon; after the X, Y and Z directions are calibrated, the images collected by the dual camera are spliced and rendered to obtain a picture with a wider field of view and higher accuracy, which meets the system reliability and accuracy in practical scenarios.

[0011] The application specifically adopts the following technical solutions:

[0012] A dual-camera calibration method based on a three-dimensional line scanning laser detection system, characterized by: a three-dimensional imaging platform is built, the device height is raised to expand the field of view, and in order to ensure the accuracy while improving the detection range, a dual-camera oblique measurement method is adopted; a sawtooth calibration block is made to calibrate the Z direction of the dual camera by using a sawtooth calibration algorithm, and Halcon is used to calibrate the X and Y directions, the required coverage range and height are brought into calculation to obtain the size of the specific calibration block to be made, the sawtooth is moved uniformly several times within the required height range, and the highest feature point and the lowest feature point of the sawtooth after each movement are extracted to calculate the internal and external parameters of the camera, so as to correct the distortion of the image.

[0013] Further, based on a road detection system RIS comprising at least a 3D line array industrial camera;

[0014] Comprising the following steps:

[0015] Step S1: 3D line array industrial camera Z direction correction, and making calibration block to facilitate feature point extraction, extracting camera internal and external parameters to convert Z direction pixel coordinates into real coordinates;

[0016] Step S2: 3D line array industrial camera X, Y direction correction, according to the calibration block correction parameter, using Halcon to fit the X, Y direction data, and performing coordinate mapping on the X, Y direction to obtain real coordinates;

[0017] Step S3: three-dimensional reconstruction, efficiently splicing the images collected by the calibrated camera to form a complete and seamless image result; and displaying the splicing result by three-dimensional reconstruction method.

[0018] Furthermore, step S1 specifically includes the following steps:

[0019] Step S11: Calibration Block Design: Considering that the single camera of the road 3D inspection vehicle needs to cover a width of 1.1m and a height range of 1.5m to 1.9m, the size of the sawtooth calibration plate is calculated based on the actual required measurements and coverage area. The design of the calibration block considers factors such as angle and the number of points on the slope, and ensures that the slope of the pixel line in the image field of view is greater than 20 degrees, as shown in Formula 1:

[0020]

[0021] Where ROW represents the pixel height of a single serration, and Column represents the pixel width of a single serration;

[0022] Step S12: Feature Point Extraction: Based on the range algorithm settings, the range value is scaled according to the number of sub-pixels used; the range value is normalized by multiplying by a scaling factor; after scaling, the sensor ROI is set, and the sensor coordinate origin is defined by pixel (0, 0); when using the sensor ROI, the parameter sensorRoi is used to compensate for the offset in the x and y directions; the parameter sensorRoi consists of four values, including: width starting position, height starting position, set width, and set length; finally, the center point is found, which is the origin of the image center defined by the pinhole camera model;

[0023] Step S13: Image correction: The homography transformation matrix between the tilted imaging plane and the ideal frontal image plane is described by a rotation matrix represented by the two-dimensional Scheimpflug angle. The extracted pixel coordinates are then subjected to distortion correction to obtain the actual pixel coordinates after distortion removal, thereby obtaining the camera's intrinsic parameter coefficients.

[0024] Step S14: Coordinate transformation: Based on the camera pinhole model, the imaging process is similar to the pinhole imaging principle. The camera distortion model is established for correction to obtain the conversion of pixel values ​​to true values.

[0025] Furthermore, step S2 specifically includes the following steps:

[0026] Step S21: Data acquisition and preprocessing: After converting the 16-bit depth 3D image acquired by the CCD camera into 8-bit, the image data is associated with the position of the actual object by mapping the pixel coordinates on the image to coordinates in 3D space.

[0027] Step S22: Data segmentation: First, separate the objects of interest from the background in the acquired image, then use a concatenation function to connect them into complete regions, thereby obtaining a more comprehensive target region. Finally, use the select_shape function to select the parts that conform to specific shape features from the concatenated regions.

[0028] Step S23: Data Fitting: Use the reduce_domain function to extract the data region of interest, use the min_max_gray function to obtain the minimum and maximum pixel values ​​within the selected region to calculate the range of X and Y dimensions, and use the calculated range values ​​of X and Y dimensions to represent the real data of X and Y dimensions.

[0029] Furthermore, step S3 specifically includes the following steps:

[0030] Step S31: Image acquisition and processing: Through the camera's internal and external parameters, an accurate mapping between image coordinates and real-world coordinates is achieved; after correction, the depth information of the image is further considered to map the image to real-world coordinates, and the three-dimensional real-world coordinates are mapped to 16-bit depth values ​​to enhance the visual effect of the image.

[0031] Step S32: Depth Information Enhancement: By comparing with the original pixel values ​​of a single image, the 16-bit depth value mapped from the real value is limited to the range of 1800-2000 pixel values ​​to avoid unnecessary amplification of road surface elevation difference information, thus ensuring the visual quality and interpretability of the final image;

[0032] Step S33: Height difference correction: Based on the overlapping area, crop the left duplicate image of the right image, obtain the pixel value difference between the left and right images in each row of the column to be stitched, and take the average height deviation value of all rows; so that the height value of each pixel in the right image is increased by the average height deviation value, thereby eliminating the height difference between the images.

[0033] Compared to existing technologies, this invention and its preferred solution expand the field of view by building a self-constructed 3D imaging platform and increasing the height of the device. In order to ensure accuracy while increasing the detection range, a dual-camera oblique measurement method is adopted. A self-made sawtooth calibration block is used to calibrate the dual cameras in the Z direction using a sawtooth calibration algorithm, while Halcon is used to calibrate the X and Y directions. Finally, the dual-camera calibration data is fused and rendered to obtain an image with a wider field of view and higher accuracy, which meets the system reliability and accuracy requirements in practical scenarios. Attached Figure Description

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0035] Figure 1 This is a schematic diagram of the appearance of the road detection system used in an embodiment of the present invention, wherein (a) the composition of the RIS; and (b) the RIS is installed on the detection vehicle.

[0036] Figure 2 This is a design example diagram of the calibration block according to an embodiment of the present invention;

[0037] Figure 3 The diagram shows the X and Y fitting mapping results of an embodiment of the present invention, where (a) is the mapping result in the x direction and (b) is the mapping result in the y direction.

[0038] Figure 4 This is a three-dimensional reconstruction image of the splicing result in an embodiment of the present invention. Detailed Implementation

[0039] To make the features and advantages of this patent more apparent and understandable, specific embodiments are provided below, along with accompanying drawings, for detailed explanation:

[0040] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0041] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0042] In this example, the overall technical solution is as follows:

[0043] (1) Non-traditional camera illumination modes

[0044] To achieve a better balance between target detection accuracy and speed, this example employs a dual-camera oblique-view method. Traditional camera calibration typically uses vertical illumination; however, this example utilizes an unconventional approach: oblique illumination from both cameras. This innovative method significantly expands the field of view while maintaining accuracy, offering new possibilities for applications in various fields.

[0045] (2) Calibration algorithm for oblique camera based on sawtooth calibration block

[0046] This example introduces a novel sawtooth calibration block design during camera calibration, which is effectively used for calibrating cameras operating at an angle. By inputting the required coverage area and height into the calculations, the size and dimensions of the specific calibration block are determined. The sawtooth is then moved uniformly 10 times within the desired acquisition height range. The highest and lowest feature points of the sawtooth are extracted after each movement to calculate the camera's intrinsic and extrinsic parameters for image distortion correction. This provides greater flexibility and accuracy for camera calibration, adapting to the needs of different applications.

[0047] (3) Image stitching and rendering

[0048] The average height difference at the stitching point of the two cameras is found by mapping the true values ​​after dual-camera correction to the range of 1800-2000. After processing the image on one side as a whole, the images are stitched together and rendered, which improves the accuracy of detection and positioning.

[0049] Specifically, this example provides a calibration method for a dual-camera system based on a 3D detection system, such as... Figure 1 As shown, the hardware components of the road detection system (RIS) on which it is based include a 3D linear industrial camera, a laser emitter, an incremental rotary photoelectric encoder (DMI), a global positioning system module (GPS), and a host device; it completes the corresponding road information collection by being installed on a surveying vehicle.

[0050] The method includes the following steps:

[0051] Step S1: Z-axis calibration of 3D linear scan industrial camera. Make calibration blocks to facilitate feature point extraction. Use algorithms to extract camera intrinsic and extrinsic parameters and convert Z-axis pixel coordinates into real coordinates.

[0052] Step S2: X and Y direction calibration of 3D linear scan industrial camera. Based on the calibration block calibration parameters, Halcon is used to fit the X and Y direction data and perform coordinate mapping in the X and Y directions to obtain the true coordinates.

[0053] Step S3: 3D reconstruction. The calibrated images acquired by the camera are efficiently stitched together to form a complete and seamless image result. The 3D reconstruction method is then demonstrated on the stitched result.

[0054] In this example, the specific implementation method is as follows:

[0055] The Z-axis calibration of the 3D linear scan industrial camera described in step S1, which involves creating a calibration block to facilitate feature point extraction, and converting the Z-axis pixel coordinates into true coordinates by extracting the camera's intrinsic and extrinsic parameters using an algorithm, includes the following steps:

[0056] Step S11: Calibration block design: as follows Figure 2As shown, considering that a single camera on a road surface 3D inspection vehicle needs to cover a width of 1.1m and a height range of 1.5m to 1.9m, the dimensions of the sawtooth calibration plate are calculated based on the actual required measurements and coverage area. The angle of the calibration block should not be too small, and the number of points on the slope should be as large as possible (greater than 100). The current X-axis resolution of the camera is 1mm / pixel. The distance between the points on the slope is 1*100 = 100mm, and the spacing between the sawtooth blocks is 100*2 = 200mm. The design is based on a 90° angle, and the sawtooth height is 100mm. The FindPoint function in the SDK requires the pixel line slope to be greater than 20 degrees in the image field of view, as shown in Formula 1 below.

[0057]

[0058] Where ROW represents the pixel height of a single serration, and Column represents the pixel width of a single serration, which is greater than 20 degrees.

[0059] Step S12: Feature Point Extraction: Based on the range algorithm settings, the range values ​​are scaled according to the number of sub-pixels used. The range values ​​are normalized by multiplying by a scaling factor. After scaling, the sensor ROI is set, with the sensor coordinate origin defined by pixel (0, 0). When using the sensor ROI, the parameter sensorRoi is used to compensate for offsets in the x and y directions. The parameter sensorRoi consists of four values ​​(width starting position, height starting position, set width, and set length). Finally, the center point is found, which is the origin (also called the principal point) of the image center defined by the pinhole camera model.

[0060] Step S13: Image Correction: The homography transformation matrix between the tilted imaging plane and the ideal frontal image plane is described by a rotation matrix represented by the two-dimensional Scheimpflug angle. The extracted pixel coordinates are then subjected to distortion correction to obtain the actual pixel coordinates after distortion removal. This is used to obtain the camera's intrinsic parameter coefficients, which are then stored in an XML file for subsequent extraction and use.

[0061] Step S14: Coordinate transformation: Based on the camera pinhole model, the imaging process is similar to the pinhole imaging principle. The camera distortion model is established for correction to obtain the conversion of pixel values ​​to true values.

[0062] Step S2 involves calibrating the X and Y axes of the 3D linear industrial camera. Based on the calibration block parameters, Halcon is used to fit the X and Y axis data, and coordinate mapping is performed on the X and Y axes to obtain the true coordinates. This includes the following steps:

[0063] Step S21: Data Acquisition and Preprocessing: The 16-bit depth 3D image acquired by the CCD camera is converted to 8-bit; the calibration file is loaded using the cx_3d_calib_load function, resulting in an object hCalib containing calibration information; the cx_3d_range2calibratedABC function is used to map the pixel coordinates on the image to coordinates in 3D space, thus associating the image data with the position of the actual object.

[0064] Step S22: Data segmentation: First, the objects of interest in the acquired image are separated from the background. With the help of the concatenation function, these parts can be connected into complete regions to obtain a more comprehensive target region. Finally, the select_shape function is used to select the parts that conform to specific shape features from the concatenated regions.

[0065] Step S23: Data Fitting: The `reduce_domain` function is used to extract the data region of interest. The `min_max_gray` function is used to obtain the minimum and maximum pixel values ​​within the selected region to calculate the X and Y dimension ranges. The calculated X and Y dimension ranges are then used to represent the true X and Y dimension data. The fitting mapping effect is as follows: Figure 3 As shown.

[0066] In step S3, 3D reconstruction involves efficiently stitching together the calibrated images acquired by the camera to form a complete, seamless image result. The 3D reconstruction method for the stitched result is also demonstrated, such as... Figure 4 As shown. Includes the following steps:

[0067] Step S31: Image acquisition and processing: Accurate mapping between image coordinates and real-world coordinates is achieved through the camera's internal parameters (such as focal length, principal point, etc.) and external parameters (such as rotation matrix, translation matrix, etc.); After correction, the depth information of the image is further considered to map the image to real-world coordinates, and the three-dimensional real-world coordinates are mapped to 16-bit depth values ​​to enhance the visual effect of the image.

[0068] Step S32: Depth Information Enhancement: By comparing with the original pixel values ​​of a single image, the 16-bit depth value mapped from the real value is limited to the range of 1800-2000 pixel values ​​to avoid unnecessary amplification of road surface elevation difference information, thus ensuring the visual quality and interpretability of the final image.

[0069] Step S33: Height difference correction: Based on the overlapping area, crop the left duplicate image of the right image, obtain the pixel value difference between the left and right images in each row of the column to be stitched, and take the average height deviation value of all rows; so that the height value of each pixel in the right image is increased by the average height deviation value, thereby eliminating the height difference between the images.

[0070] Preferably, this example, combined with a self-developed road detection system, proposes a precise, novel, and multi-technology integrated 3D camera calibration method, which improves the success rate of oblique camera calibration. This method meets the system's reliability and accuracy requirements in practical scenarios.

[0071] Meanwhile, this example fully considers the impact of complex traffic road environments on the robustness of the recognition model. This example specifically establishes experimental road sections and uses different test blocks and simulated standard potholes to simulate real road surface defects, demonstrating strong generalization ability.

[0072] Specifically, this example addresses the need to cover a complete four-meter lane on a real-world road surface while maintaining accuracy. It proposes an unconventional camera illumination method: traditional camera calibration typically uses vertical illumination, but this example employs an unorthodox approach—oblique illumination from two cameras. This innovative method significantly expands the field of view while ensuring accuracy, offering new possibilities for applications in multiple fields. To address the increased complexity of calibration due to severe distortion in oblique cameras, this example proposes a serrated feature point selection method. This involves moving the serrated edges equidistantly ten times within the detection range to select feature points on the calibration block at each movement. This allows for the calculation of camera intrinsic and extrinsic parameters, and the output is an XML file for easy access. Furthermore, to address excessive noise and uneven layering after image stitching, this example further improves the stitched image: first, pixel values ​​are mapped to real-world values; then, the average height difference at the stitching point is found to adjust one side of the image; finally, the images are stitched together, thus improving the quality of the stitched image and achieving better results.

[0073] This example calibrates the camera in the Z direction based on the sawtooth calibration algorithm; it uses Halcon to preprocess the X and Y direction data, performs threshold segmentation, and fits the segmented values ​​with the corrected XML file; it maps the true values ​​after dual-camera calibration to the range of 1800-2000 to find the average height difference at the stitching point of the two cameras, processes the image on one side as a whole, and then stitches and renders the images; thus improving the accuracy of detection and localization.

[0074] This patent is not limited to the above-described preferred embodiments. Anyone can derive other forms of dual-camera calibration methods based on a three-dimensional line-scan laser detection system under the guidance of this patent. All equivalent changes and modifications made within the scope of this patent application shall fall within the scope of this patent.

Claims

1. A dual-camera calibration method based on a three-dimensional line-scan laser detection system, characterized in that: A three-dimensional imaging platform is built, and a double-camera oblique measurement method is adopted; a sawtooth calibration block is made to cooperate with a sawtooth calibration algorithm to calibrate the Z direction of the double camera, and Halcon is used to calibrate the X and Y directions; the required coverage range and height are brought into calculation to obtain the size of the specific calibration block to be made; the sawtooth is moved uniformly within the required height range, and the highest feature point and the lowest feature point of the sawtooth after each movement are extracted to calculate the internal and external parameters of the camera, so as to correct the distortion of the image; A road detection system RIS based on at least a 3D line array industrial camera; The method comprises the following steps: Step S1: 3D line array industrial camera Z correction, and making calibration block to facilitate feature point extraction, extracting camera internal and external parameters to convert Z direction pixel coordinates into real coordinates; Step S2: 3D line array industrial camera X, Y correction, according to the calibration block correction parameter, using Halcon to fit the X, Y direction data, and performing coordinate mapping on the X, Y direction to obtain real coordinates; Step S3: three-dimensional reconstruction, efficiently splicing the images collected by the corrected camera to form a complete and seamless image result; and displaying the splicing result by three-dimensional reconstruction method; Step S1 specifically comprises the following steps: Step S11: calibration block design: considering that the single camera of the road three-dimensional detection vehicle needs to cover a width of 1.1 m, and the height range needs to be ensured to be between 1.5 m and 1.9 m, the size of the sawtooth calibration plate is calculated according to the actual measurement and coverage range; the factors considered in the calibration block design include angle and number of inclined surfaces, and the pixel straight line slope is imaged to be greater than 20 degrees in the image field of view, as shown in formula 1: (1) Wherein, ROW represents the pixel height of a single sawtooth, and Column represents the pixel width of a single sawtooth; Step S12: feature point extraction: according to the range algorithm setting, the range value is scaled according to the number of sub-pixels used; the range value is normalized by multiplying the scaling factor range; after scaling, the sensor ROI is set, and the sensor coordinate origin is defined by the pixel (0, 0); when using the sensor ROI, the parameters sensorRoi are used to compensate for the offset in the x and y directions; the parameter sensorRoi consists of four values, including: width start position, height start position, set width, and set length; finally, the center point is found, that is, the origin of the image center is defined according to the pinhole camera model; Step S13: image correction: the rotation matrix represented by the two-dimensional Scheimpflug angle is used to describe the homographic transformation matrix between the tilted imaging plane and the ideal orthoview plane, the pixel point coordinates extracted are corrected for distortion to obtain the actual pixel point coordinates after removing distortion, so as to obtain the internal parameter coefficient of the camera; Step S14: coordinate conversion: based on the camera pinhole model, the camera distortion model is established for correction to obtain the conversion from pixel value to real value.

2. The double-camera calibration method based on the three-dimensional line scanning laser detection system according to claim 1, wherein: Step S2 specifically comprises the following steps: Step S21: data acquisition and preprocessing: after converting the 16-bit depth three-dimensional picture collected by the CCD camera into 8-bit, the pixel coordinates on the image are mapped to the coordinates in the 3D space to realize the association of the image data and the actual object position; Step S22: data segmentation: first, the object of interest in the collected image is separated from the background, then connected into a complete region using the connection function, so as to obtain a more comprehensive target region, finally, the select_shape function is used to select the part meeting the specific shape characteristics from the connected region; Step S23: data fitting: the reduce_domain function is used to intercept the data region of interest, the min_max_gray function is used to obtain the minimum and maximum pixel values in the selected region to calculate the X and Y dimension range, and the calculated X and Y dimension range values are used to represent the real data of X and Y dimensions.

3. The dual-camera calibration method based on the three-dimensional line-scan laser detection system according to claim 2, characterized in that: Step S3 specifically comprises the following steps: Step S31: image acquisition processing: through the internal and external parameters of the camera, accurate mapping between the image coordinates and the real world coordinates is realized; after correction, the depth information of the image is further considered to map the image to the real world coordinates, and the three-dimensional real world coordinates are mapped to 16-bit depth values; Step S32: depth information enhancement: through comparison with the original pixel value of a single picture; the 16-bit depth value mapped by the real value is limited in the range of 1800-2000 pixel values; Step S33: height difference correction: according to the range of the overlapping area, the left repeated image of the right image is cropped to obtain the pixel value difference of each row of left and right images of the column to be spliced, and the height average deviation value of all rows is taken; the height value of each pixel in the right image is increased by the average height deviation value to eliminate the height difference between the images.

Images